pranavk.me

Dialog tunneling - Part 2

2018-04-12T00:00:00+00:00

This post is in continuation with the dialog tunneling post I talked about last time here. The last post talked about what we wanted to achieve and showcased it using a test tool in C++, gtktiledviewer. The real aim was to integrate this feature in LibreOffice Online so that people can use the awesome features that already exist in the LibreOffice desktop version.

Since last couple of months, we have been polishing it to look nicer, kill minor inconsistencies left & right, and worked on tunneling the modal dialogs as well where we faced interesting problems to deal because of the collaborative editing in Online.

From the implementation point of view, the API is also now more generic and simplified. We earlier used to have different LOK callbacks to notify about the dialog invalidation and other controls like combo boxes, color picker, etc. Now we have a single callback that handles all of these. This is possible because now we assign unique window ID to each of these entities (the main dialog frame as well as their child controls). So all of them can talk to Online independently. The client (Online) of course have to manage the parent-child relationships on its own (which is indicated only when the windows are first created - emit the ‘created’ callback). Since we were also interested in tunneling other things like autofilter menus, spelling suggestion context menus, etc., this simplification of API helped immensely in achieving that goal.

Tunneling modal dialogs caused some grief when we started to test them with collaborative editing. When we were launching a modal dialog in Online, the dialog would start its own main-loop inside an already running application-level main-loop. This is okay. But when the other user in the same document opens same dialog (collaboratively edit it), this dialog would launch its own main-loop on the same stack. So now the first user cannot close its dialog because its main-loop is one level up. Here’s the illustration of what I want to say, if the text above was confusing:

...
ExecuteDialog2-loop()
...
ExecuteDialog1-loop()
...
Application-level main-loop()

How do you close the Dialog1 without closing the Dialog2? Just don’t let dialogs have their own main-loop. Rather execute them asynchronously. Show them on the screen and tunnel them to Online. Kendy did a great job here in converting the dialogs to execute asynchronously along with necessary infrastructure changes in dialog execution code.

Moving the dialogs to execute asynchronously helped solving another problem - whole Online getting freezed with a launch of the dialog. Online has a single thread to interact with LibreOffice core; so, when we launch a modal dialog from Online, the thread will block and wait for the dialog execution to finish (closing of dialog). And this would freeze Online because now your key and mouse events wouldn’t go to core and won’t be processed, not because core cannot process it but because the thread that is responsbile for passing those events to core is blocked.

We have been continously polishing this feature to improve the user experience for our Online users. Michael and Kendy localized the dialogs and did some theming work on tunneled dialogs so that they suit our Online theme better.

It was an interesting journey overall. You can find all the patches here. Big thanks to Collabora for sponsoring this work and all the team members involved. You can find all the improvements talked above in the latest CODE.

Dialog Tunnelling

2017-11-29T00:00:00+00:00

So I’m finally resurrecting this blog to life after a long time.

I’m simply going to talk about what I’ve been currently working on in Collabora Online or LibreOffice Online, as part of my job at Collabora.

In our quest to bring more features available to our users editing documents in the browser, we are attacking something that contains the majority of the features in LibreOffice – the dialogs. One of the complaints that power users make in Online is that it lacks advanced features: they cannot add coloured borders in their paragraphs, manage tracked changes/comments, correct the spelling and grammar in the document, etc. The question before us is how do we bring these functionalities to the cloud at your disposal in your browser tab?

We really don’t want to write another million lines of code in Javascript to make them available in your browser and then dealing with separate set of bugs for time to come.

So we decided to come up with a plan to just tunnel all the hard work that developers have done for the past couple of decades: come up with appropriate infrastructure to open the dialog in headless mode, paint them as a bitmap in the backend, and tunnel then image to you in the browser. And then add life to them by tunnelling your mouse/key events as well which will invalidate and update the new image you are seeing the browser. Don’t worry; we are not sending the whole dialog image back to your browser every time. Only the part that needs updating in the dialog is sent back to the browser saving us precious time and network bandwidth improving your UX.

The current state of the project looks really promising. Not just the modeless dialogs, we are able to tunnel the modal ones as well which is not something we had expected earlier.

Since text is boring, here’s a preview that shows dialog tunnelling in action in our test tools, GtkTiledViewer. The integration with Online is ready too and undergoing some final polishing. But it’s not something you’d have to wait for too long; we are polishing a big refactor to LibreOffice core master to install the dialog infrastructure needed for integration. Now you will be able to do pretty much all the things in Online (and in CODE version 3.0 soon to be released) that you’ve always wanted to do.

Here are the slides from the talk I delivered on the same topic in our annual LibreOffice Conference in Rome this year.

Update on Libreoffice and GNOME integration

2016-01-23T00:00:00+00:00

It’s been a long time I have talked about the project that I started with GSoC 2015 some time back. We reached at pretty much exciting results by the end of the summer where we could see the integration working pretty well with LibreOffice. We finished and merged all the major work on the Libreoffice side alongwith just-made-it-work integration with gnome-documents. Things were still in the development stage for gnome-documents, and we needed good amount of effort to get it merged upstream.

Things moved pretty slow on the integration because from time to time during the integration I would realise that I had missed something in LO, and I need to fix it before I could move forward with the integration. I would then jump back to LO, fix it, and switch back to gnome-documents, and so on. But it was only till Dec 2015 that things were slow. I suspect this thing to be the turning point. Bastien, [Debarshi] (https://debarshiray.wordpress.com/), and Cosimo held the string from gnome-documents side, and started working with my earlier WIP work/patches for gnome-documents. There were many issues that needed to be fixed for the proper ready-to-merge LO integration, and better user experience. But I was lucky now that I had the experts to take care of it.

As I expected, there were still some minor fixes to be done on LO side which they found out during the integration. Bastien would report LO related bugs; I would fix them in LO; David would help with the reviews, and build and ship the package for us. As I write this, I am glad and feeling proud that we are now there with most of the work already merged upstream in gnome-documents. With LO 5.1 release just few weeks away, gnome-documents seems to be all set to integrate LO with its next major release, 3.20.

Here is the screencast I made running gnome-documents master with LO 5.1.0 rc2 koji build on fedora. There are bug fixes that couldn’t make to LO 5.1, and some more that you will uncover as you use this. :) So, it might take little more time (one more release, maybe) while it settles down. Note that LibreOfficeKit API (to expose LO functionality) is still unstable, but the widget is quite usable in the view-only mode, and that is how we have integrated it in gnome-documents for now.

Screencast is here :

Earlier, gnome-documents was converting these non-supported formats into PDFs using unreliable command unoconv, which sometimes would not give good results especially with spreadsheets where it would scramble things up during the conversion. With the use of this new widget, now, you would see the documents as they exactly are, unless you do not have LibreOffice installed on your box in which case it would pop up an error asking you to install LibreOffice.

If you want to try it out yourself, you need to build atleast LibreOffice 5.1.0 rc2 and gnome-documents master branch. If you are on fedora, you can use this koji build to install LO 5.1.0 rc2 on your box, and you can use jhbuild to build gnome-documents from master. Running gnome-documents now should automatically make it to use LibreOffice in the background.

I would like to thank all the people, aforementioned, Bastien, Debarshi, and Cosimo to finish the GNOME integration with LibreOffice. On the LibreOffice side, Miklos and Michael with whom I started this GSoC project, and David Tardon who has helped us ship the LO package with much needed fixes in time.

While I am at it, I also want to announce about my new job at Collabora Productivity where I would be hacking on LibreOffice full-time. It would be an exciting and wonderful learning experience for me. I am greatly looking forward to it.

LibOCon 2015 -- Aarhus

2015-09-28T00:00:00+00:00

I spent my last week with the LibreOffice community talking, hacking and altogether enjoying a lot with them. It was my first LibreOffice conference, and I have brought home a lot of learning this time. I relish each moment that I spent amongst such awesome people around me.

The conference was very well organized in the beautiful city of Aarhus, in Denmark (little bit colder than I had anticipated). As they say, more than a city, Aarhus was a feeling. The local organizing team left no stone unturned to take care of the participants, and everything went smoothly. I also really liked the newly constructed venue where the event was organized.

More than anything else, it was pleasure meeting my GSoC mentors, Michael Meeks and Miklos Vajna, in person, and presenting my work before LibreOffice community. Hacking together with them was a wonderful learning experience for me, especially hacking in the train with Miklos, and trying to fix some bugs. It was also a good experience meeting faces behind IRC nicks, talking to them and sharing ideas, and grow your social skills, altogether (I hope I did) – :).

At last, I would like to thank The Document Foundation for sponsoring this conference for me, without which it would not have been possible for me to come that far and gain this experience. Aforementioned, attending this conference was a good learning experience which has motivated me to excel as a programmer, and a will to keep this relationship strong with LibreOffice in future.

GSoC Wrapup Report

2015-08-21T00:00:00+00:00

This is a GSoC wrap up report of everything I did this summer. Though, my official GSoC organization was LibreOffice, the work also involved GNOME directly, and Mozilla indirectly. Some of you, who are already familiar with my work might find this post repetitive since I would be repeating many of the things here that I have already talked about in my earlier posts.

Before I started working on it as a summer of code project, the widget:

used to render all tiles in the loaded document
used to make calls to the LibreOffice core in the same UI thread
was not introspectable
was gtk2
and needed such minor fixes here and there

Tiled Rendering

I started with implementing tiled rendering in the widget. First plan was to reuse the Mozilla tiled rendering code. After analyzing this and discussing it, we scraped that plan as it was quite infeasible. Better approach was to understand how tiled rendering is implemented and write our own class that handles the tiles for you. We took ideas from Mozilla tiled rendering code, and GeglBuffer.

We started with a small TileBuffer class, and it gradually improved as widget started to demand more from it. There is still some scope of improvement in TileBuffer class which is already on my radar. At the moment it exposes enough API for the widget to work smoothly.

Only render visible tiles

Aforementioned, the widget used to render all of the tiles even if they are not visible on the screen. This was a huge bottleneck to the widget performance. Why render millions of tiles in a large document if most of them are not even visible to the user?

We changed this to demand-based model. Only render tiles when there is a demand for them, that is, when the user wants to see them. Also keep caching these tiles with the help of TileBuffer so that next time user asks for these tiles, it doesn’t have to render the tile again.

Free the main thread

All the LibreOffice operations, like, rendering new tiles, selecting a paragraph, bold the selected text etc. is a lot of work and if done in the main thread would not give a very smooth user experience. So, we decided to move all such heavy operations (all libreoffice calls) to a new dedicated worker thread whose job is to perform the libreoffice calls and return the result to the main thread. As a further optimization, we used a thread pool with a single thread so that we don’t have to create a new thread everytime. It is to be noted here that we used a thread pool of single thread because libreoffice is almost single-threaded, and it would be useless to create multiple worker threads calling to the same libreoffice core instance. Further, thread pool with a single thread would automatically queue the LibreOffice operations for us, and execute them when thread becomes free.

As of this writing, we have moved all LibreOffice calls but one to the worker thread. This has signficantly improved the widget performance.

One of the aim of this project was to integrate it with gnome-documents. GNOME Document is written in javascript, and the widget is in C++. GObject Introspection came to our rescue here. We started making this widget more GObject friendly, added necessary comments/annotations, and finally we were able to use this widget from any of the language bindings including javascript.

Gtk3 port

Since we wanted to integrate the widget with gnome-documents (gtk3), we modified the widget to use gtk3. This also gave a new look and feel to the widget.

Ship introspection files with LibreOffice

Installing the introspection files (.gir/.typelib) into their standard location on user’s computer is something that doesn’t fit well into the LibreOffice installation model. The current plan is to let the distributions execute a script (create_tree.sh) provided by LibreOffice that would generate and install the introspection files (.gir/.typelib) into their standard location. You would most likely see this script being used by distributions with LO 5.1

Integrating it with GNOME Documents

For few days, I used a sample javascript application that uses this widget to show and edit documents. This was useful for debugging. Gradually, I started to write code for gnome-documents to use this widget. I had to change the integration model a few times. Finally, I ended up writing a new class to handle the libreoffice widget, and the integration works quite well now.

If you are interested in trying this widget out, you can checkout my gnome-documents feature branch here. However to make this work, you have to generate the introspection files manually. I have created a (wiki page)[https://wiki.documentfoundation.org/Development/Integrating_LOKDocView_and_GNOME_Documents] that should help you in this regard.

ETA in GNOME Documents

All the work on the widget is already in LO master which will become LibreOffice 5.1 to be released around January 2016. This means we still got some time to make improvements in widget, if any, till LO 5.1 freeze in November 2015. My hope is that we should be able to see the widget integrated in gnome-documents in 3.20, if everything goes well.

Here is the screencast I made where gnome-documents is using the libreoffice widget to show open documents, while still using evince view to show the pdf documents.

Overall, this was the best project that I ever worked on till now. I learnt a lot this summer. I would like to thank my GSoC mentors - Michael Meeks, Miklos Vajna for their continous support and ideas throughout the summer. I would also like to thank Debarshi Ray and Cosimo Cecchi for their invaluable suggestions that helped me finally integrate the widget with GNOME Documents. Its still not in master, but it works well, and I hope to see it soon in master. Also thanks to Mozilla community for helping me with tiled rendering concepts.

GUADEC 2015 in Gothenburg

2015-08-10T00:00:00+00:00

This year GUADEC was organized in the wonderful city of Gothenburg in Sweden. It was my pleasure attending this conference for the second time, and meet several awesome contributors in FOSS community again. Discussions with several contributors helped me get a wonderful insight of various important topics in both GNOME, as well as the LibreOffice project.

As a Summer of Code student this time for LibreOffice, I also gave a talk about my (project)[http://pranavk.github.io/open-source/initial-preview-of-libreoffice-integration-with-gnome-documents/] with Markus Mohrhard. You can download the slides (here)[https://mmohrhard.files.wordpress.com/2015/08/integrating-libreoffice-with-gnome-documents.pdf].

I would like to thank whole of the GUADEC organizing team for organizing this event smoothly. Last but not the least, a many thanks to TDF for sponsoring my travel and accomodation without which all of this would not have been possible. I am looking forward to have a long and exciting journey ahead with GNOME and Document Foundation.

FUDCon Pune 2015

2015-07-03T00:00:00+00:00

This year FUDCon, held at Pune, last week was my first ever FUDCon, and my first steps in to awesome Fedora community. This was also the first conference where I delivered a full-fledged talk about ‘Automating UI testing’ presenting some of the work I did in automating the UI tests for gnome-photos. The talk was more about how they can make their UI tests automated.

I also talked about ‘Integrating LibreOffice with your applications’ in a barcamp talk sharing and discussing ideas with few people, presenting what I am up to in this project in LibreOffice, and how they can take advantage by either directly using the new, evolving LibreOfficeKit API, or by using the new Gtk3 widget in their applications. I talked about how I am acheiving this using tiled rendering, and how I (with Michael and Miklos) am planning to enhance this in future by incoporating the support for opengl, efficient tile management, and multi-threaded support.

Besides that, it was a wonderful opportunity for me to meet new people contributing to Fedora project, and sharing ideas with them. I now have a better idea of how I can contribute more to Fedora, and feel motivated enough to continue my contributions. I have made quite a few friends who, I think, would be happy to help me if I plan to get started with any of the Fedora teams, and I do plan to involve myself in few more interesting teams in future sparing time out of my regular work.

Last but not the least, I would like to thank all the organizers for making this event possible. They have been working hard for months, and have had many sleepless nights just to make sure everything remains on track. I would also like to thank them for sponsoring my stay and travel, without which I would not have been able to attend the event.

Initial preview of LibreOffice integration with gnome documents

2015-06-22T00:00:00+00:00

I managed to integrate LibreOfficeKit’s LOKDocView widget with gnome-documents, finally. Here is the screencast for the same.

There are still lot of improvements that I need, for example, we need to have a new editing overlay now in gnome-documents so that you you can do operations like bold, italics, underline, search, switch to edit mode and view mode, and few other things. There are also crashes right now which possibly are because I haven’t yet written robust code to nicely separate the currently used EvinceView and the newly used LOKDocView.

On the other hand, I have few ideas to improve the widget backend, for example, improving the tile buffer backend by rendering nearby tiles to increase the visual coherence. Right now, it only renders the visible tiles, but it would be good to render the tiles nearby the boundary of the visual region so that the scrolling is smooth.

I would be working on improving upon this in next coming weeks.

Introspecting LOKDocView, the LibreOffice widget

2015-06-15T00:00:00+00:00

This is in continuation of my work under LibreOffice. For the past few days, I have been working on restructuring the widget, LOKDocView, to make it introspectable. I also ported the widget to gtk3 from gtk2, so applications can now start thinking of using it.

To test its introspectability, I wrote a simple test application in javascript making use of this widget. Here is the small screencast I made using the widget from javascript.

The widget still needs more polishing to provide a sane, minimal, still useful, API to consumers and to hide the still unstable LibreOfficeKit API. So, we would be improving that in coming weeks. We also plan to try GtkGLArea instead of currently used GtkDrawingArea for the widget to enable openGL while rendering tiles, and hence increase the performance. The backend currently use tilebuffer, taking few ideas from gegl-buffer, and Mozilla’s tiled buffering logic. I also hope to make further improvement in this backend to increase the widget’s performance making use of efficient algorithms.

Here is the feature branch for LOKDocView.

Integrate Libreoffice with gnome-documents

2015-06-05T00:00:00+00:00

This year I am working on integrating Libreoffice with gnome-documents. gnome-documents currently only supports viewing documents. It indirectly makes use of poppler library to render PDF documents. To show any other format such as .docx, .odt, .ods, it first converts the document into a PDF format using unreliable unoconv command, and then render these PDFs using poppler. Hence, this also disables gnome-documents from editing editable formats.

As part of my GSoC 2015 under Libreoffice, my main aim would be to improve an existing widget under Libreoffice, LOKDocView, and integrate the widget with gnome-documents. LOKDocView make calls to Libreoffice core using LibreOfficeKit. The current LOKDocView implementation needs a little furnishing. It also doesn’t support efficient tiled rendering, which is very essential especially at larger zoom levels (You never want the application to render zillions of tiles for you at 500x even if you don’t want to see majority of them). One of the improvements that I would be making in this widget is efficient tiled rendering so that it only renders the visible part, and while scrolling tries to reuse the already renders tiles to the best of its ability.

To improve this tiled rendering, I have modfied the widget to make use of a tile buffer taking ideas from Mozilla source code, and how it manages to do the tile buffering. The current implementation of the tile buffer I created acts simply as a cache returning already rendered tile instead of giving a new render tile command to the LO core. This helps in easy navigation of documents. You can find more about this on my feature branch.

To further improve the tiled rendering support, we are also trying to find ways of rendering the tiles on GPU, rather than on the CPU. GtkGLArea seems to be the good choice here, but is quite new, and most importantly Libreoffice is still at Gtk2 and it might take some time for it to migrate to Gtk3.

Other than that, to make the rendering even more efficient, we can employ techniques like double-buffering which I would be analysing the feasiblity of, and then implementing them in the widget.

But this widget is still not a replacement for existing poppler library. The Libreoffice can render other document formats such as odt, doc, docx etc., but it gives a terrible performance at rendering PDFs. A better approach would be to use both in gnome-documents, but only poppler for rendering PDF.

Please feel free to comment, if you have any idea/optimization regarding this.