Illusions in the Web: a real-time video editor built in HTML5

I am excited to blog about the current project I am working on and deeply enjoying. In Collabora, we have developed a tech demo of a simple video editor application running in the WebKit based web browser using HTML5 video technology. Well to be honest, not just plain HTML5 video, but we will get to that point later on. Keep in mind it is not a fully functional application, just a demo with some basic functionality like:

• Media library browser for images and videos.
• Drag and drop clips to build the media timeline.
• Set the inpoint and outpoint for each video clip.
• Real-time seamless preview of the media timeline.
• Basic touch based interface, although there is room for a lot of improvement here.

The main advantage is that all processing is done locally and all the data is kept locally too, so no need to upload all the source material to a server or download the final video from a server, which might take some time specially when working with HD resolutions. Another nice feature is that you can actually preview the final video in real-time, instead of waiting for the server to process it.

Some leftover Halloween (eye) candy

All this stuff was shown by Collabora last week in Prague, during the LinuxCon Europe and GStreamer conference. Thanks Collabora for sponsoring my visit to the conferences by the way.

Let me show a screenshot and a screencast of how the the demo looks like.

You can even try this out if you wish. Install the Ubuntu package named witivi located in zdra/prague-demo, kindly packaged by Xavier Claessens. Additionally, you will need to get some videos to enjoy it. Just contact me if you need instructions on how to make it run.

Current trends in the web

As the title of this blog post might suggest you, all this is not yet possible with current state of HTML5 video spec, so we had to cheat a little bit by adding some extensions to WebKit in order to make it real.
However, there are a lot of efforts around the multimedia on the web nowadays, so evolution could certainly make it possible in a hopefully near future. Some of the current hot spots in WHATWG and W3C seems to be WebRTC, WebAudio API and even Augmented Reality. Things are really moving fast to make web apps more and more powerful specially regarding multimedia features.

Just in case you haven’t heard before, let me introduce briefly these technologies. WebRTC aims to provide real-time communication built-in HTML5 without additional plugins, so imagine audio and video calls possible from any html application with a simple Javascript API. WebAudio allows to do sample processing and synthesis from Javascript.

An interesting proposal which I discovered recently is MediaStream API, which is being promoted by Robert O’Callahan from Mozilla, looks like it could lead to have video editing functionality in HTML5 and it tries to integrate nicely with the other existing specs like WebRTC and HTML5 video.

General architecture

There are two main parts in this demo, the extension to WebKit for video editing and the video editor webapp (the user interface). The Webkit extension has been fairly easy to implement by using the Gstreamer editing services (GES) module, so having video editing features seemed like a breeze. The video playback of the media timeline has been accomplished through a special webkit gstreamer sink using a fake url, although we are checking how to do it with blob URLs similarly to how it is done in WebRTC. Even though the current implementation is not really very intrusive into WebKit codebase, it would be better to keep the changes separate from WebKit for now since there is no standard for this yet. We still have to investigate for the best way to achieve this. But it depends on how the whole things evolves.

The video editor web application has been implemented using jQuery, jQuery UI and jQuery layout libraries. Implementing the video demo webapp has not been exactly a bed of roses, but good documentation and lots of examples from the aforementioned libs helped a lot indeed.

Show me the code

Here you have the git repos in case you want to sneak on how we did it:

* GES WebKit. To compile GES WebKit make sure you specify –enable-mediatimeline option on configure. GES WebKit depends on GES and gnonlin.

* Web Video editor demo, also know as Witivi in reference to our beloved pitivi. You can find here all the html, javascript and css magic used to build the user interface.

Future extensions

Here is a list of things that potentially could be done for the following versions:

* Transitions and effects on video clips (in progress).
* Adapt to MediaStream API.
* Move to BLOB URLs.
* Create text and apply them on videos.
* Render to a local file.
* Push video to server.
* Media timeline support for multiple layers.
* Integrate with webrtc to do some collaborative video editing.
* And many many more …

Related work [Update]

Apart of this demo where Gustavo Boiko and I have been working on (with contributions from Alvaro Soliverez and Abner Silva), there are some very cool related demos developed by Collabora, all of them were shown at LinuxCon in Prague. We hope you enjoy them as much as we do:

* IM client running in the Webkit browser. An IM client with chat and video calls running in the browser mostly done in Javascript and HTML using Telepathy framework.

* Video call plugin on Media Explorer. A telepathy plugin to media explorer to be able to call our contacts, integrated nicely in media explorer UI.

Stay tuned.