Illusions in the Web: a real-time video editor built in HTML5

November 2, 2011

I am excited to blog about the current project I am working on and deeply enjoying. In Collabora, we have developed a tech demo of a simple video editor application running in the WebKit based web browser using HTML5 video technology. Well to be honest, not just plain HTML5 video, but we will get to that point later on. Keep in mind it is not a fully functional application, just a demo with some basic functionality like:

  • Media library browser for images and videos.
  • Drag and drop clips to build the media timeline.
  • Set the inpoint and outpoint for each video clip.
  • Real-time seamless preview of the media timeline.
  • Basic touch based interface, although there is room for a lot of improvement here.

The main advantage is that all processing is done locally and all the data is kept locally too, so no need to upload all the source material to a server or download the final video from a server, which might take some time specially when working with HD resolutions. Another nice feature is that you can actually preview the final video in real-time, instead of waiting for the server to process it.

Some leftover Halloween (eye) candy

All this stuff was shown by Collabora last week in Prague, during the LinuxCon Europe and GStreamer conference. Thanks Collabora for sponsoring my visit to the conferences by the way.

Let me show a screenshot and a screencast of how the the demo looks like.

You can even try this out if you wish. Install the Ubuntu package named witivi located in zdra/prague-demo, kindly packaged by Xavier Claessens. Additionally, you will need to get some videos to enjoy it. Just contact me if you need instructions on how to make it run.

Current trends in the web

As the title of this blog post might suggest you, all this is not yet possible with current state of HTML5 video spec, so we had to cheat a little bit by adding some extensions to WebKit in order to make it real.
However, there are a lot of efforts around the multimedia on the web nowadays, so evolution could certainly make it possible in a hopefully near future. Some of the current hot spots in WHATWG and W3C seems to be WebRTC, WebAudio API and even Augmented Reality. Things are really moving fast to make web apps more and more powerful specially regarding multimedia features.

Just in case you haven’t heard before, let me introduce briefly these technologies. WebRTC aims to provide real-time communication built-in HTML5 without additional plugins, so imagine audio and video calls possible from any html application with a simple Javascript API. WebAudio allows to do sample processing and synthesis from Javascript.

An interesting proposal which I discovered recently is MediaStream API, which is being promoted by Robert O’Callahan from Mozilla, looks like it could lead to have video editing functionality in HTML5 and it tries to integrate nicely with the other existing specs like WebRTC and HTML5 video.

General architecture

There are two main parts in this demo, the extension to WebKit for video editing and the video editor webapp (the user interface). The Webkit extension has been fairly easy to implement by using the Gstreamer editing services (GES) module, so having video editing features seemed like a breeze. The video playback of the media timeline has been accomplished through a special webkit gstreamer sink using a fake url, although we are checking how to do it with blob URLs similarly to how it is done in WebRTC. Even though the current implementation is not really very intrusive into WebKit codebase, it would be better to keep the changes separate from WebKit for now since there is no standard for this yet. We still have to investigate for the best way to achieve this. But it depends on how the whole things evolves.

The video editor web application has been implemented using jQuery, jQuery UI and jQuery layout libraries. Implementing the video demo webapp has not been exactly a bed of roses, but good documentation and lots of examples from the aforementioned libs helped a lot indeed.

Show me the code

Here you have the git repos in case you want to sneak on how we did it:

* GES WebKit. To compile GES WebKit make sure you specify –enable-mediatimeline option on configure. GES WebKit depends on GES and gnonlin.

* Web Video editor demo, also know as Witivi in reference to our beloved pitivi. You can find here all the html, javascript and css magic used to build the user interface.

Future extensions

Here is a list of things that potentially could be done for the following versions:

* Transitions and effects on video clips (in progress).
* Adapt to MediaStream API.
* Move to BLOB URLs.
* Create text and apply them on videos.
* Render to a local file.
* Push video to server.
* Media timeline support for multiple layers.
* Integrate with webrtc to do some collaborative video editing.
* And many many more …

Related work [Update]

Apart of this demo where Gustavo Boiko and I have been working on (with contributions from Alvaro Soliverez and Abner Silva), there are some very cool related demos developed by Collabora, all of them were shown at LinuxCon in Prague. We hope you enjoy them as much as we do:

* IM client running in the Webkit browser. An IM client with chat and video calls running in the browser mostly done in Javascript and HTML using Telepathy framework.

* Video call plugin on Media Explorer. A telepathy plugin to media explorer to be able to call our contacts, integrated nicely in media explorer UI.

Stay tuned.


Open Collaboration Services and libattica on MeeGo

October 16, 2010

I am going to write about my most recent contributions to libattica. Libattica, is a library in KDE implementing a client for the Open Collaboration Services protocol (OCS). OCS is a Free Desktop specification, with the purpose of integrating web communities and web based services into desktop applications. This is very general though, more specifically, it allows users to browse and share content like applications, wallpapers, etc. All this done with the typical social features, a user can write comments, rate content, contact other people, access knowledge base, etc.

Thanks to Intel hiring Collabora for sponsoring all the work I have done in libattica so far. The efforts have been mainly directed to make use of the OCS protocol in the MeeGo Garage project. The MeeGo Garage Client uses libattica to connect with the OCS servers. OCS is actually quite large, and we are just using using a small part of it (just the content and the comments modules), and I would even say we are using it in a way it was not originally meant, but that in the end it matched the requirements we had.

I also want to give thanks to Collabora, the company I work for, for sponsoring my attendance to Akademy 2010, my first Akademy actually, and I hope to be there every year :). I met in Finland a lot of nice people in th KDE world, and we had several meetings regarding OCS stuff with the people interested in it, Frank Karlitschek, Frederik Gladhorn, Daniel Wilms, Henri Bergius, Cornelius Schumacher among many others. In particular we made good progress on drafting next version of OCS spec, with several updates needed by MeeGo Garage project and other projects. The OCS draft spec 1.6 is online. Most of the features are already implemented and tested and will be ready to use in upcoming release of libattica 0.2.0. Just to mention some features where I put some effort:

  • Comments (for content and other items)
  • New scale for voting
  • GPG fingerprint and signatures
  • New download type to describe a content specified by a package name and repository
  • Summary field in content
  • Icons in content items
  • Licenses
  • Video files in content items
  • Home page entries
  • Distributions

Not only that, but there were a lot of ideas generated that had to be postponed to OCS version 2.0 version, which I hope we can start to work soon.

Frank has been very helpful in providing test servers available for me to test the client, and implement in the server the parts of the spec I needed to test the client. Fredrik was very helpful in making me become a KDE contributor.

Finally, libattica seems will be part of MeeGo, since it is a Qt-only library, no real dependency on other KDE stuff. In case you want to dig more on OCS and/or libattica you can check the following places:


Follow

Get every new post delivered to your Inbox.

%d bloggers like this: