WebM vs H.264 Showdown

A few weeks ago I was asked to write a bio - explain a bit about myself in a few lines. I hate writing bios, all that faux third-personage - I never really know what to write.

The reason a bio was required was that I’d been offered a position as a Knight-Mozilla sponsored fellow with Al Jazeera - the happy end-result of a Knight-Mozilla news challenge that I have been actively enjoying over the last few months. It was agreed that I would work slightly less than full-time hours and on a largely remote basis and so I was glad to accept the fellowship knowing I could still dedicate time to my family and other interesting projects (not that I would classify my family as a project you understand). Working with Al Jazeera is of course an experience not to be missed. I couldn’t have imagined that my life would take such a turn a year ago or so, but I’m very glad that it did and I put it all down to my (broadly directionless) approach to working life.

I consider myself very lucky that because of my sedentary lifestyle I need very little money to get by. This has allowed me to follow my interests while working for our small ‘company’ Happyworm for the last 10 years. I consider this a very real success. We created a popular open source project and foster a fairly large community — from this many opportunities have arisen. Recently for example, I was asked by the W3C to run an online audio/video course and this has been a fantastic experience and now I have the opportunity to find out the world of journalism works and hopefully contribute. All this would probably not have occurred if I hadn’t simply thrown caution to the wind and followed by interests.

There are two of us now working for Happyworm, we used to be three but our web designer (also my partner) decided that at least one of us should have a steady income and generously offered to take up full-time work at a local council. So it’s just the two of us working with 3rd party designers when we need them. I’m based just outside Florence and Mark P’s in the heart of Edinburgh, although we are very different, we have similar requirements and a similar history - Mark P left a well paid job as a CMOS camera chip designer to come and work on web stuff.

This finally brings me to the title of this post, when writing my bio I foolishly used a little known, possibly non existent word, by describing Happyworm as tiny altrepreneurial web agency.

I stumbled on the word altrepreneur some years ago here : http://www.mylifecoach-online.com/MLC%20newsletter%20May%202005.htm#authentic

“The Altrepreneur, like their colleague the Entrepreneur, runs one of the 3 Million Micro Businesses in operation in the UK today. However unlike the Entrepreneur, with a financial and career focus, the Altrepreneur is doing it for entirely different reasons.

This article chimed very strongly with me - my objectives and the decisions I made to leave a well paid job, set up Happyworm, move to another country, be my own boss and follow my own interests where possible. We now have two children, who I am fortunate enough to see a lot of. Recently I decided to look after the 9 month old Anna in the mornings for 4 days of the working week and then work from 14:30 until midnight with a 3 hour or so (I don’t time it) break for family dinner and games. I grow vegetables, cook at least once a day and am involved in the local community centre - finally I feel like I am achieving that mythical work-life balance.

So to me the word altrepreneurial seemed a perfect concise way of explaining what I did and how I see Happyworm. Incidentally Happyworm turned 10 years old in October. You might think 10 years is pretty good going for a small company but the truth is we would never have lasted so long if we were in it for the money, we’ve had good spells but also our fair share of dry spells where we worked on open source, brewed our own beer and patched our own clothes (or at least I did). Turns out following our interests and making jPlayer has been much more of a success than we could have imagined, I think we’re approaching half a million downloads and perhaps the best measure - a community of around two and a half thousand.

Despite the money, like most people I’m not really happy working long hours on projects I’m not interested in (although through necessity I’ve done my fair share), I don’t think actually I’m any good at something my heart isn’t in. Maybe I’ve been spoiled, but the most important thing for me is to enjoy my work and so life, the money is always a secondary consideration and that’s why, contrary to what you may see written in my bio, I work for a small altreprenurial web agency - not entrepreneurial. Damn you auto-correct!

This notebook has been lying on my desk for the best part of a decade.

Some perspectives on the Knight-Mozilla News Technology Fellowship :

My Life as a Startup - The Road not Taken by The Guardian fellow - Nicola Hughes

Journalism in the Open: the 2011/12 Knight-Mozilla Fellows by Project Header Upper - Dan Sinker

Knight-Mozilla names news technology fellowship winners Journalism.co.uk

“O ye’ll tak’ the high road, and Ah’ll tak’ the low (road)
And Ah’ll be in Scotlan’ afore ye.” (The Bonnie Banks o’ Loch Lomond)

The web audio community are a vibrant bunch. No sooner had the standard <audio> API been established, than developers were clamouring for more. Just playing audio wasn’t enough, we wanted to analyse, react to and manipulate our audio. Happily, the browser makers obliged with first Mozilla, then Google producing enhanced web audio APIs for their browsers - the only problem was, they were two very different implementations. The Audio Data API implemented in Firefox exposed the data at a fairly low level, while Webkit’s Web Audio API provided a higher level abstraction providing a number of predefined functions. Luckily, it didn’t take long for the JavaScript community to react and start bridging the gap between the two, by writing libraries that provided a common API, libraries such as sink.js which smooths over low level differences. In turn, sink.js was used by ‘higher level’ libraries like audiolib.js - (a general purpose audio toolkit) and Audiolet (which provides a more musically intuitive API, with similar objectives to Webkit’s in-browser solution). There are many others, such as XAudioJS which sports a Flash® and base64 data url wav generation fallback, older projects like dynamic.js that just provides a Flash® fallback for the Audio Data API and DSP.js a Digital Signal Processing Library.

People really love messing about with audio.

Notice that the process of creating all this cool functionality didn’t come about from a W3C spec. Similarly, the Advanced Audio APIs were not the result of a W3C think-tank, but from two competing visions of what an advanced audio API should look like. Now it looks like the Web Audio API will be implemented in Safari as well as Chrome.

Once you create compelling functionality, developers will immediately start to use it. It may be experimental but developers will start to rely on it to make cool stuff. Cutting edge technology is seductive like that. I’m surer than sure that the Web Audio API has been well researched and has taken much inspiration from tried and tested APIs that exist outwith of our lovely browser based world (Apple’s Core Audio Frameworks, I believe), but I’m not convinced that you can really tell what web developers need or want until you give them something to play with.

Mozilla’s approach was to expose a very comprehensive low level API, which potentially allows JavaScript developers to create all the functionality of Webkit’s Web Audio API and then some. As a result we get libraries like JSMad cropping up. What does JSMad do? Significantly, it allows you to play MP3s in Firefox*. Is JavaScript fast enough? Apparently so. This was a ‘this changes everything’ moment for me and since then a similar approach has been taken by pdf.js and more recently Broadway.js which decodes H.264 on the fly.

*Neither Firefox or Opera support MP3 natively due to patent concerns.

I’m not saying Mozilla’s Audio Data API is perfect, there are issues with audio using the same thread as the UI and synch issues with multiple streams. However this is being addressed in the MediaStreams Processing proposal and it’s worth taking a look at it, even if it’s just for an insight into what future implementations could look like.

I’m digressing. The point is, if browser makers expose the low level API, developers will quickly come in and start writing libraries on top of that API. As is often the case, the developer community will start making things that the browser makers had never even considered. It makes sense, there are many more web developers than browser developers. Sure, web developers will bridge the gaps and polyfill over the cracks, which let’s face it, has been the only reasonable way of going forward with HTML5, but crucially they will also make new libraries that other developers can use - and all of this at very high rates of turnaround. Of course, the common-or-garden JavaScript developer has a series of enormous advantages over the browser API developer or the standards bodies that seek to define these APIs. I’m gonna name three here:

1. Strong community — Web developers have a huge active and open community to draw from.
   
   
2. Lower barrier to entry — The barrier of participation once something is put on something like github is virtually zero.
   
   
3. Room to manouevre — Nothing web developers write is ever set in stone, JavaScript represents a much more fluid abstraction than the less flexible native browser code.
   
   

Ok, so bear with me here, and this is more of a question than a proposal - What if we separate concerns between browser makers and web developers when it comes to creating standards? Browser makers could concentrate on security, privacy, performance and exposing low level API’s in such a way that web developers can start to build libraries and APIs in the fluid, dynamic, iterative and extremely reactive manner that the web as a media allows. Once these libraries reach an acceptable level of adoption, browser makers can get together and decide which of these features they want to adopt based on tried and tested use cases, and yes make it a standard and build it into the browser. Wouldn’t we move forward more quickly that way? And as a bonus, no browser would be left behind as we’d be building the polyfills along the way.

In short, what I’m saying is that if the standard bodies put their energy into defining low level APIs, the high level APIs will look after themselves, or rather the community will look after them. After all it seems that the W3C themselves want a more community based approach to standards and besides we all know that bottom-up trumps top-down, right?

Outside my flat is an open space that the local council didn’t quite know what to do with, I’m sure they considered adding basket-ball hoops, concrete tables, a kid’s playground and all kinds of things. As it turned out they created a decent flat surface and pretty much left it as that. The users of this space, mostly children, decided this was a perfect space for playing soccer and improvised the space to include a hand drawn goal and pitch markings. If the council really wanted to make something permanent, they could take inspiration from this and create real goals and solid pitch markings.

It’s probably too late to change the Webkit implementation of the Web Audio API significantly, but I would strongly urge the developers of it to include a more comprehensive low level API in future releases. What’s the worst that could happen?

A big thanks to Jussi Kalliokoski and Dustin Blake for helping with this post and deep respect and gratitude to all those hacking on audio.

➸ Follow me on Twitter

It was only a year ago for example, when I started tinkering with jPlayer to couple text and audio and was asked to show off a very small demo at the first Mozilla Festival in Barcelona last year.

A few months later I was introduced to and became involved with something known as Hyperaudio and was given the opportunity to create a couple of proof-of-concept demos : Denmark Radio’s Hyperdisken Demo and the Radiolab/Soundcloud collaboration ( a shout-out to Henrik Moltke for doing much of the groundwork for these demos and my colleagues at Happyworm without which they just wouldn’t have been possible).

Further on through the arc of this epic year, I took part in the Mozilla Knight Journalism challenge where I was encouraged to research and blog some ideas on a tool I called the Hyperaudio Pad. Happily I was flown over to Berlin with other like minded MoJo peeps, for a week of intense discussion, collaboration and hacking and managed to get the first semblance of an actual product out of the door.

Which brings me in a round-about way to this year’s Mozilla Festival, where I will again be demoing and chatting about the coupling of text and media and the other things that Hyperaudio is (or could be) about. This all at the Science Fair on Friday, 4th of November and a Hyperaudio Workshop / Design Challenge on Saturday, 5th.

The Science Fair will be all about demos, chatting to people interested in the general Hyperaudio concept and making sure I don’t spill my drink over my laptop. Meanwhile in the workshop we hope to put the Hyperaudio Pad through it’s paces and actually make something with it, while gathering ideas and maybe hacking on some of them into the bargain. It should be a fun mix of hacks, hackers, designers, audio buffs and basically anybody curious enough to be involved. The session will be fairly free flow but the plan so far is to:

• Briefly chat about and demo the Hyperaudio concept, what it can do and with whom we can collaborate.
• Breakout into groups to come up with new ideas, applications and designs.
• Reconvene and talk about the group’s ideas.
• Breakout into groups to work on these ideas, whether it is creating a programme, UI mockups, story-boarding functionality or even developing small demos.
• Creating something we can show other people - hopefully a program created with the Hyperaudio Pad but also some new ideas.

Hopefully we can cram this all into 3 hours. Folk from the BBC, Universal Subtitles, Sourcefabric and other cool people are planning to be in attendance, which should make for some interesting crossovers and fantastic opportunities to consult and maybe even influence organisations in a position to use the stuff we’re making. Additionally the BBC are providing us with some fantastic media to work with.

All will be revealed over the weekend, book your place and get your eyes and ears ready to be part of the Hyperaudio experience.

Imagine

Imagine that making your own video or audio news programs was really easy.

Imagine if you could pull together news stories from various sources, combine them and publish the results in minutes.

Imagine that the story you made came complete with a transcript, was fully accessible and could be picked up by search engines.

Imagine that you could easily embed your story into a web page and that others could incorporate your story into new stories.

Imagine that viewers of you story could share any part of it in seconds.

Imagine all this and you are imagining the functionality of the Hyperaudio Pad.

Listen to a use case.

Watch the screencast.

Hypertranscripts

Currently, the process of putting together a media program, the process will probably involve some fairly complex audio/video editing software. The result - a fairly ’static’ representation. The whole process is time consuming, wasteful and inflexible.

By changing the way we think about media, we can create a new way of manipulating it. The Hyperaudio Pad allows quick manipulation of media and takes advantage of a form of media representation referred to as a hypertranscript.

When we tell a story we invariably use the spoken word, the spoken word can be transcribed, and once transcribed can be used as an accurate reference to the media it is associated with.

Hypertranscripts are part of a broader concept dubbed Hyperaudio by Henrik Moltke, they are transcripts hyper-linked to the media they represent, a type of fine-grained subtitle, they are separate entities from the media they describe, existing as a word-level aligned references expressed in HTML.

Tools and services exist to create Hypertranscripts, word level timing can even be approximated from subtitles. In the past I have used libraries such as Popcorn.js, jPlayer and data from the Universal Subtitles project.

The good news is that Hypertranscripts already exist, here are a few examples:

Screencast [youtube] of the Danish Radio Demo

Screencast [youtube] of the Radiolab Demo.

Quick and simple demo I put together.

Demos by others include: Minnesota Public Radio Demo, Voxallead News and Liris Interactive Transcript

Using Hypertranscripts we can:

• Navigate through the media via the text.
• Select and play parts of the media by highlighting the relevant parts of the text.
• Share parts of the media creating URLs that contain start and end points.
• Make our media more discoverable (via search engines)
• Make our media more accessible.
• More easily visualize the media.
• Combine, embed and easily integrate media (using simple tools)

The Hyperaudio Pad

The Hyperaudio Pad is a tool to allow users to easily assemble spoken word based audio or video from Hypertranscripts.

It will allow users to create their media by simply copying and pasting text from existing transcripts into a new transcript. These Hypertranscripts link to the actual media that they transcribe, so when parts of the transcripts are copied the references to their associated media are also copied.

Note that the media itself is not copied, just references to parts of it, the result is mash-up of video or audio and an associated representative transcript.

The tool will be :

• Web based, cross-browser, cross-platform (work on mobile devices) and open source.
• Simply presented, taking cues from minimalist distraction-free tools such as iAWriter.
• Intuitive and extremely easy to use, taking advantage of the text editing paradigm.
• Extensible - written in such a way that it can easily be built upon.

The interface should be clear and minimal, the user can search for and choose from existing hypertranscripts, open them, play them and copy parts of them to their ‘document’ all from within the same page.

Edit Mode

Sketch of the Hyperaudio Pad in Edit Mode

Playback Mode

Sketch of the Hyperaudio Pad in Playback Mode

[video to come]

Who is this tool for?

• Newsroom journalists who need to very quickly assemble media from different existing sources and perhaps adapt the result as news unfurls.
• Podcasters, citizen journalists and mediabloggers who want a free and easy way of putting together media programs or newscasts.
• Anyone who wants their resulting media programs to include a transcript for increased accessibility and visibility.
• Anybody who is happy for their results to be taken apart, re-combined or referenced by others.

How does it work ?

Hypertranscripts are essentially defined in HTML. The simplest possible example for the use by a tool such as the Hyperaudio Pad could be:

WORD-FROM-THE-TRANSCRIPT

Example:

burgeoning

Note, we can add to this format should we wish to include meta and other data, for example :

WORD-FROM-THE-TRANSCRIPT

(from Julien Doran’s MetaFragment proposal document)

Transcripts will contain a series of these marked up words each individual word containing enough data to describe it’s associated media and at which time in the media it occurs.

Note that copying the Hypertranscript into any text editor will result in valid HTML and so can be used in other tools and applications external to the Hyperaudio Pad.

It is likely that the user’s resulting transcript will reference several different pieces of media and so it may beneficial for the application to ‘load’ and possibly cache the various media sources in advance for smoothest possible playback. We can achieve this by parsing the transcript after every save.

Future developments could include a simple scripting language that users can insert between transcripts to make transitions smoother and even the possibility to add background music or sound effects for example:

[fade out over 5 seconds]

or

[background fade in 'threatening music' over 3 seconds]

and then at the bottom

[reference 'threatening music' http://soundcloud.com/some.mp3]

You may say I’m a Dreamer

In order to start using the Hyperaudio Pad to its full potential we first need our media to be transcribed and to do this we need to make tools that make this easier and if possible free. However I believe that hypertranscripts deliver so many benefits that the incentive to transcribe media is high.

The Hyperaudio Pad is just one of many tools that could be built on the underlying Hypertranscript platform. To build this platform we should collaborate with others to :

• establish a standard markup for transcripts
• make the platform easy to build upon and enhance as new technologies become viable
• create a community with a pioneering spirit
• develop an ecosystem to facilitate the creation of a toolkit

During the course of this project it’s been a pleasure to find many other participants interested in the general theme of describing/transcribing media and utilizing the result. Over the last couple of weeks I have been collaborating with Julien Dorra, Samuel Huron, Nicholas Doiron and Shaminder Dulai. The excitement has been palpable and although only communicating virtually it felt like we were sparking off each other.

So I guess at least as far as this dream is concerned, I’m not the only one.