The Worm Hole

Drumbeat Demo – HTML5 Audio Text Sync

Sunday, December 5th, 2010

Last month I had the pleasure of travelling to Barcelona to participate in Mozilla’s Drumbeat festival (of which more details are to come).

I very much wanted to demo the capabilities of HTML5 audio and so set about creating a demo in keeping with the theme of the festival – ‘Learning, Freedom and the Web’. I ended up with a very rough prototype of a web app that synchronised audio to text, word for word, more accurately it provided an interface that allowed a person to synchronise the audio to text and then demonstrated a couple of things that were made possible once this synchronisation had taken place.

So what was possible?

The first thing I found was that once I had the timings I could easily create a mechanism to control the audio from the text. Now by clicking on individual words I could jump to the corresponding part of the audio, useful for navigating audio and also potentially as an aid to learning language. I took this a little further by allowing the user to highlight areas of the text and having just that part of the text played back, which was um, an interesting exercise*.

*This feature is very very experimental and needs some love.

Finally I added a bit of razzmatazz, tacking on ‘Image Overlay Mode’, which is really a text overlayed on image mode, but that was a bit too wordy. To achieve image overlay mode in my limited time, I used canvas, however I’m aware that CSS3 is probably a better fit for this type of ‘animation’.

The code is very crude – it really was just flung together in a desperate rush to get it working for the Drumbeat Science Fair, so I hesitate to say feel free to take it and do with it what you will. But please do consider all demos posted on our blog open source and dual licensed under the MIT and GPL licenses.

Once again the jPlayer library came in handy, providing a useful abstraction and ensuring that the solution works on various platforms.

The demo.

Instructions :

1. In Sync Mode – press play on the player and use the space bar or sync button to synchronise the words with the audio.

2. Switch to Playback Mode to see the words synched to the audio. Click on the words to play the audio from that point. Try selecting areas of text.

3. Hit Image Overlay Mode if you are so inclined and have a canvas enabled browser.

4. Try Hack mode if you want to adjust any part of the timings and/or words.

Alternatively watch a screen capture of the demo :

mp4 | ogv

(Flash version coming soon, honest.)

Feedback and ideas on this demo are particularly appreciated. I’m also interested in possible uses (apart from karaoke ) and perhaps other projects that I could collaborate with to make something genuinely useful.

Thanks to @elmook, @f1lt3r, @sroux, @aulentina and others who gave feedback and encouragement. Special thanks to @bluetezza for the original idea.

Mark B