javascript
Drumbeat Demo - HTML5 Audio Text Sync
Last month I had the pleasure of travelling to Barcelona to participate in Mozilla’s Drumbeat festival (of which more details are to come).
I very much wanted to demo the capabilities of HTML5 audio and so set about creating a demo in keeping with the theme of the festival - ‘Learning, Freedom and the Web’. I ended up with a very rough prototype of a web app that synchronised audio to text, word for word, more accurately it provided an interface that allowed a person to synchronise the audio to text and then demonstrated a couple of things that were made possible once this synchronisation had taken place.
So what was possible?
The first thing I found was that once I had the timings I could easily create a mechanism to control the audio from the text. Now by clicking on individual words I could jump to the corresponding part of the audio, useful for navigating audio and also potentially as an aid to learning language. I took this a little further by allowing the user to highlight areas of the text and having just that part of the text played back, which was um, an interesting exercise*.
*This feature is very very experimental and needs some love.
Finally I added a bit of razzmatazz, tacking on ‘Image Overlay Mode’, which is really a text overlayed on image mode, but that was a bit too wordy. To achieve image overlay mode in my limited time, I used canvas, however I’m aware that CSS3 is probably a better fit for this type of ‘animation’.
The code is very crude - it really was just flung together in a desperate rush to get it working for the Drumbeat Science Fair, so I hesitate to say feel free to take it and do with it what you will. But please do consider all demos posted on our blog open source and dual licensed under the MIT and GPL licenses.
Once again the jPlayer library came in handy, providing a useful abstraction and ensuring that the solution works on various platforms.
The demo.
Instructions :
1. In Sync Mode - press play on the player and use the space bar or sync button to synchronise the words with the audio.
2. Switch to Playback Mode to see the words synched to the audio. Click on the words to play the audio from that point. Try selecting areas of text.
3. Hit Image Overlay Mode if you are so inclined and have a canvas enabled browser.
4. Try Hack mode if you want to adjust any part of the timings and/or words.
Alternatively watch a screen capture of the demo :
(Flash version coming soon, honest.)
Feedback and ideas on this demo are particularly appreciated. I’m also interested in possible uses (apart from karaoke
) and perhaps other projects that I could collaborate with to make something genuinely useful.
Thanks to @elmook, @f1lt3r, @sroux, @aulentina and others who gave feedback and encouragement. Special thanks to @bluetezza for the original idea.
Mark B
HTML5 Media, Seeking and the Buffered Attribute
It can be very exciting playing with new technologies, HTML5 media being a case in point. The spec is still evolving and although native audio and video have only been about for little over a year in any useable form, we are already seeing browsers makers pushing the envelope and developers rushing to create new libraries.
We aim to incorporate features into the jPlayer library as they become available. Recently we have been looking at browser’s ability to jump to a point in a track that has not yet downloaded. A seeking of sorts. All the major HTML5 supporting browsers allow this type of seek (with the exception of Safari for Windows), but at the moment it seems only Chrome and Safari (both mobile and desktop versions) have taken this a step further by implementing the buffered attribute, although Firefox 4 also does, it is still in beta.
The buffered attribute allows us to determine what parts of a media track have been buffered so that we can seek or skip directly to that part without the need to pause.
More info on the buffered attribute can be found in this article HTML5 video ‘buffered’ property available in Firefox 4
Mark B
Add a Stylish Audio Player to your Blog Without using Plugins
Admittedly, it’s a little crude, but it’s simple and it works.
If you’ve ever wanted to add an audio player to your blog post and didn’t want to go to the trouble of installing a specific plugin to do so, you can always embed jPlayer within an iFrame. The one-liner you need to achieve this would probably look a lot like this:
which could give you something like this :
Sure, it’s not that efficient, you may end up loading jQuery twice if it’s already included as part of your blog, but there’s the added advantage that the page will render progressively and you have a nice little sandbox to play in should you just want to alter the player.
I used to hate iFrames and shudder at their very mention, but I must admit I’m coming around to them.
Mark B
A Simple and Robust jQuery 1.4 CDN Failover in One Line
Google, Yahoo, Microsoft and others host a variety of JavaScript libraries on their Content Delivery Networks (CDN) - the most popular of these libraries being jQuery. But it’s not until the release of jQuery 1.4 that we were able to create a robust failover solution should the CDN not be available.
First off, lets look into why you might want to use a CDN for your JavaScript libraries:
1. Caching - The chances are the person already has the resource cached from another website that linked to it.
2. Reduced Latency - For the vast majority the CDN is very likely to be much faster and closer than your own server.
3. Parallelism - There is a limit to the number of simultaneously connections you can make to the same server. By offloading resource to the CDN you free up a connection.
4. Cleanliness - Serving your static content from another domain can help ensure that you don’t get unnecessary cookies coming back with the request.
All these aspects are likely to add up to better performance for your website - something we should all be striving for.
For more discussion of this issue I highly recommend this article from Billy Hoffman : Should You Use JavaScript Library CDNs? which includes arguments for and against.
The fly in the ointment is that we are introducing another point of failure.
Major CDNs do occasionally experience outages and when that happens this means that potentially all the sites relying on that CDN go down too. So far this has happened fairly infrequently but it is always good practice to keep your points of failure to a minimum or at least provide a failover. A failover being a backup method you use if your primary method fails.
I started looking at a failover solution for loading jQuery1.3.2 from a CDN some months ago. Unfortunately jQuery 1.3.2 doesn’t recognise that the DOM is ready if it is added to a page AFTER the page has loaded. This made making a simple but robust JavaScript failover solution more difficult as the possibility existed for a race condition to occur with jQuery loading after the page is ready. Alternative methods that relied on blocking the page load and retrying an alternative source if the primary returned a 404 (page not found) error code were complicated by the fact that the Opera browser didn’t fire the "load" event in this situation, so when creating a cross-browser solution it was difficult to move on to the backup resource.
In short, writing a robust failover solution wasn’t easy and would consume significant resource. It was doubtful that the benefit would justify the expense.
The good news is that jQuery1.4 now checks for the dom-ready state and doesn’t just rely on the dom-ready event to detect when the DOM is ready, meaning that we can now use a simpler solution with more confidence.
Note that importantly the dom-ready state is now implemented in Firefox 3.6, bringing it in line with Chrome, Safari, Opera and Internet Explorer. This then gives us great browser coverage.
So now in order to provide an acceptably robust failover solution all you need do is include your link to the CDN hosted version of jQuery as usual :
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.4/jquery.min.js"></script>
and add some JS inline, similar to this:
<script type="text/javascript">
if (typeof jQuery === 'undefined') {
var e = document.createElement('script');
e.src = '/local/jquery-1.4.min.js';
e.type='text/javascript';
document.getElementsByTagName("head")[0].appendChild(e);
}
</script>
You are then free to load your own JavaScript as usual:
<script type="text/javascript" src="my.js"></script>
It should be well noted however that the above approach does not "defer" execution of other script loading (such as jQuery plugins) or script-execution (such as inline JavaScript) until after jQuery has fully loaded. Because the fallbacks rely on appending script tags, further script tags after it in the markup would load/execute immediately and not wait for the local jQuery to load, which could cause dependency errors.
One such solution is:
<script type="text/javascript">
if (typeof jQuery === 'undefined')
document.write('script type="text/javascript" src="/local/jquery-1.4.min.js"><\/script>');
</script>
Not quite as neat perhaps, but crucially if you use a document.write() you will block other JavaScript included lower in the page from loading and executing.
Alternatively you can use Getify’s excellent JavaScript Loading and Blocking library LABjs to create a more solid solution, something like this :
<script type="text/javascript" src="LAB.js">
<script type="text/javascript">
function loadDependencies(){
$LAB.script("jquery-ui.js").script("jquery.myplugin.js").wait(...);
}
$LAB
.script("http://ajax.googleapis.com/ajax/libs/jquery/1.4/jquery.min.js").wait(function(){
if (typeof window.jQuery === "undefined")
// first load failed, load local fallback, then dependencies
$LAB.script("/local/jquery-1.4.min.js").wait(loadDependencies);
else
// first load was a success, proceed to loading dependencies.
loadDependencies();
});
</script>
Note that since the dom-ready state is only available in Firefox 3.6 + <script> only solutions without something like LABjs are still not guaranteed to work well with versions of Firefox 3.5 and below and even with LABjs only with jQuery 1.4 (and above).
This is because LABjs contains a patch to add the missing "document.readyState" to those older Firefox browsers, so that when jQuery comes in, it sees the property with the correct value and dom-ready works as expected.
If you don’t want to use LABjs you could implement a similar patch like so:
<script type="text/javascript">
(function(d,r,c,addEvent,domLoaded,handler){
if (d[r] == null && d[addEvent]){
d[r] = "loading";
d[addEvent](domLoaded,handler = function(){
d.removeEventListener(domLoaded,handler,false);
d[r] = c;
},false);
}
})(window.document,"readyState","complete","addEventListener","DOMContentLoaded");
</script>
(Adapted from a hack suggested by Andrea Giammarchi.)
So far we have talked about CDN failure but what happens if the CDN is simply taking a long time to respond? The page-load will be delayed for that whole time before proceeding. To avoid this situation some sort of time based solution is required.
Using LABjs, you could construct a (somewhat more elaborate) solution that would also deal with the timeout issue:
<script type="text/javascript" src="LAB.js"></script>
<script type="text/javascript">
function test_jQuery() { jQuery(""); }
function success_jQuery() { alert("jQuery is loaded!");
var successfully_loaded = false;
function loadOrFallback(scripts,idx) {
function testAndFallback() {
clearTimeout(fallback_timeout);
if (successfully_loaded) return; // already loaded successfully, so just bail
try {
scripts[idx].test();
successfully_loaded = true; // won't execute if the previous "test" fails
scripts[idx].success();
} catch(err) {
if (idx < scripts.length-1) loadOrFallback(scripts,idx+1);
}
}
if (idx == null) idx = 0;
$LAB.script(scripts[idx].src).wait(testAndFallback);
var fallback_timeout = setTimeout(testAndFallback,10*1000); // only wait 10 seconds
}
loadOrFallback([
{src:"http://ajax.googleapis.com/ajax/libs/jquery/1.4/jquery.min.js", test:test_jQuery, success:success_jQuery),
{src:"/local/jquery-1.4.min.js", test:test_jQuery, success:success_jQuery}
]);
</script>
To sum up - creating a truly robust failover solution is not simple. We need to consider browser incompatabilities, document states and events, dependencies, deferred loading and time-outs! Thankfully tools like LABjs exist to help us ease the pain by giving us complete control over the loading of our JavaScript files. Having said that, you may find that in many cases the <script> only solutions may be good enough for your needs.
Either way my hope is that these methods and techniques provide you with the means to implement a robust and efficient failover mechanism in very few bytes.
A big thanks to Kyle Simpson, John Resig, Phil Dokas, Aaoran Saray and Andrea Giammarchi for the inspiration and information for this post.
Related articles / resources:
3.http://aaronsaray.com/blog/2009/11/30/auto-failover-for-cdn-based-javascript/
4.http://snipt.net/pdokas/load-jquery-even-if-the-google-cdn-is-down/
5.http://stackoverflow.com/questions/1447184/microsoft-cdn-for-jquery-or-google-cdn
6.http://webreflection.blogspot.com/2009/11/195-chars-to-help-lazy-loading.html
7.http://zoompf.com/blog/2010/01/should-you-use-javascript-library-cdns/
HTML5 - The Revolution will not be Televised
The Seeds of Change
There’s a lot of buzz being generated about HTML5 just now and I for one welcome the discussion that it has provoked. I’d always kept half an eye on the ongoing controversy regarding whether it was better to use the XHTML or HTML 4.01 standard. To add fuel to this fiery debate, as HTML5 gained traction it was announced that work on XHTML 2 spec would be discontinued. Many people felt that this vindicated the HTML 4 camp’s arguments and that XHTML was dead. The truth of course was slightly more complicated as HTML5 can be reasonably presented as XHTML. Either way we now seem to have one standard to unite behind which brings us closer to designer’s HTML utopia, where markup can be written once and work across all browsers. I believe a critical point has been reached.
So what advantages does HTML5 offer? Well it basically provides an open framework for a richer user experience. To name a few features without creating a huge list; it supports audio, video, vector based graphics and animation, geolocation and drag and drop. Check out the spec for more info.
Browser vendors are only now starting to implement some of the HTML5 features within their latest and greatest releases. Safari 4, Firefox 3.5, Chrome 3 and even Opera 10 to a greater or lesser extent now support HTML5. Internet Explorer being the obvious ‘elephant in the room’. It’s true that the HTML5 spec has yet to be finalised and depending on who you believe will not be finalised until 2022, but it seems this is less important than it sounds. What we are starting to see is something relatively new, the web development community getting behind a standard and actively pushing it forward. Control lies, now more than ever in the hands of web developers, the end-users, if you like, of the standards. It might seem futile to adopt a standard before it is finished, especially given that probably less than 10% of the installed browser base is currently taking advantage of it in any meaningful manner, but it’s worth considering that Google have adopted HTML5 as the markup of choice for their up and coming Wave product and also consider that Webkit is now starting to support HTML5 and that it is the rendering engine used by Chrome, Android, Safari and PalmPre OS and presumably the recently announced Google OS.
Corporation and Community
It seems that it’s not so much about the corporations anymore, more about the community. Not all browsers support HTML5, so what does the community do about it? It creates ‘patches’ for these browsers. These patches are usually written in JavaScript and aim to introduce HTML5 compliance to browsers that don’t support it. HTML5 introduces many new aspects and behaviour and various authors are working on different aspects of the spec, Dean Edwards is making fantastic progress on making Web Forms 2.0 work across all browsers. Erik Arvidsson has done some great work creating a library for emulating the Canvas tag on Internet Explorer as have others, Jacob Rask is working on HTML5 CSS Reset and then there’s people like us who hope to make contributions to the smaller aspects of the HTML5 spec such as audio. This isn’t a unified effort, at least not yet. But a common binding force that unites them is that they are all Open Source solutions, so anyone could come along, bundle them together and create a comprehensive patch for any browser. Of course the more comprehensive the support the more complex things become and I imagine Internet Explorer 6 support is the worst case scenario.
The Fly in the Ointment
To diverge and talk about Internet Explorer 6 for a moment. IE6, to put it kindly, does things ‘in its own unique way’, and for this reason is the bane of many web designer’s lives. Some time ago I was experimenting with creating custom tags for IE6 and found out that although it is possible to implement them, it goes about this completely differently to any other browser. I’m guessing that being able to deal with custom tags is essential if you have to deal with tags that aren’t implemented in a particular browser. I’m not sure whether the current crop of ‘patches’ are supporting IE6 but I can certainly imagine that if they do, they’ve had to go around the houses to do so. IE6 unfortunately has a large install base inside many large corporations. Many companies rolled out intranet applications when IE6 was standard on the corporate desktop and so were targeted specifically to that one browser. It’s hard for a company to justify re-writing these applications to work on any other browser. The “if it ain’t broke - don’t fix it” mentality is a strong one, especially where spending money is concerned. While web sites and internet based applications continue to work, corporations will have no incentive to upgrade or provide another browser for surfing the web with. There are signs though that this will change in the near future. Whole movements have sprung up against IE6, large and established sites have discontinued support or are thinking seriously about dropping it. The IE6 legacy cannot go on for ever.
There is however a price to pay for using these ‘patches’. Undoubtedly browsers that don’t support HTML5 natively will run more slowly using the patches to support it. They do anyway you might counter - people use them just the same. JavaScript is light, it can be compressed, it can be cached, it can be hosted on CDNs. I personally don’t think this a real issue and if users find that it is, they can always upgrade. Accessibility is the other key issue that needs to be addressed, but I will leave that for another discussion.
May we Live in Interesting Times
If you think about it for a minute, what is going on now, happening right under our very noses is nothing short of a revolution, a seismic shift in power towards the community and away from the browser vendors, the consequences of which cannot be underestimated. The W3C has loosened its grip on the HTML specs and we now have the WHATWG community, and it appears that most browser makers are listening attentively to the new combined web design and standards community. It’s all about the community. It doesn’t seem to even matter when a spec will be finalised, significant chunks of it are being implemented now, people are using them and the community is developing for them. Slowly but very surely we are approaching web designer’s nirvana, where not only does a modern markup language incorporating many new and needed features finally exist, but importantly an environment is being created where designers have the possibility to implement these features once, knowing that in some way or another, they can make them work on all target browsers. All of this powered by the web development community who are finally taking control of their own destiny.
¡Viva La Revolución!
Further reading:
Previous Posts
- Flash vs HTML5 Video and the Codec thing
- Altrepreneurial vs Entrepreneurial and Why I’m going to Work with Al Jazeera
- HTML5 Audio APIs - How Low can we Go?
- Hyperaudio at the Mozilla Festival
- The Hyperaudio Pad - a Software Product Proposal
- Introducing the Hyperaudio Pad (working title)
- Accessibility, Community and Simplicity
- Build First, Ask Questions Later
- Further Experimentation with Hyper Audio
- Hyper Audio - A New Way to Interact
- P2P Web Apps - Brace yourselves, everything is about to change
- A few HTML5 questions that need answering
- Drumbeat Demo - HTML5 Audio Text Sync
- HTML5 Media, Seeking and the Buffered Attribute
- Add a Stylish Audio Player to your Blog Without using Plugins
Tag Cloud
-
Add new tag
AJAX
apache
Audio
band
buffered
Canvas
CDN
chrome
community
custom tags
firefox
flash
gig
HTC
HTML5
Hyper Audio
internet explorer
java
javascript
journalism
jPlayer
jQuery
jscript
LABjs
leopard
media
Mozilla
opera
opera mini
osx
P2P
Popcorn.js
poster
prototyping
rewrite
safari
Scaling
simplicity
SoundCloud
timers
tomcat
video
Web Apps
web design