HTML5
Audio and Video

Lessons from the Trenches



mark boas (@maboa)

jPlayer.org

happyworm.com

Better Software, Florence, June 2011

Why HTML5 audio and video?

  • ☆ open standards
  • ☆ transparency
  • ☆ better page integration
  • ☆ better integration with other elements

Broad browser support

  • ☆ Android 2.3 +
  • ☆ BlackBerry 6
  • ☆ Firefox Mobile
  • ☆ iOS4 +
  • ☆ Opera Mini / Mobile
  • ☆ Chrome 10 +
  • ☆ Firefox 3.6 +
  • ☆ Internet Explorer 9
  • ☆ Opera 10 +
  • ☆ Safari 5 +

Fallbacks

For non-supporting browsers you can use a fallback or polyfill.


Simplest possible fallback :

					<audio src="media/Bubble.ogg" controls>
						<a href="media/Bubble.ogg">grab Bubble</a> 
					</audio>
				

A bit more code ...

Video example :

					<video width="480" height="270" 
					src="media/Big_Buck_Bunny_Trailer.ogv" 
					preload controls poster="images/elvis.png">
					</video>

HTML5 audio and video spec - some content attributes

  • ☆ src - url of media
  • ☆ autoplay
  • ☆ loop
  • ☆ controls - display the controls
  • ☆ muted
  • ☆ preload |none|metadata|auto|
  • ☆ type
  • ☆ height / width (video only)

HTML5 audio and video spec - some more attributes

  • ☆ currentTime
  • ☆ volume
  • ☆ muted
  • ☆ duration

HTML5 audio and video spec - some events

  • ☆ play
  • ☆ pause
  • ☆ ended
  • ☆ progress
  • ☆ timeupdate

Audio codec browser support

Browser Ogg MP3 AAC WAV
FireFox 3.6 +
Safari 5 +
Chrome 6 + ✓*9 +
Opera 10.5 +
Internet Explorer 9
iOS3 +
Android 2.3 +
			<audio controls>
			   <source src="elvis.mp3"  type='audio/mpeg; codecs="mp3" ' >
			   <source src="elvis.oga" type='audio/ogg; codecs="vorbis" ' >
			</audio>

Note : jsmad.org is an mp3 decoder in JS for Firefox4 + & Chrome10 +

Video Codec browser support

Browser Theora Vorbis Ogg H.264 AAC MP4 WebM
FireFox 3.5 + 4 +
Safari 3.0 +
Chrome 5.0 + 6.0 +
Opera 10.5 + 10.6 +
Internet Explorer 9.0 + 9.0 +VP8 codec
iOS 3 +
Android 2.0 + 2.3 +
			<video width="320" height="240" controls>
			  
			  
			  
			</video>

A Note about Safari

Safari relies on Quicktime to handle media. This means that if Quicktime isn't installed (Windows), Safari won't play web based audio/video.

Also, issues exist that mean audio will intermittently fail to play in Snow Leopard 10.6.3 through 10.6.7. Windows Safari remains unaffected.

happyworm.com/blog/2010/08/17/safari-requires-quicktime-for-html5-based-media/

isflashdeadyet.com/tests/safari-html5/

Test your browser

Get Your Mime Types Straight

.htaccess examples


  • ☆ AddType audio/mpeg .mp3
  • ☆ AddType audio/mp4 .m4a
  • ☆ AddType audio/ogg .ogg
  • ☆ AddType audio/wav .wav
  • ☆ AddType video/ogg .ogv
  • ☆ AddType video/mp4 .mp4
  • ☆ AddType video/webm .webm
  • ☆ AddType video/mp4 .m4v

Containers

Audio

  • ☆ OGA : Vorbis audio
  • ☆ MP4 : AAC

Video

  • ☆ WebM : VP8 video and Vorbis audio
  • ☆ OGV : Theora video and Vorbis audio
  • ☆ MP4 : H.264 and AAC

Encoding

Better results with Constant Bitrate. 44.1kHZ works well.

Tools :

Streaming

Only shoutcast and icecast servers supported just now.

Issues with older (HTML5) browsers esp on Linux.

MP3 streams seem to work better. M4A (AAC) have problems.

trygve-lie.com/blog/entry/html_5_audio_element_and

Buffering and Seeking

Autobuffer replaced by preload.

Preloading not supported in all older HTML5 browers (Opera 10/Firefox 3.6).

Buffered not always supported (Opera, PlayBook).

Not all HTML5 browsers allow byte-range seeking. Safari (Windows). You can still seek within the downloaded content, it downloads from the start until the end, (as does Flash).

Firefox 4 does not always give the final progress event when buffered == 100%.

When dealing with timeranges figuring out the % loaded can be difficult.

In general, due to incomplete implementations/specifications (at the time) it is difficult to create a loading progress bar that works on all browsers.

Using HTML5 audio for games / sound effects

Not easy. May not produce satisfactory results.

Issues :

  • ☆ looping is not reliable on all browsers
  • ☆ latency can be an issue
  • ☆ some browsers deal with small clips badly

Possible solution : audio sprites

remysharp.com/2010/12/23/audio-sprites/

Flash Free Audio (Opera Blog)

Detection - Part I

What and how can we detect support for audio/video, codecs and methods ?

canPlayType method returns probably, maybe or empty string. (It won't return "yes" - a MIME type alone isn't enough for the browser to say.)

Example 1 - detecting support for the media element

		var myAudio = document.createElement('audio');
		alert(!!myAudio.canPlayType); // true or false

Warning canPlayType used to return "no". Older browsers that haven't been updated may still return "no".

Detection - Part II

Example 2 - detecting codec (more specific = better)

		var myAudio = document.createElement('audio');
		alert(myAudio.canPlayType('audio/mp3')); //ok
		alert(myAudio.canPlayType('audio/mpeg; codecs="mp3"')); // better

Example 3 - detecting buffered property

		alert(myAudio.buffered === "object");

Warning Opera has a media.buffered object, but it does nothing. myAudio.buffered.length > 0 helps detect actual use.

(jplayer.org/HTML5.Media.Event.Inspector)

Mobile

Mobile browsers can act differently because devices have :

  • ☆ metered internet bandwith
  • ☆ limited cpu, memory and power (battery)
  • ☆ prioritised hardware controls

Watch Out!

  • ☞ autoplay will often not work
  • ☞ volume changes through web api may have no affect
  • ☞ cannot play more than one file simultaneously
  • ☞ video media may be forced into a different window*

*tablets tend to play in page, while phones will play video full screen. This full screen player cannot be closed without user interaction.

Quick note on Android

Android 2.2 <video> only.

Android 2.3 <video> and <audio>.

Android also requires metadata to load before playing (otherwise you need to reset src).

Advanced Audio APIs
(Going Over The Top)

Firefox 4 + : Audio Data API
wiki.mozilla.org/Audio_Data_API

Chrome 10 + : Web Audio API
chromium.googlecode.com/svn/trunk/samples/audio/


Advanced APIs let you react to and manipulate the audio data.

But they have different APIs, as a standard has yet to be established.

However libraries like audiolib.js github.com/jussi-kalliokoski provide a common API.

What's Around the Corner?

WebRTC - Real-time Audio and Video Communication.
sites.google.com/site/webrtc/

Device API - Access to microphone and camera. APIs
w3.org/2009/dap/

Audio API Enhancement and Standardisation -
w3.org/2011/audio/

Media Fragments - Download and link to fragments of media
w3.org/2008/WebVideo/Fragments/WD-media-fragments-spec/

Media Annotations - Add related data to media
w3.org/2008/01/media-annotations-wg.html

WebVTT - Mark up external text track resources
whatwg.org/specs/web-apps/current-work/webvtt.html

Libraries can help
("Medic!")

A lot of different audio and video libraries out there.

Some free, some less free.

jPlayer jPlayer.org is completely free (GPL/MIT), has a large community and is regularly maintained.

It allows you to make both audio and video players that look the same in every browser.

An interesting library that we've had fun using is Popcorn.js. popcornjs.org Popcorn allows you to trigger events at certain points in the media.

Further Reading

Thank you!

@maboa - jPlayer.org

happyworm.com/blog