Audio & Video on the Web

It's all happening!

Web Native Audio & Video - The Stats


Simple Example ( Markup )

  <video controls >
    <source src="elvis.mp4" type="video/mp4">
    <source src="elvis.webm" type="video/webm">

But what about IE8 ??!

  <video controls >
    <source src="elvis.mp4" type="video/mp4">
    <source src="elvis.webm" type="video/webm">
    <!-- add your fallback solution here -->

Simple Example ( JavaScript )

  var audio = document.createElement('audio');

Establishing Support


Which reminds me ...

You may have to touch the server.

Video MIME Types / Content Types

Add .htaccess or similar

Audio MIME Types / Content Types

Add .htaccess or similar

OMG how many formats are there?

almost ...

WebM Support

MP4 Support

Audio Codec Support (MP3, OGG)

Almost identical situation as video.

In general, Mozilla deciding to support whatever the underlying OS supports, Opera moving to Blink (will likely support the same codecs as Chrome).

Libraries and other Fallbacks

You can use jPlayer and other libraries to help you deal with browser inconsistencies.

They can allow you to :

Video - Going Deeper

You can use Canvas to manipulate video.

Canvas allows you access to the pixel data of each frame.

Essentially you can grab each frame, manipulate it and re-draw it.

Video Fingerprint Demo

Video - Seriously.js

Seriously.js is a real-time, node-based video compositor for the web. Inspired by professional software such as After Effects and Nuke, Seriously.js renders high-quality video effects, but allows them to be dynamic and interactive.

The Fly in the Ointment
Same-origin Policy

"The same-origin policy restricts how a document or script loaded from one origin can interact with a resource from another origin."

When trying to access video from Canvas, JS and video must be on same domain. (Unless CORS enabled).

Audio - Going Deeper

You can create a WAV file on the fly by supplying Base 64 encoded data directly to the source.

Speak.js is a great example of what you can do with this approach.

Audio - Going Deeper
Advanced Audio APIs

In the beginning ...

Audio - Going Deeper
Advanced Audio Libraries

There Can Be Only One!

And the Winner is
The Web Audio API

The Web Audio API is a high-level JavaScript API for processing and synthesizing audio in web applications.

It takes a node based approach, each node performs an audio function and connected together to define the overall audio rendering.

It can be as simple as this ...

Source : Getting Started with Web Audio API (HTML5Rocks)

or as complex as this ...

Source : Web Audio API spec (

Web Audio API Support

Web Audio API - Demo 1
Three Audio

Web Audio API - Demo 2
Perceptive Media

Audio and Video - Going Deeper
Transcripts break audio out of its black box

And make it ...

Audio and Video - Going Deeper

Audio and Video - Going Deeper



Popcorn.js + jPlayer

A winning combo?

jPlayer Popcorn Player

Audio and Video - Going Deeper
Hyperaudio Pad

screen cam and github

The elephant in the room

How do we create word-aligned timed transcripts?

Third party transcription services (paid)

Third party transcription services (free)

What's around the corner?

Lots of new media related web technology:


A technology to facilitate P2P comms.


Useful for ..

WebRTC - MediaStream Demo

Webcam Toy

Opus Audio Codec

ORBX.js, a downloadable HD codec

and (soon) floating point coding, better color depth, better intra-frame coding, a more parallelizable design

Today I Saw The Future Brendan Eich

Audio Codecs in JavaScript

DRM and Watermarking

*Digital watermarking aims record and expose the origin of the video.

Mobile is Getting Better

But watch out for older mobile browsers.

Live Streaming is Possible

Media Fragments

The three main parts are :

Web Speech API

Chrome Only Demo


Thanks for listening, you can find us...

More info Hyperaudio Google Group