Audio & Video on the Web

It's all happening!

Web Native Audio & Video - The Stats

• ◼◼◼◼◼◼ Supports
• ◼ Mixed
• ◼ Does Not Support

source: statcounter.com

Simple Example ( Markup )

  <video controls >
    <source src="elvis.mp4" type="video/mp4">
    <source src="elvis.webm" type="video/webm">
  </video>

But what about IE8 ??!

  <video controls >
    <source src="elvis.mp4" type="video/mp4">
    <source src="elvis.webm" type="video/webm">
    <!-- add your fallback solution here -->
  </video>

Simple Example ( JavaScript )

  var audio = document.createElement('audio');
  audio.setAttribute("src","elvis.mp3");

Establishing Support

  canPlayType("audio/ogg")

Which reminds me ...

You may have to touch the server.

Video MIME Types / Content Types

Add .htaccess or similar

• AddType video/ogg .ogv
• AddType video/mp4 .mp4
• AddType video/webm .webm
• AddType video/mp4 .m4v

Audio MIME Types / Content Types

Add .htaccess or similar

• AddType audio/mpeg .mp3
• AddType audio/mp4 .m4a
• AddType audio/ogg .ogg
• AddType audio/wav .wav

OMG how many formats are there?

almost ...

WebM Support

MP4 Support

Audio Codec Support (MP3, OGG)

Almost identical situation as video.

In general, Mozilla deciding to support whatever the underlying OS supports, Opera moving to Blink (will likely support the same codecs as Chrome).

Libraries and other Fallbacks

You can use jPlayer and other libraries to help you deal with browser inconsistencies.

They can allow you to :

• Achieve a consistent player in all browsers
• Fallback to Flash for unsupported codecs

Video - Going Deeper

You can use Canvas to manipulate video.

Canvas allows you access to the pixel data of each frame.

Essentially you can grab each frame, manipulate it and re-draw it.

Video Fingerprint Demo

Video - Seriously.js

Seriously.js is a real-time, node-based video compositor for the web. Inspired by professional software such as After Effects and Nuke, Seriously.js renders high-quality video effects, but allows them to be dynamic and interactive.

https://github.com/brianchirls/Seriously.js

seriously.org

The Fly in the Ointment
Same-origin Policy

"The same-origin policy restricts how a document or script loaded from one origin can interact with a resource from another origin."

When trying to access video from Canvas, JS and video must be on same domain. (Unless CORS enabled).

Audio - Going Deeper

You can create a WAV file on the fly by supplying Base 64 encoded data directly to the source.

Speak.js is a great example of what you can do with this approach.

Audio - Going Deeper
Advanced Audio APIs

In the beginning ...

• Audio Data API (Firefox 4+)
• Media Stream API
• Web Audio API (WebKit)

Audio - Going Deeper
Advanced Audio Libraries

• audiolib.js
• KievII
• Howler.js

There Can Be Only One!

And the Winner is
The Web Audio API

The Web Audio API is a high-level JavaScript API for processing and synthesizing audio in web applications.

It takes a node based approach, each node performs an audio function and connected together to define the overall audio rendering.

It can be as simple as this ...

Source : Getting Started with Web Audio API (HTML5Rocks)

or as complex as this ...

Source : Web Audio API spec (w3.org)

Web Audio API Support

Web Audio API - Demo 1
Three Audio

acko.net/files/three-audio/

Web Audio API - Demo 2
Perceptive Media

FutureBroadcasts.com

Audio and Video - Going Deeper
Transcripts break audio out of its black box

And make it ...

• Navigable
• Searchable
• Shareable

Audio and Video - Going Deeper
Hyperaudio

Audio and Video - Going Deeper
Technology

• Popcorn.js ↬
• jPlayer ↬
• jQuery ↬ / Zepto.js (soon)↬

Popcorn.js

• ✔ Light
• ✔ Modular
• ✔ Time based events built in
• ✔ Active community
• ✔ Library independent
• ✔ IE8+ compatible to an extent

jPlayer

• ✔ Light
• ✔ Modular
• ✔ Skinnable
• ✔ Active community
• ✔ Flash fallback built in
• ✔ IE6+ compatible

Popcorn.js + jPlayer

A winning combo?

• support IE8 using a custom player
• use one audio/video format
• masks browser differences

jPlayer Popcorn Player

Audio and Video - Going Deeper
Hyperaudio Pad

screen cam and github

The elephant in the room

How do we create word-aligned timed transcripts?

• Use third-party services
• By hand
• Both

Third party transcription services (paid)

• $ 3PlayMedia ↬
• $ Koemei ↬
• $ DotSub ↬
• $ Dragon Speech Engine ↬
• $ PlyMedia ↬
• $ Ramp ↬
• $ SpeakerText ↬
• $ VoiceBase (freemium) ↬

Third party transcription services (free)

• ☮ Amara (formerly Universal Subtitles) ↬
• ☮ CMU Sphinx ↬
• ☮ Shout Toolkit ↬
• ☮ Something that hasn't been made yet!

What's around the corner?

Lots of new media related web technology:

• WebRTC
• Web Speech API
• Media Fragments

WebRTC

A technology to facilitate P2P comms.

• MediaStream API accessing camera and mic
• PeerConnection API connecting media streams
• DataChannel API real-time P2P transfer of data

WebRTC

• Designed for browser-to-browser comms
• Handles live streaming of audio and video
• Facilitates the streaming of any type of data

Useful for ..

• Voice Calling / Video Chat
• P2P File sharing
• Games

WebRTC - MediaStream Demo

Webcam Toy

Opus Audio Codec

• Mandatory part of the WebRTC standard
• Support for both constant bit-rate (CBR) and variable bit-rate (VBR)
• Audio bandwidth from narrowband to fullband
• Dynamically adjustable bitrate, audio bandwidth, and frame size
• Good loss robustness and packet loss concealment (PLC)

ORBX.js, a downloadable HD codec

• Written in JS and WebGL
• No encumbered-format
• IP-blind runtimes
• 25% better compression than H.264
• Adaptive bit-rate while streaming

and (soon) floating point coding, better color depth, better intra-frame coding, a more parallelizable design

Today I Saw The Future Brendan Eich

Audio Codecs in JavaScript

• JSMad (MP3)
• ALAC.js
• FLAC.js
• AAC.js

Audiocogs.org

DRM and Watermarking

• Defend the Open Web: Keep DRM Out of W3C Standards (EFF.org)
• DRM in HTML5 is a victory for the open Web, not a defeat (arstechnica.com)
• Watermarks: A better DRM than DRM itself?
  (zdnet.com 2005)

*Digital watermarking aims record and expose the origin of the video.

Mobile is Getting Better

But watch out for older mobile browsers.

• Cannot autoplay
• May not be able to affect volume
• May not be able to play simultaneous audio
• Will most likely not preload

Live Streaming is Possible

• Pausing streams will cause audio to buffer. Reconnect to keep live.
• Streams can break, and the browser generally recovers, but detection and reconnection is faster.
• Only Flash can handle RTMP streams.
• Chrome does not like the ICY response headers (often used by SHOUTcast). AAC will break.

Media Fragments

The three main parts are :

• Spatial Dimension - eg #xywh=160,120,320,240
• Temporal Dimension - eg #t=9,20
• Track - eg #track='audio'

Web Speech API

• An API for speech based input
• Currently Chrome only
• Server based

Chrome Only Demo

Resources

• HTML5 Audio — The State of Play (html5doctor.com)
• video + canvas = magic | HTML5 Doctor (html5doctor.com)
• Getting Started with Web Audio API (html5rocks.com)
• Opus Interactive Audio Codec (opus-codec.org)
• More thoughts on the Opus Codec (html5audio.org)
• Broadway: An H.264 Decoder in JavaScript (badassjs.com)
• Audio Codecs in JS (audiocogs.org)
• WebRTC (developer.mozilla.org)
• Real-time communication without plugins (html5rocks.com)
• Open Peer and the Frontiers of WebRTC (bocoup.com)
• What is Hyperaudio? (appsfuel.com)
• Making HTML5 Audio Actually Work on Mobile (open-lab.com)

Thanks for listening, you can find us...

• Mark Boas / @maboa
• Mark Panaghiston / @thePag
• hyperaud.io / @hyperaud_io
• jPlayer.org / @jPlayerOrg
• happyworm.com / @happyworm

More info Hyperaudio Google Group