Audio/Video 2.0 Update – Media Modal

The latest major updates to Audio/Video before Beta were related to editing shortcodes in the medial modal. TinyMCE placeholders for audio and video have been added like so:

00-placeholders

When you click the placeholder, the media modal will now be launched. And when you get there, there are some new ways to manage HTML5 playback without writing anything by hand.

01-audio-details

Add Multiple Sources

MediaElement.js provides cross-browser support for many extensions by providing Flash and Silverlight files that will bridge the browser support gap when necessary. Ideally, as many browsers as possible would be using native HTML5 playback. To ensure this, you need to specify multiple versions of your files. Since 3.6, the audio and video shortcodes have supported this by allowing you specify multiple extensions as attributes: mp3="yolo.mp3" ogg="yolo.ogg" wma="yolo.wma", etc.

A quick and easy way to encode your files is to use FireFogg (works in Firefox only). Here are some instructions for audio and video files:

HTML5 Audio

  • Click “Make web video”
  • Select a File (your iTunes files on a Mac are in: ~/Music/iTunes/iTunes Media/Music – Pick a tune!)
  • Click “Advanced Options”
  • Uncheck Video, make sure Audio is checked
  • Format: Ogg (Theora/Vorbis)
  • Set quality to 10.0
  • Optionally add metadata
  • Click “Encode” – make sure to change the extension in the file prompt from “.ogv” to “.ogg”

HTML5 Video

  • Click “Advanced Options”
  • Make sure Audio and Video are both checked
  • Format: choose Ogg (Theora/Vorbis) or WebM (VP8/Vorbis)
  • Optionally add metadata
  • Click “Encode”
  • (Repeat these steps for the format you didn’t select this time)

There is now a workflow to make adding these extra sources to shortcodes easy:

02-audio-add-ogg

Multiple sources are specified now. Make sure to click blue “Update” button to save your shortcode back to the editor.

03-audio-sources

Video Workflow

Here is a video workflow, assuming your shortcode has no attributes and you click it:

04-video-empty

Add each video source:

05-video-add-source

Optionally add a poster image:

06-video-sources

Subtitles

If you’re feeling CRAZY, add some Subtitles! Here’s a post about them. They’re pretty cool. Probably make sure your web server is serving .vtt files as text/vtt.

07-webvtt

Add your subtitles using our easy workflow:

08-video-add-subtitles

When you’ve added them, MediaElement knows to support them out of the box:

09-video-click-subtitles

Boom. Subtitles while your video plays:

10-video-show-subtitles

When you add your subtitles, if will show you a list of “track” elements you have added, you will still need to set your language manually – all will default to English. The idea is that you can add a track element per language. Tracks get stored as the body of your video shortcode.

tracks

Testing + Tweaks

Now comes the time for testing and refining, while fixing esoteric bugs. PLEASE HELP! Put your UI + UX hats on, if you wear that kind of hat.

#audio, #media, #mediaelement, #video

Audio / Video 2.0 Update – Playlists

Previously: https://make.wordpress.org/core/2014/01/29/audiovideo-2-0-update/

It has been a slow and steady burn of commits related to media and necessary for playlists:
[27059], [27063], [27907], [27100], [27127], [27209], [27212], [27213], [27214], [27215]

A few items for conversation….

Thumbnails for Audio and Video

We have been parsing the ID3 tags in audio and video files on upload since 3.6. ID3 tags, in many cases, contain the binary bits for an image related to the item (album cover, still, etc). On upload, if your theme and the relevant pseudo post type for attachment:audio or attachment:video registered support for thumbnails AND your current theme supported it, we would upload the image bits and attach to the media item as its featured image.

So basically it was a hidden feature, because it required you to jump through hoops to turn it on.

add_post_type_support( 'attachment:audio', 'thumbnail' );
add_post_type_support( 'attachment:video', 'thumbnail' );
add_theme_support( 'post-thumbnails', array( 'post', 'attachment:audio', 'attachment:video' ) );

On top of that, if you switch themes, and the theme doesn’t support thumbnails for audio or video, the images will no longer appear alongside the media on the Edit Media page. Weird.

Playlists are best enjoyed with images, videos are best enjoyed with poster images. Soundcloud is doing some awesome things with images in their embeds – see a few on the homepage here: http://highforthis.com. Moral of the story: I think this support should be on by default. Alternately, we could add that code to every default theme’s functions.php, but then what if you switch themes…

Playlist UI

As I mentioned previously, the UI for playlists needs to be:
1) Adaptable
2) Extensible
3) Generic

Translation: it needs to show up in a theme without much drama, inherit the style of the theme, respond the theme’s $content_width, all while allowing you to completely override this behavior. So, what I have is an ultra generic design controlled by settings in the media modal:

Playlist Settings
I have tested this in the last 5 default themes:

Twenty Fourteen

Screen Shot 2014-02-20 at 3.47.59 PM

Twenty Thirteen

Screen Shot 2014-02-20 at 3.48.58 PM

Twenty Twelve

Screen Shot 2014-02-20 at 3.49.22 PM

Twenty Eleven

Screen Shot 2014-02-20 at 3.50.10 PM

Twenty Ten

Screen Shot 2014-02-20 at 3.50.49 PM
I would like to drop this code in soon, but I wanted to give an opportunity for feedback. All of this can easily be iterated upon once it goes in.

Documentation of 3.5 Media Code

This is ongoing – there has been a lot of code churn in the Backbone code, by myself and others, I’ll be picking this back up once that settles down.

#audio, #media, #playlists, #video

Audio / Video 2.0 – codename “Disco Fries”

Some history:

I wanted to do a Make post on my wants for Audio / Video in 3.9 to solicit feedback and spark some discussion about what the community wants / needs / doesn’t want / doesn’t need. Adding audio / video in 3.6 was a great first step, but there are some things we can do to continue to modernize Media and give our huge user base even more ways to display and manage their content. We can also make some changes that help developers navigate the new world of MediaElement.js, Backbone, and Underscore.

First Things First: New Icons

#26650 Replace media file type icons with Dashicons

There are some lingering icons in the admin that don’t look as pretty as their MP6ify’d brethren

Document the “new” Media code introduced in 3.5

In 3.5, we overhauled Media. @Koop produced some beautiful code, and a LOT of it. Raise your hand if you’ve ever dived in and tried to program against it? Raise you hand if you understand how all of it works? Me neither. As a community, we need to help each other learn what it is and what it does. Documentation will go a long way in getting us all on the same page. Do we have a documentation standard for JS? We need one. While this isn’t the easiest place to start, it is a necessary one. I would be happy to spend time on this, as I have spent many hours recently reading the code and learning how it works. The main files: media-editor.js, media-views.js, media-models.js

Support subtitles for Video

#26628 Use the content of a video shortcode when provided.

This ticket speaks for itself, and already has a patch.

Generate audio/video metadata on-demand

#26825 Add ability to generate metadata for audio / video files on-demand

Add “Playlist” and “Video Playlist” shortcodes

#26631 Add a “playlist” shortcode

Adding inline players for audio and video was a great first step. How do I add music to my site? Just upload an MP3 and drop the URL on a line by itself. Done. Or use the audio shortcode. This works most of the time, but can be a little clunky if you want to share an album of your tunes. MediaElement doesn’t “support” playlists out of the box, but MediaElement is JavaScript, and with JavaScript and little UI elbow grease, we can EASILY support playlists.

My ticket already contains a patch, but is still considered a work in progress. I think the playlist shortcode should produce markup that does the following:

  • Works out of the box with any existing theme: the HTML should be semi-bulletproof. Many of the Player libraries make heavy use of DIVs instead of items that might be overridden easily with CSS: LIs and the like.
  • Gives the developer total control if they want to write their own implementation
  • Exposes enough data to the page so the themer/dev can make their own decision regarding display of album cover, track meta, captions, etc.

My current implementation drops data onto the page for each playlist inline. A wrapper div “.wp-playlist” will have a script tag in it with type=”application/json”. I do this so that if ‘wp-playlist.js’ is unenqueue’d, the developer still has the data necessary to write their own implementation. The data is reachable in JS like so:

var data = $.parseJSON( el.find('script').html() );

My current UI for playlist is a basic one, and uses Backbone Views to render the tracklist on load and update the “current” playing track view. There are 2 camps of people when it comes to “JS on the frontend” – one who doesn’t like it (others) and one who says “who cares” (me). One of the reasons I am posting this at the beginning is so we can flesh out issues like this.

Abstract Gallery logic into “Collection” logic that Galleries, Playlists, etc can use with minimal registration code

I have already done a first pass at this in the playlist shortcode patch. It goes like this: a “gallery” is really a “collection” of attachments of type “image.” A “playlist” is really a “collection” of attachments of type “audio.” So they should extend or be instances of a “collection”-type class. Currently, the Gallery code has to be dupe’d. By abstracting this code, Gallery, Playlist, Video Playlist, + any other “collection” of media type can be registered minimally.

Other Ideas

  • In our playlist JS code, emit events that others can hook into – maybe a video playlist is: News clip, ad, news clip, ad, etc. When emitting events before / after an ad, the dev could disable next/prev buttons
  • Make a playlist embeddable on other sites via an iframe or embedded markup
  • Register an endpoint for audio / video that will expose the “embed code” via oEmbed

Thoughts?

#audio, #backbone, #jquery, #media, #mediaelement, #underscore, #video

Audio / Video support in Core

Post Formats are a big feature in WordPress 3.6. What you may not know is: there is now native support for Audio and Video in core! There has been great support for embeds by way of WP_Embed and oEmbed providers for a while, but, if you wanted to play an MP3 from your Media Library, you had to install a plugin. Supporting audio and video in core gives bands, podcasters, vloggers, et al the ability to easily and beautifully expresses themselves through sounds and moving pictures without using an external service.

How does this work?

At the core of the experience is the fantastic library, MediaElement.js. MediaElement is the facade layer that gives us maximum file support and cross-browser compatibility. While some libraries require a Flash-only solution to make your media work cross-environment, MediaElement lets you use HTML5 audio / video tags in every browser, and, only when necessary, will use a Flash or Silverlight plugin in the background to make incompatible media work. Translation, things like this: <audio> tag works in old IE, Windows Media files work in Chrome.

MediaElement uses the same HTML markup, regardless of playback implementation, and you can use CSS to skin the players.

Shortcodes

MediaElement’s great, but we don’t want to be locked in to one external library forever. Instead of using MediaElement-specific markup everywhere, we expose audio and video markup through shortcodes: [audio] and [video].

For the following scenarios:

  • I have an old post that has a video in the Media Library attached to it, and I want to use the new shortcode: [video]
  • I have the URL for a video, from the Media Library or external, that I want to play:
    [video src="video-source.mp4"]
  • I have a source URL and fallbacks for other HTML5-supported filetypes:
    [video width="600" height="480" mp4="source.mp4" ogv="source.ogv" webm="source.webm"]

Same goes for audio:

  • I have an old post that has an audio file in the Media Library attached to it, and I want to use the new shortcode: [audio]
  • I have the URL for an MP3, from the Media Library or external, that I want to play: [audio src="audio-source.mp3"]
  • I have a source URL and fallbacks for other HTML5-supported filetypes:
    [audio mp3="source.mp3" ogg="source.ogg" wav="source.wav"]

Shortcodes focus on the “what” and abstract the “how.” If you want to use a library that is not MediaElement, you can! Just look at what to filter: here

Embeds

There are also new embed handlers for audio and video. Using them is easy as dropping a media link on a line by itself in the editor:

http://my.mp3s.com/cool/songs/coolest.mp3

I like this song because it is really cool!

Works for both audio and video with URLs matching the allowed (and filterable) list of extensions (see: here and here)

Admin

Using the new post formats UI, it is even easier to get directly at the audio and video in your Media Library. When selecting, the media modal opens to your library, filtered by media type.

Metadata

In previous versions of WP, you could upload audio and video, but we were not generating metadata like we do for images. In 3.6, using the getID3 library, we are able to extract data from audio and video like cover art, song length, artist, album, song title, genre, codec, etc. It’s pretty great. We will soon be exposing more of this data in the admin as well, along with inline previews on the Edit Media page:

Themers

Themers can get in on the action, too, using structured-post-formats in their theme (Twenty Thirteen is a great place to look). The admin gives users flexibility when associating media with a post. the_post_format_audio() and the_post_format_video() will automagically retrieve and output your media in the front end.

#audio, #media, #mediaelement, #post-formats, #shortcodes, #video