How does this work?

At the core of the experience is the fantastic library, MediaElement.js. MediaElement is the facade layer that gives us maximum file support and cross-browser compatibility. While some libraries require a Flash-only solution to make your media work cross-environment, MediaElement lets you use HTML5 audio / video tags in every browser, and, only when necessary, will use a Flash or Silverlight plugin in the background to make incompatible media work. Translation, things like this: <audio> tag works in old IE, Windows Media files work in Chrome.

MediaElement uses the same HTML markup, regardless of playback implementation, and you can use CSS to skin the players.

Shortcodes

MediaElement’s great, but we don’t want to be locked in to one external library forever. Instead of using MediaElement-specific markup everywhere, we expose audio and video markup through shortcodes: [audio] and [video].

For the following scenarios:

• I have an old post that has a video in the Media Library attached to it, and I want to use the new shortcode: [video]
• I have the URL for a video, from the Media Library or external, that I want to play:
  [video src="video-source.mp4"]
• I have a source URL and fallbacks for other HTML5-supported filetypes:
  [video width="600" height="480" mp4="source.mp4" ogv="source.ogv" webm="source.webm"]

Same goes for audio:

• I have an old post that has an audio file in the Media Library attached to it, and I want to use the new shortcode: [audio]
• I have the URL for an MP3, from the Media Library or external, that I want to play: [audio src="audio-source.mp3"]
• I have a source URL and fallbacks for other HTML5-supported filetypes:
  [audio mp3="source.mp3" ogg="source.ogg" wav="source.wav"]

Shortcodes focus on the “what” and abstract the “how.” If you want to use a library that is not MediaElement, you can! Just look at what to filter: here

Embeds

There are also new embed handlers for audio and video. Using them is easy as dropping a media link on a line by itself in the editor:

http://my.mp3s.com/cool/songs/coolest.mp3

I like this song because it is really cool!

Works for both audio and video with URLs matching the allowed (and filterable) list of extensions (see: here and here)

Admin

Using the new post formats UI, it is even easier to get directly at the audio and video in your Media Library. When selecting, the media modal opens to your library, filtered by media type.

Metadata

In previous versions of WP, you could upload audio and video, but we were not generating metadata like we do for images. In 3.6, using the getID3 library, we are able to extract data from audio and video like cover art, song length, artist, album, song title, genre, codec, etc. It’s pretty great. We will soon be exposing more of this data in the admin as well, along with inline previews on the Edit Media page:

Themers

Themers can get in on the action, too, using structured-post-formats in their theme (Twenty Thirteen is a great place to look). The admin gives users flexibility when associating media with a post. the_post_format_audio() and the_post_format_video() will automagically retrieve and output your media in the front end.

UI / Administration

Main ticket: #19570

@melchoyce refreshed the layout wireframes and we ran them by @jenmylo for a gut check. The wireframes linked are for reference to see progression – quite a bit will be different, notably that the collapsing content editor will not actually do that and we probably won’t be able to get so adventurous with the status format. We are going to do the following and should aim for a commit addressing a good portion of this in the next several days:

• Remove the separate meta and field display for the image and gallery formats. Though it’s nice to guide a user through via UI, the image format did not really pose any issues in the user testing, and the whole uncoupled multiple gallery shift in 3.5 makes a separate field less sensible. With good parsing, just the content area should be enough for both. Image also still has the featured image metabox, which is confusing. One thing that we should try doing is having the “Add Media” button open the modal in the “Create Gallery” panel by default if in the gallery format. Knowing about galleries at all was an issue in user testing, not related to the post format.
• @lessbloat proposed a dropdown selector, in these mockups placed by the title: http://make.wordpress.org/core/2013/02/22/post-formats-ui-update-221/#comment-8182. Anything is better than tabs for this interaction, and I’m sorry that I used them even temporarily as they clearly became a distraction, so we are going to keep the dropdown idea, but try putting it where the screen icon is, per suggestion from Jen.
• @melchoyce is making some changes to the icons she’s been working on: http://core.trac.wordpress.org/ticket/19570#comment:78. The standard post icon will be the pin, the gallery icon will become the image icon (since images are not necessarily photos), and the gallery icon should be something like the image icon with a stack behind it.
• @lessbloat is going to experiment with what can be done to make TinyMCE less intrusive in some formats where it’s not as much of a focal point, such as quote, audio, and video. Have warned him about what it means to re-initialize TinyMCE, but teeny mode seems that it would be ideal, and perhaps hide the Add Media button / move it into the TinyMCE toolbar instead, especially for the audio/video formats, so it doesn’t confuse the user as to what they should be using for which.
• Title should be somehow de-emphasized for some formats: aside, status, and image come to mind. A toggle to edit comes to mind, maybe like the way permalink slugs are edited. Titles need to be filled in based on content for those formats, a la P2. Might be nice to try to do something smart with image, like detecting an alt or title attribute for the fill-in, but at the very least, something better than (no title) or Auto Draft. This is less important for a second commit, but needs a body.
• Audio and video should have inline previews, whether it’s of oEmbed response, rendered HTML (sanitized per user cap, of course), or a media library item. We should hook up the media modal if possible, especially with the insert from URL panel. Related:Â #15490 (see farther below for more details on the ticket). Needs a body or bodies.
• Screen option: no longer sure we actually need this with the less obtrusive dropdown for format selection, but if we keep it, it should just hide the switcher.
• No-JS: probably means a select dropdown (which should be the non-visible input that is changing with JS on) with a submit button next to it that saves a draft with that format, which will thereby change the screen once it loads again. Not sure what that means for a published post, though. Ideas about execution in that regard are very welcome. Needs a body or bodies.
• Accessibility: need to check on ordering when tabbing through fields and make sure things like labels are done correctly. Also should make sure the switcher is accessible. Needs a body or bodies.

Other tickets and tasks:

• #23198 – TinyMCE class. @azaozz said he will take a look and get it polished and committed. Looking for it to be there on init as well as dynamically changing with the format. Hooray for editor styles
• #16047 – list table display/filtering. Once we have icons, we should drop those into place next to the post title, with each icon linking to filter the list table down to that format, and lose the column that was added. “Standard” (the pin) should not show.
• #23539 – previewing. Format should be dealt with the way category is (taxonomy); formats with meta fields are trickier to deal with given that post meta is not stored separately for revisions. Not sure where this stands.

Bonus: @ryelle is working on a dashboard widget for one-click entry into writing a post in a given format, which should help with discoverability. Have suggested that work on this be done as a plugin for a couple reasons – so patches don’t go stale and so that if it doesn’t quite make it to core for one reason or another (whether that’s functionality or decision that it’s not for core), it’s available as something usable for those who want it and/or for user testing and data collection.

Data

• #23347 — Theme fallback output. We need to remove gallery and image fallbacks along with removing them as separate meta. This needs some more work regarding more/excerpts and probably pagination as well – see http://core.trac.wordpress.org/ticket/23347#comment:41 for more details. Seems that placement before/after needs to be different per different formats – link should default to after as it is now, but audio, video, and quote should probably be before the_content. Needs a body.
• #23282 – HTML5 audio / video and #23673 — video / audio metadata. These need to drop in together, and we should do it soon and let it soak. We can test patches all day, but nothing beats getting it into trunk for a wider audience and set of environments. There are concerns about needing jQuery on the front, but in the last dev chat we decided to at least drop it in and let it soak. Needs lead dev eye and blessing for commit ASAP, please.
• #23726 — Show Audio and Video previews on Edit Attachment screen. Pretty much comes with the above.
• Provided that the link and quote formats continue to use a separate field for URL, they should probably be added to the pingback URLs. See #18506.
• Meta doesn’t delete, because it’s not passed in $_POST if empty. Likely not a particularly complicated fix.

Parsing

Tracking ticket: #23725

So many thanks to @wonderboymusic for leading the charge here. These need a commit very soon so that Twenty Thirteen can move on with practical usage. The biggest use cases are going to be image and gallery formats, as well as new theme, old data. Parsing functions will be available for use by devs, and in core should be used as fallbacks in the get_post_format_meta() function.

• #22960 — Extract images from content, attached to post, gallery, or arbitrary string. Has unit tests and patch. Needs testing and a lead dev eye and blessing.
• #23570 — Extract a URL from an arbitrary string, HTML, or the current $post. Has unit tests and patch. Needs testing and a lead dev eye and blessing.
• #23572 — Parse Audio / Video data out of arbitrary content or a post. Has patch, needs unit tests. Needs #23282 to go in first.
• #23625 — Parse chat as structured data. Has patch and unit tests, but needs to be scaled back and just allow for added formats (regex patterns) via plugins. Suggest that it just be split by lines, ignoring empties, with anything before the first colon becoming the label for that line (screen name, speaker, whatever you want to call it).

Documentation

• Will need screen help text, possibly/probably its own panel. If we keep the screen option for post formats, would be smart to hide the panel if toggled off.
• Codex will need attention. I know of http://codex.wordpress.org/Post_Formats. Seems that user documentation needs to be separated from theme dev documentation.

Otherwise related

• #15490 – oEmbed preview for insert from URL in media modal. We found in user testing that users go back to this panel for all sorts of things, including YouTube URLs and the like. There’s a working patch from @jtsternberg that could use a little finessing. The Ajax action is probably going to be helpful with audio/video inline previews on the edit post screen.

Our next steps for the actual building of the UI are:

• Hook up the media modal to the gallery field.
• Doing some preview magic for audio/video, also possibly hooking up the media modal.
• Adding icons for each selector (currently tabs) and some of the inputs.
• Experimenting with interactions for the selector itself – it still seems that a selector that collapses on itself once a format has been chosen, rather than tabs that always show everything, will be better.
• Hiding the title and/or the content for some of the post types. @melchoyce’s wireframes show which formats should have the title hidden by default; note that we should always allow the title to be edited, even if hidden under a toggle. The same goes for the content – can be hidden for some of the post types where the content serves more as a description or commentary on the other data provided for the format (image, gallery, audio, video, and quote come to mind).

Related to hiding the title, we need to add a way for titles to be smartly auto-generated/populated for those formats where a title isn’t prominently “required”. Some of the ones that also don’t need content may be served well with some smart detection if possible, e.g. the title of a video or image.

After discussion about old and new themes and content, we’ve chosen to go a little more aggressive with compat output. We had previously attempted using the current add_theme_support( ‘post-formats’ ) implementation to determine whether or not compat output should appear, but it quickly became apparent that it would create less-than-ideal situations with themes that supported post formats in the original style but have not updated. We’ve now introduced add_theme_support( ‘structured-post-formats’ ) for themes to indicate that they handle the metadata for a given format(s) and that core need not add fallback output. See the commit message on r23467 for more details.

The next step for fallback output is to accommodate new-style themes (that is, themes that specifically handle metadata) with old-style content. The Codex has recommendations for content entry/handling that will help us figure out what might have previously been done for post format support. @wonderboymusic has started work on functions to parse out relevant format data from the content itself: #23570 (URL), #23572 (audio/video), and #22960 (getting images from a gallery shortcode, especially the first). We’ll need more for other formats, but this is a start. While that is going on, we’ve been discussing what the theme output layer would look like and how de-duplication would work. “Cutting” the parsed-out data from the content itself before display is, again, a bit aggressive, but seems likely to be the best way forward (noting that it would be on display only, not actually altering the user’s content). See post-dev chat IRC logs for more details: https://irclogs.wordpress.org/chanlog.php?channel=wordpress-dev&day=2013-02-20&sort=asc#m560171

Finally, note that we’ve created a Post Formats component in Trac. There will sometimes be some overlap between components, but especially for the 3.6 cycle, let’s put anything that relates to getting this feature working in that component so we have it handy.

@beaulebens is working on the fallback output on #23347, and @sabreuse is joining @rachelbaker to work on the edit form and switcher on #19570. @melchoyce has updated wireframes she’ll be linking to in a comment here. TBD: no-JS treatment, will also need to be keeping an eye on accessibility.

Also, user testing was done with core as-is, and I’ve got videos to watch and analyze for post formats using Crowd Favorite’s UI and fallback code. These are extremely revealing and are really helping us identify pain points as well as setting a baseline for measuring any improvements we hope to make.

• Office hour log
• Dev chat log

We’re going to go ahead and get patching so we can test and iterate, and still looking for somebody interested in creating icons. @rachelbaker will be working on the form itself and switching. Would love a volunteer or two on data retrieval and theme usage functions, but will likely be looking at that myself in the next couple of days.

Office hour log

Discussion point 1: format switcher, with an example from @lessbloat: http://f.cl.ly/items/1t09071U2v1E2x2s3X2i/post-formats.png (we would use “Standard” as opposed to “Text” for that label). We will not be blocking the current behavior/ability to just start writing a post when you enter the add new screen, i.e. no forced selection of a post format before you can start writing. There is some concern about user confusion in that one might think that you can/should specify data for multiple formats in a single post if the switcher is always viewable. A viable idea seems to be that all formats are shown when you load the add new post page, which then collapse into one item (the selected one) upon a specific selection. Editing an existing post, including drafts, will only show the one format selected. Clicking on the selected format would allow for switching of the format via a to-be-determined interaction. There’s also the idea that a screen option would be provided to turn off the switcher entirely should the user so desire, just as the “Format” metabox can be turned off now.

Action: Call for icons, one per post format. No dimensions determined yet; keep in mind that 2x versions will be needed.

Discussion point 2: data structure needs, based on this really great start on wireframes by @melchoyce: http://make.wordpress.org/core/2013/01/22/post-formats-ui-update-121/#comment-7674. Proposed:

• Media URL/embed code/shortcode (basically, text), to be shared between audio and video
• Link URL, to be shared between link and image
• Quote content
• Quote source
• Gallery, which would be a shortcode that likely needs to default to [gallery] for backcompat; would use shortcode with list of IDs for new style
• Image, which could be an attachment ID or URL (this is separate from featured image)

Sharing data fields (storage TBD, likely post meta) would mean that switching between formats that are similar in data needs would retain that field. Action: discuss proposed data structure in comments; pros/cons of sharing fields between formats, anything missed, etc. Also discuss anything related to the wireframes; @melchoyce was working on some discussed tweaks (although she may be understandably behind due to computer theft ), and there will likely be more that we’ll want to do.

Finally, as a tightly-related item, @wonderboymusic started work on the possibility of an HTML5 audio/video player in core: #23282. Testing and feedback more than welcome.

Full chat log

A large portion of our office hour today was spent discussing a script for user testing that we can use throughout this process to help us evaluate the changes proposed. We need to use a theme that includes all ten formats, doesn’t do anything to the admin UI, and preferably does something to the front end display for each. One that I know of is San Kloud, although it has some deprecated function usage (hey @kovshenin – want to fix that? ); other suggestions welcome.

Since the first part of the issue here is discoverability, the script needs to not hand-hold the format selection or entry process. Here’s a rough outline of what it might look like – discuss any changes in the comments, and @lessbloat will help us get testing. Some of these are particularly difficult to describe in a non-leading manner. Through each of these, it would be good to have the user narrate what they want or expect will happen for each of these formats. A little fuzzy since we’re not going to point them at the format picker and a given tester may not be used to blogging, but the more narration of thinking/expectations, the better.

• Standard/text: Add a blog post entitled X, with the following text. Publish and view the post.
• Aside: Add a brief blog post without a title, or what might be referred to as an “aside”. Publish and view.
• Status: Add a blog post that represents your current status as (provide text), the way you might on a site like Facebook or Twitter. Publish and view.
• Chat: Add a chat blog post where the content is a chat transcript (provide something to copy-paste). Publish and view.
• Quote: Add a quote blog post with X quote attributed to Y. Publish and view.
• Link: Add a link blog post with X text linking to [URL]. Publish and view.
• Image: Add an image blog post and insert X image into the content editor. Publish and view.
• Gallery: Add an image gallery blog post with X images (this probably needs step-by-step instructions for creating a gallery). Publish and view.
• Video: Add a video blog post with [something from YouTube]. Paste the URL on its own line without linking it. Publish and view.
• Audio: Add an audio blog post with [something from SoundCloud]. Paste the URL on its own line without linking it. Publish and view.
• View your site. (If they chose formats, it should show the differences pretty clearly in a post archive view, hence the importance of choosing an appropriate theme.)

Full IRC chat log

Had our first office hours chat today to get started – great discussion happening on a lot of fronts, especially theme compat. We identified two distinct tasks to get us through the weekend.

The first, and most crucial piece, is thinking about/identifying what users are doing when creating a post using a post format and turning that into (likely annotated) wireframes/storyboards. That is to say – what sort of UI might we want want to see on the add/edit post screen? We’ll want to think about things like what makes it easy for a user to select a format, what a user might expect to see on the edit screen for a given format (no title, or the title is de-emphasized, or what images show in that gallery, etc.), and what happens when switching between formats. Some existing examples are CF Post Formats and WordPress.com. We also shouldn’t ignore what Tumblr does for its various formats. @alexkingorg gave us a demo link for CF and @beaulebens was kind enough to take some screenshots:

• CF Post Formats (go to theme admin)
• WP.com: standard, image, video, quote, link
• Tumblr: text (standard), image, quote, link, chat, audio, video

Please post in a comment here with anything you come up with. Don’t worry about anything beyond the add/edit post screen for the moment. Also don’t worry about what data exactly is or is not represented – base your ideas on user workflows. The workflows and UI will heavily influence any structured data. There are also some other things to keep in the back of your mind as possible influences for anything core UI-related: how might a layout you’re imagining adapt to various screen sizes and devices, and are there pieces that can be used as the basis for more generalized core UI components.

The second piece for now is functions (probably using regex, and I know) for pulling out relevant data for post formats as appropriate, e.g. the first URL in the content for the link format, oEmbed URL or appropriate HTML element for audio/video formats, etc. What might or might not be done with the data hasn’t been determined yet, but we can at least start here. Some existing examples are the twentyeleven_url_grabber() function and what they use on WordPress.com. Nobody has claimed this task. For now, go ahead and upload any patches to the main ticket, #19570.

There are also some related existing tickets that could use testing/+1′s or patches:

• #16047 – separate column for post format in the list table
• #23198 – pass the post format as a class to TinyMCE
• #15323 – list table filter by post format
• #15514 – add to the cat-tag convertor plugin

Read the full IRC log here.