Widget Improvements in WordPress 4.9

On the heels of adding TinyMCE rich editing to the Text widget and the media widgets in 4.8, there are another round of improvements coming to the Text widget and Video widget in 4.9, among other improvements to widgets.

Shortcodes in Text Widget

One very longstanding request—for over 8 years—has been to support shortcodes in the Text widget (#10457). This is finally implemented in WordPress 4.9. It is no longer required to have plugins and themes do add_filter( 'widget_text', 'do_shortcode' ). Core now will do_shortcode() at the widget_text_content filter (added in 4.8) in the same way it is applied in the_content at priority 11, after wpautop() and shortcode_unautop(). If a plugin has added do_shortcode() to widget_text then this filter will be suspended while the widget runs to prevent shortcodes from being applied twice. If a Text widget is in legacy mode then it will manually do_shortcode() as well.

One reason for the long delay with adding shortcode support in Text widgets was due to many shortcodes looking for a global $post when they run. Since the global $post varies depending on whatever the main query is, the shortcodes in a Text widget could render wildly different on different templates of a site. The solution worked out was to temporarily nullify the global $post before doing the shortcodes so that they will consistently have the same global state, with this global $post then restored after the shortcodes are done. So if you have shortcodes that depend on a global $post—or call get_post()—then you should make sure that they short-circuit when $post is null in order for them to behave properly if used in the Text widget.

As of [42185] this nullification of $post is only done for archive (non-singular) queries; for singular queries, the $post will instead be set to be the current main queried post via get_queried_object(). This ensures that the global $post is consistent and explicit. This setting of the $post global while applying filters (and shortcodes) is also now implemented for the Custom HTML widget. Additionally, to ensure that gallery shortcodes that lack ids do not end up listing out every attachment in the media library, a shortcode_atts_gallery filter has been added which makes sure the shortcode’s id attribute is set to -1 when the widget is rendered on archive templates. This allows you to embed the gallery for any currently queried post in the sidebar. You should make sure such a Text widget is not displayed on an archive template by either adding it exclusively to a sidebar that appears on singular templates, or by using a feature like Jetpack’s Widget Visibility to hide the widget on non-singular templates.

Media in Text Widget

One reason why shortcode support in the Text widget was needed in this release is because 4.9 also allows media to be embedded in the Text widget (#40854). There is now the same “Add Media” button in the rich Text widget as on the post editor, allowing you to add images, galleries, videos, audio, and other media. To support these, core also needed to support shortcodes like captionaudiovideo, and gallery. Note there are also dedicated widgets (Image, Audio, Video, and Gallery) for these media types as well.

Having separate media-specific widgets helps with discovery and allows us to provide streamlined interfaces for each media type. For example, the Image widget now has a field specifically for supplying the link URL (see #41274), and the Video widget now provides more guidance to users when supplying external URLs (#42039). The media-specific widgets are closely aligned with blocks in Gutenberg; the existence of media inside the Text widget will align with eventual nested blocks in Gutenberg, and would be treated as Classic Text blocks in any future migration from widgets to blocks.

Embeds in Text Widget and Video Widget

One shortcode not mentioned above is embed. This one was more difficult to support because oEmbeds have not been supported anywhere other than post content. This was because there were dependencies on having a post as context for the sake of caching, as the responses to oEmbed requests get stored in postmeta. However, as of #34115 if there is no post as context the oEmbeds will now get cached in an oembed_cache custom post type instead. Since a Text widget will explicitly nullify the global $post while shortcodes are processed, this means oEmbeds will get cached in this custom post type. Similarly to how do_shortcode() now applies in the widget_text_content filter like it applies on the_content, so too now WP_Embed::autoembed() and WP_Embed::run_shortcode() both also now run on widget_text_content.

In WordPress 4.8 the Video widget was introduced with support for displaying an uploaded video file, a YouTube video, or a video from Vimeo. Each of these were displayed using MediaElement.js. Just as oEmbeds are now able to be displayed in the Text widget, so too now the Video widget has been expanded to support any oEmbed provider for video. See #42039.

Theme Styling Changes

As with the previously-introduced media widgets (#32417) and the new Gallery widget (#41914), some themes will need to be updated to ensure the proper styling is applied to media and embeds that appear in the widget area context, since previously they would only appear in post content. Please follow #42203 and #41969 for style changes that are made to the core bundled themes, as you may need to make similar changes to your themes.

Improved Theme Switching

A longstanding difficulty with widgets has been where they end up when switching from one theme to another. With #39693 this experience is improved in 4.9 by having logic that is able to better map widgets between the themes’ widget areas. As noted by @obenland in [41555], there are three levels of mapping:

  1. If both themes have only one sidebar, they gets mapped.
  2. If both themes have sidebars with the same slug (e.g. sidebar-1), they get mapped.
  3. Sidebars that (even partially) match slugs from a similar kind of sidebar will get mapped. For example, if one theme as a widget area called “Primary” and another theme has “Main” then the widgets will be mapped between these widget areas. Similarly, widgets would get mapped from “Bottom” to “Footer”.

The names for the widget areas used for the mapping groups were obtained by gathering statistics from all the themes on WordPress.org.

Widget Saved State on Admin Screen

With #23120 there is now an indication for whether or not changes to a given widget has been saved on the widgets admin screen. (Widgets in the Customizer already had a saved state by virtue of being registered as regular settings.) When first opening a widget, the button will say “Saved” and appear disabled. Once a change is made to the widget (e.g. a change event triggered), then the button will become enabled and say “Save”. If you try leaving the admin screen at this point, an “Are you sure?” message will appear alerting that if you leave your changes will be lost. If you cancel, then the first widget with unsaved changes will be scrolled into view, expanded, and focused. Upon hitting “Save” the spinner will appear and then upon a successful save it will switch to “Saved” and become disabled. The “Close” link has been changed to “Done” and it only appears when the changes have been saved. Note that the HTML5 checkValidity method will now be called on the widget form prior to attempting to submit, and submitting will be blocked if it returns false. If you have JavaScript-based fields in the widget, make sure that you trigger change events whenever changes are written into any hidden inputs; this was already a requirement for widgets in the Customizer.

Related Tickets

  • #10457: Parse shortcodes in text widgets by default
  • #23120: There should be indication that widget settings have been saved
  • #34115: oEmbed not working on author page without posts
  • #38017: Add widget instance to remaining widget argument filters
  • #39693: Fix missing assignment of widgets on theme switch
  • #40442: Widgets: Rename “Custom Menu” widget to “Menu”
  • #40854: Allow media to be embedded in Text widget
  • #41274: Improve discoverability of link URL in Image widget.
  • #41610: Widgets: Change “close” to “done?”
  • #41914: Widgets: Add gallery widget
  • #41969: Ensure Gallery widget is styled properly across widget areas in bundled themes
  • #42039: Widgets: Enable oEmbed support for Video widget
  • #42203: Ensure media & embeds in Text widget are styled properly across widget areas in bundled themes

See full list of tickets in the Widgets component with the 4.9 milestone.

#4-9, #dev-notes, #feature-oembed, #media, #media-widgets, #widgets

Dev Chat Summary: September 20th (4.9 week 8)

This post summarizes the dev chat meeting from September 20th (agendaSlack archive).

4.9 schedule

  • 1 week until the feature project merge deadline, 2 weeks until Beta 1
  • Drafting and scheduling (#39896) has designs and is working through development
  • Gallery widget (#41914) is now available in the Core Media Widgets feature plugin, so please test that as we plan to merge it into core next week
  • Updating a plugin via ZIP (#9757) or drag/drop (#24579) is not getting any traction, so they will likely be punted
  • Review needed on #34115 to bring the ability to use oEmbed outside the context of posts
    • This allows the Text widget to have embeds in it, as well as to be able to remove the restriction on the Video widget to only show YouTube and Vimeo
  • Long list of Code Editor tickets that could use contributors
    • If you’re interested and capable with JS and playing with CodeMirror, #41873 is a great place to start
  • Any tickets related to goals in the 4.9 Goals post should be prioritized
  • Bug scrub post will be published with dates/times to scrub features and enhancements ahead of the Beta 1 deadline. If you’re interested in helping run a scrub, then please let @jbpaul17 know.
  • User testing needed on #39693, especially running it through a battery of theme switching tasks; the goal is to improve the “it just works” experience
    • Testing steps include installing a theme, populating sidebars with widgets, switching to another theme, and checking how the widgets appear in the newly switched theme
    • If you switch back to the old theme, the sidebar widgets get restored
    • Testing should be done when switching themes via the WP admin themes page, and also via live preview in the Customizer
    • Themes that have varying numbers of sidebars would be the key to test with. Switching from a theme with 2 sidebars, to one with 4, to another with 1, to another with 5, and so on.
    • If you want to have a public test environment set up with the patch to test, then please ping @westonruter

Editor update

General announcements

#4-9, #core, #core-editor, #dev-chat, #feature-oembed, #media-widgets, #security, #summary

New Embeds Feature in WordPress 4.4

WordPress has been operating as an oEmbed consumer for quite some time now, allowing users to easily embed content from other sites. Starting with version 4.4, WordPress becomes an oEmbed provider as well, allowing any oEmbed consumer to embed posts from WordPress sites.

Here’s how that looks:

Feature Plugin Merge Proposal: oEmbed

In order to achieve this, WordPress’ oEmbed consumer code has been enhanced to work with any site that provides oEmbed data (as long as it matches some strict security rules). For security, embeds appear within a sandboxed iframe – the iframe content is a template that can be styled or replaced entirely by the theme on the provider site.

Related ticket: #32522

What That Means for Developers

First of all, this new feature means that any post (or basically any public post type) will now be embeddable. If you’re using pretty permalinks, the embeddable content will be available at example.com/your-post/embed/.

If you’re a developer, make sure you do not add an embed rewrite endpoint yourself!

Functions and Hooks

Here are the four most useful functions related to embeds:

  • get_post_embed_url() — Retrieves the URL to embed a specific post in an iframe, e.g. https://make.wordpress.org/core/2015/10/28/new-embeds-feature-in-wordpress-4-4/embed/
  • get_post_embed_html() — Retrieves the full embed code for a specific post.
  • get_oembed_endpoint_url() — Retrieves the oEmbed endpoint URL for a given permalink, e.g. https://make.wordpress.org/core/?oembed=true&url=<url>. This is used to add the oEmbed discovery links to the HTML <head> for single posts.
  • get_oembed_response_data() — Retrieves the oEmbed response data for a given post, according to the oEmbed specification.

Of course the return values of these functions are all filterable, making it easy for you to customize this new feature.

Customizing The Output

The embed template mentioned earlier can be customized similarly to any theme template file. Use embed_head and embed_footer to add custom code in the beginning and the end of the template.

Note that an X-WP-embed:true header will be sent when that template is used, so you can easily target embedded posts in your application.

Further Improvements

We’re currently tweaking the embeds functionality to sort out the last few bugs. Here’s a short list of tickets that are being worked on and their purposes:

  • #34204 — Improved support for IE7+. This will also introduce a enqueue_embed_scripts hook that can be used to enqueue JavaScript and CSS.
  • #34451 — Improved support for embedding posts on non-WordPress sites
  • #34278 — Add support for embeds in the theme template hierarchy, allowing you to override the template easily in your theme.
  • #34207 — Leverage the REST API structure for the oEmbed endpoint.
  • #34462 — Add a <blockquote> fallback for the iframe.

Disabling The Feature

Don’t like these enhanced embeds in WordPress 4.4? You can easily disable the feature using the Disable Embeds plugin if you really want to.

#4-4, #dev-notes, #embeds, #feature-oembed

Feature Plugin Merge Proposal: oEmbed

For the past 6 years, users have been able to embed YouTube videos, tweets and many other resources on their sites through a nifty feature called oEmbed.

Today, we (mainly me, @pento and @melchoyce) ask to consider extending this feature by merging the oEmbed API plugin into WordPress core. This plugin allows anyone to embed posts from your site by just pasting its URL. We’ve been working hard on it for months and are now eager to hear your feedback.

Purpose & Goals

While I initially built an early version of the plugin about a year ago, it was @melchoyce who kicked things off with #32522. Her idea was simple: When you can embed almost anything in a WordPress post, why aren’t we able to embed WordPress posts themselves in another WordPress post?

That’s exactly what we’re aiming for. Our goal is to allow a big portion of the web to easily and securely embed such post previews.

Have a look at this post to see the user flow for this feature (and a live demo!):

https://make.wordpress.org/flow/2015/09/12/embeds-visual-survey-of-the-oembed-feature-plugin-macnchrome-iphone-5s/

Security Considerations

Embedding content from a random source on your site depends on lots of trust. We take precautions to make the whole process as easy as possible. It’s worth noting that:

  • We use iframes with the sandbox attribute to enable extra restrictions on the content that can appear in the inline frame.
  • The host and the embedded site communicate via postMessage to allow resizing and clicking on links safely

Browser Compatibility

We successfully tested the feature with all major browsers on mobile and desktop. Since IE < 10 doesn’t support the sandbox attribute, we use the proprietary security attribute there to have similar security restrictions. This means no JavaScript inside the iframe is run, e.g. for the resizing. The most important thing, clicking, still works there though.

Long story short, the feature works with all major browsers and degrades gracefully on older IE versions.

Core Changes & Merge Implementation Details

The plugin was developed in such a way that merging it into core eventually is as straightforward as possible. We are working actively on a patch that can be added to core within the merge window.

There are two things that we need to change in core together with the merge:

  • When doing a redirect because of a changed post slug the rewrite endpoints need to be respected as well. See #33920
  • Attachment rewrite endpoints need to be fixed in core. See #19918
  • This feature only works with oEmbed discovery turned for every user, even those lacking the unfiltered_html capability. That capability check needs to be removed.

Developer Notes

This plugin adds a new /embed/ rewrite endpoint for posts, pages and attachments. We haven’t found any plugin in the directory using this endpoint, but if you already use that endpoint, you should consider renaming it or changing the oEmbed rewrite endpoint using the filters we provide.

Note: There’d be a separate post for this after the merge.

What about the REST API?

The plugin works well on WordPress 3.9+ and uses a simple class to return the oEmbed API responses. However, with the REST API officially proposed for a core merge, we built a controller class for it. This class does exactly the same and follows the REST API best practices.

We could definitely profit from the REST API and built upon it when merging into core. Not needing a fallback means no duplicated code and easier maintenance.

In case you missed it, here’s the REST API merge proposal:

https://make.wordpress.org/core/2015/09/21/wp-rest-api-merge-proposal/

Remaining Issues

There is currently one issue with Slack not displaying the oEmbed output correctly. It simply displays the JavaScript as plain-text instead of removing it. We’ve reached out to them to see if they could fix that and at least white-list WordPress.org in the meantime to properly display the embeds in Slack channels.

Besides that, there are also some small layout quirks we still need to work out. Meanwhile we’re continuing to improve the codebase and inline documentation.

Future Plans

We’re looking into improving support for different response types in addition to regular post content, depending on the feedback we receive from users.

Contributors

While I’ve been the lead developer of the plugin, a ton of valuable input and contributions have come from others in the community.

#4-4, #feature-plugins, #feature-oembed, #merge, #proposal

oEmbed Update: September 28th

As of today, the current version of the oEmbed feature plugin is 0.9.0. These are the biggest changes since last week:

  • There is now an is_embed conditional tag to more easily target embedded posts.
  • New: Support for embedding video and audio attachments.
  • You can now copy the whole embed code and not only the post’s URL in the sharing dialog.
  • There were major JavaScript improvements. It is more robust and also a bit faster too.
  • We now show a read more link instead of the word count in the excerpt.
  • Improved input sanitization.

These were all areas discussed in last week’s chat and I’m happy with what we achieved. The plugin has been installed on WordPress.com, although it has been temporarily deactivated 🙁. Nonetheless I’m looking forward to publishing another post regarding the plugin tomorrow…

There are only some smaller issues left, so now is a great time to test and review the plugin! Our weekly chat is today (September 28 2015 9pm UTC) in the #feature-oembed Slack channel

Note: We received some feedback regarding weird link previews in Slack for WordPress.org links. This is a Slack bug and not our fault. We already reached out to them and hope it gets fixed soon.

As always, the latest version of the plugin is available on the plugin repository. Errors and suggestions can be either reported on GitHub or our #feature-oembed Slack channel.

Next chat: October 5 2015 9pm UTC

#embeds, #feature-plugins, #feature-oembed, #update

oEmbed Update: September 22nd

Today I just tagged version 0.8.0 of the oEmbed feature plugin. Despite the small version number the plugin saw some major improvements compared to last week. The changelog:

  • Added translations for Spanish, French, Italian, Finnish, Hebrew. There are now 13 translations in total!
  • Majorly improved embed handling: There’s now a 404 view for non-existent posts and we redirect to the right URL when post slugs change. The latter needs to be fixed in core though, see #33920.
  • Improved REST API support thanks to a review from the team
  • Better support for multisite and WordPress installs in a subdirectory
  • Now the plugin properly supports emojis too! 😀

This was only possible because of great feedback from plenty of users. The plugin is running smoothly on lots of sites now, including WordPress.org itself! It will also ship to WordPress.com this week if everything goes as planned.

We’re very happy with the plugin’s current form. Our priority tasks are now reviewing inline documentation and making sure the plugin is as secure as possible. If these are areas you’d like to work on, your help would be highly appreciated!

As always,the latest version of the plugin is available on the plugin repository. Errors and suggestions can be either reported on GitHub or our #feature-oembed Slack channel.

Next chat: September 28 2015 9pm UTC

#embeds, #feature-plugins, #feature-oembed, #update

oEmbed Update: September 16th

We’re constantly refining the oEmbed feature plugin and have been receiving some very good feedback over past seven days.

The following things changed in the plugin compared to last week:

  • There are now 8 translations: Danish (Denmark), German (Germany), German (Switzerland), Japanese, Greek, English (Australia), Arabic and Arabic (Morocco).
  • Full Right-to-Left language support.
  • Various performance improvements.
  • Dashicons are now included as inline SVGs in the CSS for better IE compatibility.
  • Added new filters to the embed template, allowing for better customization

Besides that I wrote a post on make/flow about the oEmbed plugin and also pinged several teams asking for feedback. Following best practices is very important to us.

In our weekly chat on Monday we discussed several things:

  • The plugin is active on the Make blogs, but some server settings need to be changed for it to work properly.
  • We learned that the plugin runs great on several sites, even with +100k daily users!
  • To make sure the plugin works under any conditions, go give it a try and report any bugs to us.
  • There was even some “homework” for next week:

    To help test the plugin, take a couple of minutes to sit down with someone and watch them use the plugin. Take some notes, then post details and findings in a GitHub issue.

We opened a couple of issues according to the feature plugin checklist to help us get ready for the core merge window.

As always,the latest version of the plugin is available on the plugin repository. Errors and suggestions can be either reported on GitHub or our #feature-oembed Slack channel.

Next chat: September 21 2015 9pm UTC

#embeds, #feature-plugins, #feature-oembed, #update

oEmbed Update: September 9th

Another week, another status update for the oEmbed feature plugin!

The following things changed in the plugin compared to last week:

  • Further accessibility improvements (props @afercia, @joedolson) by removing headings from the iframe content.
  • Better embeds for image attachments
  • thumbnail_url, thumbnail_width and thumbnail_height are now sent in the oEmbed endpoint response if available.
  • Major refactoring to make an possible core merge easier (props @pfefferle)
  • Many bug fixes thanks to improved test coverage. About 90% of the codebase is covered now!

On Monday there was also our weekly chat. The following things were discussed:

  • @pento will be working on implementing the plugin WordPress.com, giving us a larger userbase with hopefully great feedback.
  • In general, we need more testers. So if you have a website, no matter if large or small, go give it a try!
  • The core merge window is getting closer. We opened a couple of issues according to the feature plugin checklist to help us get there.
  • We’ll reach out to various core teams to get feedback on different aspects of the plugin, like inline docs, accessibility, user flow, etc.

As always,the latest version of the plugin is available on the plugin repository. Errors and suggestions can be either reported on GitHub or our #feature-oembed Slack channel.

Next chat: September 14 2015 9pm UTC

#embeds, #feature-plugins, #feature-oembed, #update

oEmbed Feature Plugin Update

After kicking off the oEmbed feature plugin a couple of weeks ago, it’s high time for another status update.

In case you have missed it, the oEmbed API plugin makes WordPress an oEmbed provider, allowing you to embed blog posts just like YouTube videos or tweets. Of course everything happens with security and ease-of-use in mind.

oEmbed Feature Plugin

Embedding a post is super simple!

We made some great progress over the last few weeks. The highlights are:

  • Improved test coverage, which led to many fixed bugs
  • Auto-resizing of the embedded iframe so it looks great on every screen
  • It seamlessly integrates with the REST API, but also works perfectly without it

The plugin is very stable so far. We’re looking into bringing it to WordPress.com for testing, but of course we also need your help to bring this further! Download the plugin from the repository — play with it, break it, and help us fixing all bugs that may appear. We’re always looking for areas to improve.

We’re now mainly working on getting it into shape for an eventual core merge proposal and implementing the different oEmbed response types. This means supporting embedding attachment posts and posts with different post formats.

Please, test and report both errors and suggestions either on GitHub or our #feature-oembed Slack channel. Anyone is welcome to join us!

Next chat: September 7 2015 9pm UTC

#embeds, #feature-plugins, #feature-oembed, #updates

oEmbed Chat Summary – July 20th, 2015

Yesterday we held our first weekly chat in #feature-oembed. Hooray! There were quite a few participants already, which is great.

Logs can be found here: https://wordpress.slack.com/archives/feature-oembed/p1437426031000036

Summary:

  • There’s a proof-of-concept oEmbed implementation in the develop branch on GitHub (demo):
  • We agreed on keeping the embeds simple and minimally styled.
  • As expected, there was quite a discussion about the direction to pursue. There are basically two ways for doing this:
    1. WordPress has an oEmbed endpoint and returns HTML people can embed
    2. WordPress has no endpoint. We scrape the referenced websites to get data for a preview. Kinda like Facebook, Slack or Twitter show previews for links.
  • We decided on first finishing the HTML, as we need that anyway. After that we can focus on the next steps.

Development happens on GitHub, where we’ll be filing a couple of issues to work on until Monday. Anyone is welcome to contribute to the plugin.

Next chat: Monday, July 27, 2015 21:00 UTC

#chats, #embeds, #feature-plugins, #feature-oembed, #updates