Customize Changesets (formerly Transactions) Merge Proposal

This is a merge proposal and overview of Customize Changesets (#30937), a project formerly known and proposed as Customizer Transactions back in January 2015. The customizer is WordPress’s framework for doing live previews of any change on your site. One of the biggest problems the customizer faces right now is that changes are ephemeral. If you navigate away, you lose what you are working on. Additionally, you can not share proposed changes with others, nor can you take the changes you are working on and save to continue working later.

Imagine a WordPress user named Tina. Tina is building a website for her daughter’s band. The band has been getting more and more popular, and Tina wants to experiment with some new widgets in the sidebar but also wants to be able to share the proposed solutions with her husband Matthew. Right now, Tina and Matthew would need to be in the same room to collaborate. Or Tina would need to take screenshots, but that kind of defeats the purpose of live preview. If only there was a way to make customizer changes persistent without publishing them.

Introducing Customize Changesets

Customize changesets make changes in the customizer persistent, like autosave drafts. Users can make changes to one theme and switch to another in the customizer without losing the changes upon switching. A customizer session can be bookmarked to come back to later or this URL can be shared with someone else to review and make additional changes (the URLs expire after a week without changes). The new APIs make possible many new user-facing features in future releases and feature plugins, including saving long-lived drafts, submitting changesets as pending for review, scheduling changes, seeing the previewed state on the frontend without being in an iframe, sharing preview URLs with others who do not have customizer access, and others.

Customize changesets allow each change you make in the customizer for a given live preview session to be persistent in the database. A unique identifier (a UUID like f67efbbf-c663-4271-ab1c-95ce1d447979) for each live preview session is generated and as soon as a change is made, the change setting value is sent in an Ajax request to be written into a custom post type whose post_name is the UUID for the customizer session. Once the changes have been written into the changeset post, then any request to WordPress (including to the REST API) can be made with a customize_changeset_uuid query param with the desired UUID and this will result in the customizer being bootstrapped and the changes from that changeset being applied for preview in the response. The unique UUID means that customizer sessions can be sent to other users and also that they can be used as query parameters on the front end.

Design and Technical Decisions

For the initial core merge, no UI changes are being proposed. The feature will only be exposed as the new query parameter on the URL. Adding a UI to this feature will happen in a future release. As such, the proposal for customize changesets is similar to the proposal for including the REST API infrastructure: it provides a foundation for new core features in future releases and a platform for plugins to add new features. Nevertheless, while the customize changesets patch doesn’t introduce any new features it does fix several long-standing issues related to incompatibilities between JavaScript running on the site’s frontend when previewed in the customizer. Under the hood, the customize changesets patch touches on many of the lowest level pieces of the customizer. Please check out the Customize Changesets Technical Design Decisions to see what is happening under the hood.

Testing

Please test! If you use any plugins that extend the customizer, please ensure that there aren’t any regressions. The patch is intended to be fully backwards compatible and users shouldn’t notice any difference in normal use. Two things to look for when testing is as soon as you make a change, you should see a customize_uuid query param added to the URL. You should be able to reload and find your changes persist (note the AYS dialog is retained because there is no UI yet for listing changesets). Also, when you navigate around the preview it should feel much more natural like normal browsing as opposed to having a fade effect. Otherwise, previewing settings that require refresh should still work as normal, as will settings that preview with JavaScript and selective refresh.

The patch is in a GitHub pull request and you can apply the patch via:

grunt patch:https://github.com/xwp/wordpress-develop/pull/161

If you’re using the Git repo from develop.git.wordpress.org then you can check out the branch directly via:

git remote add -f xwp https://github.com/xwp/wordpress-develop.git && git checkout xwp/trac-30937

I’d appreciate code review feedback directly on the pull request. For any revisions to the patch, please open a pull request to that trac-30937 branch if possible.

The Future

In future releases we can explore new UIs to take advantage of the new capabilities that changesets provide. New UIs can provide a way to schedule changes, the ability to undo the last change, show an audit log (revision history) for changes, collaborative editing of a customizer changeset, and so on. Future feature projects will explore many of these and feature plugins will start to prototype them.

Thanks to @jorbin who contributed to this proposal post.

#4-7, #customize, #feature-projects, #proposal

The Road to 4.7 Beta 1

For the last 55 days, trunk has been in an alpha state and open for all commits. In 15 days this will change and the first beta for 4.7 will be released. During beta, no new enhancements or feature requests should be committed to WordPress core. While some may desire to wait for just one more thing, that is a rabbit hole which is inconsistent with the WordPress philosophies. Here is a list of things that need to happen during the next 15 days in order to get ready for Beta 1 on October 26th.

The deadline for merging feature projects into WordPress core is in 8 days. There are currently three proposals for projects to merge published with several additional proposals actively being worked on. All committers and active contributors should review these proposals (I’ll add links to additional proposals when they are made):

There are 60 enhancements and feature requests milestoned for 4.7.  Of these, 21 have no owner.  At the end of this week, all enhancements and feature requests without an owner will be punted.  Tickets will also be regularly evaluated this week.  Over the next two weeks, tickets will be evaluated during both scheduled and unscheduled bug scrubs, and tickets not moving forward will be removed from the milestone. The next scheduled bug scrub will be at Wednesday, October 12, 2016 17:00 UTC.

During this week’s dev chat, we will discuss all the proposals that have been made thus far. If additional meetings for considering any of the proposals before the merge deadline are needed, this will be decided then.

#4-7, #proposal

Feature Proposal: Better theme customizations via custom CSS with live previews

When people ask “why WordPress?”, some of the most common answers center around flexibility for users of all kinds, whether they’re building their sites primarily through code or UI. Let’s take the story of a user who does a little of both – we’ll call her Becky.

Becky is a pretty savvy user. She knows that you’re supposed to make child themes instead of hacking on a theme directly, because updates can wipe out your changes. She found that out the hard way when she first started using WordPress – she hardly knew what CSS or PHP were, but she knew there was a theme editor in the admin and that she could make tweaks to colors or remove the author byline pretty easily without having to figure out this FTP stuff. Later on, most colors could be changed with the customizer so having a child theme just to remove an author byline seemed like overkill, but it was certainly better than having it reappear every time her site updated, especially with auto updates turned on.

After a couple years with the same theme on her personal site, Becky felt it was time to change things up. She was pleasantly surprised to find some new features that made getting a theme set up a lot easier, especially when live previewing them. Still, though, that pesky author byline remained, and since her last child theme copied a template to get rid of the byline, she would have to set up a whole new one to do it again. Then Becky found an “Edit CSS” option and realized she could hide things using CSS without having to go through the entire child theme process. Now, it turns out that those CSS tweaks didn’t come with live previewing, and that functionality was provided by a certain plugin, but Becky got what she needed to get done a lot faster than she would have otherwise, and ended up with the site she wanted.

This isn’t one specific story, but it is a combination of user stories many have heard, witnessed, or even personally experienced. You could replace Becky with @helen and it would be 100% accurate. The theme editor is a dangerous but useful entry point to more deeply customizing your site – rather than outright removing it and cutting off that introduction not just to WordPress code but to the concept of web development at large, why not provide a far safer and more user-friendly alternative? This post will explain why custom CSS with live previewing is valuable for WordPress and propose an implementation for inclusion in 4.7.

Proposed solution: Custom CSS with live preview

When bridging the gap between advanced user and beginning developer, desired changes are typically visual tweaks, such as changing a font size or hiding something, that are theme-specific. These sorts of changes should not require that users take risks editing live files that might white screen their sites or jump immediately into developer-facing tasks such as using FTP. Therefore, the scope of this feature has been defined as a custom CSS editor that leverages the customizer for a user-friendly live preview experience. This live preview allows for users to try various tweaks to a theme before saving and setting their changes live.

There are hundreds of thousands (if not millions) of users making use of custom CSS plugins or other themes/plugins that have custom CSS capabilities, and the frequent suggestion of CSS fixes in support forums justify a core need for this functionality. When plugins and themes interact in unexpected ways, CSS snippets are often an efficient solution to fixing the particular problem on particular sites.

The CSS editor takes inspiration from the many plugins offering similar solutions, but with an updated approach that offers instant live previewing in the customizer. The proposal for 4.7 looks like this:

custom-css-proposal-demo

Notably, previewing CSS in the customizer allows the site to be navigated and previewed on different sized devices by leveraging existing core features, allowing users to visualize the impact of their changes across their site. Error messages are shown for common syntax mistakes to help users ensure that their CSS is formatted properly before saving.

In future releases, the interface can be iterated on to further improve usability. The long-term design vision provides functionality such as revisions, syntax highlighting, and in-preview selector helpers, and can be implemented iteratively over time (click through for the full version):

customizer-css-i2

CSS would be stored in a custom post type (without admin UI), with a post stored for each theme. The editor would be used to supplement and override theme styles rather than editing them directly, as users have long been advised that directly editing files may lead to lost changes during updates. Instead, custom CSS safely stays available through updating and switching themes, starting fresh for each new theme. Projects such as customize changesets (#30937) and revisions for customizer settings (#31089) would bring future enhancements to this feature and further leverage the opportunities that come with storing the data in post objects.

This is proposed as core functionality rather than remaining as plugin territory because it is designed as the first step toward a next generation of the existing theme editor in core, with a more refined feature set and safer, more user-oriented focus. The theme editor is not proposed to be removed at this time, though with the introduction of this feature it likely makes sense to introduce more friction before accessing the editor (#31779).

Documentation

To improve the user experience further, it is critical that a link to documentation and resources for learning CSS be included with useful help text. This could initially be the “CSS” codex page but would ideally live on a user or developer handbook of some sort eventually (perhaps the theme developer handbook?). This help text must be much more succinct than the help tab on the existing theme editor, conveying what CSS is, where to learn about specific rules, and explaining that it’s specific to each theme in only a few lines.

Help is needed to create a resource for using custom CSS in WordPress, and locate it on WordPress.org. There are some related resources on make/training and WordPress.com has a good introductory page that they may be willing to contribute. Translated versions will eventually be needed as well. If anyone is interested in improving this aspect of the feature, which will presumably live on WordPress.org, please comment on this post.

Security, Capabilities, and Multisite

While the proposal includes basic validation, it is not possible to fully sanitize CSS. For this reason, a new meta capability will be introduced for managing css, unfiltered_css. By default, this is mapped to the unfiltered_html capability.

Site administrators on multisite networks do not have the unfiltered_html capability by default. A plugin that remaps unfiltered_css to a different capability can be created to provide this access on multisite, where custom CSS is especially useful given the need to restrict the number of themes and child themes in the network. This is an area of potential evolution over time.

Related Customize API Improvements

There are a couple of customizer API improvements introduced as part of the implementation of custom CSS in the customizer. A new “Code Editor” customizer control (WP_Customize_Code_Editor_Control) is used for the CSS editor and can also be utilized in plugins and elsewhere in the future. It currently handles line numbers and basic code styling, and will eventually add enhancements such as syntax highlighting.

Additionally, the WP_Customize_Section class has a new “description_hidden” parameter, which locates the section description in the section header behind the help icon toggle (“?”), functioning in the same manner as the customizer panel descriptions.

Contributors

@johnregan3 is leading development of this project, based on initial work by myself (@celloexpressions). @folletto is leading design efforts, with a focus on the long-term growth of the feature for maximum usability.

The implementation takes inspiration from many of the numerous plugins and services that implement custom CSS, specifically including:

  • Simple Custom CSS (@johnregan3)
  • Modular Custom CSS (@celloexpressions)
  • WordPress.com Custom CSS in the design upgrade (Automattic)
  • Jetpack (Automattic)

Testing, Feedback, and Next Steps

Your help is needed in giving feedback on this proposal and testing the feature! To test, please apply the patch either via Trac or the PR (helpful reminder: grunt patch handles both) and try some custom CSS in the customizer using various themes.

Pending approval of this proposal, the next steps will be to finalize and commit the patch on #35395. Code review is ongoing in the GitHub PR linked on the ticket. Feedback on the feature in general and the specific implementation is encouraged via the comments on this post, with any more technical implementation discussion happening on the Trac ticket or GitHub PR.

#4-7, #css, #customize, #proposal, #themes

Feature Proposal: A New Experience for Discovering, Installing, and Previewing Themes in the Customizer

This is the feature merge proposal for the new themes experience in the customizer introduced with #37661. Here’s an overview of the current proposed UI:

Customizer themes design and user flow mockup

Customizer themes design and user flow mockup by @folletto.

A theme is the most fundamental aspect of customizing a site. This project seeks to unify the theme-browsing and theme-customization experiences by introducing a comprehensive theme browser and installer directly in the customizer.

Walkthrough of the latest patch on #37661.

Walkthrough of the latest patch on #37661.

Background & History

The customizer originated as a tool for previewing and customizing themes and as such, was closely integrated into the theme browsing experience in wp-admin when it was introduced in WordPress 3.4. The theme browser and installer were rewritten in WordPress 3.8 and 3.9, respectively, offering a fast JavaScript-based way to explore, install, and switch themes.

Eventually, as the customizer’s role grew to that of a framework for live-previewing any change to a site, it became apparent that it would benefit from a more direct way to switch themes, without entering the wp-admin context. The Customizer Theme Switcher plugin was created, and after some refinement, merged into WordPress 4.2. However, while it initially included external links to install themes in the admin, these were eventually removed due to the jarring experience of unexpectedly leaving the customizer.

Currently, there is no indication that additional themes can be installed when viewing available themes in the customizer. For new users, it may take quite a bit of time to discover the ability to install themes, via wp-admin, or they may give up on WordPress before making this discovery. This is a usability dead-end where a user’s flow is disrupted in the process of discovering, installing, previewing, and activating themes, both on initial site setup and when considering a redesign.

When the theme switcher plugin was developed, the team made preliminary plans for a theme installation interface as a second phase of the project. Specifically, it would leave the “preview” context of the customizer but retain the same identity in the user experience. @folletto helped develop this initial concept in early 2015.

Technical Constraints & Requirements

There have been several technical limitations preventing theme installation in the customizer from being addressed previously. Most notably, such an interface requires “shiny” ajax-based theme installation, updates, and deletion, so that the user flow can persistently stay in the customizer themes interface rather than jumping to separate “installing” views. This is now possible with phase 2 of “Shiny Updates” in WordPress 4.6. Additionally, expansions of the customizer JavaScript and JS-templated controls APIs to better support dynamically-registered controls were needed to support theme installation within the customizer framework, and these were fully fleshed out for the customizer menus interface introduced in WordPress 4.3. With these technical constraints eliminated, theme installation in the customizer is now possible without additional significant improvements to the underlying themes or customizer APIs.

The customizer must currently be completely reloaded from PHP to preview a different theme. To perform a theme switch without a reload, theme-defined settings, sections, and controls would need to be updated dynamically with JavaScript. While the customizer internals have been working toward making this possible for some time, significant work remains to make inline theme switches possible. Therefore, changes to this part of the theme-switching workflow are out of scope for the current project, which focuses on the user-facing flow.

The biggest usability block that this limitation causes is that unsaved changes are lost when the theme is switched. Unsaved changes are currently handled by prompting users with an are-you-sure notice in the browser before making the switch. Unfortunately, limitations in JavaScript require the loading indicator to be hidden after the user decides to stay on the page or to continue to the new theme, causing confusion. In the new interface, this is further mitigated by displaying a warning that there are unsaved changes, with an inline button to save and publish them, at the top of the interface. With customize changesets (transactions) (#)30937, a “save draft” option could also become possible in the future, allowing changes to be saved (potentially automatically) without being published between theme previews.

Previewing Themes

One of the biggest challenges with theme installation in wp-admin, and opportunities in the customizer, is previewing themes. Currently, a customizer-like frame displays a preview hosted on WordPress.org, with limited content. Rather than opening this potentially-disorienting similar but different interface, the proposed flow de-emphasizes the distinction between installed and available themes. The primary action for available themes is now “Install & Preview”, which installs the theme and live previews it in one step.

Users can now see any theme on their site with their content and play with its options in the customizer in one click. If they decide it’s the wrong theme for their site, the themes panel can be quickly reopened and another theme selected and previewed with no harm done. A secondary action allows themes to be installed without instantly previewing, so that the installed themes tab can become a personal theme library of sorts, where users can save themes that they might want to try on their site. Installed themes being a filter along with the available theme headings unifies the previously-disorienting separation of themes and add-new themes on separate screens, with confusingly-separate search and header (add new/upload theme) functionality.

Proposed Themes Interface

Due to the tight integration with the existing system, with the existing theme control and section as well as internal elements in the customizer manager and theme details template requiring moderate modifications, this project was implemented as a patch and cannot be reasonably converted into a plugin and back. The patch has been available on trac for six+ weeks, with iterations continuing to improve and polish the new experience.

The technical implementation continues adapting the concepts present in the backbone.js-based themes experience in wp-admin to leverage the customizer API. With the themes experience natively built on the customizer framework, it should be much easier for developers to improve and maintain the core experience in the future as well as extending the core experience in a structured way.

A few highlights of the proposed details:

  • Installed themes are no longer loaded every time the customizer is opened, resulting in potentially significant performance improvements by only calling wp_prepare_themes_for_js() when needed. This also allows themes in the customizer to be fully disabled with remove_panel( 'themes' ).
  • The themes experience is unchanged on the top level of the customizer, but selecting the change theme button now opens a panel that fills the entire screen, as the preview is not relevant when considering a theme change.
  • The UI diverges somewhat from what is found in the theme installer in wp-admin (which will remain), particularly around the filters.
  • The theme details view is unified between installed and available themes; clicking on a screenshot opens the details view to match the admin UI.
  • Primary buttons are used where clicking them takes you away from the current page (ie, for previews); secondary buttons are used elsewhere.
  • The loading strategy attempts to balance performance with wait time by loading theme data from Ajax in large batches (100 themes) and following up by rendering screenshots as they become visible (as the existing interface does).

Usability Testing

Four usability tests have been conducted so far. The full test screencasts are available on Make/Design, alongside key takeaways. These tests expose a lot of largely-known issues with themes and the customizer in general, but did not reveal any significant issues with the proposed new theme browser. Because the tests were conducted in-person with a limited set of volunteers, additional testing with a broader user base would be ideal.

There has been design feedback since the user testing was conducted, resulting in some significant changes. @karmatosed has volunteered to coordinate additional testing in the next week to verify that the changes haven’t introduced usability regressions, and to test with a broader audience. Check out the call for user testing on make/design to help out here.

A visual record on a phone of the revised design has been posted on make/flow.

Extensibility

Because the new interface is built entirely on the customizer API, third-party plugins should now be able to integrate much more easily. This means that other theme marketplaces (including commercial themes) could realistically be browsed (and maybe even installed) from within WordPress, while leveraging the core UI exactly.

The presentational flexibility is available via the customizer API (with custom theme sections for other theme sources, and theme controls for individual themes), but there are likely some gaps in the ability to do this seamlessly in the internals. If anyone is interested in building this sort of functionality, please evaluate whether any additional hooks are needed so that they can launch alongside the new feature.

Review and Approval

In addition to a general core approval of this proposal, the following sign-offs are required before the feature could be approved for merge, based on the applicable elements of this list:

  • Flow (and mobile) review (see also an initial post)
  • Docs review
  • Security audit
  • Polyglots/i18n review
  • Design/UX review – tentative approval has been provided from @karmatosed and @folletto (with additional input from others in last week’s design meeting) with an expectation that minor adjustments will continue to be required. General design feedback is still welcome, but major changes are unlikely to be feasible at this point.
  • Accessibility review – @afercia completed an initial review, with the issues fixed in a subsequent patch. A comprehensive final review would be a good idea as well, since there have been significant design changes.
  • Code review – to be handled by @westonruter once the patch is otherwise deemed “ready” based on review from other teams.

To test, update to latest trunk and apply the latest patch on #37661. On your test site, open the customizer and “change” the theme. Try out the various filters, browse themes, and install and preview them. Also test the inline update and deletion functionality.

To meet the feature merge deadline for 4.7 (10/19), reviews from various teams and any corresponding iterations need to be completed by October 12th, leaving a week for final code review and commit. General feedback and specific reviews and action items should be provided as comments on this post.


Update: this feature was merged in 4.9.

#4-7, #4-9, #customize, #proposal, #shiny-updates, #theme-switcher, #themes

Feature Proposal: Content Authorship in Menus, with Live Preview

The current navigation menus system is built around a paradigm that every menu item must be associated with an existing piece of content. However, this is problematic for new users, who may find themselves with the opportunity to build a menu before creating any content. #34923 seeks to improve this experience and eliminate this usability “dead end” by adding the ability to create new post objects (most notably pages) within the menus interface in the customizer.

content-authorship-nav-menus

Purpose & Goals

While this feature is aimed primarily at new users setting up their site for the first time, it may also be useful for users that are restructuring a site, or even if they want to add a page here or there and add it to their menu before filling out its contents. It’s important to note that posts and pages added here are stubs – only their title is filled out, and the content will be added later via the post editor. For that reason, most existing sites would probably maintain their existing workflows to create new pages via the post editor, publish them, and then add them to menus. This is of course still fully supported and the new proposal seeks to provide an alternative approach that may be better for different use cases such as new sites.

Because the feature is proposed to be located in the customizer, it also fully supports live preview. The live preview component can build user trust and confidence by letting users preview and interact with their site as changes are made, before they’re published.

Technical Considerations

To allow new content to be created in the customizer, posts are created with the auto_draft status. When the user saves & publishes in the customizer, these newly-created posts are transitioned to be published. In the customizer preview, the status is modified to protected to allow the posts to be previewed.

Ideally, term creation should also be a part of the menus UI. Unfortunately, terms to not currently have a status field, and implementing that or looking at alternative approaches is something that will take more thought. Thus, this feature is being built with future support for terms in mind, but without support currently.

By default, the ability to add new content depends on the appropriate capabilities and the show_in_nav_menus parameter when registering post types. Additionally, a new filter is proposed to allow authorship in menus to be disabled on a per-post-type basis.

Testing

User testing has been requested and there are several tasks that could be tested:

  • Create a new menu for a brand new site by creating pages and adding them to a menu, assign the menu to a location and publish it.
  • On an existing site, create new pages to add to an existing menu.
  • On a complex site with multiple menus create a new page once then add it to multiple menus.
  • On sites with custom post types, create new custom posts for any of the scenarios above.

Test with the latest patch on #34923 and make sure the customize posts plugin is not activated (it has conflicts). If anyone is able to conduct user tests that would be much appreciated.

Customize Posts Plugin

Part of the inspiration for this feature comes from the Customize Posts plugin, which has the ability to live-preview posts and post meta in the customizer. Only a very small portion of the plugin would make its way into core as part of the nav menus content authorship feature. However, the proposal is currently to establish the wp.customize.Posts namespace for future expansions of post-related functionality in the customizer. The existing plugin would extend this core namespace, and other plugins could do so as well.

There are currently no plans to consider the Customize Posts plugin for full merge into core; however, stay tuned for an upcoming feature project kick-off that will seek to explore the future of live preview in WordPress at a broader level.

Feedback & Next Steps

The latest patch is currently seeking feedback for accessibility, design, code review, docs, and general comments. Please test it and leave your feedback as comments on this post or the ticket, #34923. Given current timing the ticket is likely to target the 4.7 milestone.

One remaining piece to implement is a mechanism to inform the user that their new posts have been published and provide links to edit the posts in the admin. This could take the form of a notifications area in the customizer or preview, or happen inline within the menus UI as a notice using the customizer setting validation feature that’s new in 4.6. UX feedback and ideas for the potential approach here are needed, and it could also probably work to ship without this part for now.

#customize, #menus, #proposal

Proposal: More Shiny Updates

The Bleak screen of Sadness™ 😢 that users encounter when installing/updating/deleting plugins or themes is a terrible experience WordPress users. It’s not timely anymore and doesn’t reflect the values WordPress strives to adhere to. Instead, WordPress needs a simpler and more straight forward experience when installing, updating, and deleting items.

That’s why the Shiny Updates Team is proposing a merge of the Shiny Updates plugin into WordPress 4.6 💥. We’re eager to hear feedback from WordPress core contributors and users alike.

Old plugin install process

Existing plugin install process, showing The Bleak Screen of Sadness.

Purpose & Goals

The Bleak screen of Sadness™ is disruptive to user workflows, pulling them out of the context of plugins or themes, and dropping them into a screen filled with technical details that most users don’t care about or don’t understand. Shiny Updates deals with these details behind the scenes, maintaining the context of the triggered actions and leaving users with clear actions and results.

This caters to two core principles of WordPress, designing for the majority, and striving for simplicity. Users don’t really care about the internal process of installing or updating themes and plugins. Listing out these technical steps for them is unnecessary at best.

With Shiny Updates these actions also don’t require a page reload anymore, which creates a simpler workflow without context changes and lets users achieve their goals of an enhanced WordPress experience quicker.

We also revamped the whole Dashboard -> Updates page to improve simplicity and make the process of updating translations and WordPress core shiny as well. 🎉

Project Background

Whether it was through the update mechanism available under Dashboard -> Updates or the automatic updates introduced in version 3.7, WordPress has always been encouraging users to update their sites to the newest versions.

Plugin updates have been made shiny in WordPress 4.2, but now we want to extend this to other areas as well. Shiny Updates v2 improves the update process for themes, translations and even WordPress itself, as well as install and delete workflows for plugins and themes.
As of today, the plugin has been downloaded about 8,000 times and is actively installed on over 1,000 WordPress sites. We’ve gotten input from many users and core committers through GitHub and during regular meetings in the #feature-shinyupdates.

You can read more about the shiny updates flow with various visual records on make/flow, where we also shared results of the various user tests we did. Doing multiple rounds of user testing has really shaped the whole project and helped us refine the plugin and improve the overall usability of installing updates in WordPress.

Implementation Details

Shiny Updates builds upon the shiny plugin updates feature already existing in core, which basically consists of some JavaScript and Ajax callbacks for updating plugins in the background. As such, it can easily be extended by the new JavaScript parts of Shiny Updates. All new JavaScript functionality is available under the wp.updates umbrella.

Here’s it looks like in action:

In addition to that, we propose a revamped updates overview under Dashboard -> Updates. It’s simpler, more elegant, more shiny:

Shiny Updates Table

With Shiny Updates, the Dashboard -> Updates page gets a much needed overhaul

Relevant Core Tickets

Merging Shiny Updates into core would resolve a long list of outstanding trac tickets related to updates, including #31529, #31530, #31531, #31532, #31534, #31535, #31773, #33637 and #35032. All tickets related to Shiny Updates can be found here.

Remaining Issues

There are a few remaining bugs on GitHub, which will be resolved by Friday, June 3rd. Since the revamped updates table relies on plugin icons being returned by the Plugins API, the API needs to be changed as part of the plugin directory update. The new directory will launch well before the 4.6 release, so that shouldn’t be a big deal. As a bonus, this change would also enable us to fix #30186.

Contributors and Feedback

This is a proposal and is subject to revision based on your feedback. If you haven’t already tried out the plugin, please download and install it from WordPress.org or the comfort of your WordPress admin. You can review the current code and leave feedback at the project’s GitHub repository or in #feature-shinyupdates on Slack.

Thanks a lot to everyone who has been contributing to this plugin since its inception, especially @obenland for leading this project, @adamsilverstein for his numerous contributions, @mapk for helping with testing and UX, and @ocean90 for giving valuable feedback despite being super busy with leading 4.6.

So far we’ve received positive feedback from different core teams like the accessibility and design teams, and we have reached out to @drew who will review the docs once a core patch is ready.

#4-6, #feature-plugins, #merge, #proposal, #shiny-updates

Proposal: Make/Core Post Guidelines

During the 4.3 retrospective, one commonly mentioned theme was make/core posts and improving their style and language. All active contributors should review this post and comment with your feedback. This is a DRAFT and a PROPOSAL, it is not a whack from above with Mjölnir. After revising, this document will live in the core handbook.

Introduction

The Make WordPress Core Blog (make/core) is the official blog of the WordPress core team. It is read by thousands of people, many of which don’t know the intricacies of WordPress core or that don’t fully understand the process of how core is developed. In order to ensure the best experience possible for the developer community, these guidelines are developed with the reader in mind. The goal of most make/core posts is two-fold: to generate feedback from the developer community (including testing) and to ensure developers know about changes and can plan accordingly.

When to write a post

All API changes should have a post written on the make/core blog, ideally no later than the first week of the beta period.   Examples of API changes that should be announced include, but are not limited to: new filters, new actions, changed order of hooks, substantial enhancements to queries (The Boone Gorges Rule), Doing a ticket binge on a component, changing the purpose of a parameter in a hook, and new general helper functions.

Grouping related changes into one post is fine. For example, a single post called “Customizer changes in WordPress 4.2” is fine, rather than individual changes about each commit. If in doubt, discuss your posting plans during the weekly dev chat.

Changes should be announced as early as feasible. There is almost always more feedback on make/core posts than in Trac tickets. This feedback is important to the development process.

Peer Review

It is strongly encouraged to ask a committer to proofread a post before publishing it. This is to ensure that you look your best when publishing a post. Merge Proposals should always be read by the release lead (or a designee) before posting. Release dev notes should be read by the release lead (or a designee) before publishing.

Style and Substance

First Person pronouns should be avoided. As the blog is the official voice of the core team, you should keep your personal thoughts out of the body of the post and instead put them in comments.

In general, the word “We” should be avoided without it being very clear who the group is you are speaking about being a member of.

Posts that are designed as Requests for Comments or are draft roadmap in nature should make sure to highlight that it is a draft proposal. Highlight this in the title and in the opening paragraph.

Many people reading make/core don’t speak English as a first language (or in the case of Jorbin, don’t read or write in any known language). Keep that in mind when choosing words. It’s better to keep it simple than it is to sound smart. In general, the tone should be similar to WordPress: Friendly.

Meeting Announcements

When announcing a meeting, you should always use the time shortcode so that visitors see when a meeting is in their local time. You should also include what slack channel the meeting will take place in.

Tags

Almost all posts are related to a specific component, please tag them with the component name.
For announcements related to a specific release, please tag them with the release number.
For announcements of API changes, including additions, please tag them with dev-notes.
For posts related individual projects and initiatives, please tag them consistently with a project name.

#makecore, #proposal, #this-is-so-meta

Feature Plugin Merge Proposal: oEmbed

For the past 6 years, users have been able to embed YouTube videos, tweets and many other resources on their sites through a nifty feature called oEmbed.

Today, we (mainly me, @pento and @melchoyce) ask to consider extending this feature by merging the oEmbed API plugin into WordPress core. This plugin allows anyone to embed posts from your site by just pasting its URL. We’ve been working hard on it for months and are now eager to hear your feedback.

Purpose & Goals

While I initially built an early version of the plugin about a year ago, it was @melchoyce who kicked things off with #32522. Her idea was simple: When you can embed almost anything in a WordPress post, why aren’t we able to embed WordPress posts themselves in another WordPress post?

That’s exactly what we’re aiming for. Our goal is to allow a big portion of the web to easily and securely embed such post previews.

Have a look at this post to see the user flow for this feature (and a live demo!):

https://make.wordpress.org/flow/2015/09/12/embeds-visual-survey-of-the-oembed-feature-plugin-macnchrome-iphone-5s/

Security Considerations

Embedding content from a random source on your site depends on lots of trust. We take precautions to make the whole process as easy as possible. It’s worth noting that:

  • We use iframes with the sandbox attribute to enable extra restrictions on the content that can appear in the inline frame.
  • The host and the embedded site communicate via postMessage to allow resizing and clicking on links safely

Browser Compatibility

We successfully tested the feature with all major browsers on mobile and desktop. Since IE < 10 doesn’t support the sandbox attribute, we use the proprietary security attribute there to have similar security restrictions. This means no JavaScript inside the iframe is run, e.g. for the resizing. The most important thing, clicking, still works there though.

Long story short, the feature works with all major browsers and degrades gracefully on older IE versions.

Core Changes & Merge Implementation Details

The plugin was developed in such a way that merging it into core eventually is as straightforward as possible. We are working actively on a patch that can be added to core within the merge window.

There are two things that we need to change in core together with the merge:

  • When doing a redirect because of a changed post slug the rewrite endpoints need to be respected as well. See #33920
  • Attachment rewrite endpoints need to be fixed in core. See #19918
  • This feature only works with oEmbed discovery turned for every user, even those lacking the unfiltered_html capability. That capability check needs to be removed.

Developer Notes

This plugin adds a new /embed/ rewrite endpoint for posts, pages and attachments. We haven’t found any plugin in the directory using this endpoint, but if you already use that endpoint, you should consider renaming it or changing the oEmbed rewrite endpoint using the filters we provide.

Note: There’d be a separate post for this after the merge.

What about the REST API?

The plugin works well on WordPress 3.9+ and uses a simple class to return the oEmbed API responses. However, with the REST API officially proposed for a core merge, we built a controller class for it. This class does exactly the same and follows the REST API best practices.

We could definitely profit from the REST API and built upon it when merging into core. Not needing a fallback means no duplicated code and easier maintenance.

In case you missed it, here’s the REST API merge proposal:

https://make.wordpress.org/core/2015/09/21/wp-rest-api-merge-proposal/

Remaining Issues

There is currently one issue with Slack not displaying the oEmbed output correctly. It simply displays the JavaScript as plain-text instead of removing it. We’ve reached out to them to see if they could fix that and at least white-list WordPress.org in the meantime to properly display the embeds in Slack channels.

Besides that, there are also some small layout quirks we still need to work out. Meanwhile we’re continuing to improve the codebase and inline documentation.

Future Plans

We’re looking into improving support for different response types in addition to regular post content, depending on the feedback we receive from users.

Contributors

While I’ve been the lead developer of the plugin, a ton of valuable input and contributions have come from others in the community.

#4-4, #feature-plugins, #feature-oembed, #merge, #proposal

WP REST API: Merge Proposal

Hello everyone! This is the post you’ve all been waiting for. 🙂

We on the REST API team (myself, @rachelbaker, @joehoyle, @danielbachhuber, and newest member and core committer @pento) would like to propose merging the REST API into WordPress core. We’ve been working a while on this, and think it’s now ready to get your feedback.

This is our first iteration of the proposal, and we’re actively looking for feedback. If you have thoughts on the project, or on this proposal, let us know! Only with your feedback can we make progress. 🙂

What is the REST API?

The REST API is a nice and easy way to get at your data in WordPress externally, whether that’s from JavaScript in a theme or plugin, mobile and desktop applications, or importing and exporting data. The API offers up all core data types (posts, terms comments, and users), plus support for meta and revisions; we’ve got plans to eventually have access to everything the admin and frontend have access to.

The REST API differs from existing WordPress APIs in that it is explicitly designed from the ground up for modern mobile and browser usage, using the lightweight and widely-supported JSON data serialization format with a modern REST interface. Both of these are already familiar to most developers: JSON is a subset of JavaScript intended purely as a data interchange format, and REST is a set of best practices around HTTP. Both are supported natively by almost every programming language and platform.

Why do we need a new API?

WordPress already has external APIs: XML-RPC, designed for desktop clients; Atom and RSS feeds, designed for post syndication; and the venerable admin-ajax, designed for Ajax requests in the admin and frontend. These APIs all serve different purposes, but often have a great deal of overlap. In addition, these have all been stretched beyond their original intentions: XML-RPC now contains site management tools, RSS has been extended into the WXR export format, and admin-ajax is the catch-all of any sort of browser-server communication in plugins and themes.

The REST API builds upon the heritage of these APIs to provide better support today for using these, as well as laying the groundwork for expanded use in the future.

XML-RPC is the closest analogue to the REST API in terms of usage and capabilities. Originally designed back in 1998 to allow desktop clients to create and edit posts on blogs, WordPress has extended this with both other specifications (such as MetaWeblog) and with its own proprietary additions. Fundamentally, XML-RPC is built around Remote Procedure Calls (RPC), essentially a way of calling a function externally. It then uses XML to serialize the data for passing back and forth.

Unfortunately, XML serialization can be problematic at times. XML has lots of power, but support for custom entities and DOCTYPEs can cause parsing problems and security attacks, including billion laughs, and XXE exploits. (Currently, WordPress has to disable parts of the XML parser and apply regular expression replacements to be able to parse these safely.) XML is also very verbose, and represents data in a way which doesn’t map easily to programmatic data structures. JSON on the other hand is both concise and well-represented in memory, as it’s based on JavaScript’s native syntax.

The admin-ajax API is also very commonly used in WordPress, albeit typically only by plugins and themes. This is a very lightweight API that essentially acts as a mini-router. Typical usage of this API uses JSON, since it’s a browser-based API, but all usage is completely custom. A lot of the usage of this involves retrieving or updating posts on-the-fly, but due to its nature as simply a framework, these are done in completely different ways. This doesn’t lead itself to extensibility, and requires a lot of duplication every time developers want to get data in or out. We don’t want to replace all of admin-ajax though, since some use cases don’t map exactly: UI-related interactions or things like the Heartbeat API are great examples of this.

The REST API can help to supplant these uses. Our aim is to eventually replace the XML-RPC API completely, to act as a secondary import/export format, and to replace most (but not all) uses of admin-ajax. The REST API offers an easier to use interface than the existing solutions.

Why this project?

We’ve been working on this project ever since the first WordPress Contributor Summit back in 2012. Since then, we’ve had lots of feedback from core developers, the community at large, and further beyond in the form of client developers. We believe that the REST API has an immense amount of experience behind it, and plenty of viewpoints have contributed to the project’s direction.

The API has seen great usage in the community, from various mobile apps to large news sites. This usage has helped to battle-test and prove out the API. In the process, we’ve found plenty of bugs, and some security issues. Thanks to this feedback, the API is incredibly stable and secure. (The most recent security bugs we fixed were relatively minor bugs.)

We also designed the API from the ground-up to be part of core, following a core-like mentality to our processes. The API is intended to be both a great feature and a base to build off in plugins. We undertook a significant refactoring and partial rewrite in version 2 to make this extensibility even better. This open process also means that most of the design decisions are documented publicly as we’ve engaged stakeholders to gauge feedback.

We believe these pieces combined make this a fantastic feature for WordPress core, and we hope you all agree. 🙂

What’s the plan?

The plan we’re aiming for is a two part merge of the API. For the first stage, the infrastructure code would be merged into wp-includes and made available for plugins and themes. This is an internal API only, but offers an “API construction kit” for developers to use. For the second stage, the endpoints would be merged, and the API would be enabled for sites by default.

This plan splits the API into two parts, infrastructure and endpoints:

  • Stage One: Infrastructure: The infrastructure is the code responsible for routing requests and handling the “meta” layer of the API, including JSON serialisation/deserialisation, linking, embedding, and REST best practices. This adds a simplified routing layer outside of WP’s rewrites, allowing non-query-var rewrites easily, acting as a base for building APIs inside WordPress.
  • Stage Two: Endpoints: These are where much of the complexity of the API lies, as they’re responsible for mapping data from the external JSON format to the internal data structures, and vice versa. The “business” logic of integrating with WordPress is almost entirely contained within the endpoints. These are the more complex part of the API, as they require using deep APIs in WordPress, and handling security and privacy concerns.

With this plan, we would aim to have the infrastructure merged in 4.4, and the endpoints merged one release later in 4.5.

The slow nature of this plan allows a longer review time on the API for core committers. It also gives extra time for reviewing the endpoints, since they would be delayed one release.

Merging the infrastructure now would allow third-party code to begin using the API to build upon it, including migrating from existing custom code. It would also help to increase developer confidence in the API (as it represents a commitment by the project towards a REST API).

In this plan, the first stage would not include any of the base controllers (such as the posts controller). This may limit the utility of the infrastructure for plugins and themes, however as the endpoints would be merged a cycle later, it’s expected that this wouldn’t have a huge impact.

The infrastructure of the API is approximately 2700 lines of code (roughly a third of the API codebase), and the endpoints make up the remaining 5500 lines of code.

What would happen after merge?

After merging the REST API into core, we’d spend approximately two weeks partying and celebrating. 🙂

Once we’re done with the parties, one major question would be how we manage the API in the future. The existing management and contribution process via GitHub has been extremely successful, as we’ve had 61 people’s pull requests merged into the API. Contribution via GitHub is especially useful for the API, as it’s a heavily developer-focussed project, and is aimed at external (non-WordPress) developers. On the other hand, all other contribution to WordPress is done via SVN and Trac, so integrating with this process is important for existing developers, as well as core’s general processes. We need to ensure the API is an integral part of core, not a separate project.

Given the team’s experience with GitHub as well as Trac, we can bring the best of both worlds by helping integrate the two. This would also improve contribution for WordPress as a whole, and benefit the whole community. This will be especially important as we encourage more contributions from the wider community (client authors, for example). We think we can make good progress here, and we’d love to try and help improve the process. 🙂

In addition to the project management, there are still further API projects we need to tackle. Authentication is the most important of these, as a huge focus on OAuth and similar would be needed to make the API more useful for external clients. Currently, we haven’t had enough time to spend on this as well as managing the API project, however the API is now reaching a finalised stage, so this should be able to improve quickly. Centralised authentication is a huge part of this, as the regular OAuth registration process is cumbersome for a distributed system like WordPress.

Important note: We don’t believe authentication is required for the API merge, and we treat it as a separate project. The authentication is a completely separate system to the API itself. This is something we’d give more time to in the future, but we want to focus on shipping the API for now.

Let’s go!

This is our merge plan for the API, however it’s not finalised. If you’ve got comments, thoughts, suggestions, opinions, or words of encouragement, let us know. We’d love to know what you think. Thank you all, you’re wonderful, and stay golden.

#feature-plugins, #json-api, #merge, #proposal, #rest-api

Taxonomy term metadata proposal

During a recent meeting focused on the taxonomy roadmap, a number of contributors discussed the possiblility of adding metadata for taxonomy terms to WordPress #10142. Assuming we decide to build termmeta, a major hurdle will be compatibility with the many plugins – both publicly available and privately developed – that already implement their own version of termmeta. We plan to do an in-depth scan of all public plugins that conflict with the core implementation (function names, database table schema, etc) and to mount an outreach and documentation project to help affected plugin developers move to the core system. Our goal is zero fatal errors or lost data on WP installations.

But before moving forward with research of existing plugins, we need a fairly clear sense of what the core implementation would, in fact, look like. These details will serve as the basis for a set of heuristics developers can use to tell whether their plugins will conflict with what goes into core.

Toward that end, below is the outline of a proposed implementation of termmeta.

  1. Database
    1. Term metadata will be stored in a new database table, called {$wpdb->prefix}termmeta (wp_termmeta, for convenience in this proposal).
    2. The termmeta table name will be stored at $wpdb->termmeta.
    3. The wp_termmeta schema will mirror the schema for postmeta and usermeta tables:
      CREATE TABLE $wpdb->termmeta (
          meta_id bigint(20) unsigned NOT NULL auto_increment,
          term_id bigint(20) unsigned NOT NULL default '0',
          meta_key varchar(255) default NULL,
          meta_value longtext,
          PRIMARY KEY (meta_id),
          KEY term_id (term_id),
          KEY meta_key (meta_key($max_index_length))
      ) $charset_collate;
  2. New API functions
    New API functions will closely mirror those for postmeta and usermeta. They will primarily be wrappers for the _metadata() family of functions. We may also build in some fault tolerance for when $term_id does not identify a unique term (ie, because shared terms have not yet been split by the 4.3 upgrade routine).
    1. add_term_meta( $term_id, $meta_key, $meta_value, $unique = false )
    2. delete_term_meta( $term_id, $meta_key, $meta_value = '' )
    3. get_term_meta( $term_id, $meta_key, $single = false )
    4. update_term_meta( $term_id, $meta_key, $meta_value, $prev_value = '' )
    5. update_termmeta_cache( $term_ids )
  3. Other API improvements
    1. Termmeta cache priming for term-fetching functions: get_term(), get_term_by(), get_terms(), wp_get_object_terms(). See _prime_post_caches() for how this works for postmeta.
    2. A 'update_term_meta_cache' argument for functions that fetch multiple terms (get_terms(), wp_get_object_terms()) that will allow developers to prevent cache priming described in 3.1 above. As in the case of posts, 'update_term_meta_cache' will default to false.
    3. A new 'meta_query' argument for get_terms() and wp_get_object_terms().

Please note that this is a draft. Details are subject to revision based on feedback from contributors. If you see something missing or amiss in the above, please leave a note below. (Please also note that this doesn’t mean that termmeta is a done deal. Sketching the implementation is part of the evaluation process.) Thanks in advance for your feedback!

#proposal, #taxonomy