XML Sitemaps Feature Project Proposal

Note: a follow post was published with more recent information about this project.

While web crawlers usually discover pages from links within the site and from other sites, sitemaps supplement this approach by allowing crawlers to pick up all URLs included in the sitemap and learn about those URLs using the associated metadata.

Today, WordPress coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. does not generate XML Sitemaps by default, affecting a high number of WordPress websites search engine discoverability. 4 out of the top 15 plugins on WordPress pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party repository currently ship with their own implementation of XML sitemaps, pointing to a universal need for this feature and a great potential to join forces.

This post proposes integration of XML Sitemaps to WordPress Core as a feature project. The proposal was created as a collaboration between Yoast*, Google** and various contributors.

Proposed Solution

In a nutshell, the goal of the proposal is to integrate basic XML Sitemaps in WordPress Core and introduce an XML Sitemaps APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. to make it fully extendable. Below is a diagram of the proposed XML Sitemaps structure:

XML Sitemaps will be enabled by default making the following content types indexable

  • Homepage
  • Posts page
  • Core Post Types (Pages and Posts)
  • Custom Post Types
  • Core Taxonomies (Tags and Categories)
  • Custom Taxonomies
  • Users (Authors)

Additionally, the robots.txt file exposed by WordPress will reference the sitemap index.

Developers

An XML Sitemaps API will be introduced as part of the integration allowing extensibility. At a high level, below is a list of the ways the XML Sitemaps may be manipulated via the API:

  • Add extra sitemaps and sitemap entries
  • Add extra attributes to sitemap entries
  • Provide a custom XML Stylesheet
  • Exclude a specific post type from the sitemap
  • Exclude a specific post from the sitemap
  • Exclude a specific taxonomyTaxonomy A taxonomy is a way to group things together. In WordPress, some common taxonomies are category, link, tag, or post format. https://codex.wordpress.org/Taxonomies#Default_Taxonomies. from the sitemap
  • Exclude a specific term from the sitemap
  • Exclude a specific author from the sitemap
  • Exclude a specific authors with a specific role from the sitemap

Non Goals

While the initial XML Sitemaps integration will fulfill search engines minimum requirements and cover most WordPress content types, below is a list of features which will not be included in the initial integration:

  • Image sitemaps
  • Video sitemaps
  • News sitemaps
  • User-facing changes like UIUI User interface controls to exclude individual posts or pages from the sitemap
  • XML Sitemaps caching mechanisms

i18ni18n Internationalization, or the act of writing and preparing code to be fully translatable into other languages. Also see localization. Often written with a lowercase i so it is not confused with a lowercase L or the numeral 1. Often an acquired skill.

The XML Sitemaps will leverage standard internationalization functionality provided by WordPress core.

Since there are plans by WordPress leadership to officially support multilingual websites in WordPress, the XML Sitemaps will be flexible enough to list localized content in the future as per web development best practices.

What’s next?

Your thoughts on this proposal would be greatly valued. Please share your feedback, questions or interest in collaboration by commenting on this post. After that we can decide on how to best proceed with this proposed project and set up a meeting on SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. to kick things off.

* @joostdevalk, @omarreiss, @jonoalderson, @herregroen

** @swissspidy @albertomedina @westonruter @flixos90 @tweetythierry

#feature-plugins, #feature-projects, #proposal, #seo, #sitemaps, #xml-sitemaps

Proposal: More Shiny Updates

The Bleak screen of Sadness™ 😢 that users encounter when installing/updating/deleting plugins or themes is a terrible experience WordPress users. It’s not timely anymore and doesn’t reflect the values WordPress strives to adhere to. Instead, WordPress needs a simpler and more straight forward experience when installing, updating, and deleting items.

That’s why the Shiny Updates Team is proposing a merge of the Shiny Updates plugin into WordPress 4.6 💥. We’re eager to hear feedback from WordPress core contributorsCore Contributors Core contributors are those who have worked on a release of WordPress, by creating the functions or finding and patching bugs. These contributions are done through Trac. https://core.trac.wordpress.org. and users alike.

Old plugin install process

Existing pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party install process, showing The Bleak Screen of Sadness.

Purpose & Goals

The Bleak screen of Sadness™ is disruptive to user workflows, pulling them out of the context of plugins or themes, and dropping them into a screen filled with technical details that most users don’t care about or don’t understand. Shiny Updates deals with these details behind the scenes, maintaining the context of the triggered actions and leaving users with clear actions and results.

This caters to two coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. principles of WordPress, designing for the majority, and striving for simplicity. Users don’t really care about the internal process of installing or updating themes and plugins. Listing out these technical steps for them is unnecessary at best.

With Shiny Updates these actions also don’t require a page reload anymore, which creates a simpler workflow without context changes and lets users achieve their goals of an enhanced WordPress experience quicker.

We also revamped the whole Dashboard -> Updates page to improve simplicity and make the process of updating translations and WordPress core shiny as well. 🎉

Project Background

Whether it was through the update mechanism available under Dashboard -> Updates or the automatic updates introduced in version 3.7, WordPress has always been encouraging users to update their sites to the newest versions.

Plugin updates have been made shiny in WordPress 4.2, but now we want to extend this to other areas as well. Shiny Updates v2 improves the update process for themes, translations and even WordPress itself, as well as install and delete workflows for plugins and themes.
As of today, the plugin has been downloaded about 8,000 times and is actively installed on over 1,000 WordPress sites. We’ve gotten input from many users and core committers through GitHub and during regular meetings in the #feature-shinyupdates.

You can read more about the shiny updates flow with various visual records on make/test, where we also shared results of the various user tests we did. Doing multiple rounds of user testing has really shaped the whole project and helped us refine the plugin and improve the overall usability of installing updates in WordPress.

Implementation Details

Shiny Updates builds upon the shiny plugin updates feature already existing in core, which basically consists of some JavaScriptJavaScript JavaScript or JS is an object-oriented computer programming language commonly used to create interactive effects within web browsers. WordPress makes extensive use of JS for a better user experience. While PHP is executed on the server, JS executes within a user’s browser. https://www.javascript.com/. and Ajax callbacks for updating plugins in the background. As such, it can easily be extended by the new JavaScript parts of Shiny Updates. All new JavaScript functionality is available under the wp.updates umbrella.

Here’s it looks like in action:

In addition to that, we propose a revamped updates overview under Dashboard -> Updates. It’s simpler, more elegant, more shiny:

Shiny Updates Table

With Shiny Updates, the Dashboard -> Updates page gets a much needed overhaul

Relevant Core Tickets

Merging Shiny Updates into core would resolve a long list of outstanding tracTrac An open source project by Edgewall Software that serves as a bug tracker and project management tool for WordPress. tickets related to updates, including #31529, #31530, #31531, #31532, #31534, #31535, #31773, #33637 and #35032. All tickets related to Shiny Updates can be found here.

Remaining Issues

There are a few remaining bugs on GitHub, which will be resolved by Friday, June 3rd. Since the revamped updates table relies on plugin icons being returned by the Plugins APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways., the API needs to be changed as part of the plugin directory update. The new directory will launch well before the 4.6 release, so that shouldn’t be a big deal. As a bonus, this change would also enable us to fix #30186.

Contributors and Feedback

This is a proposal and is subject to revision based on your feedback. If you haven’t already tried out the plugin, please download and install it from WordPress.orgWordPress.org The community site where WordPress code is created and shared by the users. This is where you can download the source code for WordPress core, plugins and themes as well as the central location for community conversations and organization. https://wordpress.org/ or the comfort of your WordPress adminadmin (and super admin). You can review the current code and leave feedback at the project’s GitHub repository or in #feature-shinyupdates on SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/..

Thanks a lot to everyone who has been contributing to this plugin since its inception, especially @obenland for leading this project, @adamsilverstein for his numerous contributions, @mapk for helping with testing and UXUX User experience, and @ocean90 for giving valuable feedback despite being super busy with leading 4.6.

So far we’ve received positive feedback from different core teams like the accessibility and design teams, and we have reached out to @drew who will review the docs once a core patchpatch A special text file that describes changes to code, by identifying the files and lines which are added, removed, and altered. It may also be referred to as a diff. A patch can be applied to a codebase for testing. is ready.

#4-6, #feature-plugins, #merge, #proposal, #shiny-updates

Feature Plugin Merge Proposal: oEmbed

For the past 6 years, users have been able to embed YouTube videos, tweets and many other resources on their sites through a nifty feature called oEmbed.

Today, we (mainly me, @pento and @melchoyce) ask to consider extending this feature by merging the oEmbed API plugin into WordPress coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress.. This pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party allows anyone to embed posts from your site by just pasting its URLURL A specific web address of a website or web page on the Internet, such as a website’s URL www.wordpress.org. We’ve been working hard on it for months and are now eager to hear your feedback.

Purpose & Goals

While I initially built an early version of the plugin about a year ago, it was @melchoyce who kicked things off with #32522. Her idea was simple: When you can embed almost anything in a WordPress post, why aren’t we able to embed WordPress posts themselves in another WordPress post?

That’s exactly what we’re aiming for. Our goal is to allow a big portion of the web to easily and securely embed such post previews.

Have a look at this post to see the user flow for this feature (and a live demo!):

https://make.wordpress.org/flow/2015/09/12/embeds-visual-survey-of-the-oembed-feature-plugin-macnchrome-iphone-5s/

Security Considerations

Embedding content from a random source on your site depends on lots of trust. We take precautions to make the whole process as easy as possible. It’s worth noting that:

  • We use iframes with the sandbox attribute to enable extra restrictions on the content that can appear in the inline frame.
  • The host and the embedded site communicate via postMessage to allow resizing and clicking on links safely

Browser Compatibility

We successfully tested the feature with all major browsers on mobile and desktop. Since IE < 10 doesn’t support the sandbox attribute, we use the proprietary security attribute there to have similar security restrictions. This means no JavaScriptJavaScript JavaScript or JS is an object-oriented computer programming language commonly used to create interactive effects within web browsers. WordPress makes extensive use of JS for a better user experience. While PHP is executed on the server, JS executes within a user’s browser. https://www.javascript.com/. inside the iframeiframe iFrame is an acronym for an inline frame. An iFrame is used inside a webpage to load another HTML document and render it. This HTML document may also contain JavaScript and/or CSS which is loaded at the time when iframe tag is parsed by the user’s browser. is run, e.g. for the resizing. The most important thing, clicking, still works there though.

Long story short, the feature works with all major browsers and degrades gracefully on older IE versions.

Core Changes & Merge Implementation Details

The plugin was developed in such a way that merging it into core eventually is as straightforward as possible. We are working actively on a patchpatch A special text file that describes changes to code, by identifying the files and lines which are added, removed, and altered. It may also be referred to as a diff. A patch can be applied to a codebase for testing. that can be added to core within the merge window.

There are two things that we need to change in core together with the merge:

  • When doing a redirect because of a changed post slug the rewrite endpoints need to be respected as well. See #33920
  • Attachment rewrite endpoints need to be fixed in core. See #19918
  • This feature only works with oEmbed discovery turned for every user, even those lacking the unfiltered_html capability. That capability check needs to be removed.

Developer Notes

This plugin adds a new /embed/ rewrite endpoint for posts, pages and attachments. We haven’t found any plugin in the directory using this endpoint, but if you already use that endpoint, you should consider renaming it or changing the oEmbed rewrite endpoint using the filters we provide.

Note: There’d be a separate post for this after the merge.

What about the REST APIREST API The REST API is an acronym for the RESTful Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. It is how the front end of an application (think “phone app” or “website”) can communicate with the data store (think “database” or “file system”) https://developer.wordpress.org/rest-api/.?

The plugin works well on WordPress 3.9+ and uses a simple class to return the oEmbed APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. responses. However, with the REST API officially proposed for a core merge, we built a controller class for it. This class does exactly the same and follows the REST API best practices.

We could definitely profit from the REST API and built upon it when merging into core. Not needing a fallback means no duplicated code and easier maintenance.

In case you missed it, here’s the REST API merge proposal:

https://make.wordpress.org/core/2015/09/21/wp-rest-api-merge-proposal/

Remaining Issues

There is currently one issue with SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. not displaying the oEmbed output correctly. It simply displays the JavaScript as plain-text instead of removing it. We’ve reached out to them to see if they could fix that and at least white-list WordPress.orgWordPress.org The community site where WordPress code is created and shared by the users. This is where you can download the source code for WordPress core, plugins and themes as well as the central location for community conversations and organization. https://wordpress.org/ in the meantime to properly display the embeds in Slack channels.

Besides that, there are also some small layout quirks we still need to work out. Meanwhile we’re continuing to improve the codebase and inline documentation.

Future Plans

We’re looking into improving support for different response types in addition to regular post content, depending on the feedback we receive from users.

Contributors

While I’ve been the lead developer of the plugin, a ton of valuable input and contributions have come from others in the community.

#4-4, #feature-plugins, #feature-oembed, #merge, #proposal

WP REST API: Merge Proposal

Hello everyone! This is the post you’ve all been waiting for. 🙂

We on the REST APIREST API The REST API is an acronym for the RESTful Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. It is how the front end of an application (think “phone app” or “website”) can communicate with the data store (think “database” or “file system”) https://developer.wordpress.org/rest-api/. team (myself, @rachelbaker, @joehoyle, @danielbachhuber, and newest member and coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. committercommitter A developer with commit access. WordPress has five lead developers and four permanent core developers with commit access. Additionally, the project usually has a few guest or component committers - a developer receiving commit access, generally for a single release cycle (sometimes renewed) and/or for a specific component. @pento) would like to propose merging the REST API into WordPress core. We’ve been working a while on this, and think it’s now ready to get your feedback.

This is our first iteration of the proposal, and we’re actively looking for feedback. If you have thoughts on the project, or on this proposal, let us know! Only with your feedback can we make progress. 🙂

What is the REST API?

The REST API is a nice and easy way to get at your data in WordPress externally, whether that’s from JavaScriptJavaScript JavaScript or JS is an object-oriented computer programming language commonly used to create interactive effects within web browsers. WordPress makes extensive use of JS for a better user experience. While PHP is executed on the server, JS executes within a user’s browser. https://www.javascript.com/. in a theme or pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party, mobile and desktop applications, or importing and exporting data. The APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. offers up all core data types (posts, terms comments, and users), plus support for metaMeta Meta is a term that refers to the inside workings of a group. For us, this is the team that works on internal WordPress sites like WordCamp Central and Make WordPress. and revisionsRevisions The WordPress revisions system stores a record of each saved draft or published update. The revision system allows you to see what changes were made in each revision by dragging a slider (or using the Next/Previous buttons). The display indicates what has changed in each revision.; we’ve got plans to eventually have access to everything the adminadmin (and super admin) and frontend have access to.

The REST API differs from existing WordPress APIs in that it is explicitly designed from the ground up for modern mobile and browser usage, using the lightweight and widely-supported JSONJSON JSON, or JavaScript Object Notation, is a minimal, readable format for structuring data. It is used primarily to transmit data between a server and web application, as an alternative to XML. data serialization format with a modern REST interface. Both of these are already familiar to most developers: JSON is a subset of JavaScript intended purely as a data interchange format, and REST is a set of best practices around HTTPHTTP HTTP is an acronym for Hyper Text Transfer Protocol. HTTP is the underlying protocol used by the World Wide Web and this protocol defines how messages are formatted and transmitted, and what actions Web servers and browsers should take in response to various commands.. Both are supported natively by almost every programming language and platform.

Why do we need a new API?

WordPress already has external APIs: XML-RPC, designed for desktop clients; Atom and RSS feeds, designed for post syndication; and the venerable admin-ajax, designed for Ajax requests in the admin and frontend. These APIs all serve different purposes, but often have a great deal of overlap. In addition, these have all been stretched beyond their original intentions: XML-RPC now contains site management tools, RSS has been extended into the WXR export format, and admin-ajax is the catch-all of any sort of browser-server communication in plugins and themes.

The REST API builds upon the heritage of these APIs to provide better support today for using these, as well as laying the groundwork for expanded use in the future.

XML-RPC is the closest analogue to the REST API in terms of usage and capabilities. Originally designed back in 1998 to allow desktop clients to create and edit posts on blogs, WordPress has extended this with both other specifications (such as MetaWeblog) and with its own proprietary additions. Fundamentally, XML-RPC is built around Remote Procedure Calls (RPC), essentially a way of calling a function externally. It then uses XML to serialize the data for passing back and forth.

Unfortunately, XML serialization can be problematic at times. XML has lots of power, but support for custom entities and DOCTYPEs can cause parsing problems and security attacks, including billion laughs, and XXE exploits. (Currently, WordPress has to disable parts of the XML parser and apply regular expression replacements to be able to parse these safely.) XML is also very verbose, and represents data in a way which doesn’t map easily to programmatic data structures. JSON on the other hand is both concise and well-represented in memory, as it’s based on JavaScript’s native syntax.

The admin-ajax API is also very commonly used in WordPress, albeit typically only by plugins and themes. This is a very lightweight API that essentially acts as a mini-router. Typical usage of this API uses JSON, since it’s a browser-based API, but all usage is completely custom. A lot of the usage of this involves retrieving or updating posts on-the-fly, but due to its nature as simply a framework, these are done in completely different ways. This doesn’t lead itself to extensibility, and requires a lot of duplication every time developers want to get data in or out. We don’t want to replace all of admin-ajax though, since some use cases don’t map exactly: UIUI User interface-related interactions or things like the Heartbeat API are great examples of this.

The REST API can help to supplant these uses. Our aim is to eventually replace the XML-RPC API completely, to act as a secondary import/export format, and to replace most (but not all) uses of admin-ajax. The REST API offers an easier to use interface than the existing solutions.

Why this project?

We’ve been working on this project ever since the first WordPress Contributor Summit back in 2012. Since then, we’ve had lots of feedback from core developers, the community at large, and further beyond in the form of client developers. We believe that the REST API has an immense amount of experience behind it, and plenty of viewpoints have contributed to the project’s direction.

The API has seen great usage in the community, from various mobile apps to large news sites. This usage has helped to battle-test and prove out the API. In the process, we’ve found plenty of bugs, and some security issues. Thanks to this feedback, the API is incredibly stable and secure. (The most recent security bugs we fixed were relatively minor bugs.)

We also designed the API from the ground-up to be part of core, following a core-like mentality to our processes. The API is intended to be both a great feature and a base to build off in plugins. We undertook a significant refactoring and partial rewrite in version 2 to make this extensibility even better. This open process also means that most of the design decisions are documented publicly as we’ve engaged stakeholders to gauge feedback.

We believe these pieces combined make this a fantastic feature for WordPress core, and we hope you all agree. 🙂

What’s the plan?

The plan we’re aiming for is a two part merge of the API. For the first stage, the infrastructure code would be merged into wp-includes and made available for plugins and themes. This is an internal API only, but offers an “API construction kit” for developers to use. For the second stage, the endpoints would be merged, and the API would be enabled for sites by default.

This plan splits the API into two parts, infrastructure and endpoints:

  • Stage One: Infrastructure: The infrastructure is the code responsible for routing requests and handling the “meta” layer of the API, including JSON serialisation/deserialisation, linking, embedding, and REST best practices. This adds a simplified routing layer outside of WP’s rewrites, allowing non-query-var rewrites easily, acting as a base for building APIs inside WordPress.
  • Stage Two: Endpoints: These are where much of the complexity of the API lies, as they’re responsible for mapping data from the external JSON format to the internal data structures, and vice versa. The “business” logic of integrating with WordPress is almost entirely contained within the endpoints. These are the more complex part of the API, as they require using deep APIs in WordPress, and handling security and privacy concerns.

With this plan, we would aim to have the infrastructure merged in 4.4, and the endpoints merged one release later in 4.5.

The slow nature of this plan allows a longer review time on the API for core committers. It also gives extra time for reviewing the endpoints, since they would be delayed one release.

Merging the infrastructure now would allow third-party code to begin using the API to build upon it, including migrating from existing custom code. It would also help to increase developer confidence in the API (as it represents a commitment by the project towards a REST API).

In this plan, the first stage would not include any of the base controllers (such as the posts controller). This may limit the utility of the infrastructure for plugins and themes, however as the endpoints would be merged a cycle later, it’s expected that this wouldn’t have a huge impact.

The infrastructure of the API is approximately 2700 lines of codeLines of Code Lines of code. This is sometimes used as a poor metric for developer productivity, but can also have other uses. (roughly a third of the API codebase), and the endpoints make up the remaining 5500 lines of code.

What would happen after merge?

After merging the REST API into core, we’d spend approximately two weeks partying and celebrating. 🙂

Once we’re done with the parties, one major question would be how we manage the API in the future. The existing management and contribution process via GitHubGitHub GitHub is a website that offers online implementation of git repositories that can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged be the repository owner. https://github.com/ has been extremely successful, as we’ve had 61 people’s pull requests merged into the API. Contribution via GitHub is especially useful for the API, as it’s a heavily developer-focussed project, and is aimed at external (non-WordPress) developers. On the other hand, all other contribution to WordPress is done via SVNSVN Subversion, the popular version control system (VCS) by the Apache project, used by WordPress to manage changes to its codebase. and TracTrac An open source project by Edgewall Software that serves as a bug tracker and project management tool for WordPress., so integrating with this process is important for existing developers, as well as core’s general processes. We need to ensure the API is an integral part of core, not a separate project.

Given the team’s experience with GitHub as well as Trac, we can bring the best of both worlds by helping integrate the two. This would also improve contribution for WordPress as a whole, and benefit the whole community. This will be especially important as we encourage more contributions from the wider community (client authors, for example). We think we can make good progress here, and we’d love to try and help improve the process. 🙂

In addition to the project management, there are still further API projects we need to tackle. Authentication is the most important of these, as a huge focus on OAuth and similar would be needed to make the API more useful for external clients. Currently, we haven’t had enough time to spend on this as well as managing the API project, however the API is now reaching a finalised stage, so this should be able to improve quickly. Centralised authentication is a huge part of this, as the regular OAuth registration process is cumbersome for a distributed system like WordPress.

Important note: We don’t believe authentication is required for the API merge, and we treat it as a separate project. The authentication is a completely separate system to the API itself. This is something we’d give more time to in the future, but we want to focus on shipping the API for now.

Let’s go!

This is our merge plan for the API, however it’s not finalised. If you’ve got comments, thoughts, suggestions, opinions, or words of encouragement, let us know. We’d love to know what you think. Thank you all, you’re wonderful, and stay golden.

#feature-plugins, #json-api, #merge, #proposal, #rest-api

Fast previewing changes to Menus in the Customizer

The foundation of the CustomizerCustomizer Tool built into WordPress core that hooks into most modern themes. You can use it to preview and modify many of your site’s appearance settings. is built on the concept of the “setting”, an abstract representation of anything in WordPress: options, theme mods, posts, terms, and so on. Settings are linked to controls which are responsible for manipulating them. When a setting is changed it gets pushed into the preview window so that the modification can be reviewed before being published live.

There are two ways (transports) for how a setting can be pushed into the preview: refresh and postMessage. The refresh transport does a full page refresh with the modified settings applied, and so it relies on PHPPHP The web scripting language in which WordPress is primarily architected. WordPress requires PHP 7.4 or higher to apply the setting changes. The refresh transport is designed to be compatible with all themes, but for heavy sites refresh can be excruciating slow and taxing on the server. For this reason, the postMessage transport was introduced for previewing changes. This transport uses JavaScriptJavaScript JavaScript or JS is an object-oriented computer programming language commonly used to create interactive effects within web browsers. WordPress makes extensive use of JS for a better user experience. While PHP is executed on the server, JS executes within a user’s browser. https://www.javascript.com/. to apply the changes instantly in the preview without any Ajax requests or PHP calls. This is great, but themes have to opt-in to postMessage transport for settings because they must also include JavaScript code which duplicates the logic which is already being executed on the server by PHP. The postMessage transport violates the DRY principle.

With that background out of the way…

Partial Refresh of Menus

In WordPress 4.3 menu management is being added to the Customizer, extending the types of objects that the Customizer can manipulate to include nav_menu taxonomyTaxonomy A taxonomy is a way to group things together. In WordPress, some common taxonomies are category, link, tag, or post format. https://codex.wordpress.org/Taxonomies#Default_Taxonomies. terms and nav_menu_item posts. We wanted this menu management experience to be fast so it had to avoid the slow full-page refresh transport. Nevertheless, the postMessage transport wasn’t possible to use either since we wouldn’t be able to duplicate all of the PHP menu walker logic and theme/pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party extensions for rendering menus purely in JavaScript. We also wanted to enable fast previewing of menu changes by default. So we implemented a postMessage/refresh hybrid approach which uses postMessage to sync the menus settings to the preview, and then the preview does an Ajax request to render just the contents of the menu container, and this Ajax response is then inserted into the DOM to replace the previous menu container. The technical name for this approach we have been calling “partial refresh”, but you can call it “fast preview”.

When the Customizer determines it can’t do a partial refresh, it falls back to performing a full page refresh. Conditions for why a menu change will get a full page refresh include:

  • if the menu was just assigned to a location for the first time or removed from being assigned to a location, since the theme may have some changes to the layout that it may need to perform;
  • or if wp_nav_menu() is called with with echo being false, since the theme may be doing some string replacement in this case which we can’t replicate in the partial-refresh Ajax request;
  • or if wp_nav_menu() is called with with a non-JSONJSON JSON, or JavaScript Object Notation, is a minimal, readable format for structuring data. It is used primarily to transmit data between a server and web application, as an alternative to XML. serializable fallback_cb, such as a closure or a object method—this arg has to be empty or a string;
  • or if wp_nav_menu() is called with with a walker object instance, as opposed to a class name string;
  • or if wp_nav_menu() is called without a theme_location arg and the menu arg is not an term ID or a menu object;
  • or if the menu selected in a Custom Menu widgetWidget A WordPress Widget is a small block that performs a specific function. You can add these widgets in sidebars also known as widget-ready areas on your web page. WordPress widgets were originally created to provide a simple and easy-to-use way of giving design and structure control of the WordPress theme to the user. is changed, since partial refresh for widgets is not yet supported (however, see feature plugin)—subsequent changes to the menu will result in fast partial refreshes if the above conditions

When a menu is being updated via partial refresh, the menu container will receive a classname customize-partial-refreshing, which by default changes the opacity to 25%.

Theme Support

The fast preview for menu changes is being enabled by default so that theme authors don’t have to opt-in to the functionality like they do for instant preview (postMessage transport). In general this should be fine. However, if the theme includes some JavaScript which dynamically manipulates the rendered menu container, such as adding event handlers for expanding/collapsing submenus, then the theme would need to include some JSJS JavaScript, a web scripting language typically executed in the browser. Often used for advanced user interfaces and behaviors. to re-apply the modifications once the menu is partial-refreshed, as the old DOM for the menu container gets replaced.

The Twenty Fifteen theme actually uses JS to expand/collapse submenu items, so it needed a patchpatch A special text file that describes changes to code, by identifying the files and lines which are added, removed, and altered. It may also be referred to as a diff. A patch can be applied to a codebase for testing. in 4.3 to re-setup a partial-refreshed main nav menu. In addition to firing the menu container DOM setup logic once at jQuery.ready(), it now invokes the functionality whenever the menu is partial-refreshed. See the initMainNavigation() JS function.

So in general, if your theme needs to dynamically initialize menus with JavaScript, the pattern for the initialization code should be:

jQuery(function($) {
	function initMainNavigation( container ) {
		/* set up container... */
	}
	initMainNavigation( $( '.main-navigation' ) );

	$( document ).on( 'customize-preview-menu-refreshed', function( e, params ) {
		if ( 'primary' === params.wpNavMenuArgs.theme_location ) {
			initMainNavigation( params.newContainer );
			/* optionally sync a previous menu state from params.oldContainer... */
		}
	});
});

The params being passed to the event handler consists of the following properties:

  • newContainer: jQuery object containing the new menu container element retrieved from Ajax; this is what you would manipulate to initialize.
  • oldContainer: the previous jQuery object holding the element for the replaced menu container; this is useful if there is any state in the old menu that should persist in the new menu, such as which submenus are expanded (as in Twenty Fifteen).
  • wpNavMenuArgs: The array of arguments passed to wp_nav_menu() in the template, such as template_location.
  • instanceNumber: The index for which wp_nav_menu() call being updated.

The Future

Partial refresh is an “experimental transport” introduced here specifically for menus. In 4.4 we hope to generalize the framework so that any setting can be setup for fast preview, starting with widgets (see #27355). There is a Customize Partial Refresh feature pluginFeature Plugin A plugin that was created with the intention of eventually being proposed for inclusion in WordPress Core. See Features as Plugins. that implements widget partial refresh in a betaBeta A pre-release of software that is given out to a large group of users to trial under real conditions. Beta versions have gone through alpha testing in-house and are generally fairly close in look, feel and function to the final product; however, design changes often occur as part of the process. state. This partial refresh functionality I’ve also identified as being critical for Customizer Transactions.

#4-3, #customize, #dev-notes, #feature-plugins, #feature-selective-refresh, #kickoff, #menu-customizer, #menus, #partial-refresh, #proposal

Feature Plugin Merge Proposal: Menu Customizer

The Customizer team is proposing to merge the Menu Customizer plugin into coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. for WordPress 4.3. In this post, I’ll outline the purpose and history of this project, as well as highlighting the improvements that we have made.

Purpose & Goals

The purpose of the Menu CustomizerCustomizer Tool built into WordPress core that hooks into most modern themes. You can use it to preview and modify many of your site’s appearance settings. project is to move navigation menu management from the WordPress adminadmin (and super admin) to the Customizer. In the process, we hope to offer an updated design with improved user flow, a mobile-first interface, improved accessibilityAccessibility Accessibility (commonly shortened to a11y) refers to the design of products, devices, services, or environments for people with disabilities. The concept of accessible design ensures both “direct access” (i.e. unassisted) and “indirect access” meaning compatibility with a person’s assistive technology (for example, computer screen readers). (https://en.wikipedia.org/wiki/Accessibility), rebuild the administration UIUI User interface from the ground up to be JavaScriptJavaScript JavaScript or JS is an object-oriented computer programming language commonly used to create interactive effects within web browsers. WordPress makes extensive use of JS for a better user experience. While PHP is executed on the server, JS executes within a user’s browser. https://www.javascript.com/.-driven, solve long-standing problems with the current implementation (#14134), and clarify the purposes and capabilities of the menus feature. Additionally, Menu Customizer contributes significantly to the long-term goal to move all appearance functionality and, really, everything that could benefit from live previewing, from the admin to the Customizer.

Background

Menu Customizer started out as my Google Summer of Code 2014 Project. The initial proposal and revised schedule highlight the initial goals and provide good perspective for where we’ve come in the past year. See also the periodic posts here on make/core for updates. Development has happened on GitHub since the project opened to the community.

Core APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. Improvements

As I began developing this feature in pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party form, it became clear that the core Customizer API would need a lot of improvements to support something as complex as menus. Countless tickets have worked towards this goal over the past year, from the addition of the concept of Panels ( #27406) to JSJS JavaScript, a web scripting language typically executed in the browser. Often used for advanced user interfaces and behaviors./Underscore-templated controls ( #29572), and now, full support for dynamically-added sections and controls ( #30737, #30738, and #30741).

Developers are still realizing the full potential of the Customizer API, and Menu Customizer pushes the boundaries of what can be done here pretty far. One of the goals with our approach is to bring as much functionality that should be natively available in the API into core as possible. With the improvements made here already, as well as the future potential to continue abstracting functionality like the add-menu-items fly-out panel or the ability to add screen options in Customizer Panels, Menu Customizer broadens the potential for developers to extend the Customizer to do anything, in core, plugins, and themes.

Contributors

In the initial GSoC project, I (@celloexpressions) developed the plugin from scratch, using Widgets in the Customizer as the design basis, with @ethitter and @obenland serving as my mentors. When the project was opened to the community for contributions, several designers and developers stepped up to help. Code contributors to date include @westonruter, @valendesigns, @voldemortensen, @adamsilverstein, and @kucrut. @designsimply and @folletto have also put in a tremendous amount of time helping with design and usability.

Plugin Overview

I highly recommend trying the plugin, which currently requires the latest version of 4.3 alpha. @designsimply has prepared a video demo:

I’ve posted a comparison of the mobile menus flow in the admin and the Customizer on make/flow, and @designsimply has also posted flows there (more flows with more recent versions of the plugin on trunktrunk A directory in Subversion containing the latest development code in preparation for the next major release cycle. If you are running "trunk", then you are on the latest revision. are still needed). Usability testing has been conducted on usertesting.com, with results posted on make/design. As further refinements are made, additional testing and feedback can be incorporated to make the new experience as polished as possible.

A couple of technical details: each menu is a Customizer section, and new menus can be added (dynamically adding new Customizer sections and controls in the process). Menu items are Customizer controls. To maximize scalability, menu items are all rendered using a single JS template, only when their containing menu section is expanded. The add-menu-item panel loads available menu items on an as-needed basis via ajax. The plugin uses several custom Customizer objects including a custom panel that implements screen options, two custom sections (menus, for lazy-loading of menu items, and new menus, which is rendered as a button toggle), and several custom controls. But everything is built off of the core Customizer API.

A summary of some key improvements that the plugin includes:

  • Modernized, simplified, and more compact UI
  • Mobile-first design that leverages the Customizer
  • Scalable, JavaScript-driven and avoids performance issues ( #14134)
  • All menus easily accessible in one place, without page reloads
  • Live previews of active menus as they are edited
  • Menu locations that can be set from the main panel or while editing
  • Global search that includes all post types and terms in all taxonomies
  • Quick-delete for deleting several items sequentially
  • “Original” item links open directly into the live preview
  • The Customizer API can be used to hook into the experience in countless ways with plugins. Support for additional menu item fields can be added much more easily now in a future release, potentially leveraging the Fields API

Core Tickets Fixed

Menu Customizer fixes numerous tickets on core tracTrac An open source project by Edgewall Software that serves as a bug tracker and project management tool for WordPress.. This is not an exhaustive list, but covers many bigger ones:

  • #14134: Menus item are limited to 16 item and will not save more than that
  • #28138: Updating menu item requires passing all of a menu item’s data to wp_update_nav_menu_item() (The plugin steps around this, we can actually fix it in core on merge)
  • #32218: Remove title attribute option in Menu Editor (off by default)
  • #19272: Add FilterFilter Filters are one of the two types of Hooks https://codex.wordpress.org/Plugin_API/Hooks. They provide a way for functions to modify data of other functions. They are the counterpart to Actions. Unlike Actions, filters are meant to work in an isolated manner, and should never have side effects such as affecting global variables and output. to Nav Menu Support Themes Text (can modify via Customizer API)
  • #21603: Add ability to delete multiple menu items
  • #16828: Add filter on initial_meta_boxes for nav menu Probably fixed, all are shown currently, which could use improvement but it will default to more being shown at least
  • #19464: Auto add do_action for menu in admin (can use Customizer API)
  • #31391: Make the list of registered nav menus (locations) filterable (can use Customizer API)
  • #32440: on Menu page, turn posts by default on “view options”
  • #18517: Visual Feedback for Nav Menu UI

The Plan for the Menus Component

This project has a very explicit goal of not just adding menu management to the Customizer, but also removing the existing admin page in the process. The menu management screen has significant, fundamental problems in its implementation and will never scale (see #14134) without a significant refactoring along the lines of what we’ve done with the Customizer. Additionally, the new UI in the Customizer is considerably more polished than the admin screen and already includes numerous features and bugfixes proposed for core (see above). Ultimately, the new UI provides a much better experience for all users, including desktop, mobile, accessibility, etc.

The plan for the “removal” of the old menus admin screen is as follows:

  • Immediately and officially “deprecate” it: wind down any ongoing development efforts and focus all new administration-focused Menus component work on the new UI in the Customizer. Update trac tickets accordingly, and add a “Manage in Customize” link to the existing screen. Any existing tickets proposing enhancements or new features for menu administration would be required to be adapted to the Customizer version, with the (discouraged) option of also making changes to the old screen.
  • Point the “Menus” link in the admin bar to Menus in the Customizer in 4.3. Remove that menu from the admin bar in 4.4 in favor of a top-level Customize link, and put something more useful in its place (as all of its core links will point to the Customizer now).
  • Retain the admin screen codebase, along with existing links to it throughout the admin.
  • In WordPress 4.5 or 4.6, remove all core links (including admin menu) to the Menus admin screen, or point them to the Customizer. This would likely coincide with a similar change for Widgets and Themes to use the Customizer versions exclusively, once full feature-parity is achieved with the Customizer versions of the other features (Menus has it now). At this point the admin screen would be accessible only by plugin-added links or for users who cannot access the Customizer (no-js, IE7, IE8&9 with domain mapping, a very small percentage of users overall).
  • The admin screen and related code would likely not be removed entirely from core in the foreseeable future, and critical bugs or security issues would still be addressed. New feature development and enhancements would be restricted to the Customizer version.

The above plan is fairly aggressive, to eliminate any ambiguity about future plans and intentions and to avoid the potential for mass trac ticketticket Created for both bug reports and feature development on the bug tracker. rot. The fact that the Menus component has no maintainers and has not received significant attention since the 3.6 release indicates that there is a general lack of developer interest in dealing with the mess that the current system is. I am willing to step up as a component maintainer for Menus if the above plan is implemented.

Ongoing Work

We have a few issues left that work working on. Notably, @westonruter has proposed refactoring the way menu item settings are handled, along with menu creation and deleted, and has begun work there, but wouldn’t finish until after a core merge due to time constraints and integration with core code. @adamsilverstein is working on improving drag & drop to work with sub menus. There are also several minor issues remaining on GitHub, which would either be handled in the next couplle days or after merge (many issues have been punted to after a potential merge already).

#customize, #feature-plugins, #menu-customizer, #menus, #merge, #proposal