WP REST API: Merge Proposal

Hello everyone! This is the post you’ve all been waiting for. 🙂

We on the REST API team (myself, @rachelbaker, @joehoyle, @danielbachhuber, and newest member and core committer @pento) would like to propose merging the REST API into WordPress core. We’ve been working a while on this, and think it’s now ready to get your feedback.

This is our first iteration of the proposal, and we’re actively looking for feedback. If you have thoughts on the project, or on this proposal, let us know! Only with your feedback can we make progress. 🙂

What is the REST API?

The REST API is a nice and easy way to get at your data in WordPress externally, whether that’s from JavaScript in a theme or plugin, mobile and desktop applications, or importing and exporting data. The API offers up all core data types (posts, terms comments, and users), plus support for meta and revisions; we’ve got plans to eventually have access to everything the admin and frontend have access to.

The REST API differs from existing WordPress APIs in that it is explicitly designed from the ground up for modern mobile and browser usage, using the lightweight and widely-supported JSON data serialization format with a modern REST interface. Both of these are already familiar to most developers: JSON is a subset of JavaScript intended purely as a data interchange format, and REST is a set of best practices around HTTP. Both are supported natively by almost every programming language and platform.

Why do we need a new API?

WordPress already has external APIs: XML-RPC, designed for desktop clients; Atom and RSS feeds, designed for post syndication; and the venerable admin-ajax, designed for Ajax requests in the admin and frontend. These APIs all serve different purposes, but often have a great deal of overlap. In addition, these have all been stretched beyond their original intentions: XML-RPC now contains site management tools, RSS has been extended into the WXR export format, and admin-ajax is the catch-all of any sort of browser-server communication in plugins and themes.

The REST API builds upon the heritage of these APIs to provide better support today for using these, as well as laying the groundwork for expanded use in the future.

XML-RPC is the closest analogue to the REST API in terms of usage and capabilities. Originally designed back in 1998 to allow desktop clients to create and edit posts on blogs, WordPress has extended this with both other specifications (such as MetaWeblog) and with its own proprietary additions. Fundamentally, XML-RPC is built around Remote Procedure Calls (RPC), essentially a way of calling a function externally. It then uses XML to serialize the data for passing back and forth.

Unfortunately, XML serialization can be problematic at times. XML has lots of power, but support for custom entities and DOCTYPEs can cause parsing problems and security attacks, including billion laughs, and XXE exploits. (Currently, WordPress has to disable parts of the XML parser and apply regular expression replacements to be able to parse these safely.) XML is also very verbose, and represents data in a way which doesn’t map easily to programmatic data structures. JSON on the other hand is both concise and well-represented in memory, as it’s based on JavaScript’s native syntax.

The admin-ajax API is also very commonly used in WordPress, albeit typically only by plugins and themes. This is a very lightweight API that essentially acts as a mini-router. Typical usage of this API uses JSON, since it’s a browser-based API, but all usage is completely custom. A lot of the usage of this involves retrieving or updating posts on-the-fly, but due to its nature as simply a framework, these are done in completely different ways. This doesn’t lead itself to extensibility, and requires a lot of duplication every time developers want to get data in or out. We don’t want to replace all of admin-ajax though, since some use cases don’t map exactly: UI-related interactions or things like the Heartbeat API are great examples of this.

The REST API can help to supplant these uses. Our aim is to eventually replace the XML-RPC API completely, to act as a secondary import/export format, and to replace most (but not all) uses of admin-ajax. The REST API offers an easier to use interface than the existing solutions.

Why this project?

We’ve been working on this project ever since the first WordPress Contributor Summit back in 2012. Since then, we’ve had lots of feedback from core developers, the community at large, and further beyond in the form of client developers. We believe that the REST API has an immense amount of experience behind it, and plenty of viewpoints have contributed to the project’s direction.

The API has seen great usage in the community, from various mobile apps to large news sites. This usage has helped to battle-test and prove out the API. In the process, we’ve found plenty of bugs, and some security issues. Thanks to this feedback, the API is incredibly stable and secure. (The most recent security bugs we fixed were relatively minor bugs.)

We also designed the API from the ground-up to be part of core, following a core-like mentality to our processes. The API is intended to be both a great feature and a base to build off in plugins. We undertook a significant refactoring and partial rewrite in version 2 to make this extensibility even better. This open process also means that most of the design decisions are documented publicly as we’ve engaged stakeholders to gauge feedback.

We believe these pieces combined make this a fantastic feature for WordPress core, and we hope you all agree. 🙂

What’s the plan?

The plan we’re aiming for is a two part merge of the API. For the first stage, the infrastructure code would be merged into wp-includes and made available for plugins and themes. This is an internal API only, but offers an “API construction kit” for developers to use. For the second stage, the endpoints would be merged, and the API would be enabled for sites by default.

This plan splits the API into two parts, infrastructure and endpoints:

  • Stage One: Infrastructure: The infrastructure is the code responsible for routing requests and handling the “meta” layer of the API, including JSON serialisation/deserialisation, linking, embedding, and REST best practices. This adds a simplified routing layer outside of WP’s rewrites, allowing non-query-var rewrites easily, acting as a base for building APIs inside WordPress.
  • Stage Two: Endpoints: These are where much of the complexity of the API lies, as they’re responsible for mapping data from the external JSON format to the internal data structures, and vice versa. The “business” logic of integrating with WordPress is almost entirely contained within the endpoints. These are the more complex part of the API, as they require using deep APIs in WordPress, and handling security and privacy concerns.

With this plan, we would aim to have the infrastructure merged in 4.4, and the endpoints merged one release later in 4.5.

The slow nature of this plan allows a longer review time on the API for core committers. It also gives extra time for reviewing the endpoints, since they would be delayed one release.

Merging the infrastructure now would allow third-party code to begin using the API to build upon it, including migrating from existing custom code. It would also help to increase developer confidence in the API (as it represents a commitment by the project towards a REST API).

In this plan, the first stage would not include any of the base controllers (such as the posts controller). This may limit the utility of the infrastructure for plugins and themes, however as the endpoints would be merged a cycle later, it’s expected that this wouldn’t have a huge impact.

The infrastructure of the API is approximately 2700 lines of code (roughly a third of the API codebase), and the endpoints make up the remaining 5500 lines of code.

What would happen after merge?

After merging the REST API into core, we’d spend approximately two weeks partying and celebrating. 🙂

Once we’re done with the parties, one major question would be how we manage the API in the future. The existing management and contribution process via GitHub has been extremely successful, as we’ve had 61 people’s pull requests merged into the API. Contribution via GitHub is especially useful for the API, as it’s a heavily developer-focussed project, and is aimed at external (non-WordPress) developers. On the other hand, all other contribution to WordPress is done via SVN and Trac, so integrating with this process is important for existing developers, as well as core’s general processes. We need to ensure the API is an integral part of core, not a separate project.

Given the team’s experience with GitHub as well as Trac, we can bring the best of both worlds by helping integrate the two. This would also improve contribution for WordPress as a whole, and benefit the whole community. This will be especially important as we encourage more contributions from the wider community (client authors, for example). We think we can make good progress here, and we’d love to try and help improve the process. 🙂

In addition to the project management, there are still further API projects we need to tackle. Authentication is the most important of these, as a huge focus on OAuth and similar would be needed to make the API more useful for external clients. Currently, we haven’t had enough time to spend on this as well as managing the API project, however the API is now reaching a finalised stage, so this should be able to improve quickly. Centralised authentication is a huge part of this, as the regular OAuth registration process is cumbersome for a distributed system like WordPress.

Important note: We don’t believe authentication is required for the API merge, and we treat it as a separate project. The authentication is a completely separate system to the API itself. This is something we’d give more time to in the future, but we want to focus on shipping the API for now.

Let’s go!

This is our merge plan for the API, however it’s not finalised. If you’ve got comments, thoughts, suggestions, opinions, or words of encouragement, let us know. We’d love to know what you think. Thank you all, you’re wonderful, and stay golden.

#feature-plugins, #json-api, #merge, #proposal, #rest-api