K. Adam White 11:15 pm on October 13, 2016
Tags: 4.7 ( 124 ), rest-api ( 107 )

Merge Proposal Discussion: REST API Content Endpoints

There are discussion meetings and office hours in #core-restapi at 2016-10-14 14:00UTC and 2016-10-14 19:00UTC on Friday the 14th. Our next team meeting is on 2016-10-17 14:00UTC. Please attend some of all of these, because

We are meeting at 2016-10-18 01:00 UTC to make a decision on this merge proposal!

To that end, the below discussion points will be updated regularly, please leave comments on this post or join the conversation in #core-restapi.

Yesterday at the dev chat the APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. Team proposed the Content API Endpoints for merge in WordPress 4.7. There was popular support for this feature but as @jorbin and @helen noted that the lack of dissent suggested additional review is needed, so the API Team is continuing to seek thorough review & constructive criticism of the content endpoints, including the open questions previously shared on the week 7 and week 8 API team updates.

The merge proposal also engendered follow-up discussion in the #core-restapi channel about the benefit content endpoints bring to coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress., whether having such endpoints built in is quantifiably more beneficial than having them as a pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party, whether moving development from a plugin to core would slow development, and whether the endpoints as-proposed have been sufficiently reviewed for security and performance. We attempt to capture those questions & concerns (and the perspectives on them) below.

Security

Have the content API endpoints been thoroughly reviewed for security?

The REST APIREST API The REST API is an acronym for the RESTful Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. It is how the front end of an application (think “phone app” or “website”) can communicate with the data store (think “database” or “file system”) https://developer.wordpress.org/rest-api/. plugin has been on HackerOne for over a year with paid bounties for bugs
@barry has begun a security review

Performance

How does the API measure up against alternatives? Are there concerns about how the API could impact the servers to which it is deployedDeploy Launching code from a local development environment to the production web server, so that it's available to visitors.?

DeliciousBrains did a performance comparison with Admin AJAX and found the REST API to have a performance improvement (These tests have not yet been independently verified)
@mikeschroder notes in the comments that using the REST API in place of Adminadmin (and super admin)-Ajax will also bring speed benefits by permitting many previously-uncacheable requests to be cached.

User Feedback

Are the content endpoints sufficiently well-tested & vetted by the community?

@matt questions whether feedback is coming too late in development for concerns to be acted upon
- @rmccue notes that the v2 endpoints were created based on user feedback; REST API endpoints are being deployed by plugins and running on VIP, and the content endpoints have been in wide use across a variety of sites, leading to 90+ code contributors and far more developers’ support & feedback on the endpoints
@rmccue has also reached out to Phil Sturgeon for feedback and will follow up

Do Content Endpoints Benefit Core Development?

Will having these endpoints in core improve future core development, or solve any immediate problems?

@bradyvercher suggested that the content API endpoints would remove the need to write a variety of one-off ajax callbacks when developing future WordPress Core AJAX functionality
@westonruter notes that the customizerCustomizer Tool built into WordPress core that hooks into most modern themes. You can use it to preview and modify many of your site’s appearance settings. could dynamically create settings for posts and other kinds of content without having to wire up new admin-ajax handlers

Will Merging Negatively Impact API Development?

Will having to work through tracTrac An open source project by Edgewall Software that serves as a bug tracker and project management tool for WordPress. instead of GitHubGitHub GitHub is a website that offers online implementation of git repositories that can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged be the repository owner. https://github.com/ cause development to atrophy?

@jjj argues that merging will slow development, because GitHub-hosted plugins are not bound to WordPress release cycles and have less overhead for features to be developed and deployed for testing. @jjj requested a plan for how the REST API will be developed going forward after the merge of these endpoints that would account for the added friction.
@krogsgard countered that core increases the visibility of a project like the content endpoints
- The number of new contributors in this SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. discussion suggests that this merge proposal is bringing in new voices; whether this supports Brian’s point or not, the team is grateful for the breadth of perspectives being shared -Ed.
@rachelbaker suggested that the API endpoints are sufficiently inter-dependent with core WordPress code that maintaining the plugin separately amounts to maintaining a fork, and that such separated development is untenable long-term.
@matt hopes that a merge of these endpoints would slow release speed, but not development speed; @rmccue feels that development speed will stay the same or increase, and that tying releases to WordPress Core increases the stability and predictability of the API endpoints.
The versioning of the API endpoints supports forward compatibility

Do Content Endpoints Belong on Every WordPress Site?

What are the pros and cons to having every WordPress site have content API endpoints?

@rmccue suggests the API has networknetwork (versus site, blog) effects that can only be realized with a large install base. @krogsgard draws a comparison to RSS, the widespread availability of which enables everything from podcasting from WP to the use of apps like Feedly.
@matt suggests that the Atom API is a better analogue than RSS, which is an independent and pre-existing standard, and that network effects could be tested through inclusion in Jetpack
@joostdevalk notes that many plugins, like Yoast, have data tied to existing content such as posts and pages; either they expose the content through their own endpoints, or core does. If Core exposes content types through the API then plugins may build on top of that shared foundation, not independently reinvent the wheel. “if this doesn’t end up in core, we’ll start rolling our own API for stuff. Others will too. Interoperability won’t be there, for even the most basic stuff. I think this isn’t like RSS, I think this is much more like all of us using the same table space in MySQLMySQL MySQL is a relational database management system. A database is a structured collection of data where content, configuration and other options are stored. https://www.mysql.com/..”
- @shelob9 and @masonjames agree that merging the endpoints would create a consistent and reliable open “standard” that WordPress developers can use instead of continually reinventing how to read and edit post data over HTTPHTTP HTTP is an acronym for Hyper Text Transfer Protocol. HTTP is the underlying protocol used by the World Wide Web and this protocol defines how messages are formatted and transmitted, and what actions Web servers and browsers should take in response to various commands..
- In response to the question “what prevents you from building on the endpoints in their plugin form,” @joostdevalk went on to note that plugin dependencies would make that a viable option, but that burden currently lies on the user. Plugin installation is not frictionless.
Can these endpoints be bundled? short takeaway: no
- Woo bundled the API infrastructure before it was merged; doing so for content endpoints would require bundling prohibitively large amounts of endpoint code.
- @nerrad worries that if plugins bundle different versions of the endpoints plugin, then those plugins may conflictconflict A conflict occurs when a patch changes code that was modified after the patch was created. These patches are considered stale, and will require a refresh of the changes before it can be applied, or the conflicts will need to be resolved. if all bundled copies are not kept in sync.
  - @nerrad clarifies in the comments below that these worries also encompass the additional risk of conflicts when plugin authors each build their own versions of these content endpoints, instead of leveraging a shared standard: if two plugins each expose their own REST collection for posts, a developer working on a site with multiple such endpoints will need to decide which to extend, and will then have their extension tied to that specific plugin rather than to a core API.
@schrapel and @jorbin discussed that these content endpoints make it much easier to crawl a site, which also brings some potential performance concerns: no new content is exposed, but the process of aggregating it is easier and more easily automated.
In the comments below @foliovision believes that merging the endpoints will be the best way to assert the level of back-compatibility that mid-size agencies need in order to confidently utilize the endpoints.

Please leave thoughts, questions, concerns, comments & experience in the comments below. Thank you!

Edited 2016-10-16 to include the below comments into the body of the post

#4-7, #rest-api

FolioVision 11:27 pm on October 13, 2016

Great discussion summary Adam. As the leader of a mid-size development agency, the sooner these endpoints are in coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress., the sooner we can start to use the REST APIREST API The REST API is an acronym for the RESTful Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. It is how the front end of an application (think “phone app” or “website”) can communicate with the data store (think “database” or “file system”) https://developer.wordpress.org/rest-api/.. For smaller to medium size clients, trying to play pin the tail on a donkey on a moving train (i.e. different versions of the REST API pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party, different versions of core) is simply not economically or technologically viable.

As long as the REST API team can commit to retaining some backward compatibility if the APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. endpoints must change significantly in a couple of major releases, I can’t see any benefit to keeping REST API separate from core. So much great work has gone into creating the additional REST potential, it would be a shame to see it remain bottled up and a niche product only for huge agencies and hosts like VIP.wordpress.comWordPress.com An online implementation of WordPress code that lets you immediately access a new WordPress environment to publish your content. WordPress.com is a private company owned by Automattic that hosts the largest multisite in the world. This is arguably the best place to start blogging if you have never touched WordPress before. https://wordpress.com/ or WordPress.com.
Josh Pollock 11:39 pm on October 13, 2016

One thing I wanted to add to this is that the REST APIREST API The REST API is an acronym for the RESTful Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. It is how the front end of an application (think “phone app” or “website”) can communicate with the data store (think “database” or “file system”) https://developer.wordpress.org/rest-api/. adds standards. I and many others have written tons of custom systems for alternative ways to read and edit post data using adminadmin (and super admin)-ajax, or using custom rewrite rules.

The constant reinventing of the wheal is counter to the spirit of open-source. By agreeing to a standard, with tons of eyes on it, we can all do better with greater efficiency and security. This after all is why we use WordPress and other open-source tools, instead of making our own CMSes.
- masonjames 12:53 pm on October 14, 2016
  
  I want to affirm this point as well. Our agency looks to WordPress (coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress.) to provide an opinion. We value the decisions and respect them (even when we disagree).
  
  Web development is an incredibly competitive space with pressures in open sourceOpen Source Open Source denotes software for which the original source code is made freely available and may be redistributed and modified. Open Source **must be** delivered via a licensing model, see GPL. and otherwise. Having content end points in core ensures consistency and reliability across the WP community.
rag-guay 7:00 am on October 14, 2016

I would like to be able to edit widgets from the desktop app. So slow over the net from Thailand!
- K.Adam White 12:58 pm on October 14, 2016
  
  That’s a good point and I agree we need a solution for it. On behalf of the APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. Team, support for editing widgets is on the roadmap but is not part of what’s being proposed for merge this cycle. It is definitely a common request, and is one of our priorities for 4.8 as we expand the site management capabilities of the API!
  
  You can see some early experiments from earlier this year towards this usage here: https://github.com/WP-API/wp-api-menus-widgets-endpoints
Darren Ethier (nerrad) 10:46 am on October 14, 2016

Great summary! A lot of great back and forth in that discussion thread that isn’t always easy to capture 🙂

Just wanted to clarify that the comments I made were not only in regards to the possibility of conflicts if pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party authors bundled the plugin version of the REST APIREST API The REST API is an acronym for the RESTful Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. It is how the front end of an application (think “phone app” or “website”) can communicate with the data store (think “database” or “file system”) https://developer.wordpress.org/rest-api/. with their own plugins, but also, with the presence of the infrastructure already in WordPress coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress., there is the possibility of conflicts should plugin authors roll their own different versions of the content endpoints to support what they are doing.

Imagine plugin author A has used to api infrastructure in WP core to build custom endpoints for his plugin and his users are asking if he can add some endpoints for their posts, pages and users as well, so he goes ahead and does that. In the meantime, plugin author B has users asking her to build custom endpoints for users that support her needs as well so she adds them. Then along comes User Sally who installs and activates both plugins. She contracts mobile app developer Jimmy to build an app for her that interacts with both plugins and her website. Jimmy starts running into conflicts between what “user” endpoint to use but he manages to get it working. Then Sally tells Jimmy that she wants to distribute this app for anybody using plugin A or plugin B and Jimmy starts thinking about building a custom user endpoint in a plugin that site owners will need to install to use the app to save having to worry in the mobile app about what user endpoint to utilize.

There are definitely different technical solutions to the above example story, but the reality is, things would be MUCH simpler for all the actors in the story above if there was an accepted standard in WordPress core for the content endpoints. It prevents a lot of problems from occurring.
K.Adam White 1:00 pm on October 14, 2016

If I may make a request of the developer community: Share this with your colleagues and friends outside of WordPress. See what they like, and what they find rough; let us know what they think! We’ve been gathering outside input on this but for more is always better.
Mike Schroder 6:03 pm on October 14, 2016

I noted this in the #coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. chat this week, but from my perspective at a host, we’re excited that landing the REST APIREST API The REST API is an acronym for the RESTful Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. It is how the front end of an application (think “phone app” or “website”) can communicate with the data store (think “database” or “file system”) https://developer.wordpress.org/rest-api/. will have performance benefits — in particular, because it will mean that we can cache requests that could only go through `adminadmin (and super admin)-ajax` previously (and were thus forced to be dynamic).

This will help with scaling even if it’s found that performance for the actual time in PHPPHP The web scripting language in which WordPress is primarily architected. WordPress requires PHP 5.6.20 or higher/MySQLMySQL MySQL is a relational database management system. A database is a structured collection of data where content, configuration and other options are stored. https://www.mysql.com/. is exactly the same.

Gradually moving those requests (whether they be core related, or added with plugins) out of `admin-ajax` will also help SysEng/Support more easily spot which requests are problematic.
- Matt Mullenweg 1:15 am on October 18, 2016
  
  It probably won’t have any real-world performance benefit — the adminadmin (and super admin)-ajax benchmark was kind of flawed, and you won’t be able to do HTTPHTTP HTTP is an acronym for Hyper Text Transfer Protocol. HTTP is the underlying protocol used by the World Wide Web and this protocol defines how messages are formatted and transmitted, and what actions Web servers and browsers should take in response to various commands.-level caching unless you also support some quite-fancy invalidation.
Luke Cavanagh 9:49 pm on October 14, 2016

I only see this as positive thing to be in WP coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress..
Brian Richards 2:21 pm on October 17, 2016

I’m late to the party but I want to officially register my voice in favor of merging the content endpoints into coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress..

As @joostdevalk mentioned in SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. (and others have discussed elsewhere), there are a lot of opportunities to include APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways.-based interactions within a pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party that are stifled presently because of the dependency on the REST APIREST API The REST API is an acronym for the RESTful Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. It is how the front end of an application (think “phone app” or “website”) can communicate with the data store (think “database” or “file system”) https://developer.wordpress.org/rest-api/. as a plugin. While it does seem like a small thing to inform end-users “This plugin requires the WP REST API plugin” it’s still a burden they needn’t bear. Further, by relegating the REST API to a plugin it stifles the ability for API-based interactions across many sites. For instance, a plugin that creates a widgetWidget A WordPress Widget is a small block that performs a specific function. You can add these widgets in sidebars also known as widget-ready areas on your web page. WordPress widgets were originally created to provide a simple and easy-to-use way of giving design and structure control of the WordPress theme to the user. that shows the most recent comments across a number of sites that you don’t control (e.g. getting the latest comments from a variety make/core inside a personal dashboard … and having the ability to bookmark certain comments or posts into a “read later” list for later consumption).

As @getsource mentions above there are some fantastic wins by moving these interactions that currently depend on adminadmin (and super admin)-ajax.php into standard, cacheable API calls.

The most convincing dissenting opinion that gave me reason to pause was @JJJ’s post that merging would hinder development. I do believe that is true. However, I also believe @rachelbaker is correct in that maintaining parity within the plugin as a long-term stand-alone is implausible. Given the choice between a REST API that can adapt and change quickly at the expense of dying slowly from a thousand cuts and a REST API that changes slowly but maintains perfect core parity I will pick the latter, every time.

TL;DR: I really want to see the REST API land in core because I want to build some fantastic utilities (for myself and others) that can communicate with many sites without necessarily requiring administrative access (the ability to install the REST API plugin) within those sites.
K.Adam White 2:30 pm on October 17, 2016

I don’t think we’ve noted yet that the endpoints here don’t just encompass the built-in types, but also the endpoints that extend them — `WP_REST_Controller` provides a strong base for implementing new endpoints, and a custom post typeCustom Post Type WordPress can hold and display many different types of content. A single item of such a content is generally called a post, although post is also a specific post type. Custom Post Types gives your site the ability to have templated posts, to simplify the concept. with `show_in_rest => true` will automatically pick up behavior extending that class without any further action from the developer than a single line in `register_post_type`.

Aside from the benefits of augmenting a shared resource like the posts collection @joostdevalk mentions above, having a sharable code foundation for new custom classes has the potential to reduce a lot of endpoint class duplication and fragmentation across the pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party ecosystem.