REST API Team Meeting Notes, 2016-10-17

As a reminder, in just 30 minutes there is a meeting in #core to decide whether the REST API Content Endpoints will be merged as a part of WordPress 4.7! See the ongoing discussion of this proposal here. Note that the OAuth server is no longer proposed for merge at this time, but authentication options will be a primary focus area for the API project during the 4.8 development cycle.

The REST API Content Endpoints provide a new foundation upon which the WordPress developer community can build themes, plugins, and core feature. They represent a common standard and consistent interface across WordPress’s core content data types, and provide robust support for custom post types and meta values. These endpoints lay the foundation upon which future releases will add remote authentication options, even deeper querying abilities, and broader endpoint coverage for site management. This iterative approach fits WordPress’s development model and philosophy, advancing the project’s long-term goal of opening WordPress up to a wider developer audience and helping to ensure continued work on the REST API in the release cycles and years to come.

Meeting Notes

At today’s weekly API team meeting in core-restapi (agenda here) the team resolved all outstanding decisions milestoned for the REST API 2.0 / WordPress 4.7 merge candidate:

  • The ?filter query parameter will be removed from the REST API plugin prior to core merge, a breaking change that improves the consistency of querying the API and eliminates a set of parameters that could introduce backwards compatibility issues were they to be committed to WordPress core. A separate plugin will be published to reinstate the `filter` parameter on a strictly opt-in basis.
  • Comments on password-protected posts are being deferred as a future enhancement until a robust solution is proposed that permits the API to adequately mirror existing functionality.
  • The unfiltered_html capability should be respected by the API, and a patch will be submitted to bring the API’s behavior in line with core’s.

There are 29 tickets left in the 2.0 milestone, several of which have open pull requests already. These issues represent a mix of outstanding bugs, documentation needs and improvements that will be moved to trac should the merge proposal be accepted.

The REST API team leads would like to recognize that the content endpoints plugin now has 95 contributors: thank you and welcome to all of the new participants who have joined the project in the past week!

#4-7, #meeting-notes, #rest-api

Merge Proposal Discussion: REST API Content Endpoints

There are discussion meetings and office hours in #core-restapi at 2016-10-14 14:00UTC and 2016-10-14 19:00UTC on Friday the 14th. Our next team meeting is on 2016-10-17 14:00UTC. Please attend some of all of these, because

We are meeting at 2016-10-18 01:00 UTC to make a decision on this merge proposal!

To that end, the below discussion points will be updated regularly, please leave comments on this post or join the conversation in #core-restapi.

Yesterday at the dev chat the API Team proposed the Content API Endpoints for merge in WordPress 4.7. There was popular support for this feature but as @jorbin and @helen noted that the lack of dissent suggested additional review is needed, so the API Team is continuing to seek thorough review & constructive criticism of the content endpoints, including the open questions previously shared on the week 7 and week 8 API team updates.

The merge proposal also engendered follow-up discussion in the #core-restapi channel about the benefit content endpoints bring to core, whether having such endpoints built in is quantifiably more beneficial than having them as a plugin, whether moving development from a plugin to core would slow development, and whether the endpoints as-proposed have been sufficiently reviewed for security and performance. We attempt to capture those questions & concerns (and the perspectives on them) below.

Security

Have the content API endpoints been thoroughly reviewed for security?

  • The REST API plugin has been on HackerOne for over a year with paid bounties for bugs
  • @barry has begun a security review

Performance

How does the API measure up against alternatives? Are there concerns about how the API could impact the servers to which it is deployed?

  • DeliciousBrains did a performance comparison with Admin AJAX and found the REST API to have a performance improvement (These tests have not yet been independently verified)
  • @mikeschroder notes in the comments that using the REST API in place of Admin-Ajax will also bring speed benefits by permitting many previously-uncacheable requests to be cached.

User Feedback

Are the content endpoints sufficiently well-tested & vetted by the community?

  • @matt questions whether feedback is coming too late in development for concerns to be acted upon
    • @rmccue notes that the v2 endpoints were created based on user feedback; REST API endpoints are being deployed by plugins and running on VIP, and the content endpoints have been in wide use across a variety of sites, leading to 90+ code contributors and far more developers’ support & feedback on the endpoints
  • @rmccue has also reached out to Phil Sturgeon for feedback and will follow up

Do Content Endpoints Benefit Core Development?

Will having these endpoints in core improve future core development, or solve any immediate problems?

  • @bradyvercher suggested that the content API endpoints would remove the need to write a variety of one-off ajax callbacks when developing future WordPress Core AJAX functionality
  • @westonruter notes that the customizer could dynamically create settings for posts and other kinds of content without having to wire up new admin-ajax handlers

Will Merging Negatively Impact API Development?

Will having to work through trac instead of GitHub cause development to atrophy?

  • @jjj argues that merging will slow development, because GitHub-hosted plugins are not bound to WordPress release cycles and have less overhead for features to be developed and deployed for testing. @jjj requested a plan for how the REST API will be developed going forward after the merge of these endpoints that would account for the added friction.
  • @krogsgard countered that core increases the visibility of a project like the content endpoints
    • The number of new contributors in this Slack discussion suggests that this merge proposal is bringing in new voices; whether this supports Brian’s point or not, the team is grateful for the breadth of perspectives being shared -Ed.
  • @rachelbaker suggested that the API endpoints are sufficiently inter-dependent with core WordPress code that maintaining the plugin separately amounts to maintaining a fork, and that such separated development is untenable long-term.
  • @matt hopes that a merge of these endpoints would slow release speed, but not development speed; @rmccue feels that development speed will stay the same or increase, and that tying releases to WordPress Core increases the stability and predictability of the API endpoints.
  • The versioning of the API endpoints supports forward compatibility

Do Content Endpoints Belong on Every WordPress Site?

What are the pros and cons to having every WordPress site have content API endpoints?

  • @rmccue suggests the API has network effects that can only be realized with a large install base. @krogsgard draws a comparison to RSS, the widespread availability of which enables everything from podcasting from WP to the use of apps like Feedly.
  • @matt suggests that the Atom API is a better analogue than RSS, which is an independent and pre-existing standard, and that network effects could be tested through inclusion in Jetpack
  • @joostdevalk notes that many plugins, like Yoast, have data tied to existing content such as posts and pages; either they expose the content through their own endpoints, or core does. If Core exposes content types through the API then plugins may build on top of that shared foundation, not independently reinvent the wheel. “if this doesn’t end up in core, we’ll start rolling our own API for stuff. Others will too. Interoperability won’t be there, for even the most basic stuff. I think this isn’t like RSS, I think this is much more like all of us using the same table space in MySQL.”
    • @shelob9 and @masonjames agree that merging the endpoints would create a consistent and reliable open “standard” that WordPress developers can use instead of continually reinventing how to read and edit post data over HTTP.
    • In response to the question “what prevents you from building on the endpoints in their plugin form,” @joostdevalk went on to note that plugin dependencies would make that a viable option, but that burden currently lies on the user. Plugin installation is not frictionless.
  • Can these endpoints be bundled? short takeaway: no
    • Woo bundled the API infrastructure before it was merged; doing so for content endpoints would require bundling prohibitively large amounts of endpoint code.
    • @nerrad worries that if plugins bundle different versions of the endpoints plugin, then those plugins may conflict if all bundled copies are not kept in sync.
      • @nerrad clarifies in the comments below that these worries also encompass the additional risk of conflicts when plugin authors each build their own versions of these content endpoints, instead of leveraging a shared standard: if two plugins each expose their own REST collection for posts, a developer working on a site with multiple such endpoints will need to decide which to extend, and will then have their extension tied to that specific plugin rather than to a core API.
  • @schrapel and @jorbin discussed that these content endpoints make it much easier to crawl a site, which also brings some potential performance concerns: no new content is exposed, but the process of aggregating it is easier and more easily automated.
  • In the  comments below @foliovision believes that merging the endpoints will be the best way to assert the level of back-compatibility that mid-size agencies need in order to confidently utilize the endpoints.

Please leave thoughts, questions, concerns, comments & experience in the comments below. Thank you!

Edited 2016-10-16 to include the below comments into the body of the post

#4-7, #rest-api

REST API Team Update, 4.7 Week 8

Summary: Beta 15 has been released, there are open questions that would benefit from your feedback, and the Content API Endpoints and OAuth Server are being proposed for merge as distinct, separate enhancements to the existing WordPress REST API infrastructure.

REST API v2 Beta 15 released

The 15th beta release of the REST API content endpoints plugin was released on October 7. This release builds on top of the recent Beta 14 to…

  • Add support for Post Meta, Term Meta, User Meta and Comment Meta within their parent endpoints
  • Introduce a settings endpoint to allow key site setting values to be retrieved & modified using the API
  • Introduce query parameters to query for posts that are NOT IN one or many terms of specific taxonomies
  • Resolve bugs, including bad comparison logic when updating comments.

Please try it out and report any outstanding issues; the REST API project gained its 90th code contributor this week and the team is deeply grateful for the energy and support of the broader WordPress community in testing out this merge-candidate plugin!

New Questions & Discussion Items

Items which have arisen through final ticket triage & review on which the team seeks feedback:

  1. Should the `filter` shim should be removed prior to merge? It is the majority position of the API team that `filter` be deprecated to dramatically improve the simplicity and consistency of API query functionality
  2. How should comments be handled for password-protected posts? Should the password be passed as a query parameter with the PUT/POST request, or is there a better option?
  3. Should the API match core’s logic when users with the `unfiltered_html` capability are creating or updating Posts or Comments?

Meeting Notes

At the weekly team meeting on October 10 the group reviewed open issues in the 2.0 milestone, which represents the candidate for our merge proposal shared last week.

Meeting attendees agreed to review open issues and pull requests individually, and to reconvene on Tuesday at 1500UTC to ensure all priority tickets had an owner.

At that meeting on October 11, the team reviewed the incoming feedback around the OAuth plugin (linked above). While the API team feels that having a built-in authentication solution provides a much-needed service, particularly to developers building mobile and desktop applications, the design and usability feedback we have received does indicate that the plugin needs more work.

OAuth’s place in the Merge Proposal

The API Team believes that the identified issues are resolvable, that the OAuth plugin is on track and that it should still be considered for merge in 4.7. However, after discussion within the team, input from @matt, and advice from @aaroncampbell and other core committers, we have edited our merge proposal to submit the Content API Endpoints and OAuth server as separate merge candidates. The API Team proposes both components for merge, but we submit the content endpoints for consideration independently of the OAuth1 server.

Content Endpoints Without OAuth

The Content API endpoints are stable, well-tested, and in wide production use across a variety of applications. Theme and plugin developers will benefit from having canonical, well-tested API endpoints in core, which may be used to query WordPress both from PHP code and from JavaScript applications running on the front-end or admin of WordPress. Sharing the endpoints for core data types enables increased consistency of what data is exposed and how it is persisted across different plugins, improving consistency and shortening development time by using . These themes and plugins have full read and write access to the API using the existing cookie & nonce authentication.

Mobile and desktop applications can leverage these same endpoints in a read-only capacity to create a variety of powerful reader-oriented applications and tools that expand the capability of what WordPress can do today, such as a unified reader for Make WordPress blogs and other experiments hypothesized by @jorbin.

Should OAuth 1 not be accepted for 4.7, secure write access for these external applications would still be only a plugin install away; and while having an OAuth server in core will provide a canonical approach for authenticating from remote applications, depending on the needs of a specific site or specific client application other authentication schemes may actually be preferable. Plugins exist for JWT Authentication and of course OAuth 2, and should OAuth 1 not be accepted for 4.7 these plugins may still be installed to enable an external application to opt-in to secure write access to your WordPress site.

In Summary

The API team submits for 4.7 merge consideration two enhancements to the REST API infrastructure: the Content API Endpoints for core WordPress datatypes, and an OAuth server which will reduce the setup time needed to securely interact with those endpoints from outside of WordPress. We believe these enhancements are each individually sufficiently tested and mature to meet the quality and security standards of WordPress Core, and each individually provides wide-reaching benefit to WordPress developers, and through them to the authors, readers & publishers of the web.

#4-7, #rest-api

REST API Merge Proposal, Part 2: Content API

Hi everyone, it’s your friendly REST API team here with our second merge proposal for WordPress core. (WordPress 4.4 included the REST API Infrastructure, if you’d like to check out our previous merge proposal.) Even if you’re familiar with the REST API right now, we’ve made some changes to how the project is organised, so it’s worth reading everything here.

(If you haven’t done so already, now would be a great time to install the REST API and OAuth plugins from WordPress.org.)

A brief history of the REST API

The REST API was created as a proof-of-concept by Ryan McCue (hey, that’s me!) at the WordPress Contributor Summit in 2012, but the project kicked off during the 2013 Google Summer of Code. The end result was Version 1.0, which grew into a community supported initiative that saw adoption and provided for a solid learning platform. The team used Version 1 to test out the fundamental ideas behind the API, and then iterated with Version 2, which made some major breaking changes, including explicit versioning, the introduction of namespacing for forwards compatibility, and a restructure of the internals. Version 2 also led to the infrastructure of the REST API being committed to WordPress core in 4.4.

This infrastructure is the core of the REST API, and provides the external interface to send and receive RESTful HTTP requests. Since shipping in 4.4, the infrastructure is now used by WordPress Core for oEmbed responses, and by plugins like WooCommerce and Jetpack, enabling anyone to create their own REST API endpoints.

The team has also been hard at work on the API endpoints. This has included core changes to WordPress to support the API, including deeper changes to both settings and meta.

Today the REST API team is proposing the inclusion of a collection of endpoints that we term the “Content API” into WordPress Core.

Proposals for Merge

Content Endpoints

For WordPress 4.7 the API team proposes to merge API endpoints for WordPress content types. These endpoints provide machine-readable external access to your WordPress site with a clear, standards-driven interface, allowing new and innovative apps for interacting with your site. These endpoints support all of the following:

  • Content:
    • Posts: Read and write access to all post data, for all types of post-based data, including pages and media.
    • Comments: Read and write access to all comment data. This includes pingbacks and trackbacks.
    • Terms: Read and write access to all term data.
    • Users: Read and write access to all user data. This includes public access to some data for post authors.
    • Meta: Read and write access to metadata for posts, comments, terms, and users, on an opt-in basis from plugins.
  • Management:
    • Settings: Read and write access to settings, on an opt-in basis from plugins and core. This enables API management of key site content values that are technically stored in options, such as site title and byline.

This merge proposal represents a complete and functional Content API, providing the necessary endpoints for mobile apps and frontends, and lays the groundwork for future releases focused on providing a Management API interface for full site administration.

Content API endpoints support both public and authenticated access. Authenticated access allows both read and write access to anything your user has access to, including post meta and settings. Public access is available for any already-public data, such as posts, terms, and limited user data for published post authors. To avoid potential privacy issues we’ve taken pains to ensure that everything we’re exposing is already public, and the API uses WordPress’ capability system extensively to ensure that all data is properly secured.

Just like the rest of WordPress, the Content API is fully extensible, supporting custom post meta, as well as allowing more complex data to be added via register_rest_field. The API is built around standard parts of WordPress, including the capability system and filters, so extending the API in plugins should feel as familiar to developers as extending any other part of WordPress.

This Content API is targeted at a few primary use cases, including enhancing themes with interactivity, creating powerful plugin interfaces, building mobile and desktop applications, and providing alternative authoring experiences. We’ve been working on first-party examples of these, including a mobile app using React Native and a liveblogging web app, as well as getting feedback from others, including WIRED, the New York Times, and The Times of London. Based on experience building on the API, we’ve polished the endpoints and expanded to support settings endpoints, which are included as the first part of the Management API.

Authentication

The API Infrastructure already in WordPress core includes support for regular cookie-based authentication. This is useful for plugins and themes that want to use the API, but requires access to cookies and nonces, and is hence only useful for internal usage.

To complement the Content Endpoints, for WordPress 4.7 the API team also proposes merging the REST API OAuth 1 server plugin into WordPress Core. This plugin provides remote authentication via the OAuth 1 protocol, allowing remote servers and applications to interact securely with the WordPress API.

OAuth is a standardised system for delegated authorisation. With OAuth, rather than providing your password to a third-party app, you can authorise it to operate on your behalf. Apps are also required to be registered with the site beforehand, which gives site administrators control over third-party access. Access to these apps can be revoked by the user if they are no longer using the app, or by a site administrator. This also allows apps with known vulnerabilities to have compromised credentials revoked to protect users.

We’ve chosen OAuth 1 over the newer OAuth 2 protocol because OAuth 1 includes a complex system for request signing to ensure credentials remain secure even over unsecured HTTP, while OAuth 2 requires HTTPS with a modern version of TLS. While it is strongly encouraged for sites to use HTTPS whenever possible (Let’s Encrypt makes it easier than ever to do so), WordPress itself does not require HTTPS and we do not believe WordPress should make HTTPS a requirement for using the API. The additional complexity that OAuth 1 adds can be easily supported by a library, and many such libraries already exist in most programming languages. OAuth 1 remains supported around the web, including for the Twitter API, and we also provide extensive documentation on using it.

Authentication Beyond 4.7

One issue with OAuth over direct username and password authentication is that it requires applications to be registered on the site. For centralized OAuth servers this wouldn’t be a problem, but the distributed nature of WordPress installations makes this tough to handle: your application must be independently registered with every WordPress site it connects to. If you’ve ever had to create a Twitter or Facebook app just to use an existing plugin on your site, you’ll know this can be a less-than-optimal experience for users.

To solve this distribution problem, we’ve created a solution called brokered authentication. This allows a centralised server (called the “broker”) to handle app registration and to vouch for these apps to individual sites. It simplifies app registration by allowing app developers to register once for all sites, and improves security by allowing the broker to vet applications and revoke them across the entire network. The system is designed to allow multiple brokers; while the main broker is run at apps.wp-api.org, organisations can run their own broker for internal usage, and developers can run a broker locally for testing.

While the broker system has been running live at apps.wp-api.org for months, we want to stay conservative in our approach to the API, especially where security is concerned. We are therefore proposing brokered authentication for WordPress 4.8 to ensure we have further time to continue testing and refining the broker system. In addition, this will require an installation of the broker on a centralised server to act as the canonical broker for out-of-the-box WordPress. While apps.wp-api.org is currently acting in this role, this is currently hosted by a third-party (Human Made) on behalf of the API team. For long-term usage the broker should instead be hosted on WordPress.org, alongside the existing plugin and theme repositories. This migration will take time but we remain committed to continuing to develop and support the broker.

After Merge

After merging the REST API, the team plans to continue developing the API as before. We expect that integrating the REST API into WordPress core will bring additional feedback, and we plan on incorporating this feedback through the rest of the 4.7 cycle.

During the remaining parts of this release cycle and through into the 4.8 cycle, additional work will go into other parts of the API. This includes further work and refinement on the broker authentication system, including work on WordPress.org infrastructure. Additionally, we plan to continue working on the Management API endpoints, including theme and appearance endpoints to support the Customiser team. Both of these components will be maintained as separate feature projects on GitHub until they’re ready for merge into core.

The team remains committed to supporting the API in core, and the Content API will switch from GitHub to Trac for project management and contributions. This same process occurred for the API Infrastructure in WordPress 4.4.

Reviews and Feedback

With this merge proposal, we’re looking for feedback and review of the project. In particular, we’re focussing on feedback on the security of the API and OAuth projects, and are also reaching out to specific people for reviews. (We take the security of the API seriously, and bug reports are welcomed on HackerOne at any time.) Design and accessibility reviews for the OAuth authorisation UI are also welcomed to ensure we maintain the high standards of WordPress core.

Both the REST API plugin and the OAuth plugin are available on WordPress.org, and issues can be reported to the GitHub tracker for the API and the OAuth plugin respectively. We have released a final beta (Beta 15 “International Drainage Commission”) which includes the meta and settings endpoints.

With Love from Us

As always, this is a merge proposal, and is not final until 4.7 is released. We’re eager to hear your thoughts and feedback; the comments below are a perfect place for that, or you can pop along to one of our regular meetings. We’re also always available in the #core-restapi room on Slack.

We’d like to thank every single one of our contributors, including 88 contributors to the main repository and 23 contributors to the OAuth repository. Particular thanks goes to my (@rmccue) wonderful co-lead Rachel Baker (@rachelbaker), our 2.0 release leads Daniel Bachhuber (@danielbachuber) and Joe Hoyle (@joehoyle), and our key contributors for the 4.7 cycle: Adam Silverstein (@adamsilverstein), Brian Krogsgard (@krogsgard), David Remer (@websupporter), Edwin Cromley (@chopinbach), and K. Adam White (@kadamwhite). Thanks also to the core committers helping us out through the 4.7 cycle, including Aaron D. Campbell (@aaroncampbell) and Aaron Jorbin (@aaronjorbin), and to the fantastic release lead, Helen Hou-Sandí (@helen).

Thanks also to everyone who has used the REST API, and to you for reading this. We built the REST API for you, and we hope you like it.

With love, The REST API Team

#feature-plugins, #json-api, #merge-proposals, #rest-api

REST API Team Update (4.7 week 7)

Our two (pt1) part (pt2) (split due to team travel) weekly meeting is now complete, and we have some new updates from the API team!

Team review & commentary is requested on “Open Questions on Settings” and “Open Questions on Meta”, below.

Settings

This Pull Request for adding Settings support to the API, resolving one of the few remaining major content gaps in the API, is almost ready to be merged into the rest-api feature plugin. Settings are exposed as keys on the resource `/wp/v2/settings`, e.g.:

{
    title: 'WordPress Develop',
    description: 'Just another WordPress site',
    url: 'http://src.wordpress-develop.dev',
    email: 'admin@local.dev',
    timezone: '',
    date_format: 'F j, Y',
    time_format: 'g:i a',
    start_of_week: 1,
    language: 'en_US',
    use_smilies: true,
    default_category: 1,
    default_post_format: '0',
    posts_per_page: 10
}

This resource is only accessible or modifiable if you are authenticated as a user with the `manage_options` capability.

Open Questions on Settings

Team feedback is requested on the below, and on the settings proposal in general.

  1. What do we do if a setting value in the database does not match the type used in `register_setting`?
    • This becomes a problem when there is serialized data in the `option` value. Right now we are forcing this to be a `null` value in the rest api responses. The problem with this approach is that sending that response data back to the settings endpoint will cause the deletion of that option value (as `null` in this case is essentially a delete operation).
  2. Having support for `object` recursion and `array` types in the Schema would be very nice, allowing settings that are either objects or arrays.
    • We propose that does not need to be resolved prior to merge, and may be added in a subsequent cycle.
  3. Currently, reading all options at `/wp/v2/settings` is behind a `manage_options` cap check: we need to work out if we want to publicly expose option values for some settings.
    • If we do this, we’d need a way in `register_setting` (or via some kind of `read_setting` meta cap) to work out which settings should be exposed publicly (or at specific permissions checks).
    • Site title and description are still available on the root `/wp-json/` resource index; several other settings here can be inferred from other responses or site content. We propose that this resource be left as requiring `manage_options` caps, and move establishing a mechanism in core to register a setting as public be handled in a subsequent development cycle.

Setting Registration

For your settings to be exposed in the REST API, you need to register them. WordPress includes a register_setting function which is not usually required to get/set options, but is required for API support.

To register your field, simply call register_setting and set the show_in_rest flag to true.

register_setting( 'group', 'option_name', array(
    'show_in_rest' => true,
));

register_setting Arguments

register_setting is part of WordPress core, but isn’t heavily used in most parts of core, so you may not have seen it before. Here are the options relevant to the REST API:

  • show_in_rest – Should the setting be exposed in the API? true for default behaviour, or an array of options to override defaults (see below).
  • type – This is the type of data the setting is. Can be `string`, `number`, or `boolean` (so far).
  • description – Human-readable description of what the field is for. The API exposes this in the schema.
  • sanitize_callback – Callback to sanitize the value before it’s stored in the database. See the "sanitize_{$object_type}_meta_{$meta_key}" filter.

REST API-Specific Arguments

The show_in_rest parameter defaults to false, but can be set to true to enable API support. Out of the box, the API makes some assumptions about your setting, but you can override these by setting show_in_rest to an options array instead:

register_setting( 'group', 'option_name', array(
    'show_in_rest' => array(
        'name' => 'fieldname'
    )
));

The options you can set are:

  • name – The key exposed via the REST API. For example, if your setting starts with an underscore, you may want to remove this for the API.
  • default – The default value for the setting via the REST API if it’s not set in the database.
  • schema – The JSON Schema options for the field. Exposed via OPTIONS requests to the post. This is generated automatically based on other parameters, but can be specified manually if you’d like to override it.

Settings Defaults

We’d also love feedback on ticket #38176, this adds the ability to specify a default value for an option via `register_setting()`. As you can see from the above, we are currently defining the `default` value within
the `show_in_rest` options, as there’s no way to do this in the Settings API. We’d like for there to be a way to do this in core, so `get_option()` could default to the value set in `register_setting( … [ default => ‘myvalue’ ])`
when no default is specific in `get_option`.

Meta

Meta is also ready to be merged into the plugin, and also could use review from the broader WordPress team. Similar to custom post types and taxonomies, and as discussed previously, only meta that opts-in to API support is available through the REST API.

Open Questions on Meta

  1. Right now, all meta registered through the mechanism below is assumed to be visible wherever its parent object appears. Meta values registered to a public post will be visible on that post without authentication.
    • Is it acceptable to expose registered meta fields on objects in this way, where the meta visibility will match the visibility of the parent object?
  2. As described in this comment, when you have multiple of the same value stored in a meta key, it will delete all matching values. For example, if you had a meta value [ 11, 13, 13, 13, 13 ] and attempted to remove one 13, it would result in the value [ 11 ].
  • Should multiple instances of the same value be able to be stored in a `single => false` meta field?

Meta Registration

For your meta fields to be exposed in the REST API, you need to register them. WordPress includes a register_meta function which is not usually required to get/set fields, but is required for API support.

To register your field, simply call register_meta and set the show_in_rest flag to true.

register_meta( 'post', 'field_name', array(
    'show_in_rest' => true,
));

Note: register_meta must be called separately for each meta key.

register_meta Arguments

register_meta is part of WordPress core, but isn’t heavily used in most parts of core, so you may not have seen it before. Here are the options relevant to the REST API:

  • show_in_rest – Should the field be exposed in the API? true for default behaviour, or an array of options to override defaults (see below).
  • description – Human-readable description of what the field is for. The API exposes this in the schema.
  • single – Is this field singular? WordPress allows multiple values per key on a single object, but the API needs to know whether you use this functionality.
  • sanitize_callback – Callback to sanitize the value before it’s stored in the database. See the "sanitize_{$object_type}_meta_{$meta_key}" filter.
  • auth_callback – Callback to determine whether the user can write to the field. See the "auth_post_{$post_type}_meta_{$meta_key}" filter.

REST API-Specific Arguments

The show_in_rest parameter defaults to false, but can be set to true to enable API support. Out of the box, the API makes some assumptions about your meta field, but you can override these by setting show_in_rest to an options array instead:

register_meta( 'post', 'field_name', array(
    'show_in_rest' => array(
        'name' => 'fieldname'
    )
));

The options you can set are:

  • name – The key exposed via the REST API. For example, if your meta key starts with an underscore, you may want to remove this for the API.
  • schema – The JSON Schema options for the field. Exposed via OPTIONS requests to the post. This is generated automatically based on other parameters, but can be specified manually if you’d like to override it.

Using Meta over the API

Meta is added to existing resources as a new meta field. For example, to get the meta for a single post, fetch the post at /wp/v2/posts/{id}, then look in the meta field. This is a key-value map for the registered fields.

To update a field, simply send back a new value for it.

Setting a field to null will cause the value to be removed from the database. If a field has a default value set, this essentially “resets” the value to the default (although no default will be stored in the database).

For meta fields which can hold multiple values, this field will be a JSON list. Adding or removing items from the list will create or remove the corresponding values in the database.

Other Updates

  • Add a {taxonomy}_exclude query parameter to query for resources NOT IN a particular taxonomy term_id
  • Decide against supporting full tax_query functionality via filter[tax_query]
  • Defer supporting auto-draft to a future cycle, as it did not have the expected benefits and the applicability of that state to the API is not generally understood

#4-7, #rest-api

WordPress REST API – 2.0 Beta 14

Hey folks! I’m excited to announce the release of Beta 14 of the REST API. It’s been a while since our last release, beta 14 is jam packed with general improvements, bug fixes and general refinement to polish the feature plugin before core merge.

Get it at WordPress.org or GitHub. View all changes on GitHub; here are the highlights:

  • Add support for password protected posts. There is now a protected attribute in the content and excerpt fields in post response. To view password protected posts via the API, use the password query parameter to provide the post’s password.
  • Allow returning an error from field update callbacks.
    Simply have your update_callback return a WP_Error.
  • Add relevance orderby to posts endpoint
  • Ability to order by slug, email and url on the users endpoints.
  • Add sticky parameter to the posts endpoint.
  • Update the wp-api.js client from the client-js repo.
  • Many many more

Thanks to all the contributors and special thanks to @chopinbach, @kadamwhite and @websupporter for their effort this release.

We’ll be kicking Beta 15 which will be focused on meta support for posts, terms, comments and users as well as a brand new settings endpoint. These are currently in review awaiting feedback, check out the meta pull request and the settings pull request and let us know what you think.

In addition we will be publishing a merge proposal for WordPress 4.7 this week, stay tuned!

#feature-plugins, #json-api, #rest-api

REST API Meeting Agenda for Oct 3 2016

REST API: Meeting Agenda for Oct 3, 2016

The weekly meeting for the API team will be on Monday at 2016-10-03 14:00 UTC in #core-restapi. On the agenda:

  • Update on Settings from @joehoyle
  • Update on Meta from @rmccue
  • 4.7 Merge Plan TODOs
    • Decision on supported authentication methods
    • Progress on outside review
    • Timeboxed open floor
  • Review open PRs
  • Find owner for open tickets, e.g. some OAuth/Cookie conflicts

#4-7, #rest-api

API Team Update

API Team Update Sept. 27 2016

The API team met yesterday on slack for our weekly update, which this week was predominately focused on follow-up from last Thursday’s comprehensive GitHub issues scrub.

Since the previous API team update the group has been making steady progress through the gameplan outlined in that post. To give a quick update on a few key “quirky” issues that had been identified:

Password-Protected Posts

As noted in last week’s dev chat, password-protected posts will be included in collections with their content set to '', and the content can be viewed by passing ?password=XXXXX as a query or GET parameter when querying for a specific post. Query parameters are not an ideal solution: Authorization headers are out because you can’t have multiple authorization schemes in one request; cookies don’t afford enough control to browser clients, and custom headers aren’t respected by caches. See GH issue #2701 for more background, and check out the open pull request to review the specifics of the implementation.

Sticky Posts

After much debate there is now a path forward for handling sticky posts as well. Following this open pull request, sticky posts are included in the /wp/v2/posts collection, but are not given special treatment in terms of ordering—a sticky post will, by default, be displayed ordered by date or whatever orderby has been set for the request. The parameter ?sticky=true may be passed to return only sticky posts; ?sticky=false may be passed to exclude sticky posts from the response.

There is ongoing discussion around how the API could surface posts in the “normal loop order,” with stickies on top, followed by non-sticky posts. @jorbin will propose a follow-up enhancement that could be added in to the API in a later cycle. See GH issue #2210 and associated slack discussion for more commentary.

Meta Support

A pull request is open on the API meta endpoints repository to add support for registering meta with the API using the main register_meta function. This PR includes a comprehensive readme explaining how meta handling works. Once merged & reviewed, this functionality will be PR’d against the primary REST API plugin repository.

Site Settings (Options) Support

A pull request has merged into the API site endpoints repository which adds support for accessing and manipulating site settings through the API. These endpoints will be PR’d against the core REST API plugin repository this week.

Discussion items for 9/28 dev chat

This project is not quite done yet, and in tomorrow’s dev chat the API team will be looking to the broader WordPress team for input on three priority issues that need a decision:

Upcoming Meetings

Please join us in #core-restapi on chat.wordpress.org for our next two group meetings:

#4-7, #rest-api, #team-update

WordPress REST API update

tl;dr

There’s a renewed push going on right now to try and get what is being termed “content endpoints” into WordPress core with the 4.7 release.

In the first core development meeting of the 4.7 cycle, @helen identified a series of tasks that would need to be analyzed and acted upon to be able to make a new proposal for core inclusion, including identifying existing blockers. There is a team of people actively working on these items, and your participation is wanted!

New meeting times

Regular meetings have been changed to take place at 14:00 UTC Mondays, with bug scrubs at 14:00 UTC Thursdays — all in #core-restapi. So the next meeting is Monday, September 12th, 14:00 UTC.

Fuller story and action items

If you are not caught up on the state of the WordPress REST API, the infrastructure for the API went into WordPress 4.4. Since that time, several prominent plugins are using the infrastructure to create their own REST APIs.And now the feature project (not in core) consists of core endpoints and authentication mechanisms. The four primary types of resources that have been developed are for: posts (and other post types), users, comments, and terms. There is also a Google spreadsheet where you can list sites you know of running V2 of the REST API plugin in production.

The proposal, if the various criteria are met, would request core inclusion for what we’re calling “content endpoints”, and “management endpoints” would be part of a subsequent release cycle. With the content endpoints, website developers would have tools at their disposal to build websites in whatever programming language they choose, using data from WordPress. Additionally, certain types of applications would also be able to create experiences for managing WordPress content — though not complete WordPress site management the way you can from the WordPress admin.

The primary focus areas for core inclusion of the WordPress REST API — as initially defined by Helen, and then expanded on in the core dev chat meeting — are as follows:

  1. Rigorously test 4.6 and trunk compatibility and resolve any issues that may be found.
    1. Includes reviews by current component maintainers for existing endpoints.
    2. For example: WP_Post_Types and other new objects, need compatibility.
  2. Identify and resolve some of the final “quirky” issues (e.g. password-protected posts).
  3. Create support for meta.
    1. By “meta support” in the API, we refer to meta values that have been registered by a developer. Ideally via register_meta( ...., array( 'show_in_rest' ) ). For clarity, this excludes arbitrary meta storing (i.e. a client arbitrarily using the WordPress database)
  4. Create support for options – this is not “content” per say, but imagine an app where you can’t change your site title and tagline.
    1. Needs significant clarification, definition of what should be achieved
    2. Repo for site endpoints: https://github.com/WP-API/wp-api-site-endpoints
      1. Discuss architecture for how this would work (like, site endpoints w/ object of settings) https://github.com/WP-API/WP-API/issues/816
  5. Establish a forward compatibility plan, particularly around how to avoid/minimize plugin and theme conflicts. IE: namespacing and documentation / protected stuff. Relate to current general WP best practices (IE: custom field named “likes” – possible vs good idea). Need document to outline our views.
  6. Dedicated reviews from developers with deep experience in security and REST APIs, ideally including some of the non-WP PHP community.
    1. Identify and request reviews by specific developers & subjects.
      1. Security
      2. Core compatibility
      3. Integration
      4. Consumption
      5. Non-WP developers / fresh eyes
  7. Identify current authentication options, and their viability for inclusion in 4.7. Document flows of implementing and using each.
    1. Cookie auth
    2. Basic auth
    3. oAuth 1
    4. oAuth 2
  8. Establish and document data with performance comparisons – speed, bandwidth, etc – against admin-ajax, XML-RPC, etc. (Might just require education, as really all these are pretty similar). Identify and address performance related issues on project GitHub repo.
  9. Recruit and assign new / excited contributors
  10. Align existing docs with primary project. Outline documentation needs, and create them.
    1. Get volunteers to take on specific tasks
    2. Decide where these docs go, both now and in the future

Specific initiatives

There are several more specific initiatives to work on, many of which we’ve tasked out and assigned, but plenty that could use more input.

Docs Initiatives

  • Inline docs will eventually be merged into core inline docs, so any prep there should be roadmapped along with the rest of the 4.7 planning
  • User docs on consuming the API (e.g. doing things with the routes it provides from external systems, like uploading media) are needed and should live in the docs-v2 repo for now. See issues list for current user-facing documentation needs.
  • User docs on extending within the context of the API plugin (how to add routes, how to lock down access to auth’d users) are needed and should live in the docs-v2 repo for now
  • Docs on making endpoints with the infrastructure currently in core should live within the developer handbook

Task Assignments

  • Ping component maintainers to see what testing they’ve done w/ API (@Krogsgard)
  • Password game plan: relies on #16483: Blank content, rendered string, title however it is done in core now, 401 for individual posts, content to include protected:truein return object, and (maybe) give users that can edit posts access to content in response, consider Authorization header options. (@rmccue)
  • A pass at registered settings. @joehoyle has tackled this with a first draft, along with #37885.
  • Document feature detection as it works today (http://v2.wp-api.org/guide/discovery/). Establish best practice for extending with new endpoints. Establish best practice for modifying existing objects.
  • Documentation needed: compare register_rest_field() vs register_meta() and document best practice for when register_rest_field() may still be preferable. But generally encourage usage of register_meta()
  • Consideration: With register_rest_field, maybe force a namespace a la register_rest_route
  • Contact implementors from @joehoyle‘s API-in-use list to get feedback on their experience with the API (@krogsgard)
  • Document that name, description, url and home are the options already available in index as read-only. Consider change of this with global options endpoint.
  • Develop personas for user groups that interact with the API (@jorbin)
  • Interface testing for cookie and oauth1 implementations. Recruit UI/Design help. (@krogsgard and @kadamwhite)
  • Determine auth_callback (needs different name, has a conflict right now) necessity within register_meta(), as someone may want meta exposed, but only for authenticated users, and there is no cap system for read_meta.>
  • Review https://github.com/WP-API/WP-API/issues/2558 for performance gain.
  • Review https://github.com/WP-API/WP-API/issues/1625 for awkward data handling w/ client-js

Bug scrubs

The first bug scrub of this cycle took place today, and we were able to go through all open issues on GitHub that do not have a label, and label them, plus add at least some level of context. Our priority for future meetings will be to ensure that we have assigned bugs to appropriate people, and go back through and ensure we have milestones assigned to various tickets. We’ll have an open floor period during each regular meeting to discuss particular issues.

Get involved

If you have any interest in the API, your help and insights are wanted! You can join Chat.WordPress.org in the #core-restapi room to sit in and watch, or jump in to various discussions. Also, if you just want to play with the plugin and report back your experience, that’d also be super helpful.

One group of core contributors we really need feedback from are component maintainers. The team working on the REST API would like your input on how well the API currently interacts with your component, how it can improve, and to identify trouble areas that would need to be addressed both for initial core inclusion of the API, and down the road.

Thanks for listening!

#4-7, #rest-api

Enhancing register_meta() in 4.6

Note: Note: The direction of register_meta() has changed since this was published. Please see the most recent developer note explaining these changes.

In 4.6, register_meta() expands to support the registration of meta keys and what to expect from those keys. See #35658 for the discussion around this change.

The behavior of register_meta() is similar to register_post_type() in that the registration of this data is stored in the global scope. This makes an object’s meta data more accessible to parts of core and extending code.

Why make this change

In 4.5 and earlier, WordPress provided no method to explicitly register a meta key’s public state. Instead, register_meta() provided “protected” and “authenticated” meta. This can be used effectively for some things, particularly to determine what meta appears in the Custom Fields metabox when editing a post. It can not be used as a basis for arbitrarily showing meta keys and values to unauthenticated visitors.

What does this change involve

A global variable, $wp_meta_keys, contains all registered meta keys.

The function signature of register_meta() has changed to support 3 arguments, the last being an array. That array should contain data about the meta with these key/values:

  • sanitize_callback, a callable method to provide sanitization. This is the new version of the current 3rd parameter in register_meta().
  • auth_callback, a callable method to provide authorization. This is the new version of the current 4th parameter in register_meta().
  • object_subtype, a string containing an object subtype slug. If there is no object subtype, meta will not be registered and a WP_Error will be returned instead.
  • type, a string indicating what type of data the meta value will be. This is intentionally not restricted to a specific list of data types, however full names should be used when possible. (e.g. boolean, integer)
  • description, a string containing a basic description of the meta value.
  • single, whether code retrieving meta for this key should expect a single or multiple values when using something like get_post_meta().
  • show_in_rest, whether this should be shown as part of a post’s meta endpoint in the WordPress REST API. Note that this should be treated as experimental until the WordPress REST API provides support for meta.

The register_meta_args filter is available to add support for additional arguments. This need to explicitly whitelist further arguments is to reserve the right for core to add further arguments in the future; should you choose to use this filter, you should be prepared to follow along with further development. Your testing with trunk or betas is greatly appreciated!

By default, only WordPress core object types (post, user, term, comment) can be registered. To add support for custom object types, use the wp_object_types filter. This whitelisting is similar to the above.

Object sub types provide specific registration of meta keys to a type of data.

  • A standard WordPress post has an object type of “post” and an object subtype of “post”.
  • A custom post type registered with a slug of “my_cpt” has an object type of “post” and an object subtype of “my_cpt”.
  • A WordPress user has an object type of “user” and an object subtype of “user”.
  • A standard WordPress comment has an object type of “comment” and an object subtype of “comment”.
  • A standard WordPress category term has an object type of “term” and an object subtype of “category”.
  • A custom taxonomy registered with a slug of “my_tax” has an object type of “term” and an object subtype of “my_tax”.

Meta keys must be explicitly registered for each object type and subtype combination.

Additional helper functions get_registered_metadata(), get_registered_meta_keys(), unregister_meta(), and registered_meta_key_exists() have been added to make the innards of the global data more accessible.

The $wp_meta_keys variable should not be altered directly. It is possible that its structure will change in the future.

Any code currently using register_meta() and expecting pre-4.6 behavior will continue to work as is. Please report any breaks in compatibility that might be found.

Example

Here’s what registering meta looked like in 4.5. This meta key has sanitization and authorization callbacks.

register_meta( 'post', 'my_meta_key', 'sanitize_my_meta_key', 'authorize_my_meta_key' );

The above code will continue to work in 4.6, though will not be considered completely registered. The callbacks will be registered, but the key will not be added to the global registry. A WP_Error object will be returned with this explanation.

Here’s what registering meta looks like in 4.6. This meta key will have sanitization and authorization callbacks, and be registered as public for the WordPress REST API.

$args = array(
    'object_subtype' => 'post',
    'sanitize_callback' => 'sanitize_my_meta_key',
    'auth_callback' => 'authorize_my_meta_key',
    'type' => 'string',
    'description' => 'My registered meta key',
    'single' => true,
    'show_in_rest' => true,
);
register_meta( 'post', 'my_meta_key', $args );

If you are currently using register_meta() and would like to maintain support for older versions of WordPress, the best method will be to check for the registration of the sanitization and authorization callbacks after calling register_meta() and then registering them manually if not present. This manual registration is all that register_meta() was previously doing. Continuing from the example above:

// Pre-WordPress 4.6 compatibility
if ( ! has_filter( 'sanitize_post_meta_my_meta_key' ) ) {
    add_filter( 'sanitize_post_meta_my_meta_key', 'sanitize_my_meta_key', 10, 4 );
}

if ( ! has_filter( 'auth_post_meta_my_meta_key' ) ) {
    add_filter( 'auth_post_meta_my_meta_key', 'authorize_my_meta_key', 10, 6 );
}

What’s next

While the initial work on this is largely driven by the need to move the WordPress REST API forward, its scope is not limited to solving the problem of public access to metadata. By having information about the meta keys used in a code base, it becomes much easier for other pieces of code to make decisions about what can be done with this data.

This data can one day be useful to everything from the Customizer, the editing experience for all object types, the Fields API, and many plugins that rely on metadata. Work will continue on transforming this $wp_meta_keys object and the methods surrounding it to be flexible and explicit. As WordPress core becomes familiar with register_meta() and more confident in the approach, the set of default arguments can and likely will expand to include more data about registered meta keys as required.

Feedback

Please leave any feedback you have on this new iteration of register_meta() in the comments below. Test it out in your plugins and themes to be sure it is working in a backwards compatible way. Try out the new function signature and register meta for objects and their subtypes. This continues to be a work in progress and feedback is important.

#4-6, #dev-notes, #fields-api, #options-meta, #rest-api