REST API team meeting agenda for February 6

The REST API team will be meeting at 21:00 UTC in #core-restapi. Please note the updated time, which has been moved to 21:00 UTC to better match the schedules of the majority of the REST API leads & contributor team.

On the agenda:

  • focus for 4.7.3 and beyond: project Trello board review & discussion
  • establish owners to complete the docs migration from wp-api.org to developer.wordpress.org
  • scrub tickets in the REST API trac component
  • open office hours

REST API: Agenda for November 7, 2016 Meeting

The weekly REST API meeting will happen in the core-restapi channel at 2016-11-07 14:00 UTC (Note to US attendees, the end of daylight savings time makes 14:00 UTC an hour earlier than in recent weeks).

On the agenda:

  • Documentation (in collaboration with @drew and the docs team): establish responsibilities & schedule to ensure documentation is adequately ported from wp-api.org to the WordPress REST API handbook during the coming weeks
  • WP-API plugin: establish ownership, goals & timeline for a stable release of the plugin to prepare for the 4.7 release to support existing plugin users and support the plugin-to-core transition
  • Ticket discussion & open floor

REST API: Agenda for October 24 Meeting

Here’s the agenda for Monday’s weekly meeting for the REST API project, at 2016-10-24 14:00 UTC in the #core-restapi channel in slack. The meeting will run for one hour.

  • Discuss post-merge Process Changes & the future of the WP-API GitHub Repository
  • Review Trac tickets: what hasn’t been migrated from GitHub?
  • Clarify priorities & responsibilities leading up to Beta 1
  • Open floor.

Meetings are every Monday at 1400 UTC. See you there!

#4-7, #agenda, #rest-api

REST API Team Meeting Notes, 2016-10-17

As a reminder, in just 30 minutes there is a meeting in #core to decide whether the REST API Content Endpoints will be merged as a part of WordPress 4.7! See the ongoing discussion of this proposal here. Note that the OAuth server is no longer proposed for merge at this time, but authentication options will be a primary focus area for the API project during the 4.8 development cycle.

The REST API Content Endpoints provide a new foundation upon which the WordPress developer community can build themes, plugins, and core feature. They represent a common standard and consistent interface across WordPress’s core content data types, and provide robust support for custom post types and meta values. These endpoints lay the foundation upon which future releases will add remote authentication options, even deeper querying abilities, and broader endpoint coverage for site management. This iterative approach fits WordPress’s development model and philosophy, advancing the project’s long-term goal of opening WordPress up to a wider developer audience and helping to ensure continued work on the REST API in the release cycles and years to come.

Meeting Notes

At today’s weekly API team meeting in core-restapi (agenda here) the team resolved all outstanding decisions milestoned for the REST API 2.0 / WordPress 4.7 merge candidate:

  • The ?filter query parameter will be removed from the REST API plugin prior to core merge, a breaking change that improves the consistency of querying the API and eliminates a set of parameters that could introduce backwards compatibility issues were they to be committed to WordPress core. A separate plugin will be published to reinstate the `filter` parameter on a strictly opt-in basis.
  • Comments on password-protected posts are being deferred as a future enhancement until a robust solution is proposed that permits the API to adequately mirror existing functionality.
  • The unfiltered_html capability should be respected by the API, and a patch will be submitted to bring the API’s behavior in line with core’s.

There are 29 tickets left in the 2.0 milestone, several of which have open pull requests already. These issues represent a mix of outstanding bugs, documentation needs and improvements that will be moved to trac should the merge proposal be accepted.

The REST API team leads would like to recognize that the content endpoints plugin now has 95 contributors: thank you and welcome to all of the new participants who have joined the project in the past week!

#4-7, #meeting-notes, #rest-api

REST API: Agenda for Oct 17, 2016 Meeting

The meeting to decide whether to merge the REST API Content Endpoints in WordPress 4.7 will occur at 2016-10-18 01:00 UTC in #core!

Please comment on this feedback post with your arguments for or against.

In preparation for that meeting, we will be holding bonus weekend office hours at 2016-10-15 17:00 UTC and 2016-10-16 17:00 UTC in #core-restapi.

Our usual team meeting is also happening, on Monday at 2016-10-17 14:00 UTC. The agenda for this meeting:

Merge Proposal Discussion: REST API Content Endpoints

There are discussion meetings and office hours in #core-restapi at 2016-10-14 14:00UTC and 2016-10-14 19:00UTC on Friday the 14th. Our next team meeting is on 2016-10-17 14:00UTC. Please attend some of all of these, because

We are meeting at 2016-10-18 01:00 UTC to make a decision on this merge proposal!

To that end, the below discussion points will be updated regularly, please leave comments on this post or join the conversation in #core-restapi.

Yesterday at the dev chat the API Team proposed the Content API Endpoints for merge in WordPress 4.7. There was popular support for this feature but as @jorbin and @helen noted that the lack of dissent suggested additional review is needed, so the API Team is continuing to seek thorough review & constructive criticism of the content endpoints, including the open questions previously shared on the week 7 and week 8 API team updates.

The merge proposal also engendered follow-up discussion in the #core-restapi channel about the benefit content endpoints bring to core, whether having such endpoints built in is quantifiably more beneficial than having them as a plugin, whether moving development from a plugin to core would slow development, and whether the endpoints as-proposed have been sufficiently reviewed for security and performance. We attempt to capture those questions & concerns (and the perspectives on them) below.

Security

Have the content API endpoints been thoroughly reviewed for security?

  • The REST API plugin has been on HackerOne for over a year with paid bounties for bugs
  • @barry has begun a security review

Performance

How does the API measure up against alternatives? Are there concerns about how the API could impact the servers to which it is deployed?

  • DeliciousBrains did a performance comparison with Admin AJAX and found the REST API to have a performance improvement (These tests have not yet been independently verified)
  • @mikeschroder notes in the comments that using the REST API in place of Admin-Ajax will also bring speed benefits by permitting many previously-uncacheable requests to be cached.

User Feedback

Are the content endpoints sufficiently well-tested & vetted by the community?

  • @matt questions whether feedback is coming too late in development for concerns to be acted upon
    • @rmccue notes that the v2 endpoints were created based on user feedback; REST API endpoints are being deployed by plugins and running on VIP, and the content endpoints have been in wide use across a variety of sites, leading to 90+ code contributors and far more developers’ support & feedback on the endpoints
  • @rmccue has also reached out to Phil Sturgeon for feedback and will follow up

Do Content Endpoints Benefit Core Development?

Will having these endpoints in core improve future core development, or solve any immediate problems?

  • @bradyvercher suggested that the content API endpoints would remove the need to write a variety of one-off ajax callbacks when developing future WordPress Core AJAX functionality
  • @westonruter notes that the customizer could dynamically create settings for posts and other kinds of content without having to wire up new admin-ajax handlers

Will Merging Negatively Impact API Development?

Will having to work through trac instead of GitHub cause development to atrophy?

  • @jjj argues that merging will slow development, because GitHub-hosted plugins are not bound to WordPress release cycles and have less overhead for features to be developed and deployed for testing. @jjj requested a plan for how the REST API will be developed going forward after the merge of these endpoints that would account for the added friction.
  • @krogsgard countered that core increases the visibility of a project like the content endpoints
    • The number of new contributors in this Slack discussion suggests that this merge proposal is bringing in new voices; whether this supports Brian’s point or not, the team is grateful for the breadth of perspectives being shared -Ed.
  • @rachelbaker suggested that the API endpoints are sufficiently inter-dependent with core WordPress code that maintaining the plugin separately amounts to maintaining a fork, and that such separated development is untenable long-term.
  • @matt hopes that a merge of these endpoints would slow release speed, but not development speed; @rmccue feels that development speed will stay the same or increase, and that tying releases to WordPress Core increases the stability and predictability of the API endpoints.
  • The versioning of the API endpoints supports forward compatibility

Do Content Endpoints Belong on Every WordPress Site?

What are the pros and cons to having every WordPress site have content API endpoints?

  • @rmccue suggests the API has network effects that can only be realized with a large install base. @krogsgard draws a comparison to RSS, the widespread availability of which enables everything from podcasting from WP to the use of apps like Feedly.
  • @matt suggests that the Atom API is a better analogue than RSS, which is an independent and pre-existing standard, and that network effects could be tested through inclusion in Jetpack
  • @joostdevalk notes that many plugins, like Yoast, have data tied to existing content such as posts and pages; either they expose the content through their own endpoints, or core does. If Core exposes content types through the API then plugins may build on top of that shared foundation, not independently reinvent the wheel. “if this doesn’t end up in core, we’ll start rolling our own API for stuff. Others will too. Interoperability won’t be there, for even the most basic stuff. I think this isn’t like RSS, I think this is much more like all of us using the same table space in MySQL.”
    • @shelob9 and @masonjames agree that merging the endpoints would create a consistent and reliable open “standard” that WordPress developers can use instead of continually reinventing how to read and edit post data over HTTP.
    • In response to the question “what prevents you from building on the endpoints in their plugin form,” @joostdevalk went on to note that plugin dependencies would make that a viable option, but that burden currently lies on the user. Plugin installation is not frictionless.
  • Can these endpoints be bundled? short takeaway: no
    • Woo bundled the API infrastructure before it was merged; doing so for content endpoints would require bundling prohibitively large amounts of endpoint code.
    • @nerrad worries that if plugins bundle different versions of the endpoints plugin, then those plugins may conflict if all bundled copies are not kept in sync.
      • @nerrad clarifies in the comments below that these worries also encompass the additional risk of conflicts when plugin authors each build their own versions of these content endpoints, instead of leveraging a shared standard: if two plugins each expose their own REST collection for posts, a developer working on a site with multiple such endpoints will need to decide which to extend, and will then have their extension tied to that specific plugin rather than to a core API.
  • @schrapel and @jorbin discussed that these content endpoints make it much easier to crawl a site, which also brings some potential performance concerns: no new content is exposed, but the process of aggregating it is easier and more easily automated.
  • In the  comments below @foliovision believes that merging the endpoints will be the best way to assert the level of back-compatibility that mid-size agencies need in order to confidently utilize the endpoints.

Please leave thoughts, questions, concerns, comments & experience in the comments below. Thank you!

Edited 2016-10-16 to include the below comments into the body of the post

#4-7, #rest-api

REST API Team Update, 4.7 Week 8

Summary: Beta 15 has been released, there are open questions that would benefit from your feedback, and the Content API Endpoints and OAuth Server are being proposed for merge as distinct, separate enhancements to the existing WordPress REST API infrastructure.

REST API v2 Beta 15 released

The 15th beta release of the REST API content endpoints plugin was released on October 7. This release builds on top of the recent Beta 14 to…

  • Add support for Post Meta, Term Meta, User Meta and Comment Meta within their parent endpoints
  • Introduce a settings endpoint to allow key site setting values to be retrieved & modified using the API
  • Introduce query parameters to query for posts that are NOT IN one or many terms of specific taxonomies
  • Resolve bugs, including bad comparison logic when updating comments.

Please try it out and report any outstanding issues; the REST API project gained its 90th code contributor this week and the team is deeply grateful for the energy and support of the broader WordPress community in testing out this merge-candidate plugin!

New Questions & Discussion Items

Items which have arisen through final ticket triage & review on which the team seeks feedback:

  1. Should the `filter` shim should be removed prior to merge? It is the majority position of the API team that `filter` be deprecated to dramatically improve the simplicity and consistency of API query functionality
  2. How should comments be handled for password-protected posts? Should the password be passed as a query parameter with the PUT/POST request, or is there a better option?
  3. Should the API match core’s logic when users with the `unfiltered_html` capability are creating or updating Posts or Comments?

Meeting Notes

At the weekly team meeting on October 10 the group reviewed open issues in the 2.0 milestone, which represents the candidate for our merge proposal shared last week.

Meeting attendees agreed to review open issues and pull requests individually, and to reconvene on Tuesday at 1500UTC to ensure all priority tickets had an owner.

At that meeting on October 11, the team reviewed the incoming feedback around the OAuth plugin (linked above). While the API team feels that having a built-in authentication solution provides a much-needed service, particularly to developers building mobile and desktop applications, the design and usability feedback we have received does indicate that the plugin needs more work.

OAuth’s place in the Merge Proposal

The API Team believes that the identified issues are resolvable, that the OAuth plugin is on track and that it should still be considered for merge in 4.7. However, after discussion within the team, input from @matt, and advice from @aaroncampbell and other core committers, we have edited our merge proposal to submit the Content API Endpoints and OAuth server as separate merge candidates. The API Team proposes both components for merge, but we submit the content endpoints for consideration independently of the OAuth1 server.

Content Endpoints Without OAuth

The Content API endpoints are stable, well-tested, and in wide production use across a variety of applications. Theme and plugin developers will benefit from having canonical, well-tested API endpoints in core, which may be used to query WordPress both from PHP code and from JavaScript applications running on the front-end or admin of WordPress. Sharing the endpoints for core data types enables increased consistency of what data is exposed and how it is persisted across different plugins, improving consistency and shortening development time by using . These themes and plugins have full read and write access to the API using the existing cookie & nonce authentication.

Mobile and desktop applications can leverage these same endpoints in a read-only capacity to create a variety of powerful reader-oriented applications and tools that expand the capability of what WordPress can do today, such as a unified reader for Make WordPress blogs and other experiments hypothesized by @jorbin.

Should OAuth 1 not be accepted for 4.7, secure write access for these external applications would still be only a plugin install away; and while having an OAuth server in core will provide a canonical approach for authenticating from remote applications, depending on the needs of a specific site or specific client application other authentication schemes may actually be preferable. Plugins exist for JWT Authentication and of course OAuth 2, and should OAuth 1 not be accepted for 4.7 these plugins may still be installed to enable an external application to opt-in to secure write access to your WordPress site.

In Summary

The API team submits for 4.7 merge consideration two enhancements to the REST API infrastructure: the Content API Endpoints for core WordPress datatypes, and an OAuth server which will reduce the setup time needed to securely interact with those endpoints from outside of WordPress. We believe these enhancements are each individually sufficiently tested and mature to meet the quality and security standards of WordPress Core, and each individually provides wide-reaching benefit to WordPress developers, and through them to the authors, readers & publishers of the web.

#4-7, #rest-api

REST API Team Update (4.7 week 7)

Our two (pt1) part (pt2) (split due to team travel) weekly meeting is now complete, and we have some new updates from the API team!

Team review & commentary is requested on “Open Questions on Settings” and “Open Questions on Meta”, below.

Settings

This Pull Request for adding Settings support to the API, resolving one of the few remaining major content gaps in the API, is almost ready to be merged into the rest-api feature plugin. Settings are exposed as keys on the resource `/wp/v2/settings`, e.g.:

{
    title: 'WordPress Develop',
    description: 'Just another WordPress site',
    url: 'http://src.wordpress-develop.dev',
    email: 'admin@local.dev',
    timezone: '',
    date_format: 'F j, Y',
    time_format: 'g:i a',
    start_of_week: 1,
    language: 'en_US',
    use_smilies: true,
    default_category: 1,
    default_post_format: '0',
    posts_per_page: 10
}

This resource is only accessible or modifiable if you are authenticated as a user with the `manage_options` capability.

Open Questions on Settings

Team feedback is requested on the below, and on the settings proposal in general.

  1. What do we do if a setting value in the database does not match the type used in `register_setting`?
    • This becomes a problem when there is serialized data in the `option` value. Right now we are forcing this to be a `null` value in the rest api responses. The problem with this approach is that sending that response data back to the settings endpoint will cause the deletion of that option value (as `null` in this case is essentially a delete operation).
  2. Having support for `object` recursion and `array` types in the Schema would be very nice, allowing settings that are either objects or arrays.
    • We propose that does not need to be resolved prior to merge, and may be added in a subsequent cycle.
  3. Currently, reading all options at `/wp/v2/settings` is behind a `manage_options` cap check: we need to work out if we want to publicly expose option values for some settings.
    • If we do this, we’d need a way in `register_setting` (or via some kind of `read_setting` meta cap) to work out which settings should be exposed publicly (or at specific permissions checks).
    • Site title and description are still available on the root `/wp-json/` resource index; several other settings here can be inferred from other responses or site content. We propose that this resource be left as requiring `manage_options` caps, and move establishing a mechanism in core to register a setting as public be handled in a subsequent development cycle.

Setting Registration

For your settings to be exposed in the REST API, you need to register them. WordPress includes a register_setting function which is not usually required to get/set options, but is required for API support.

To register your field, simply call register_setting and set the show_in_rest flag to true.

register_setting( 'group', 'option_name', array(
    'show_in_rest' => true,
));

register_setting Arguments

register_setting is part of WordPress core, but isn’t heavily used in most parts of core, so you may not have seen it before. Here are the options relevant to the REST API:

  • show_in_rest – Should the setting be exposed in the API? true for default behaviour, or an array of options to override defaults (see below).
  • type – This is the type of data the setting is. Can be `string`, `number`, or `boolean` (so far).
  • description – Human-readable description of what the field is for. The API exposes this in the schema.
  • sanitize_callback – Callback to sanitize the value before it’s stored in the database. See the "sanitize_{$object_type}_meta_{$meta_key}" filter.

REST API-Specific Arguments

The show_in_rest parameter defaults to false, but can be set to true to enable API support. Out of the box, the API makes some assumptions about your setting, but you can override these by setting show_in_rest to an options array instead:

register_setting( 'group', 'option_name', array(
    'show_in_rest' => array(
        'name' => 'fieldname'
    )
));

The options you can set are:

  • name – The key exposed via the REST API. For example, if your setting starts with an underscore, you may want to remove this for the API.
  • default – The default value for the setting via the REST API if it’s not set in the database.
  • schema – The JSON Schema options for the field. Exposed via OPTIONS requests to the post. This is generated automatically based on other parameters, but can be specified manually if you’d like to override it.

Settings Defaults

We’d also love feedback on ticket #38176, this adds the ability to specify a default value for an option via `register_setting()`. As you can see from the above, we are currently defining the `default` value within
the `show_in_rest` options, as there’s no way to do this in the Settings API. We’d like for there to be a way to do this in core, so `get_option()` could default to the value set in `register_setting( … [ default => ‘myvalue’ ])`
when no default is specific in `get_option`.

Meta

Meta is also ready to be merged into the plugin, and also could use review from the broader WordPress team. Similar to custom post types and taxonomies, and as discussed previously, only meta that opts-in to API support is available through the REST API.

Open Questions on Meta

  1. Right now, all meta registered through the mechanism below is assumed to be visible wherever its parent object appears. Meta values registered to a public post will be visible on that post without authentication.
    • Is it acceptable to expose registered meta fields on objects in this way, where the meta visibility will match the visibility of the parent object?
  2. As described in this comment, when you have multiple of the same value stored in a meta key, it will delete all matching values. For example, if you had a meta value [ 11, 13, 13, 13, 13 ] and attempted to remove one 13, it would result in the value [ 11 ].
  • Should multiple instances of the same value be able to be stored in a `single => false` meta field?

Meta Registration

For your meta fields to be exposed in the REST API, you need to register them. WordPress includes a register_meta function which is not usually required to get/set fields, but is required for API support.

To register your field, simply call register_meta and set the show_in_rest flag to true.

register_meta( 'post', 'field_name', array(
    'show_in_rest' => true,
));

Note: register_meta must be called separately for each meta key.

register_meta Arguments

register_meta is part of WordPress core, but isn’t heavily used in most parts of core, so you may not have seen it before. Here are the options relevant to the REST API:

  • show_in_rest – Should the field be exposed in the API? true for default behaviour, or an array of options to override defaults (see below).
  • description – Human-readable description of what the field is for. The API exposes this in the schema.
  • single – Is this field singular? WordPress allows multiple values per key on a single object, but the API needs to know whether you use this functionality.
  • sanitize_callback – Callback to sanitize the value before it’s stored in the database. See the "sanitize_{$object_type}_meta_{$meta_key}" filter.
  • auth_callback – Callback to determine whether the user can write to the field. See the "auth_post_{$post_type}_meta_{$meta_key}" filter.

REST API-Specific Arguments

The show_in_rest parameter defaults to false, but can be set to true to enable API support. Out of the box, the API makes some assumptions about your meta field, but you can override these by setting show_in_rest to an options array instead:

register_meta( 'post', 'field_name', array(
    'show_in_rest' => array(
        'name' => 'fieldname'
    )
));

The options you can set are:

  • name – The key exposed via the REST API. For example, if your meta key starts with an underscore, you may want to remove this for the API.
  • schema – The JSON Schema options for the field. Exposed via OPTIONS requests to the post. This is generated automatically based on other parameters, but can be specified manually if you’d like to override it.

Using Meta over the API

Meta is added to existing resources as a new meta field. For example, to get the meta for a single post, fetch the post at /wp/v2/posts/{id}, then look in the meta field. This is a key-value map for the registered fields.

To update a field, simply send back a new value for it.

Setting a field to null will cause the value to be removed from the database. If a field has a default value set, this essentially “resets” the value to the default (although no default will be stored in the database).

For meta fields which can hold multiple values, this field will be a JSON list. Adding or removing items from the list will create or remove the corresponding values in the database.

Other Updates

  • Add a {taxonomy}_exclude query parameter to query for resources NOT IN a particular taxonomy term_id
  • Decide against supporting full tax_query functionality via filter[tax_query]
  • Defer supporting auto-draft to a future cycle, as it did not have the expected benefits and the applicability of that state to the API is not generally understood

#4-7, #rest-api

REST API Meeting Agenda for Oct 3 2016

REST API: Meeting Agenda for Oct 3, 2016

The weekly meeting for the API team will be on Monday at 2016-10-03 14:00 UTC in #core-restapi. On the agenda:

  • Update on Settings from @joehoyle
  • Update on Meta from @rmccue
  • 4.7 Merge Plan TODOs
    • Decision on supported authentication methods
    • Progress on outside review
    • Timeboxed open floor
  • Review open PRs
  • Find owner for open tickets, e.g. some OAuth/Cookie conflicts

#4-7, #rest-api

API Team Update

API Team Update Sept. 27 2016

The API team met yesterday on slack for our weekly update, which this week was predominately focused on follow-up from last Thursday’s comprehensive GitHub issues scrub.

Since the previous API team update the group has been making steady progress through the gameplan outlined in that post. To give a quick update on a few key “quirky” issues that had been identified:

Password-Protected Posts

As noted in last week’s dev chat, password-protected posts will be included in collections with their content set to '', and the content can be viewed by passing ?password=XXXXX as a query or GET parameter when querying for a specific post. Query parameters are not an ideal solution: Authorization headers are out because you can’t have multiple authorization schemes in one request; cookies don’t afford enough control to browser clients, and custom headers aren’t respected by caches. See GH issue #2701 for more background, and check out the open pull request to review the specifics of the implementation.

Sticky Posts

After much debate there is now a path forward for handling sticky posts as well. Following this open pull request, sticky posts are included in the /wp/v2/posts collection, but are not given special treatment in terms of ordering—a sticky post will, by default, be displayed ordered by date or whatever orderby has been set for the request. The parameter ?sticky=true may be passed to return only sticky posts; ?sticky=false may be passed to exclude sticky posts from the response.

There is ongoing discussion around how the API could surface posts in the “normal loop order,” with stickies on top, followed by non-sticky posts. @jorbin will propose a follow-up enhancement that could be added in to the API in a later cycle. See GH issue #2210 and associated slack discussion for more commentary.

Meta Support

A pull request is open on the API meta endpoints repository to add support for registering meta with the API using the main register_meta function. This PR includes a comprehensive readme explaining how meta handling works. Once merged & reviewed, this functionality will be PR’d against the primary REST API plugin repository.

Site Settings (Options) Support

A pull request has merged into the API site endpoints repository which adds support for accessing and manipulating site settings through the API. These endpoints will be PR’d against the core REST API plugin repository this week.

Discussion items for 9/28 dev chat

This project is not quite done yet, and in tomorrow’s dev chat the API team will be looking to the broader WordPress team for input on three priority issues that need a decision:

Upcoming Meetings

Please join us in #core-restapi on chat.wordpress.org for our next two group meetings:

#4-7, #rest-api, #team-update