Completing the implementation of metadata registration with the REST API

A priority of the REST API team for the upcoming 5.0 release is to finally enhance WordPress metadata registration so that it can cover common API use-cases. This will help with adaptation of the REST API by making management of custom metadata in existing content endpoints easier, which should in turn also benefit any metadata-related work in Gutenberg. This post provides some insight into the surrounding issues, shows the current progress of the improvements, and highlights what still needs to be discussed.

A little history on register_meta()

The register_meta() function has been around since 3.3, but did not attract much attention with developers until WordPress 4.6 introduced enhancements to permit metadata registration for use within the REST API. As with other metadata functions, an object type (by core definitions either a “post”, “term”, “comment”, or “user”) needs to be passed to the function, alongside the meta key to register and the arguments for its behavior.

However, metadata in WordPress is often used only for a subset of one of the above four types. When dealing with post meta in a plugin for example, that post metadata commonly only applies to one or more specific post types – not to all posts of any post type. That is what currently makes register_meta( $object_type, $meta_key, $args ) unusable for the majority of REST API-related cases. Before the enhanced version was introduced in 4.6 many thoughts were put into this concern, with a subtype-aware implementation even being merged into core. That implementation was reverted before the release because it was unclear how to handle conflicts (more on that below). The ticket where all those changes were discussed is #35658.

Misconceptions about register_meta()

The improvements and their documentation caused developers to listen up, but unfortunately since the concept of an object type is rarely exposed in high-level WordPress APIs, many people misunderstood the first parameter as requiring a post type because post meta is the most common kind of metadata used by plugin developers. Some developers started using the function to register post type-specific meta keys without that function actually supporting that, as shown in this example:

Note:

This does not do what you may think it does:

register_meta( 'book', 'isbn', array( ... ) );

The code above will not register an ISBN number meta key for a “book” post type. It will register that key for a custom object type “book”—which probably doesn’t exist, unless a whole custom meta database table, a custom object database table, and surrounding setup has been implemented.

Those function calls fortunately don’t cause any harm in most cases since WordPress only contains four object types that use metadata (ignoring multisite). It becomes problematic though if the function is used to register metadata for the “post” post type.

Note:

This also does not do what you may think it does:

register_meta( 'post', 'some_meta_key', array( ... ) );

Since the “postpost type shares a name with the “postobject type, and since, again, the first parameter of register_meta() is for object type and not for a post type, register_meta() will register that meta key and arguments passed for all posts of any subtype (post type).

If you’re reading this and have used register_meta() in a way where you assumed it could register metadata for a single post type or taxonomy, at present we recommend you switch to using register_rest_field(), which is currently the only (however a little more involved) way to achieve this.

Adding subtype support for metadata behavior

A follow-up ticket #38323 aims to finally implement this missing functionality to register_meta() so that there’s an easy-to-use API to handle subtype-specific metadata in the REST API. It picks up the approach used before, where an additional object_subtype argument in the $args parameter can be used to specify the subtype the meta key should be registered with. Picking up the aforementioned example, you could register the ISBN for a “book” post type via the following code:

register_meta( 'post', 'isbn', array(
	'object_subtype' => 'book',
	...
) );

Open questions regarding subtype handling

Implementing meta registration for subtypes and modifying existing behavior to account for it is rather straightforward, and there is a patch available to experiment with the current approach. There are a couple things that need to be discussed though:

  • How should we deal with or prevent conflicts between meta keys of the same name and object type? The method currently allows to register metadata for an entire object type of course, so we cannot simply change that now and prevent those calls entirely. It appears we have four alternatives here:
    1. Allow meta keys of the same name and object type to be registered in any way. In case a meta key is registered for the current subtype, but also more generally for the object type, the behavior registered for the subtype takes precedence. Meta keys are internally stored nested under their object type and object subtype ($wp_meta_keys[ $object_type ][ $object_subtype ][ $meta_key ]). This is how the current patch on the ticket works, mainly because it is the most straightforward and flexible approach (it was also how the original code from #35658 worked). However it doesn’t deal with conflicts at all, so it is likely not preferable. An issue would be that on every update of a value, sanitization would happen for both behaviors, easily resulting in invalid data and unexpected errors. Throwing notices would be an option – but it would only help preventing conflicts, not actually prevent them.
    2. Allow meta keys of the same name and object type to be registered either per subtype OR for the entire object type. It would be based on whoever comes first takes priority. Meta keys are internally stored nested under their object type and object subtype ($wp_meta_keys[ $object_type ][ $object_subtype ][ $meta_key ]). If someone registers a meta key for an object type after someone else already registered the same meta key for a subtype of that object type, the process should be rejected and fail. If someone registers a meta key for an entire object type without the same meta key already being registered for a subtype of that object type, afterwards all meta keys for a subtype of that object type would be rejected. This prevents all possible conflicts since meta keys will only exist once per object type/object subtype combination, but still provides the flexibility of having multiple meta keys of the same name for different object subtypes. The implementation to prevent the conflicts during meta registration would be a little more involved though.
    3. Allow meta keys of the same name and object type to be registered only once in total. It would be based on whoever comes first takes priority. Meta keys are internally stored nested under their object type only, without an additional level for their object subtype ($wp_meta_keys[ $object_type ][ $meta_key ]). Whoever comes first, can register their meta key, every following request to add the same meta key for the same object type afterwards will be rejected. This prevents all possible conflicts since the data structure alone makes it impossible for two meta keys of the same name and object type to be registered. The approach generally appears to not be flexible enough: For example if two different post types were using a meta key of the same name, they wouldn’t be able to co-exist. However, based on the assumption that developers prefix all their metadata (which is probably not taught often enough), this option could very well work and be a straightforward solution.
    4. Allow meta keys of the same name and object type to be registered in any way, but use the behavior registered for the entire object type as a fallback only. In other words, if the same meta key exists with a subtype, only run that logic – for all subtypes where there is no specific behavior available, fall back to the behavior registered on the entire object type. To be precise, the difference between this approach and the first one is that in the first one the behavior (like sanitization and auth) would be executed for both, while here it only happens for the object subtype or the object type when no subtype-specific handling is registered. This is a conflict-free approach that at the same time doesn’t require any restrictions where calls to register_meta() would need to be rejected. The idea of using meta keys registered for an entire object type as fallback is slightly different from the other variants, but it seems to work well. It should still be highlighted that meta keys should be prefixed, but if that is a given, this implementation would even allow an individual plugin to provide more flexible logic for its metadata that may be needed for all post types (general behavior as fallback, subtype-specific behavior where needed).
  • Should we introduce easy-to-use wrapper functions like register_post_meta( $post_type, $meta_key, $args ) and register_term_meta( $taxonomy, $meta_key, $args ) which wrap register_meta() appropriately? The reason those might be useful is that the concept of object types and object subtypes is not commonly exposed in core’s documentation & high-level APIs (think about *_post_meta(), *_term_meta(), *_comment_meta() and *_user_meta() all wrapping the respective lower-level *_metadata() function). By adding these methods we would then recommend to only use those, and never recommend the use of register_meta() itself (as registering metadata for an entire object type is probably not desired). The only downside would be that comments and users don’t really have subtypes, so those two functions would either have a redundant first parameter or not have that parameter at all (which wouldn’t be future-proof).
  • A more specific issue: Should a comment’s comment_type value be considered a subtype? While one could argue that they are subtypes, they are treated entirely differently and do not really define how that comment works or how it can be edited. There’s also no flexible API around it to register additional comment types, and in the REST API there are no individual controllers per comment type. Currently, the patch ignores comment types, but this is still open for discussion.

Upcoming Meeting

The above items will be discussed in next week’s upcoming REST API meeting which will take place at the regular meeting time, Thursday, May 3rd, 2018 at 17:00 UTC in the #core-restapi channel. If you’re interested, please attend this meeting so that we can move forward with this ticket, as it should be included in WordPress 5.0. Also feel free to review the latest patch on #38323 and leave a comment on the ticket. While it is unlikely that this is the final implementation, a lot of it would not need to be adjusted regardless of the outcome for the above questions.

Extra credit goes to @kadamwhite for proofreading and several improvements.

#agenda, #rest-api, #team-update

API Team Update

API Team Update Sept. 27 2016

The API team met yesterday on slack for our weekly update, which this week was predominately focused on follow-up from last Thursday’s comprehensive GitHub issues scrub.

Since the previous API team update the group has been making steady progress through the gameplan outlined in that post. To give a quick update on a few key “quirky” issues that had been identified:

Password-Protected Posts

As noted in last week’s dev chat, password-protected posts will be included in collections with their content set to '', and the content can be viewed by passing ?password=XXXXX as a query or GET parameter when querying for a specific post. Query parameters are not an ideal solution: Authorization headers are out because you can’t have multiple authorization schemes in one request; cookies don’t afford enough control to browser clients, and custom headers aren’t respected by caches. See GH issue #2701 for more background, and check out the open pull request to review the specifics of the implementation.

Sticky Posts

After much debate there is now a path forward for handling sticky posts as well. Following this open pull request, sticky posts are included in the /wp/v2/posts collection, but are not given special treatment in terms of ordering—a sticky post will, by default, be displayed ordered by date or whatever orderby has been set for the request. The parameter ?sticky=true may be passed to return only sticky posts; ?sticky=false may be passed to exclude sticky posts from the response.

There is ongoing discussion around how the API could surface posts in the “normal loop order,” with stickies on top, followed by non-sticky posts. @jorbin will propose a follow-up enhancement that could be added in to the API in a later cycle. See GH issue #2210 and associated slack discussion for more commentary.

Meta Support

A pull request is open on the API meta endpoints repository to add support for registering meta with the API using the main register_meta function. This PR includes a comprehensive readme explaining how meta handling works. Once merged & reviewed, this functionality will be PR’d against the primary REST API plugin repository.

Site Settings (Options) Support

A pull request has merged into the API site endpoints repository which adds support for accessing and manipulating site settings through the API. These endpoints will be PR’d against the core REST API plugin repository this week.

Discussion items for 9/28 dev chat

This project is not quite done yet, and in tomorrow’s dev chat the API team will be looking to the broader WordPress team for input on three priority issues that need a decision:

Upcoming Meetings

Please join us in #core-restapi on chat.wordpress.org for our next two group meetings:

#4-7, #rest-api, #team-update

Metadata API project reborn: The new Fields API project

Many of you will remember that a couple of years ago, @ericlewis and a group of us set out to start a project to make sense of all of the different APIs that arose from third party plugins and themes to build custom field solutions on top of WordPress. That project was nicknamed “Metamorphosis” and subsequently went by the “Metadata UI/API” project. As plugin authors dealing with and using custom fields and content types, we had been tackling the issues of developing for multiple object types in WordPress for years prior to getting together.

Now, @helen and I are pleased to (re)introduce to you what we’re hoping to be the fruits of all of this labor: Meet the new Fields API project.

function fields_api_example_post_field_register( $wp_fields ) {

   // 1. Define a new section (meta box) for fields to appear in
   $wp_fields->add_section( 'post_type', 'my_cpt', 'my_meta_box',
      array(
         // Visible title of section
         'title'       => __( 'My Meta Box', 'mytheme' ),

         // Determines what order this appears in
         'priority'    => 'high',

         // Determines what order this appears in
         'context'     => 'side',

         // Capability needed to tweak
         'capability'  => 'my_custom_capability'
      )
   );

   // 2. Register new fields
   $wp_fields->add_field( 'post_type', 'my_cpt', 'my_custom_field',
      array(
         // Default field/value to save
         'default'    => 'All about that post',

         // Optional. Special permissions for accessing this field.
         'capability' => 'my_custom_capability',

         // Optional. Add an associated control (otherwise it won't show up in the UI)
         'control' => array(
            // Don't set 'id' and it will automatically be generated for you as
            'id' => 'mysite_my_custom_field',

            // Admin-visible name of the control
            'label'   => __( 'My Custom Field', 'mytheme' ),

            // ID of the section this control should render in (can be one of yours, or a WordPress default section)
            'section' => 'my_meta_box',

            // Control type
            'type'    => 'text'
         )
      )
   );

}
add_action( 'fields_register', 'fields_api_example_post_field_register' );

(Please note: The code above may not be the same as the final implementation, it will especially change over the next few weeks)

First of all, we split off the UI side of the project to focus solely on the API. This way, we’re not held up by anything as we move towards implementation inside of core. We can tackle each UI separately and that helps us keep the project nimble and on track.

The goals of this new API proposal will be to be implemented across WordPress objects as an API to register fields and sections for. We aim to cover these objects (not necessarily in this order):

  • Customizer (retrofitting it beneath the existing Customizer API)
  • User profile screen
  • Post editor
  • Settings screens (retrofitting it beneath the existing Settings API)
  • And other areas in the future (Comment editor, Network Settings screens [see #15691], Media modals, etc)

This API offers a great deal of standardization across all of WordPress and code reusability. It’s paramount for third-party developers, who will be able to utilize these structures in their own plugins and themes, build apps and give access to other developers so they can contextually see what a site really contains. I see the biggest use-case being able to expose the entirety of WordPress to the REST API and giving apps like the WordPress iOS app access to meta boxes and custom fields to display real editor screens with field inputs based on the control types natively.

Areas we need help currently:

  • Examples and use cases for Customizer, User profile screen, Post editor, and Settings screens (submit issues in GitHub with your ideas, coordinate with us to refine them if you want)
  • Unit tests for core
  • Testing Customizer implementation, you’ll find the core files to replace under the /implementations/ folder in the repo
  • Brainstorming implementation classes for other object types to implement in the UI, starting with User profile screen

If you’re interested in joining our merry crew which includes @helen and a few of us, hop into Slack #core-fields and join our weekly meetings every Monday (Monday 20:00 UTC 2015). You can check out the Fields API and submit PRs on GitHub, or ping us in #core-fields throughout the week with questions or concerns.

#customize, #fields-api, #options-meta, #team-update

Autosave and Post Locking Update

Today we had our first scheduled meeting.

Consensus was, for a first pass:

  • Display a simple lock/padlock icon next to locked posts within the post list screen, leaving the “Request Lock” button for the post edit screens.
  • When a post is locked, hide the Quick Edit link, and disable batch edit for the post
  • To unlock a post, the user will enter the post edit screen, then click a button to request a lock. This will trigger a request to appear for the user with the current lock, where they can decide whether or not to allow the new lock.

As far as development tasks go, @azaozz is planning on finishing a first run at the “Heartbeat” API to be committed before the end of the week. I’ll begin working on the list table changes for post listings. We’re currently looking for volunteers to aid, as there’s plenty of room for help! Additional tasks will include the UI for handling post lock requests and auto-save to local storage.

We’ll certainly chat more about this tomorrow in dev chat, and are planning another meeting this Friday (2013–01–25), at 21:00 UTC (the same time as dev chat), unless this doesn’t work for many of you looking to help. Please post here, or come to the chats if you’re interested in helping with this project, and we’ll get in touch with you for further details!

#3-6, #autosave-and-post-locking, #team-update

Team Update XML RPC on behalf of westi…

Team Update: XML-RPC (on behalf of westi)

In our second cycle, the primary focus was CRUD methods for taxonomies. The work was tracked in #18438, and covers both the core implementation and the unit tests.

We also spent more time refining the cycle 1 work (CRUD for all post types), tracked in #18429. This included fixing some date bugs (related to #15098), adding support for getting/setting post thumbnails (related to #15098), and writing unit tests.

After using the posts methods from cycle 1, it became clear that the post type information methods from #18436 were highly desirable. The old patches on that ticket were updated and aligned with the rest of the XML-RPC work for this release.

westi was not available for much of this cycle, so Marko and I worked on our own; he is now reviewing our final patches and will comment or commit shortly. In the meantime, I’m finishing up work on XML-RPC documentation for the codex, which should be ready by beta. Marko and I will also continue to expand the XML-RPC unit tests suite (now around 75 tests).

#team-update, #xml-rpc

Team Update: Browsing Buddies

@getsource and I did some in-person hacking at WordCamp Phoenix over the weekend. New patch on #19816 for multiple screenshots that seems good to go in as a first pass after a JS sanity check and probably an update to using .data() instead of .attr() tomorrow before dev chat. Will definitely need some UI/UX once that’s in, and will also likely need a little update for whatever the .org API response will be.

There is also a new patch on #19815 that adds a class member variable and removes a global (yay!) and passes data via JSON instead of parseQuery. Eyes also welcome on that.

#browsing-buddies, #team-update

We’re digging in full bore to get things…

We’re digging in full bore to get things done by freeze. #19978 is the primary ticket.

Tasks that need to happen by Wed Feb 29:

  • Finish the styling (Drew)
  • CSS file cleanup (Lance)
  • RTL stylesheet
  • Editor stylesheet

The last big missing piece for styling is post formats.

Update Wed Feb 29: see my comment below.

#bundled-theme, #team-update, #twentytwelve

Team Gandalf Update

Last week marked the end of our second cycle and the beginning of our sprint to get things done. We began by doing just that: last Friday, we merged the theme customizer into core. There’s still quite a bit to do and polish, so please keep that in mind. Development is being trac(k)ed at #19910.

In the past week, we’ve introduced better JS APIs, working preview navigation (you can click URLs and it works, hooray), namespaced attributes (which will allow us to enable form submission within previews), and more. We also uncovered and fixed a fairly insane bug that stemmed from a low-level bug in Opera’s JS engine. Good times.

Moving forward, the main goals are to integrate more setting types (headers, widgets), improve the refreshing process, polish the UI, and remove any vestigial plugin cruft. Dominik’s time will be limited over the next few weeks due to exams, so if any of this sounds exciting to you, please let me know.

#customize, #gandalf, #team-update

Team Update Bugs RPC Unfortunately nothing happened for…

Team Update: Bugs-RPC

Unfortunately nothing happened for us this cycle. Since I will be out for awhile starting in a few days it has been proposed that the two XML-RPC teams be merged together.

#team-update

Team Update: Browsing Buddies

@getsource (DH-Shredder) spent some more time over the past week working with @dkoopersmith refining the infinite scroll JS for #19815, which was committed earlier today. We briefly discussed having more results/pagination for some of the other theme-install.php tabs with @nacin (requires changes on the API end), as well as the recurring thought that perhaps the featured tab should show first. Therefore, the other patch that restricted the JS enqueue to the theme-install.php search tab only was not committed for the time being.

After reviewing some comps with @jane, we started on the display of multiple screenshots. An initial rough patch will be up on #19816 soon. We still need to hash out the details of retrieving multiple screenshots, both in get_themes() and from the .org API, and how those images will be added to the extended details div without displaying in no JS, as discussed when first scoping the feature. We also need to take into consideration what happens when the window is resized. Provided that we can get that sorted out tomorrow, it looks like we’re on target for the cycle.

I wrote the Theme Review Team and gave them an update on the anticipated screenshot sizes, which will remain at 300×225 (4:3), constrained by CSS. Gandalf functions as the large screenshot 🙂 The screenshots in the list table view will be enlarged to match, as there is more space now that details are hidden by default.

#browsing-buddies, #customize, #team-update