Brian Krogsgard 4:37 pm on October 19, 2016
Tags: 4.7 ( 124 ), rest-api ( 107 )

WordPress REST API Success Metrics

This post is part of the conditional merge of the WordPress REST APIREST API The REST API is an acronym for the RESTful Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. It is how the front end of an application (think “phone app” or “website”) can communicate with the data store (think “database” or “file system”) https://developer.wordpress.org/rest-api/..

Tracking and measuring specific success metrics is a priority for the WordPress REST API Content Endpoints project. The aim is to be able to look back at the implementation of the feature project to see how it went after the fact, and to utilize that information to inform future decision making.

A submission form will be created to help track implementations, in addition to automated scripting and analysis. Feedback will be grouped based on which metric it applies to. Developers and implementers will be polled to determine usage, where automation is a challenge.

It is very important to note prior to analyzing individual goals, that these goals and the achievement of them does not automatically indicate success, or that not achieving them means a lack of success. For instance, one metric that’s very hard to capture in goals like those that follow is the impact of a singular implementation. One highly impactful implementation could indicate a great deal of success, but would not be fully captured by the following metrics. These goals are established to help identify success metrics that would offer clear insight on the project, but they are not all encompassing or definitive alone.

Metric #1: PluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party utilization

Per the “Popular” plugins page of the WordPress Plugin Directory, the following can be determined:

21 plugins have 1m+ active installs
126 plugins have 200,000+ active installs
232 plugins have 100,000+ active installs

Many of these are not good candidates for utilizing the REST API, but there are a number of plugins that are.

The following goals are proposed:

15% utilization of plugins with more than 1 million installs.
7% utilization of plugins with more than 200,000 installs and fewer than 1 million.
3% utilization of plugins with more than 100,000 installs and fewer than 200,000.

Additionally, it is proposed to seek 5-10 examples of non-directory or otherwise prominent plugins, that may not fit the above metrics, to showcase their utilization of the APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways., and how it has benefited them. This may include commercial plugins.

Timeline for analysis: three major releases after merge (5.0 release day).

Metric #2: Distributed theme utilization

Popular themes are a bit of a misnomer in the WordPress landscape. The default themes or themes that have been around for a very long time account for a significant percentage of all active themes. And feature development isn’t as common for existing themes as it is for existing plugins.

In addition to polling free theme makers, top theme shops (according to Alexa) should be polled to determine usage of the API for theming purposes.

Additionally, coordination can occur with the Theme Review Team to identify usage of the API in newly submitted themes, to determine adoption.

The API could be useful in themes themselves, as well as complimentary plugins, which are not likely to be picked up by the plugin analysis. However, it’s difficult to know exactly how WordPress themes may use the content endpoints in a distributed theme context. It could be part of efforts within the customizerCustomizer Tool built into WordPress core that hooks into most modern themes. You can use it to preview and modify many of your site’s appearance settings., page building, custom widgets, etc.

A goal of 2% adoption of the API in new themes is proposed, determined within three major releases, and analyzed amongst distinct ecosystems:

New WordPress.orgWordPress.org The community site where WordPress code is created and shared by the users. This is where you can download the source code for WordPress core, plugins and themes as well as the central location for community conversations and organization. https://wordpress.org/ theme review submissions
Third party theme shops (A shop using the API in any capacity, likely new releases, would count as “yes”, rather than theme by theme)
Analysis of page builder or helper plugins
Distinct theme communities with “builder”, “framework”, or “starter” themes.

Metric #3: Custom development implementations

The WordPress REST API is immediately useful for custom development projects, where the developer has control over the environment. The REST API in coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. provides a vote of confidence to outside developers that they are building with an API that has the full backward compatibility and stability expected of features in the core project.

It is proposed to identify 50+ projects in production or under development utilizing the Content Endpoints, within two major releasemajor release A release, identified by the first two numbers (3.6), which is the focus of a full release cycle and feature development. WordPress uses decimaling count for major release versions, so 2.8, 2.9, 3.0, and 3.1 are sequential and comparable in scope. cycles.

The likelihood is that there will be far more consumption-only applications, versus those that manipulate content. Within the same two release timeframe, the goal is to identify at least 12 custom projects that utilize the Content Endpoints beyond a read-only use case.

Metric #4: Core development and feature projects

The WordPress REST API can aid core development and feature projects. Items like Quick Draft, Press This and a menu bar revisions browser are test cases for using the API in a core context with WordPress 4.7.

Utilization of the API for additional core features and feature projects will be tracked over the next three major release cycles. It is proposed to identify 6 core features or patches than can utilize the API, as well as 1 feature project, within three major releases.

#4-7, #rest-api

Boone Gorges 6:06 pm on October 19, 2016
Many thanks to the APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. team for work on this excellent document.

A couple of questions and comments:
- What counts as “utilization” in a theme or pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party? Will you be grepping for a certain string (‘wp-jsonJSON JSON, or JavaScript Object Notation, is a minimal, readable format for structuring data. It is used primarily to transmit data between a server and web application, as an alternative to XML.’?) in the plugin source? Relying on self-reporting of some sort?
- You propose to “identify” custom projects using the endpoints. Will these projects exclude those that are using a publicly-available plugin/theme to integrate with the API (since those projects would be counted as part of Metric #1 or #2)? Are we just collecting project names, or is the plan to get more significant details about the project? How will the data be collected? (Details aren’t necessary – just want a gesture in the direction of the mechanics.)
- Brian Krogsgard 6:21 pm on October 19, 2016
  
  @boonebgorges Good questions.
  
  What counts as “utilization” in a theme or pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party? Will you be grepping for a certain string (‘wp-jsonJSON JSON, or JavaScript Object Notation, is a minimal, readable format for structuring data. It is used primarily to transmit data between a server and web application, as an alternative to XML.’?) in the plugin source? Relying on self-reporting of some sort?
  
  We plan to scan those top few hundred plugins for “wp-json” or similar, as well as allow for self reporting through a form and potential surveys. I have several ideas on the self-reporting front that I didn’t get into very much with this post.
  
  Will these projects exclude those that are using a publicly-available plugin/theme to integrate with the APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. (since those projects would be counted as part of Metric #1 or #2)? Are we just collecting project names, or is the plan to get more significant details about the project?
  
  The way I imagine custom projects is someone is using the content endpoints within a non-distributed codebase. So no, it wouldn’t include someone using a plugin that has the endpoints — as you mention, those would be counted in #1 and #2.
  
  There’s been a floating Google Doc with information on plugin usage that’s a good start, but I want to make it easier and a nicer display than that.
  
  How will the data be collected?
  
  Self reporting (forms and surveys) mostly. We also discussed working with hosts to determine API calls but just theoretically so far.
  - Aram Zucker-Scharff 6:45 pm on October 19, 2016
    
    Just going to note here, I have a pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party that builds extensively on the WP-APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. and it doesn’t use the string “wp-jsonJSON JSON, or JavaScript Object Notation, is a minimal, readable format for structuring data. It is used primarily to transmit data between a server and web application, as an alternative to XML.” even once. Why not scan for “rest_api_init” as well, as that seems a more likely indicator of a plugin having extended the API?
    - Brian Krogsgard 7:03 pm on October 19, 2016
      
      Just a sample string to search for 🙂 Smarter people than me will help determine the appropriate scans.
Mike Nelson 6:19 pm on October 19, 2016

How were the numbers for these goals determined? Eg why not “50% utilization of plugins with more than 1 million installs.” or “1% utilization of plugins with more than 1 million installs.”?

Might it be useful to compare utilization of the REST APIREST API The REST API is an acronym for the RESTful Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. It is how the front end of an application (think “phone app” or “website”) can communicate with the data store (think “database” or “file system”) https://developer.wordpress.org/rest-api/. to other features/APIs of WP? Eg what is the adoption of the XML RPC? Usage of adminadmin (and super admin)-ajax? Usage of the settings APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways., or filesystem API, etc?

I think having a ballpark for usage of these other systems might help determine the usage goals for the REST API.
- Brian Krogsgard 6:28 pm on October 19, 2016
  
  How were the numbers for these goals determined? Eg why not “50% utilization of plugins with more than 1 million installs.” or “1% utilization of plugins with more than 1 million installs.”?
  
  I know your numbers are for example only, but to answer nonetheless…. Many of the 1M+ plugins are non-starters for the REST APIREST API The REST API is an acronym for the RESTful Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. It is how the front end of an application (think “phone app” or “website”) can communicate with the data store (think “database” or “file system”) https://developer.wordpress.org/rest-api/., so 50% of 1mm+ is actually much higher (maybe close to 100%) when you consider which would be candidates to utilize it anyway. 1% would be a non-integer less than 1 🙂
  
  The numbers are based on looking at the plugins and using our best judgement. It’s not all-encompassing, and while I would love to work with more data, these numbers have a degree of “best guesses” applied, based on collective experience in the WordPress world and familiarity with the APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. as well as pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party landscape.
  
  The reason we kept it limited to the plugin groups we did was so that we’d have a more targeted sample that’s more definitive and measurable than all plugins anywhere. Hence, that’s why we have the additional goal of 5-10 specific cases outside that plugin sample.
  
  As noted at the top of the post, we hope these numbers can help offer insight, but they are not an all-encompassing measurement of the APIs success, and one area they can miss is impact of a single use case.
  
  We don’t know exactly how the API will be used yet, but we think these goals set some realistic, but in some ways ambitious, goals that we can measure against, so that in a year or so we can look back and compare what we thought might happen versus what actually happened.
  
  And while this may be new for WordPress with this project, it could provide an interesting model for other feature projects as well.
  - Mike Nelson 12:10 am on October 20, 2016
    
    > Many of the 1M+ plugins are non-starters for the REST APIREST API The REST API is an acronym for the RESTful Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. It is how the front end of an application (think “phone app” or “website”) can communicate with the data store (think “database” or “file system”) https://developer.wordpress.org/rest-api/.
    
    meaning there’s really no obvious way for them to use it right? It might be good to have it recorded what percentage of plugins/themes etc are non-starters for REST API usage (eg so we can have a guess at the upper limit of how many will use it).
    in a year or so we can look back and compare what we thought might happen versus what actually happened.
    
    Ok so these numbers are “what we think adoption will be, if it’s ‘successful'”-ish.
    And while this may be new for WordPress with this project, it could provide an interesting model for other feature projects as well.
    
    Yeah I think certainly having some estimates of how to measure success is good, even though it’s obviously really hard to boil down success into numbers
Mike Nelson 12:14 am on October 20, 2016

Another usage that could be measured would be non-WordPress projects using the REST APIREST API The REST API is an acronym for the RESTful Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. It is how the front end of an application (think “phone app” or “website”) can communicate with the data store (think “database” or “file system”) https://developer.wordpress.org/rest-api/.. Eg a Django module that works with the REST API, or an iOSiOS The operating system used on iPhones and iPads. app, or zapier integration, etc.

This might fit under “Metric #3: Custom development implementations”; and admittedly might be difficult to track considering these developers are probably not terribly involved in the WP community.
Darren Ethier (nerrad) 5:14 pm on October 20, 2016

This might be something you’ve already considered as a part of your self-reporting. But it’d be nice if there was a dedicated place where anyone could post about something they’ve come across using the REST APIREST API The REST API is an acronym for the RESTful Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. It is how the front end of an application (think “phone app” or “website”) can communicate with the data store (think “database” or “file system”) https://developer.wordpress.org/rest-api/.. That way we could enlist (through various avenues of promotion) the WordPress community to document places where they see the REST API being used.

It’s likely that there will be developers outside the WP community who might build something using the REST API that would have no clue that the community is wanting to know about projects using it, but someone who IS aware of this and sees this developer’s work could report it.