Make WordPress Core

Updates from Ryan McCue Toggle Comment Threads | Keyboard Shortcuts

  • Ryan McCue 4:05 am on October 8, 2016 Permalink |
    Tags: , , merge-proposals,   

    REST API Merge Proposal, Part 2: Content API 

    Hi everyone, it’s your friendly REST API team here with our second merge proposal for WordPress core. (WordPress 4.4 included the REST API Infrastructure, if you’d like to check out our previous merge proposal.) Even if you’re familiar with the REST API right now, we’ve made some changes to how the project is organised, so it’s worth reading everything here.

    (If you haven’t done so already, now would be a great time to install the REST API and OAuth plugins from WordPress.org.)

    A brief history of the REST API

    The REST API was created as a proof-of-concept by Ryan McCue (hey, that’s me!) at the WordPress Contributor Summit in 2012, but the project kicked off during the 2013 Google Summer of Code. The end result was Version 1.0, which grew into a community supported initiative that saw adoption and provided for a solid learning platform. The team used Version 1 to test out the fundamental ideas behind the API, and then iterated with Version 2, which made some major breaking changes, including explicit versioning, the introduction of namespacing for forwards compatibility, and a restructure of the internals. Version 2 also led to the infrastructure of the REST API being committed to WordPress core in 4.4.

    This infrastructure is the core of the REST API, and provides the external interface to send and receive RESTful HTTP requests. Since shipping in 4.4, the infrastructure is now used by WordPress Core for oEmbed responses, and by plugins like WooCommerce and Jetpack, enabling anyone to create their own REST API endpoints.

    The team has also been hard at work on the API endpoints. This has included core changes to WordPress to support the API, including deeper changes to both settings and meta.

    Today the REST API team is proposing the inclusion of a collection of endpoints that we term the “Content API” into WordPress Core.

    Proposals for Merge

    Content Endpoints

    For WordPress 4.7 the API team proposes to merge API endpoints for WordPress content types. These endpoints provide machine-readable external access to your WordPress site with a clear, standards-driven interface, allowing new and innovative apps for interacting with your site. These endpoints support all of the following:

    • Content:
      • Posts: Read and write access to all post data, for all types of post-based data, including pages and media.
      • Comments: Read and write access to all comment data. This includes pingbacks and trackbacks.
      • Terms: Read and write access to all term data.
      • Users: Read and write access to all user data. This includes public access to some data for post authors.
      • Meta: Read and write access to metadata for posts, comments, terms, and users, on an opt-in basis from plugins.
    • Management:
      • Settings: Read and write access to settings, on an opt-in basis from plugins and core. This enables API management of key site content values that are technically stored in options, such as site title and byline.

    This merge proposal represents a complete and functional Content API, providing the necessary endpoints for mobile apps and frontends, and lays the groundwork for future releases focused on providing a Management API interface for full site administration.

    Content API endpoints support both public and authenticated access. Authenticated access allows both read and write access to anything your user has access to, including post meta and settings. Public access is available for any already-public data, such as posts, terms, and limited user data for published post authors. To avoid potential privacy issues we’ve taken pains to ensure that everything we’re exposing is already public, and the API uses WordPress’ capability system extensively to ensure that all data is properly secured.

    Just like the rest of WordPress, the Content API is fully extensible, supporting custom post meta, as well as allowing more complex data to be added via register_rest_field. The API is built around standard parts of WordPress, including the capability system and filters, so extending the API in plugins should feel as familiar to developers as extending any other part of WordPress.

    This Content API is targeted at a few primary use cases, including enhancing themes with interactivity, creating powerful plugin interfaces, building mobile and desktop applications, and providing alternative authoring experiences. We’ve been working on first-party examples of these, including a mobile app using React Native and a liveblogging web app, as well as getting feedback from others, including WIRED, the New York Times, and The Times of London. Based on experience building on the API, we’ve polished the endpoints and expanded to support settings endpoints, which are included as the first part of the Management API.


    The API Infrastructure already in WordPress core includes support for regular cookie-based authentication. This is useful for plugins and themes that want to use the API, but requires access to cookies and nonces, and is hence only useful for internal usage.

    To complement the Content Endpoints, for WordPress 4.7 the API team also proposes merging the REST API OAuth 1 server plugin into WordPress Core. This plugin provides remote authentication via the OAuth 1 protocol, allowing remote servers and applications to interact securely with the WordPress API.

    OAuth is a standardised system for delegated authorisation. With OAuth, rather than providing your password to a third-party app, you can authorise it to operate on your behalf. Apps are also required to be registered with the site beforehand, which gives site administrators control over third-party access. Access to these apps can be revoked by the user if they are no longer using the app, or by a site administrator. This also allows apps with known vulnerabilities to have compromised credentials revoked to protect users.

    We’ve chosen OAuth 1 over the newer OAuth 2 protocol because OAuth 1 includes a complex system for request signing to ensure credentials remain secure even over unsecured HTTP, while OAuth 2 requires HTTPS with a modern version of TLS. While it is strongly encouraged for sites to use HTTPS whenever possible (Let’s Encrypt makes it easier than ever to do so), WordPress itself does not require HTTPS and we do not believe WordPress should make HTTPS a requirement for using the API. The additional complexity that OAuth 1 adds can be easily supported by a library, and many such libraries already exist in most programming languages. OAuth 1 remains supported around the web, including for the Twitter API, and we also provide extensive documentation on using it.

    Authentication Beyond 4.7

    One issue with OAuth over direct username and password authentication is that it requires applications to be registered on the site. For centralized OAuth servers this wouldn’t be a problem, but the distributed nature of WordPress installations makes this tough to handle: your application must be independently registered with every WordPress site it connects to. If you’ve ever had to create a Twitter or Facebook app just to use an existing plugin on your site, you’ll know this can be a less-than-optimal experience for users.

    To solve this distribution problem, we’ve created a solution called brokered authentication. This allows a centralised server (called the “broker”) to handle app registration and to vouch for these apps to individual sites. It simplifies app registration by allowing app developers to register once for all sites, and improves security by allowing the broker to vet applications and revoke them across the entire network. The system is designed to allow multiple brokers; while the main broker is run at apps.wp-api.org, organisations can run their own broker for internal usage, and developers can run a broker locally for testing.

    While the broker system has been running live at apps.wp-api.org for months, we want to stay conservative in our approach to the API, especially where security is concerned. We are therefore proposing brokered authentication for WordPress 4.8 to ensure we have further time to continue testing and refining the broker system. In addition, this will require an installation of the broker on a centralised server to act as the canonical broker for out-of-the-box WordPress. While apps.wp-api.org is currently acting in this role, this is currently hosted by a third-party (Human Made) on behalf of the API team. For long-term usage the broker should instead be hosted on WordPress.org, alongside the existing plugin and theme repositories. This migration will take time but we remain committed to continuing to develop and support the broker.

    After Merge

    After merging the REST API, the team plans to continue developing the API as before. We expect that integrating the REST API into WordPress core will bring additional feedback, and we plan on incorporating this feedback through the rest of the 4.7 cycle.

    During the remaining parts of this release cycle and through into the 4.8 cycle, additional work will go into other parts of the API. This includes further work and refinement on the broker authentication system, including work on WordPress.org infrastructure. Additionally, we plan to continue working on the Management API endpoints, including theme and appearance endpoints to support the Customiser team. Both of these components will be maintained as separate feature projects on GitHub until they’re ready for merge into core.

    The team remains committed to supporting the API in core, and the Content API will switch from GitHub to Trac for project management and contributions. This same process occurred for the API Infrastructure in WordPress 4.4.

    Reviews and Feedback

    With this merge proposal, we’re looking for feedback and review of the project. In particular, we’re focussing on feedback on the security of the API and OAuth projects, and are also reaching out to specific people for reviews. (We take the security of the API seriously, and bug reports are welcomed on HackerOne at any time.) Design and accessibility reviews for the OAuth authorisation UI are also welcomed to ensure we maintain the high standards of WordPress core.

    Both the REST API plugin and the OAuth plugin are available on WordPress.org, and issues can be reported to the GitHub tracker for the API and the OAuth plugin respectively. We have released a final beta (Beta 15 “International Drainage Commission”) which includes the meta and settings endpoints.

    With Love from Us

    As always, this is a merge proposal, and is not final until 4.7 is released. We’re eager to hear your thoughts and feedback; the comments below are a perfect place for that, or you can pop along to one of our regular meetings. We’re also always available in the #core-restapi room on Slack.

    We’d like to thank every single one of our contributors, including 88 contributors to the main repository and 23 contributors to the OAuth repository. Particular thanks goes to my (@rmccue) wonderful co-lead Rachel Baker (@rachelbaker), our 2.0 release leads Daniel Bachhuber (@danielbachuber) and Joe Hoyle (@joehoyle), and our key contributors for the 4.7 cycle: Adam Silverstein (@adamsilverstein), Brian Krogsgard (@krogsgard), David Remer (@websupporter), Edwin Cromley (@chopinbach), and K. Adam White (@kadamwhite). Thanks also to the core committers helping us out through the 4.7 cycle, including Aaron D. Campbell (@aaroncampbell) and Aaron Jorbin (@aaronjorbin), and to the fantastic release lead, Helen Hou-Sandí (@helen).

    Thanks also to everyone who has used the REST API, and to you for reading this. We built the REST API for you, and we hope you like it.

    With love, The REST API Team

    • George Stephanis 12:17 pm on October 8, 2016 Permalink | Log in to Reply

      Regarding Authentication, as nice as OAuth 1.0a may be in some respects, I feel that we’re missing some rather obvious problems here.

      We’re expressing such concern about securing the REST API endpoints to work on HTTP — that we’re missing the fact that for sites that don’t support HTTP, normal wp-login requests or cookies could just as easily be sniffed.

      It just feels to me like we’re spending a lot of time reinforcing the back door against attackers, when the front door isn’t even locked.

      The concept of a third-party broker feels very antithetical to WordPress Core. To have to ping the third-party server for every login to check for invalidations of their applications, let alone the initial confirmation of an application … for me, it doesn’t pass the gut check.

      I’d much rather see something akin to the Application Password system we’ve devised. It has unique per-application passwords, with 142 bits of entropy, usage logging (date and ip last used), and trivial invalidation, as well as a proposed simple flow for applications to request passwords and get the generated passwords passed back.. And, as an added bonus, it works with the legacy XML-RPC API.

      Compared to that, making someone hoping to do something for API access to a single site register for a cloud broker, and then reauth with the individual site .. the UX doesn’t pass muster for me. :\

      Authentication is probably the most important key to getting the REST API correct, and it’s necessary (imho) to ship. We need to get it right.

      • marsjaninzmarsa 7:25 am on October 9, 2016 Permalink | Log in to Reply

        To have to ping the third-party server for every login to check for invalidations of their applications, let alone the initial confirmation of an application … for me, it doesn’t pass the gut check.

        Nope, you’re missing the point, third party will be pinged only on first use of new app on particular site, and next every few hours or so for invalidation (same as with checking for security updates).

      • Dion Hulse 10:40 am on October 9, 2016 Permalink | Log in to Reply

        Core. To have to ping the third-party server for every login to check for invalidations of their applications, let alone the initial confirmation of an application … for me, it doesn’t pass the gut check.

        I tend to agree here.

        I’d much rather see something akin to the Application Password system we’ve devised

        However, I feel that Application Passwords are actually a far worse solution to the problem than the various oAuth options out there, to the point that I think Application Passwords are an anti-pattern which core should never directly support. The user experience of Application Passwords are not user friendly at all, they’re a hacky solution to avoid using a proper modern application authentication mechanism.

        I could ramble for quite some time on the UX of these, but at the end of the day moving towards oAuth is going to provide a far better developer and user experience for API clients.
        Unfortunately though with all options here there’s the downside that Applications need to be registered/known/considered safe and also be able to invalidate their credentials in the event the applications key store is leaked (For example, Plugin X which relies upon a service has it’s keystore hacked, the application would have no way of informing those sites to distrust it’s previous authenticated sessions).

        In an ideal world, a central provider wouldn’t be needed, but we don’t have a decentralised platform for WordPress yet, so there’s no other mechanism for WordPresses out there to be told the sort of information they need to know.

      • Ryan McCue 8:40 pm on October 9, 2016 Permalink | Log in to Reply

        At its core, OAuth is just a standardised system for issuing app tokens. The primary things that it builds on top of this are specifically for security over HTTP, so I would be extremely reluctant to move backwards from those to a more simple token-based system.

        Regular WordPress logins are protected by two main factors: nonces and session tokens. While it is possible to sniff a raw username/password over the initial login, subsequent requests are protected, which means automated attacks are much harder. The nature of an external API means neither WP nonces nor session tokens (which are both intentionally server-only secrets) can be used to protect authentication, so we need a replacement system.

        OAuth incorporates a combination of nonce and timestamp to replace these in a similar manner, which are then combined with the secret, which we need since it’s no longer server-only. (WP nonces and session tokens have an internal secret with NONCE_SALT, the only difference is that we share them with the client for OAuth.)

        The broker system exists specifically because dynamic registration (either with OAuth or your plugin) has a number of key problems, primarily that it makes revocation useless. If the app is revoked, it can simply register again, which subverts both the site admin and W.org team if it were intentionally revoked. We looked at dynamic registration and rejected it specifically because of the issues with it.

        While the system might seem conceptually complex, it’s pretty simple to actually implement, because the broker is just regular OAuth under the hood. The entirety of the flow can be written in 200 lines of JavaScript. From the app’s perspective, it primarily Just Works, with most of the complexity hidden inside the broker implementation instead.

        The OAuth 1.0a specification is rigorously tested by security researchers and in production on huge web apps. The broker system builds on top of this, but avoids reimplementing the wheel, and instead just uses OAuth for some of the heavy lifting. Application Passwords is at its core HTTP Basic Authentication, which in its own specification 17 years ago recommended against actually using it.

        OAuth 1 isn’t perfect, but every thing in it is there for a reason. Is it perfect? No. But it provides a lot of benefit over Basic Auth for a relatively low cost.

    • Jon Brown 1:09 pm on October 8, 2016 Permalink | Log in to Reply

      Much love back at you all. This looks fabulously well thought out and complete.

    • Josh Pollock 2:42 pm on October 8, 2016 Permalink | Log in to Reply

      5 Stars!

    • fgilio 3:09 pm on October 8, 2016 Permalink | Log in to Reply

      Thanks to all of you!

    • Ahmad Awais 4:07 pm on October 8, 2016 Permalink | Log in to Reply

      I’m all in. Let’s do it this time. BTW great read Ryan.

    • Omaar Osmaan 5:55 pm on October 8, 2016 Permalink | Log in to Reply

      Thanks so much for all the hard work from the team- eager to see it get merged on core!

    • Noel Tock 7:56 pm on October 8, 2016 Permalink | Log in to Reply


    • Sergio Santos 1:45 pm on October 9, 2016 Permalink | Log in to Reply

      Thanks to all the team behind this outstanding work!

    • Matthew Eppelsheimer 9:43 pm on October 9, 2016 Permalink | Log in to Reply

      Thumbs up!

    • Weston Ruter 12:12 am on October 10, 2016 Permalink | Log in to Reply

      Additionally, we plan to continue working on the Management API endpoints, including theme and appearance endpoints to support the Customiser team. Both of these components will be maintained as separate feature projects on GitHub until they’re ready for merge into core.

      I’d like to note that customizer changesets (#30937, formerly “transactions”) allows for REST API responses to have customizations applied when the request includes a customize_changeset_uuid query param which identifies a customize_changeset post. This allows the customizer to be used with headless sites and mobile apps.

      A feature plugin needs to be started shortly that allows for these changeset posts to be CRUD’ed via the REST API as well, allowing for new customizer administrative applications to be built on top of the REST API without any need to go to customize.php.

    • Tammie Lister 8:08 pm on October 10, 2016 Permalink | Log in to Reply

      As requested in the design Slack channel, here is a design review. I took the OAuth plugin for a spin today and only focused on the design elements.

      I first of all had discoverability issues, but that was mainly due to not using this. I sort of imagine if someone is going to use this they will be doing so knowing more than I did – so that’s ok.

      The first screen is minimal but as expected when empty. It seemed to all fit the existing WP patterns. Thanks for taking the time and doing that. Should there be select all boxes though – I don’t see any actions for all.

      On the ‘add’ screen, I tried adding nothing in the fields. I was surprised when got a green message. I would expect this should be red over the success green.


      The term ‘Add Application’ but then the action button being ‘Save Consumer’, this threw me. Perhaps this is my lack of knowledge over terminology, but shouldn’t they both be the same?

      The page gets cluttered fast, I sort of want a break here visually, just a simple line would do:


      Regenerating the secret success message I felt should be by the secret. It was odd as the message appeared above my eye view. I click to regenerate lower and then this message pops up the top, feels like it should be close to add context.


      This may be a bug but when I searched for applications it tried to search users and I got the following:


    • Matt Mullenweg 8:26 pm on October 10, 2016 Permalink | Log in to Reply

      1. Given the hurdles of authentication I don’t think that bringing this into core provides benefits for WP beyond what the community gets from the plugin. I don’t believe in its current state the benefit outweighs the cost, and we should err on the side of simplicity.

      2. I am not interested in hosting the centralized brokered authentication server on WordPress.org in the 4.8 timeframe, and hesitant about the implications it has for WP more broadly. I do appreciate the thought that has been put into solving this tricky problem.

      • K.Adam White 9:39 pm on October 11, 2016 Permalink | Log in to Reply

        @matt, thanks for the comment. Can you clarify whether with (1) you mean the authentication solution, or the merge proposal for the content endpoints more broadly? There was discussion today in slack where we weren’t sure which way to interpret this point.

    • Dominik Schilling (ocean90) 10:01 pm on October 10, 2016 Permalink | Log in to Reply

      I gave the OAuth 1 plugin a try and here’s some feedback:

      • https://github.com/WP-API/OAuth1/issues/161 makes me wonder how we can integrate something with may not work on “a lot of sites”.
      • Adding Applications as a sub menu of Users doesn’t feel right to me. I just can’t imagine that 80% of our users really need this there. Twitter and Facebook have it both in the settings.
      • The UI: https://cloudup.com/c1emHXcHP4g Reusing existing elements is fine, but that looks a bit to simple and not fully developed. I’m sure that this can made better and doesn’t require a table. See Twitter/Facebook.
      • The plugin is using closures so it’s not compatible with the minimum requirements of WordPress.
      • I got a “500 Internal Server Error” error when the callback URL was invalid.
      • A few DocBlocks don’t follow the documentation standards.

      For some reasons I can’t get a successful `oauth1/access` request, I’ll try this again tomorrow…

    • Aaron D. Campbell 7:30 pm on October 11, 2016 Permalink | Log in to Reply

      The API endpoints themselves seem to be in a really good place, but I’m not sure the oAuth plugin is. Would it be reasonable to separate these into different proposals?

    • Mark Howells-Mead 2:32 pm on October 12, 2016 Permalink | Log in to Reply

      Issues aside, I’m strongly in favour of the REST API becoming a core feature as soon as it is stable.

    • Chris Smith 3:13 pm on October 12, 2016 Permalink | Log in to Reply

      +1 here as well and thanks to the team for their hard and thoughtful work.

      Based on the comments here, it does seem like it would make sense to split the proposal into the Content API and oAuth pieces for 4.7 so they can be dealt with/commented on independently as needed.

      @kadamwhite, what were the team’s thoughts on that in yesterday’s discussion?

    • Nilambar Sharma 5:32 pm on October 12, 2016 Permalink | Log in to Reply

      Really waiting for this. Lets make WordPress more awesome. It would be a great step merging new endpoints to core.

    • brad989 5:33 pm on October 12, 2016 Permalink | Log in to Reply

      Been looking forward to this for a long time. The content endpoints are an essential addition to WordPress core.

    • Chuck Reynolds 6:52 am on October 14, 2016 Permalink | Log in to Reply

      full api integration into core needs to happen eventually… API endpoints are at/near a good place. re: oath tho… meh..
      needs full support of CPT’s, which theoretically it would but… lots to test there.

    • forbze 7:16 am on October 14, 2016 Permalink | Log in to Reply

      I really want this! I’ve had to use hacky solutions in the past to repurpose content for mobile apps (I didn’t want to use web views / Mobile ui plugins).

      WordPress’s admin interface would make it really simple for me to give clients content control for their app content – rather than me having to build a custom editor UI on top of Parse / Firebase.

      I think oauth is the right solution for auth.

    • pwaring 9:23 am on October 14, 2016 Permalink | Log in to Reply

      I’m only in favour of this if the API can be disabled (ideally by default). I don’t need or want this functionality and it sounds like another remote attack vector.

      “we do not believe WordPress should make HTTPS a requirement for using the API”

      I *strongly* disagree with this. TLS uptake is increasing all the time and some browsers (e.g. Chrome) are starting to show stronger warnings for plain HTTP. Plus, if a popular application like WordPress requires HTTPS for its API, that will act as a significant nudge to hosting providers to provide HTTPS by default (a bit like moving from PHP 4 to 5).

      • Ryan McCue 9:18 pm on October 14, 2016 Permalink | Log in to Reply

        It’s super easy to disable the REST API via the `rest_enabled` filter (just return false), and there’s already a plugin for it. 🙂

        • pwaring 8:10 am on October 15, 2016 Permalink | Log in to Reply

          Whilst I’m capable of changing the code to add a filter, I don’t think that counts as ‘super easy’. It really should be a tick box somewhere in the admin settings.

          Installing a plugin to disable optional behaviour in core also seems unnecessarily complicated, plus it introduces another dependency and there’s no guarantee that the plugin author will keep it up to date.

          • Dave McHale 1:59 pm on October 18, 2016 Permalink | Log in to Reply

            I understand your concern, pwaring, but we already don’t have the ability to disable XML-RPC in core (ever since they enabled it by default in 3.5) – and since the REST API is “the same but different” I believe that’s why the new feature is following the same model. “Decisions, not options” and all that 🙂

            It’s also why I’m the author of the plugin Ryan linked to above. I think the REST API is great, and can’t wait to start using it myself *and* also see what other people build with it. But it’s not going to be for everyone, and I thought that people may want the easy ability to turn it off if they had a reason to.

            Since the plugin uses the root-level filters provided by the REST API there should very, VERY little need for actual plugin updates moving forward into the future (though I do keep an eye on the project, in case an update is needed). The only time I’ve needed to make a real functional change was when the filters themselves changed from version 1.x to 2.x of the API, before it was even merged into core. New features like the content endpoints won’t change the use of the plugin, because the REST API is still entirely disabled if you’re using the filters.

  • Ryan McCue 10:34 am on July 27, 2016 Permalink |
    Tags: , , http-api   

    HTTP API in 4.6 

    For WordPress 4.6, the HTTP API (wp_remote_request() and family) have undergone a large internal change. Rather than using a WordPress-specific HTTP library, WordPress now uses the open-source independent Requests library, developed by yours truly.

    Why Requests?

    The WP_HTTP library in previous releases has been primarily maintained by myself and @dd32, with my support time split between the two libraries. Both libraries are very similar, and code has been shared between them (when licensing permitted) in the past. Requests follows the same development philosophies as WordPress: developing for the masses with broad PHP support, and maintaining backwards compatibility.

    By switching to Requests, a library without any WordPress-specific dependencies, WordPress benefits from input from the wider PHP ecosystem and community

    Requests also has a huge number of unit tests, with test coverage at 92% of the codebase and increasing. It’s also used by other projects via the Composer ecosystem, including wp-cli’s HTTP functionality.

    What has changed?

    From your perspective as a developer, nothing should have changed visibly. You can and should continue to use WP’s HTTP functions the way you always have.

    (Note: For the 4.6 development cycle and beta 1, the HTTP functions returned an array-like object. Many plugins and themes in the real world are using direct is_array() checks, so it was decided to pull this functionality back a bit to be safe, see #37097.)

    Some new functionality has been introduced. In particular, the array returned from wp_remote_request() now includes a new http_response value, which contains a WP_HTTP_Response object (technically, WP_HTTP_Requests_Response). This shares functionality with WP_REST_Response objects introduced in WordPress 4.4, allowing common functionality to be developed for both APIs. In future releases, WordPress may introduce new WP_HTTP_Request objects as well, allowing common middleware to be used across both APIs.

    In addition, all of Requests’ features are now available in WordPress. This includes things like better HTTP standard support, case-insensitive headers, parallel HTTP requests, support for Internationalized Domain Names (like böcean901.ch), and many other internal improvements.

    Some new features are only available when using Requests::request() directly (such as parallel requests), however, these will be introduced into new WordPress-specific APIs in future releases (#37459). This release is focussing on switching internal implementation and remaining stable.

    For more background on the change, see #33055.

    • Primoz Cigler 2:11 pm on July 27, 2016 Permalink | Log in to Reply

      Great news! Reusing existing popular libraries from the broader PHP community is the way to go, especially if they follow modern dev workflow (composer etc).

      Ryan, as I am familiar with Guzzle and you are author of the Requests, would you mind sharing the main differences between these two libraries and your motivation for creating Requests over using Guzzle? Thank you!

      • Claudio Sanches 2:19 pm on July 27, 2016 Permalink | Log in to Reply

      • Ryan McCue 4:56 am on July 28, 2016 Permalink | Log in to Reply

        A few of the reasons are covered in the Why Requests doc posted by Claudio, but there are other reasons specific to WordPress.

        Guzzle is a full REST API consuming library, including a lot of tooling around using APIs, whereas Requests is a simple HTTP library. (That doesn’t make REST requests impossible or anything, it just means it doesn’t offer anything targeted at APIs.)

        Requests is also a lot closer to being a WordPress-y API, purely because my philosophy when creating it was the same as WordPress’. Although it offers more advanced APIs, at the core, Requests offers up a function which makes a request and returns some response data. It doesn’t require usage of conceptually more complex things than arrays and simple object usage.

    • Nashwan Doaqan 4:12 pm on July 27, 2016 Permalink | Log in to Reply

      Great! .. Thanks for all the effort. 🙂

    • Jacob Santos 7:35 pm on July 27, 2016 Permalink | Log in to Reply

      If I am being honest, the HTTP API was always meant to be temporary. It is one of the reasons I advocated using the functions and warned against calling the main class directly.

      It was a simple implementation to pull in the various requests, using curl and fsockopen. I believed I looked for third party libraries, but none included all of the transports or didn’t have a sensible API separation of concerns. One had everything in the single class.

      If I had a vote, I would have requested guzzle. Even if it required contributing fopen transport.

      I will say it is annoying that library authors don’t continue support for previous php versions. However, I have said my goodbyes to php5.2 a long time ago. Any host that still supports that version is really not worth the money. I suspect that they are cheap or free hosts, but damn, php5.2 is almost a decade old. It is also only 7.2% of total usage. I believe php4 was dropped also when it dropped below 10%.

      I would rather wordpress increase to php5.3 and include and support and older version of guzzle than introduce another 5.2 library. More people use and support guzzle currently than the Requests library. 600k for requests vs 17 million + 15 million for guzzle. I do not believe adding 2 more devs to an older guzzle will comvince them to support the older version, but it would allow for a better upgrade path from guzzle supporting 5.3 to the latest guzzle. Backporting and testing should also be easier.

      Requests looks too much like the original wordpress code from when dd32, peter westwood and I wrote it for my comfort. The argument could be made that it requires the less permissive GPL license since it appears to derive from the wordpress code base. Moving to a library that has a more open license with less controversy would make more sense in case someone wants to challenge the Requests license in the future.

      • Ryan McCue 4:50 am on July 28, 2016 Permalink | Log in to Reply

        I think the rest of your points are already covered above or elsewhere, but I wanted to note on the licensing: Requests was entirely written from scratch, and does not incorporate any of WP_HTTP. There have been a few patches created for WP_HTTP that have been ported across to Requests, but the patch authors (namely, me and @dd32) have agreed to licensing those contributions under the ISC license, so there’s no issue with that.

        Most contributions have been in the opposite direction: improvements in Requests have been ported to WP_HTTP. Given that I wrote those patches and contributed them to WP under the terms of the GPL, there’s also no issue with those.

      • Martin Sotirov 5:50 pm on October 19, 2016 Permalink | Log in to Reply

        I’m with you on this one, Jacob.

        WordPress should be moving towards compatibility with the rest of the PHP community instead of always doing its own separate weird thing.

    • Alex Mills (Viper007Bond) 5:50 am on August 10, 2016 Permalink | Log in to Reply

      I just wanted to say I’m really happy about this change. I was debugging why a 401 header wasn’t being parsed correctly in WordPress stable only to discover that it’s parsed perfectly when using Requests in WordPress trunk! You rock. 🙂

  • Ryan McCue 7:10 am on May 16, 2016 Permalink

    Rewrites Next Generation: Next Meeting 

    There was a bit of confusion with the last Rewrites meeting, so we never really got to kick off the conversation. Let’s give it another shot this week. The revised kick-off meeting for the Rewrites Next Generation project will hence be on May 18th 23:00 UTC.

    The agenda for this meeting is to discuss problems you see with the rewrites system, any particular use-cases you think should be considered, as well as agreeing on a broad approach for how we want to tackle the problems.

    (In case you missed it, you may want to read the introductory post for the project.)

    Hope to see you all there!

  • Ryan McCue 8:46 am on May 3, 2016 Permalink

    Proposal: Next Generation Rewrites 

    Hi everyone! Today I’d like to propose a new feature project for WordPress: Next Generation Rewrites. After proposing this at the last feature projects meeting, it’s time to flesh out the details of what this project would be.

    The aim of the project is to modernise the rewriting and routing system for the modern CMS and platform era of WordPress.

    (This project was previously proposed in a ticket on Trac, however the project is larger than a single Trac ticket and needs a larger discussion.)


    If you’ve worked with custom development on WordPress, you’ve probably registered at least one rewrite rule. Rewrites are the key to what WordPress calls “pretty permalinks”, which are URLs designed for humans rather than the server. They look like /2016/04/19/, or /about/requirements/, rather than index.php?p=42.

    As a developer, you can register your own rewrite rules as well. For example, bbPress registers URLs like /forums/topic/jjj-is-awesome/ and WooCommerce has URLs like /product/gs3/. WordPress then “re-writes” these to the internal query variables (“query vars”), which are eventually passed into the main WP Query.

    The rewrite system was initially designed to implement pretty permalinks, but has been used and abused for so much more since then. With modern plugins and web apps built on top of WordPress, it’s not uncommon to have pages that don’t need a main query, or need a specialised system of querying. As an example of this, the REST API infrastructure added in WordPress 4.4 doesn’t use the main query, as many of the endpoints don’t require querying posts at all.

    While the usage of rewrites has developed, the internals of how URLs are handled internally hasn’t changed. The system is fundamentally still designed for the blog era of WordPress. It also predates a lot of the pieces of WordPress we take for granted, such as transients and object caching.

    Project Proposal

    It’s time to change the rewrite system. Rewrite rules should be changed from a layer on top of WP Query to a fully-fledged system all of their own. To do this, I’m proposing three initial steps:

    • Decouple rewrites from the query system
    • Refactor and rework the internals to make them testable
    • Rethink rewrite storage and remove flushing

    These steps would take place under the umbrella of a new feature project, Next Generation Rewrites. This feature project would coordinate the project and people working on it across the various parts of core it touches (primarily the Rewrite Rules component, but also potentially involving UI and UX discussion and changes; fixing the “visit the Permalinks page and save to fix your site” issue, for example). This would also coordinate the team across multiple release cycles as we slowly but surely make progress. It’s important to make progress while keeping rewrites stable, as they’re a critical part of WordPress that we cannot break.

    Decoupling Rewrites from Querying

    The first step I’m proposing is to decouple rewrites from querying. Right now, rewrites for anything except querying posts involves jumping through a few hoops. This includes registering temporary query vars, or using global state to track route matching. Separating these two components will make it easier to add non-query-based rewrites, as well as allowing more powerful query rewrites.

    Currently, rewrites are registered using code like the following:

    add_rewrite_rule( '/regex/', 'index.php?var=value' );

    These rewrite rules map a regular expression to a query string, which is later parsed into an array internally. You can achieve more complex routing via regular expressions by using a special syntax in the rewrite string:

    add_rewrite_rule( '/topic/(.+)', 'index.php?topic=$matches[1]' );

    Note that while this looks like a PHP variable, it’s actually a static string that gets replaced after the regular expression. This can lead to confusion with developers who are new to the syntax, and it also can make doing more complex rewrites hard. These typically involve temporary query vars, as well as quite a bit of processing.

    Instead, I want to introduce a new WP_Rewrite_Rule object to replace the current rewrite string. This object would contain a method called get_query_vars that receives the $matches variable and returns the query vars to set. This would look something like this:

    class MyRule extends WP_Rewrite_Rule {
        public function get_query_vars( $matches ) {
            return array(
                'post_type' => 'topic',
                'meta_query' => array(
                        'key' => '_topic_name',
                        'value' => $matches[1],
    add_rewrite_rule( '/topic/(.+)', new MyRule() );

    (The exact format of the object is yet to be decided; we’ve also started discussing the possibility of passing a request object into this callback as well.)

    Using an object for this allows us to have multiple callbacks for different stages of the routing process, and starts to simplify some of the internal code in WP_Rewrite. For example, the routing code around pages that allows URLs like /parent/child/sub-child/ for child pages can be simplified and moved out of the main routing code. This also helps make the code more testable, which dovetails nicely with the third goal.

    Refactor rewrite internals

    Step 2 of changing rewrites is to make the rewrite system fully testable. Currently, a bunch of global state is mixed into the internals of rewrite matching, and the rewrite methods tend to be monolithic. @ericlewis has previously begun work on this in #18877, and this can be continued and rolled into the Next Generation Rewrites project.

    Ideally, this should happen at the same time or before the first step to allow easy checking of regressions. This is relatively boring work that won’t affect many developers, but it’s important we do it. The rewrite system is a critical part of WordPress, and it’s crucially important that it’s testable and verifiable.

    Rethink rewrite storage

    Once the first two steps are in place, we should begin reconsidering how rewrites are stored. Currently, rewrites are stored in the database in the rewrite_rules option. The option is less of an option (it’s not really configuration), and more of a caching technique. This is somewhat of a relic, as rewrites predate object caching and transients in WordPress, both of which are better techniques to handle caching in modern WordPress.

    Removing this option and changing it to use a proper caching subsystem in WordPress should fix multiple problems, including the need to flush rewrites. This should improve the UX for end users, as we should be able to remove the “links on your site don’t work, so go to the Permalinks page and save” trick (which only really requires visiting the page, not saving).

    The need to cache rewrites at all can also be reconsidered; many of the rewrite rules are simply entries in an array, and are not expensive to generate on-the-fly, while removing the caching so can make for an easier to use and more dynamic system. More expensive rules can be cached by the code generating the rule, rather than via the monolithic rewrite cache.

    This has the potential to improve the user experience and fix a class of issues, but it also has the potential to ruin performance, depending on how other code is using it. It’s important to tread carefully as we attempt to improve this, and ensure we remain compatible and performant. Introducing a two-tiered system is one approach we can consider, with the ability for plugins to opt-in to a newer, dynamic system to avoid problems with flushing.

    Why a Feature Project?

    I’m proposing Rewrites Next Generation as a feature project for several reasons. The concept and implementation are both large and complex, moreso than simply operating in an informal matter on Trac tickets. The impact of this project is also large, affecting many plugin and theme developers, as well as potential improvements to UX. Gaining feedback from all interested stakeholders is important, as this is a crucial part of WordPress.

    Getting Started

    The steps I’ve listed here are only a selection of the issues with rewrites. The rewrite system hasn’t been massively changed since its inception, so there are plenty of parts that could be changed to better suit the modern era of WordPress development. The issues here are simply the three most important that I’ve found, but I want to hear your feedback too.

    Let’s talk rewrites. I’m proposing a weekly meeting for Rewrites Next Generation, at Wednesday, 23:00 UTC with the first meeting at May 4th 23:00 UTC. If you’re interested in rewrites, or have had problems with them in the past, let’s talk and work out what we need to do to improve rewrites. (This is a non-permanent meeting to set the scope of the project; we’ll refine meeting frequency and timing at a later date.)

    After this initial discussion, we can settle on concrete goals and a timeline for the initial stages of the project. A proof-of-concept patch and ticket are available on Trac, however alternative approaches should be considered after this discussion. The short-term goal is to begin landing these improvements in trunk, with the goal of having our first changes in WordPress 4.6, which is a quick but achievable timeline.

    This project has the potential to make a large impact on developers and users, and it’s important that everyone has their say. I hope you’ll join me for the first meeting, and join in the fun of contributing to a key part of WordPress!

    Thanks for reading. <3

    • Omaar Osmaan 9:05 am on May 3, 2016 Permalink | Log in to Reply

      So excited for this!

    • Ryan Hellyer 9:28 am on May 3, 2016 Permalink | Log in to Reply

      A simpler system for adding new rewrite rules would be nice. The current system is unintuitive IMHO.

    • Liam Gladdy 9:47 am on May 3, 2016 Permalink | Log in to Reply

      Everything about this makes me happy! The initial meeting time doesn’t work well for me in the UK, but i’ll catch up the next day!

    • andreasnrb 11:21 am on May 3, 2016 Permalink | Log in to Reply

      Really interesting. The meeting time suggestion is really bad. Does not work very well for anyone in Europe really.

    • Takayuki Miyauchi 11:41 am on May 3, 2016 Permalink | Log in to Reply

      add_rewrite_rule( '/topic/(.+)', new MyRule() ); looks nice!

    • jagnew 1:15 pm on May 3, 2016 Permalink | Log in to Reply

      This would be great, I’ve actually been building a plugin to handle routing etc. based around what current PHP frameworks do these days. If its of any use (still writing the tests and docs for it): https://github.com/jasonagnew/WP-Router

      Example of using it:

      get('id') == 1)
                  return 'No';
              return $this->next($request);
      class Route_Test {
          protected $router;
          public function __construct()
              global $wp_router;
              $this->router  = $wp_router;
          public function register_protected_routes()
              $this->router->get( array(
                  'as'   => 'getTest',
                  'uri'  => 'test/{id}',
                  'uses' => array( $this, 'get' )
              ) );
          protected function register_routes()
              $this->router->group( array(
                  'prefix' => '/protected',
                  'middlewares' => array( 'Middleware_One' ),
                  'uses' => array( $this, 'register_protected_routes' )
              ) );
              $this->router->post( array(
                  'as'   => 'postTest',
                  'uri'  => '/test/{id}',
                  'uses' => array( $this, 'post' ),
                  'prefix' => ''
              ) );
              $this->router->put( array(
                  'as'   => 'putTest',
                  'uri'  => '/test/{id}',
                  'uses' => array( $this, 'put' )
              ) );
          public function get($id, WP_Request $request)
              $all = $request->all();
              return new WP_JSON_Response($all);
          public function post($id)
              return 'POST: The ID is ' . $id;
          public function put($id)
              return 'PUT: The ID is ' . $id;
      new Route_Test();
    • Joost de Valk 2:22 pm on May 3, 2016 Permalink | Log in to Reply

      Love it. Would love to contribute but that meeting time is tough indeed.

      I’d want to add another thing to the mix: if we change rewrites, we also should consider creating a `get_current_url()` type function.

      I would also be very much in favor of caching URLs *very* aggressively, to the point of even giving them their own table. Right now, `get_permalink()` is one of the slowest things on many a page load…

    • Michael Ecklund 3:10 pm on May 3, 2016 Permalink | Log in to Reply

      Sounds interesting. Would definitely like to see improvement in this area for sure. Agreed with @joostdevalk.

    • Jacob Santos 5:07 pm on May 3, 2016 Permalink | Log in to Reply

      Composition over inheritance. It is easier to go to inheritance if composition is used first. Would help if used interfaces.

      I realize the least experienced don’t understand proper interface standards. I also realize there is a good argument for dynamic programming techniques.

      Since WordPress standardizes on neither, my preference would be proper OOP using interfaces to define a specification and Composition to tie the objects together.

      Whether the specification is defined as implicit with dynamic programming or explicit with interfaces, it would better serve then piling on the current OO mess that exists in WordPress currently.

    • Ryan McCue 12:41 am on May 4, 2016 Permalink | Log in to Reply

      Apologies on the meeting time for the Europeans, but there’s unfortunately no times that work well for everyone. Any earlier would push the boundary of whether I can make it (in Australia).

      I’ll consider having a secondary meeting for EU as well. 🙂

    • Ahmad Awais 2:57 am on May 4, 2016 Permalink | Log in to Reply

      Great stuff. Looking forward to it. Would love to contribute. WP most def needs a refresh to its routing sys.

    • schlessera 6:11 am on May 4, 2016 Permalink | Log in to Reply

      I agree with all of the issues you’re describing above. I cringe whenever I have to deal with rewrites, because in the end, it is an overly complex and fragile solution to a not-so-complex problem.

      I want to ask, though, if we are to go through the troubles of “rewriting” this part of the system and deal with all the BC headaches, why not go the whole nine yards? Instead of replacing the archaic special case system with a more modern special case system, why not create a modern general purpose system, and implement the existing rewrites as one special case of that general purpose system?

      You’ve probably guessed already that I’m talking about a full-blown routing system.

      If we define an interface (or an abstract base class if we absolutely must ignore best practices for the sake of “conformance”) called `WP_Route`, we can easily have one special case `WP_Rewrite_Rule_Route` that provides a completely BC way of implementing the current behaviour. But it allows us to have alternative routing behaviour that does not easily fit into the “rewriting” mechanism, too.

      So, the optimizations you’re talking about above are still a good idea, and should all be integrated into the `WP_Rewrite_Rule_Route`. But please don’t assume that we all just use “rewrites” for our routing purposes, when we have the opportunity to make an important change here. Assuming that all routing requests need to return query vars to pass to `WP_Query` is such an assumption. What if I want to have an URL that let’s me query custom external data? Why should this need to go through `WP_Query`?

      Most of what I’m talking about is already possible with the existing system, but it always feels dirty. I would love for WordPress to embrace a generalized approach (that can still default to special case implementations), so that I don’t constantly face situations where it feels like I’m working against the system.

    • schlessera 9:31 am on May 4, 2016 Permalink | Log in to Reply

      I thought some more about how a more generalized approach could look. I have to admit, though, that I’m not familiar with all of the intricacies and nuances that might come into play, so forgive me if anything I write here is just not possible.

      I imagine having WordPress pass the request to the routing system (let’s call it `WP_Router`). `WP_Router` will then do a minimum of parsing to be able to decide what specific routing action implementation it needs to call. Possible names of the interface would be `WP_Route` or `WP_Controller` (or anything along these lines). I’ll go with `WP_Controller` for the rest of the post to clearly differentiate it from `WP_Router`.

      The specific `WP_Controller` implementation is responsible for the main query. So, whatever `WP_Controller` you’re using, your widgets and navigation and so on should not be impacted.

      The `WP_Controller` does not return query vars or something in-between, it returns a completed “View” (the rendered content or archive, for example, or alternatively a standardized object).

      For the current default WP behaviour, we could have a `WP_Rewriting_Controller`, that collects the known rewriting rules, parses them for query vars, hands them over to `WP_Query` and returns the completed `View`.

      This would force us to have the individual elements that interact for such a request be decoupled from the overall WP lifecycle (which should mostly be the case, as far as I know).

      For any other `WP_Controller` implementation, you can freely choose what elements you need. You can, for example, completely forego `WP_Query`. The only contract that WP has with a `WP_Controller` is that it gets the request, and returns the response.

      It should be possible to build this entire system to be 100% BC, apart for the few plugins that directly mess with some data outside of the API. They will probably have plenty of time to adapt, though, as this is a bigger change anyway.

    • jeffmcneill 10:30 am on May 4, 2016 Permalink | Log in to Reply

      Not sure, but the whole issue of redirections might be something that this could include. The Redirection plugin recently was updated (after a few years) and it is a mainstay for managing the renaming of urls. Whenever there is a renaming of pages and reorganization of a site there are many of these redirection entries made to keep the 301 going properly. Not sure if a page-level thing like this should be dealt with at the same time as the much larger site-level redirection thing, but they do eventually lead to the same thing: a resolved url for page(s).

    • BenRacicot 4:10 pm on May 5, 2016 Permalink | Log in to Reply

      Awesome project! Would my ticket 34136 be applicable? https://core.trac.wordpress.org/ticket/34136 There seems to be a lot of devs looking for *much* better solutions / proper, core based solutions to CPT rewrites. Please see SO links within my trac 34136

  • Ryan McCue 9:23 pm on April 6, 2016 Permalink
    Tags: , ,   

    REST API: Slashed Data in WordPress 4.4 and 4.5 

    Hi everyone. The REST API team recently discovered a bug with parameter parsing in the API infrastructure, part of WordPress 4.4. For those of you using the API infrastructure, you need to be aware of a bug fix we’re making with the API.

    The Problem

    The REST API has several types of parameters that it mixes together. These come from several sources including the request body as either JSON or URL-encoded form data ($_POST), query parameters ($_GET), the API route, and internally-set defaults. Unfortunately, due to an oversight on our behalf, these parameters can be inconsistently formatted.

    In WordPress, the superglobal request variables ($_POST and $_GET) are “slashed”; effectively, turning magic quotes on for everyone. This was originally built into PHP as a feature to help guard against SQL injection, but was later removed. Due to compatibility concerns, WP cannot change this behaviour for the superglobals. This only applies to the PHP superglobals, not to other sources of input like a JSON body or parameters in the URL. It additionally does not apply to form data on PUT or DELETE requests.

    Internally, some low-level WordPress functions expect slashed data. These functions internally call wp_unslash() on the data you pass in. This means input data from the superglobals can be passed in directly, but other data needs to be wrapped with a call to wp_slash().

    When the REST API gathers the data sources, it accidentally mixes slashed and unslashed sources. This results in inconsistent behaviour of parameters based on their source. For example, data passed as a JSON body is unslashed, whereas data passed via form data in the body is slashed (for POST requests).

    For example, the following two pieces of data are equivalent in the REST API:

    // JSON body:
    {"title": "Foo"}
    // Form-data ($_POST)
    // Both result in:
    $request->get_param('title') === 'Foo';

    However, if the data contains slashes itself, this will be inconsistently passed to the callback:

    // JSON body:
    {"title": "Foo\Bar"}
    // Results in:
    $request->get_param('title') === 'Foo\Bar';
    // Form-data ($_POST) (%3D = "\")
    // Results in:
    $request->get_param('title') === 'Foo\\Bar';

    This means that callbacks need to understand where parameters come from in order to consistently handle them internally. Specifically:

    • Data passed in the query string ($_GET, $request->get_query_params()) is slashed
    • Data passed in the body as form-encoded ($_POST, $request->get_body_params()) is slashed for POST requests, and unslashed for PUT and DELETE requests.
    • Data passed in the body as JSON-encoded ($request->get_json_params()) is unslashed.
    • Data passed in the URL ($request->get_url_params()) is unslashed.
    • Data passed as a default ($request->get_default_params()) is unslashed.

    In addition, parameters set internally via $request->set_param() are unslashed. Unit and integration tests for API endpoints typically use these directly, so the majority of tested code (such as the WP REST API plugin) assumes parameters are unslashed.

    See the related Trac Ticket #36419 for more information.

    The Solution for WordPress 4.4 and 4.5

    We are regarding inconsistently-slashed data as a major bug, and are changing the API infrastructure to ensure unslashed data. This will ensure that data is consistent regardless of the source. Callbacks will now receive unslashed data only, and can rely on this regardless of the original data source or request method.

    If you are using functions that expect slashed data in your callback, you will need to slash your data before passing into these functions. Commonly used functions that expect slashed data are wp_insert_post, wp_update_post, update_post_meta, wp_insert_term, wp_insert_user, along with others. Before passing data into these functions, you must call wp_slash() on your data.

    The fix for this issue, will be included in the WordPress 4.5 release candidates and final release. Due to the severity of the bug, we are also backporting the fix to the next minor WordPress 4.4 update. This also ensures you can update your plugins can act consistently across all versions of the REST API.

    We understand that this may inadvertently break some plugins that are expecting slashed data. Right now, it’s not possible to consistently ensure that callbacks receive slashed data, so it is likely that these plugins will already break in some conditions.

    tl;dr: if you’re using wp_insert_* or *_post_meta in your REST API callback, you need to ensure you are calling wp_slash() on data you are passing in, regardless of source.

    We apologize for this bug existing in the first place. Slashed data is a problem that has plagued WordPress for a long time, and we’re not immune to getting caught by the issue ourselves.

    • Mike Schinkel 7:16 am on April 7, 2016 Permalink | Log in to Reply

      We apologize for this bug existing in the first place. Slashed data is a problem that has plagued WordPress for a long time, and we’re not immune to getting caught by the issue ourselves.

      Given that this issue created such a major bug, wouldn’t it be a good time to take backward compatible action to slowing start eliminating this as an issue?

      • Mike Schinkel 7:35 am on April 7, 2016 Permalink | Log in to Reply

        In case of interest in addressing this moving forward:

      • Ryan McCue 3:40 am on April 8, 2016 Permalink | Log in to Reply

        As it happens, this is part of my Secret Master Plan for Unslashing, and part of why I wanted to ensure we fixed this right now.

        My Secret Master Plan:

        • Take the generic parts of WP_REST_Request and make a new WP_HTTP_Request object
        • Ensure the Request object has unslashed data
        • Pass the Request around liberally to replace the superglobals (discussed somewhat in #36292)
        • Finally introduce internals that take unslashed data, and turn the existing functions into wrapper functions.
    • NateWr 7:54 am on April 7, 2016 Permalink | Log in to Reply

      Is this change in the latest WP 4.5 RC or do we need to get the latest commits to test our apps?

      • Marius (Clorith) 12:56 pm on April 7, 2016 Permalink | Log in to Reply

        It is not available in RC-1, that’s out at this time, as it was pushed in last evening/earlier today (depending on your time zones), the goal is to get RC-2 out some time this week, and it will include this change.

        If you are running trunk you’ll have access to it at this time, alternatively you can manually apply the patch to your own install if you wish to start testing it right away (the patch from #36419)

  • Ryan McCue 4:39 am on April 4, 2016 Permalink
    Tags: ,   

    WP REST API: 2.0 Beta 13 & Roadmap 

    Hi folks! I’m here with another exciting update from the API team.

    Beta 13

    First off, we’re excited to announce 2.0 Beta 13 “yoink.adios\losers” is now available. Grab it from the plugins repo or GitHub while it’s hot. Here’s some of the key updates:

    • BREAKING CHANGE: Fix Content-Disposition header parsing. This technically breaks backwards compatibility to correctly match the header specification. (#2239)

    • BREAKING CHANGE: Use compact links for embedded responses if they are available. We now use CURIEs for sites on 4.5+, which look like wp:term (but canonicalise to the full URI relation). (#2412)

    • Updated JS client to the latest version. (#2403)

    There’s lots more changes in this release; check out the release notes or the commits for this release.


    We’ve been thinking about how to tackle the API in the coming future. We want to do the most we can to ensure you can build sites with confidence.

    Along these lines, we’re going to release a 2.0 final version in the coming months. This will be a completely stable release with guaranteed backwards compatibility for the foreseeable future. This backwards compatibility ensures that your sites can remain up-to-date with minimal maintenance or issues with upgrading.

    We originally held the software in beta for a long period to ensure that breaking changes could be rolled in if deemed necessary to move the project forward. However, the majority of these breaks occurred at the start of the 2.0 lifecycle, and the API is mostly stable at this point. Keeping the ability to break compatibility benefits only us, whereas moving to a stable release benefits everyone.

    Moving forward, version 2.0 of the WP REST API will follow a normal project release cycle. We will have minor releases in the 2.x series as new features are added, and bugfix releases in the 2.0.x series.

    As for the core merge itself, we are not submitting a merge proposal of the core endpoints for WordPress 4.6. We believe endpoints for the main WordPress objects (posts, users, comments, terms, and taxonomies) are not enough to garner the support needed for the proposal to be accepted. Our hope is that with a stable version 2.0 release, we will attract our community members that have been waiting for the endpoints to be available in core, and submit a merge proposal for the WordPress 4.7 release cycle.

    In addition to attracting more developers within our community, we are also looking to get more contributors involved with the project. As noted in previous discussions, the four of us on the API team can’t keep pace with WordPress itself without help. We’re looking to get WordPress core component maintainers involved in their relevant components, as well as new developers from outside the project. Moving forward, the API team sees our role as advisory over the API itself, with the API treated as an integral part of the component rather than maintained by a separate team. We’re also going to continue to work on our feature plugins (metadata, site/multisite, menus/widgets, and authentication) in parallel, and are looking for help on these as well. (There’s also more news regarding authentication coming very soon.)

    If you’d like to get involved with the API, please let us know. You can comment here, ping us on Slack in the #core-restapi room, or via GitHub issues. We’re looking at spending significant time onboarding new users, so if you’d like to get involved, now’s the time! Our weekly meeting is at Monday 23:00 UTC

    Thanks for catching up with us, and have a wonderful day.

    With love,

    Ryan, Rachel, Daniel, and Joe

    • adamf321 4:19 pm on April 4, 2016 Permalink | Log in to Reply

      Hey guys,
      Firstly thanks for all the great work you’ve been producing on the API! I’m the CTO of a digital agency and we’re working on a new infrastructure which relies heavily on it. I would like for us to be involved in some part in the ongoing development of the API – so please keep me up to date with the onboarding process and I will ask who in our team is interested. We have dev and design resources. To give you an early heads-up, this month is pretty hectic, but from next month I think we can make some time to get started.


    • Mike Nelson 4:40 pm on April 4, 2016 Permalink | Log in to Reply

      +1 to 2.0 stable. If we really really really want to make breaking changes in the future it can always be on a 3.0 version. For now it’s nice having something stable.

      And I’m at least interested in helping out with the basic auth plugin

    • nathangraham 10:25 pm on April 14, 2016 Permalink | Log in to Reply

      We rely heavily on the REST API as well so thanks for all of your work. I’m happy to contribute as well.

  • Ryan McCue 12:35 am on December 15, 2015 Permalink  

    WP REST API: OAuth Plugin Security Update 

    Hi everyone. This is a quick update on the OAuth 1.0a Server plugin, available on GitHub.

    Versions of the OAuth plugin prior to this commit contain a security issue during the authorization flow, regarding signature and nonce checks. Due to the OAuth architecture, it is highly unlikely this can be used to compromise a site or client application; however due to an abundance of caution, we recommend all users update to 0.2.1 immediately. (Pull the latest changes from master.)

    Thanks to @bradyvercher for responsible disclosure of this issue via HackerOne.

    • AITpro 12:54 am on December 15, 2015 Permalink | Log in to Reply

      Wow! HackerOne is awesome! What an absolutely brilliant idea. I will definitely be signing up.

  • Ryan McCue 9:52 am on December 7, 2015 Permalink
    Tags: , ,   

    WP REST API: New Tools & OAuth Updates 

    Hi everyone! I’m here today with some special news for everyone, rather than a standard release announcement. I have three things to announce instead. 🙂

    Discovery Library for PHP

    A super cool thing that you might not know about with the API is that it’s entirely discoverable. We use a relatively simple process modelled after Atom/RSS feed discovery. All you need to do is check a site’s headers for a Link header:

    Link: <http://example.com/wp-json/>; rel="https://api.w.org/"

    This allows a tonne of possibilities, including renaming the API base, or even delegating API access to someone else. (For example, WordPress.com could use https://api.wordpress.com/{site_id}/ instead of per-site APIs.)

    However, this isn’t always the easiest process to use (even if you have an advanced degree in hypothetical topology). To simplify this, we now have a discovery library for PHP 5.3+.

    To use it, add wp-api/discovery to your Composer requirements manually or run composer require wp-api/discovery in your project directory. You can then use the WordPress\Discovery\discover() function to discover the API for a given URL. (Oh hey, namespaces!)

    The library also includes a demo that you can run using PHP’s built-in server, and we’re working on getting a public version of this up on wp-api.org so you don’t even need to install it.

    This should simplify the discovery process for everyone, and I’m hoping some wonderful folks in the community will help port this to other languages as well. (Hint hint.)

    New OAuth Server

    One of the weakest points of developing with the API right now is getting authentication working. For a long time, our OAuth server plugin has languished and fallen behind as we push forward with the API. No longer is this the case!

    Simply update to the latest master. This will also be available on WordPress.org very soon.

    This new version of the OAuth server has a bunch of changes, including:

    • Full admin UI, including application management and the ability to revoke tokens
    • Ability to delete applications or regenerate their secrets
    • Callback validation process overhaul, now supporting custom protocols
    • Overhauled internals

    In addition to this, I wanted to address one of the biggest problems with the OAuth process: documentation. The documentation for the OAuth server and process has been terrible in the past. Previously, I’d indicated to people that they should just go read the generic OAuth 1.0a documentation on the web and try with that, but as it turns out, there’s no good documentation on this. (Apologies to those of you who’ve struggled in the past.)

    Thankfully, this is now fixed, with a guide included and available on Gitbook. This is an initial version of the guide, and I imagine it’ll grow and improve further in the future.

    If there’s anything missing you’d like added or improved, file an issue on the plugin repo. (The guide’s source is also included with the plugin.)

    Demo API Client

    (pause for breath)

    Combining all these pieces, there’s now a demo API client in PHP that you can download and run to try all of this out. The demo client uses the discovery library to find your site, then connects with OAuth to display your user details. Again, you can run this locally using PHP’s built-in server, and we’re working on getting a version set up on wp-api.org too.

    Internally, this uses the wonderful OAuth 1 client library by The League of Extraordinary Packages. This repo also acts as a provider for that library, allowing you to use the library in your own code to handle OAuth.

    (Both the discovery library and the demo client are open source and MIT licensed, allowing you to use them in commercial projects should you so desire.)

    Hopefully you find these pieces useful. As always, your feedback on all of this is much appreciated; we’d also love to see discovery libraries and demo clients in other languages, if anyone wants to port them across. Thanks for being great.

    • jeffikus 10:41 am on December 7, 2015 Permalink | Log in to Reply

      This is awesome, the last piece I’ve been hoping would come out soon 🙂 Major kudos for this, quick question though, I see the demo API client is in PHP, any plans for a JS demo?

      • jeffikus 10:42 am on December 7, 2015 Permalink | Log in to Reply

        Oh gosh, I just re-read the last paragraph, ignore me 🙂 i’ll try get a JS version together and post it here 🙂

    • Justin Greer 1:37 pm on December 7, 2015 Permalink | Log in to Reply

      Super excited! I am looking forward to extensions like OpenID Connect and Discovery.

    • Ahmad Awais 2:19 pm on December 7, 2015 Permalink | Log in to Reply

      Awesome sauce! Shit, I wish I could take a holiday and read the source, oh wait, I can! Goes to read the source.

    • Collizo4sky 3:12 pm on December 7, 2015 Permalink | Log in to Reply

      Great job guys. Excited to see some PHP league package get some love in the WordPress community.
      Why reinvent the wheel when it isn’t broken 😀

      Keep up the good job Ryan.

    • Joseph Scott 3:17 pm on December 7, 2015 Permalink | Log in to Reply

      it’s entirely discoverable … including renaming the API base, or even delegating API access to someone else

      Historically this hasn’t worked out that well. I’d recommend shouting this super loud in every code example and article that deals with this API if you want to overcome the lazy momentum of clients assuming the URL will always be DOMAIN/wp-json/

      • Mike Nelson 5:52 pm on December 7, 2015 Permalink | Log in to Reply

        +1 to Joseph’s comment, especially if someone wants to run v1 of the API alongside v2, in which case they would need to have one of the APIs running on a different API base.

        And +1 to these advancements in general!

        • Ryan McCue 11:59 pm on December 7, 2015 Permalink | Log in to Reply

          Fingers crossed we’ll eventually have a 1.9 that has v1 endpoints on v2 infrastructure. 🙂

          • Mike Nelson 6:50 pm on December 8, 2015 Permalink | Log in to Reply

            huh ya that could be nice. I suppose it depends on how well folks start to adopt using v2. If lots of people stay on v1 then it might be nice

      • Ryan McCue 11:58 pm on December 7, 2015 Permalink | Log in to Reply

        > Historically this hasn’t worked out that well

        Agreed, though it has worked well for feed autodiscovery, so I’m optimistic. 🙂

  • Ryan McCue 1:27 am on November 18, 2015 Permalink

    WordPress Importer Redux 

    Hi, I’m Ryan McCue. You may remember me from such projects as the REST API.

    I’m here today to talk about something a bit different: the WordPress Importer. The WordPress Importer is key to a tonne of different workflows, and is one of the most used plugins on the repo.

    Unfortunately, the Importer is also a bit unloved. After getting immensely frustrated at the Importer, I figured it was probably time we throw some attention at it. I’ve been working on fixing this with a new and improved Importer!

    If you’re interested in checking out this new version of the Importer, grab it from GitHub. It’s still some way from release, but the vast majority of functionality is already in place. The plan is to eventually replace the existing Importer with this new version.

    The key to these Importer improvements is rewriting the core processing, taking experience with the current Importer and building to fix those specific problems. This means fixing and improving a whole raft of problems:

    • Way less memory usage: Testing shows memory usage to import a 41MB WXR file is down from 132MB to 19MB (less than half the actual file size!). This means no more splitting files just to get them to import!
    • Faster parser: By using a streaming XML parser, we process data as we go, which is much more scalable than the current approach. Content can begin being imported as soon as the file is read, rather than waiting for pre-processing.
    • Resumable parsing: By storing more in the database instead of variables, we can quit and resume imports on-the-go.
    • Partial imports: Rethinking the deduplication approach allows better partial imports, such as when you’re updating a production site from staging.
    • Better CLI: Treating the CLI as a first-class citizen means a better experience for those doing imports on a daily basis, and better code quality and reusability.

    Curious as to how all of this is done? Read on!

    (More …)

    • pavot 1:36 am on November 18, 2015 Permalink | Log in to Reply

      I can’t express how happy I am about this progress!
      Thank you very much.

    • Jon Brown 1:54 am on November 18, 2015 Permalink | Log in to Reply

      Ditto. This is awesome. Having spent a lot of hours last weekend trying to debug a massive wp.com WXR/XML export (split in over 20 parts and failing on files 14-16)… this is so timely. Looking forward to testing it.

      Curious if the pull parser works with a folder full of xml files the way the WP importer did? Since as I said above, what .com dumped was a couple dozen xml files and I don’t have a convenient way to glue them back together.

      Huge thank you for attending to a long neglected corner of WordPress!

      • Ryan McCue 1:56 am on November 18, 2015 Permalink | Log in to Reply

        Curious if the pull parser works with a folder full of xml files the way the WP importer did?

        You should be able to work with split files by just importing them one after another. (You should also be able to import them in any order, theoretically.) This is something I haven’t put a lot of testing into yet though, so give it a shot and let me know!

        In the future, hopefully we’ll never need to split files again. 🙂

      • Brent Toderash 6:05 am on November 19, 2015 Permalink | Log in to Reply

        If you’re up for an adventure, on the command line you could do something like

        cat $(ls *.xml -t) > complete.xml

        to stitch the files back together in date order, then try importing the whole thing at once 🙂

    • Sakin Shrestha 2:12 am on November 18, 2015 Permalink | Log in to Reply

      Wow this is awesome… Thanks a lot 🙂

    • nicholas_io 2:13 am on November 18, 2015 Permalink | Log in to Reply

      This is Awesome! The WordPress importer as it currently stands is far from being good. I’ll be happy testing it out

    • Mike 2:20 am on November 18, 2015 Permalink | Log in to Reply

      Great stuff Ryan. Really nice performance improvements. I’ve often ran into memory limits during import.

      As an aside, has there been discussion around offline import, most specifically around media items? Such a possibility would allow for sites to be exported wholly together with all of their content items and either kept for archive or imported more easily. Such a site could have been mocked up on an unreachable development server.

      • Ryan McCue 2:24 am on November 18, 2015 Permalink | Log in to Reply

        Thanks Mike!

        We have an issue filed for that one. Deferring the actual images getting imported will get a few gains: offline imports, parallel downloads, or static content imports (i.e. image files in a zip).

        You can pass 'fetch_attachments' => false (also the default internally, but not in the CLI) into the importer class in __construct for this already. 🙂

    • Ryan Markel 2:25 am on November 18, 2015 Permalink | Log in to Reply

      Hey, Ryan.

      This is other Ryan. 🙂 We’ve been dealing with super-large imports on WordPress.com VIP for some time now and I’ve been dealing with them for a year-plus. I’d love to chat with you at some point about the kinds of things we have run into in these large imports and in other things that are edgier-case like site merges and other things that the import currently can struggle with.

      Some of what we are doing on WordPress.com is almost certainly not practical for movement to core, but we can probably provide some valuable insight on larger imports and problems that are encountered there.

      • Ryan McCue 2:49 am on November 18, 2015 Permalink | Log in to Reply

        Hey other Ryan! Let’s chat; one of the big use cases that lead to me rewriting this was huge VIP-level imports, so it’s definitely something I’m conscious of. I’d love to have your and VIP’s feedback 🙂

    • Ipstenu (Mika Epstein) 2:43 am on November 18, 2015 Permalink | Log in to Reply

      This will be a game changer for the many people who want to export from wordpress.com and import to self-hosted where they either can’t afford or aren’t able to use the migration package.

      You, sir, deserve all the pizza bites.

    • Matthew Eppelsheimer 3:15 am on November 18, 2015 Permalink | Log in to Reply

      This is so, so, great Ryan!

      Looking forward to testing this and seeing if our pet bugs in the 0.6 importer to fix “someday” still exist. I’ll bet not…

    • JakePT 3:38 am on November 18, 2015 Permalink | Log in to Reply

      My biggest frustration is the inability to import Posts only and import images at the same time. Images only come across if the entire site is exported. This is very annoying when trying to transfer a client’s blog between sites without bringing across menus and random plugin post types. I can understand the challenge since they’re only img tags and not necessarily attached to the post, but it’s definitely my biggest hangup with the importer.

    • bishless 4:59 am on November 18, 2015 Permalink | Log in to Reply

      This is great news! Thanks for your work, Ryan!

    • jeffmcneill 5:43 am on November 18, 2015 Permalink | Log in to Reply

      I’ve run into a ton of problems with importing, usually a memory or parsing issue. I’ve had to eject a lot of content from several site migrations because of this. That you are working on this critical tool for WordPress means you should be applauded, and lauded. Thank you!

    • DeBAAT 6:26 am on November 18, 2015 Permalink | Log in to Reply

      Reading the responses, you’ve definitely hit a soft spot!
      Thanks already for picking this challenge.

      I myself did have some issues with the importer lately as I was migrating some content from dev to prod.
      And this actually had to do with the GUID, so please take real good care to using this.
      I mean, a post developed on site x.dev typically has x.dev in the guid. When I try to import it into x.com, I would like to have an option whether or not to ‘translate’ the GUID from x.dev to x.com.

      The same goes for the GUID of an image. In some cases, the image may have been uploaded already in the x.com site before. So how do you discriminate between the two images?
      Hope you’ll get it working nicely.

      Reading your article, I thought you mentioned something about changing the exporter as well. I’m not sure however, so are there any plans to work on the exporter as well?

      • Ryan McCue 6:47 am on November 18, 2015 Permalink | Log in to Reply

        The main thing with the GUID is that it shouldn’t ever change. The new importer actually uses it more, so if you change the GUIDs, later imports will reimport the same data again. That said, @westonruter‘s pull request would let you write complex logic to map this back and forth, and I do know it’s a common use case. Happy to discuss further on a GH issue 🙂

        Specifically on images, there’s a flag in the options in the new importer called update_attachment_guids – the old importer always changes the GUID to the new filename, but this rewrite avoids that by default, as it makes deduplication on future imports impossible. (The reason the old importer did this is because originally the GUID contained the filename, so that’s occasionally used by older code to find the image URL.)

        There’s no real plans to change the exporter right now, but if it turns out we can improve import efficiency by changing data in exports (i.e. avoid post-processing by being careful about the element order), we’ll likely do that.

        Thanks for the feedback!

    • Ahmad Awais 6:38 am on November 18, 2015 Permalink | Log in to Reply

      I’ve had my fair share of WTF Moments™ with WordPress importer. So, believe me when I say this, I am glad someone is finally working on improving that. It’s a great news. Going to test it this weekend.

    • Sam Hotchkiss 6:51 am on November 18, 2015 Permalink | Log in to Reply

      You’re amazing, Ryan, well done.

    • Per Soderlind 6:53 am on November 18, 2015 Permalink | Log in to Reply

      Excellent Ryan, can’t wait to test it 🙂

    • Dreb Bits 7:30 am on November 18, 2015 Permalink | Log in to Reply

      Truly amazing! Will find time this weekend to test this baby! And thanks a lot initiating the movement to improve the WordPress importer 🙂

    • Ajay 7:52 am on November 18, 2015 Permalink | Log in to Reply

      This is awesome news. It’s been a long time coming. Are you planning to update the Exporter as well?

      • Ryan McCue 7:54 am on November 18, 2015 Permalink | Log in to Reply

        No major plans there yet, but if there’s improvements to be made there, I’ll be sure to take a look. 🙂

    • Morten Rand-Hendriksen 8:00 am on November 18, 2015 Permalink | Log in to Reply

      Thank you Ryan. This is a sorely needed upgrade.

    • connectr 8:22 am on November 18, 2015 Permalink | Log in to Reply

      Ooh, goodie! It always baffled me why such an important tool received so little love. The GSoC project looked hopeful, but then not much came from it. Super duper happy it’s back on the radar 🙂

    • Mario Peshev 8:37 am on November 18, 2015 Permalink | Log in to Reply

      Thanks for the great work Ryan – given the state of the original Importer for years now, that’s incredible news.

    • Omaar Osmaan 8:38 am on November 18, 2015 Permalink | Log in to Reply

      So excited about this- 😀

    • dimitris33 9:42 am on November 18, 2015 Permalink | Log in to Reply

      Great news Thanks!

    • capuderg 10:10 am on November 18, 2015 Permalink | Log in to Reply

      Great!!! So excited! Thanks Ryan for working on this!

    • Tran Ngoc Tuan Anh 10:36 am on November 18, 2015 Permalink | Log in to Reply

      This is a great news.

      I’m wondering about media import. Some themes use licenced images and don’t want to import, they usually are replaced by dummy image as a placeholder. Can we do that (maybe with a filter) with this new importer?

      Also, would it help in importing theme mods?

      • Ryan McCue 10:39 am on November 18, 2015 Permalink | Log in to Reply

        You can filter `wxr_importer.pre_process.post`, which gets passed the full post data, and go ahead and change anything in there. 🙂

        I’m not sure what the current state of theme mods is in the export data, so I can’t speak to that, but file an issue and I can take a look. 🙂

    • Primoz Cigler 10:42 am on November 18, 2015 Permalink | Log in to Reply

      Wow, incredible! Just awesome Ryan. It is clear from responses here that this was a huge pain for many people.

      Something came to my mind when I was reading this, I am not sure if it will be relevant, but still worth mentioning: would it be possible to utilize the REST API and switch to JSON instead? I believe it would make for much cleaner approach, as I was always struggling with any XML files, while JSON is so human-readable? I believe the import/export would be possible without the XML file at all, only import WP calling the export WP over the JSON calls.

      I would love to hear the feedback on my thinking and if that is something that should be considered in the future when migrating content over the web, as it sounds much leaner approach 🙂

      • Ryan McCue 10:48 am on November 18, 2015 Permalink | Log in to Reply

        As the lead developer of the API, I’m conscious of this. 😉

        There’s a number of reasons it’s better to stick with WXR/XML right now:

        • PHP has a streaming XML parser built-in, but it doesn’t have a streaming JSON parser out of the box. This means any streaming parsing would need to take place in userland (i.e. a library), which I suspect would cause performance to be worse.
        • WXR is compatible with RSS, which means the export format can be imported by a lot of other tools, not just WP. Switching to our own format might have benefits to us, but it would likely harm the wider ecosystem.
        • XML isn’t really a problem most of the time. It’s not the greatest format ever invented, but it’s reasonably readable, and there’s tonnes of tooling around it.

        With that said, if somebody wanted to try it, it’d be interesting to see at the least. 🙂 The XML parsing in this new version of the importer is contained in a few small methods, so you could probably swap that out for JSON reasonably easily.

        • Primoz Cigler 10:52 am on November 18, 2015 Permalink | Log in to Reply

          Gotcha, thank you for your swift reply!

          I don’t promise anything, but I might actually try to implement the REST API way. I will give you heads-up if I do.

    • sonjanyc 11:23 am on November 18, 2015 Permalink | Log in to Reply

      This is amazing, Ryan!! You will make so many people so very happy with this, me included!! 🙂

    • Alvaro Gois dos Santos 11:28 am on November 18, 2015 Permalink | Log in to Reply

      Wow. Awesome @rmccue, thanks!

      Let’s try that importer…

    • Damian 12:55 pm on November 18, 2015 Permalink | Log in to Reply

      Thanks Ryan, this is problem we all faced at some point. My concern is the exporter now. It will be also updated ? Currently how the exporter work is primitive. Eg: You can’t export featured images if you just export posts, you need to export the whole content.

      Or if you want to export just taxonomies becuase you have lots of them and you want to duplicate them in a new site.

      Thanks again!

      • Ryan McCue 1:35 pm on November 18, 2015 Permalink | Log in to Reply

        Right now, I don’t have any plans to work on the exporter (unless there’s performance gains to be had for the importer there), however there’s a few tickets already on that: #27048 and #17379 are the two most relevant here I think.

        At the end of the day, these things just need someone to sit down and spend some time on them. 🙂

    • Rich Tabor 1:32 pm on November 18, 2015 Permalink | Log in to Reply

      Really glad this is happening!

    • Josh Eaton 2:57 pm on November 18, 2015 Permalink | Log in to Reply

      Wait, so you’re saying I no longer need 8GB of RAM in my local VM just to import a WXR file? 🎉

    • Steven Gliebe 3:21 pm on November 18, 2015 Permalink | Log in to Reply

      This is music to my hears. Thank you for working on this!

    • Ben Doherty (Oomph, Inc) 5:33 pm on November 18, 2015 Permalink | Log in to Reply

      This is great, I’m very excited to see this development. I wonder if parallelization is also something that’s been considered, as PHP is just SLOW when walking through large files, and I found that parallelization can greatly improve overall import times, especially when munging XML files. My quick elevator pitch is that the importer spins off M processes, 1..N, each of which imports every M+Nth post from the file in parallel, and then the post-cleanup operation (translating URLs, IDs, etc) runs when all of the processes have completed.

      • Ryan McCue 11:55 am on November 19, 2015 Permalink | Log in to Reply

        Parallelisation is definitely something in my mind, but given the complex way that the object cache and database interact in concurrent requests (i.e. concurrent saves break basically everything in WP), not something I’ve put a huge amount of time into. 🙂

        I’d love to see someone have a go at it though! The new importer should let you build this as a plugin on top reasonably easily, so if you want to try it… 😉

    • pathartl 6:12 pm on November 18, 2015 Permalink | Log in to Reply

      A much needed update. I opened up #30887 a while ago when I first noticed the GUID issue, in case you wanted another ticket to close :). Great work as always.

    • Shmoo 2:26 am on November 19, 2015 Permalink | Log in to Reply

      Ryan should get a WordPress company sports car for doing this! 🚗 💨 vroom vroom..

      Can’t wait to test this.. Did you the WP Importer yesterday -10MB file #drama

    • lepardman 9:00 am on November 19, 2015 Permalink | Log in to Reply

      Awesome! Just a thought: it would be nice to have the ability to remap post types (maybe even taxonomies?) on import.

      Example: yesterday I was trying to import posts from an old site to a new one I’m working on. The site was so old that the developers used posts as a way to store the clients products and their actual news posts were just added as text to a huge list in the editor on a page. Some years after, a new developer added a new post type called news since post were “taken”. So I figured that I at least could do an export of the lastest news and the products. Then import them and switch post types, news as post and the products (old post type post) into a new post type called products. But that wasn’t possible, I just got and error that the post type news didn’t exist and the import failed.

      TL;DR: I couldn’t remap posts on import to other (nowadays more correct) post types. A nice way would be to be able to assign them to post type(s) just as you can change author (I guess a bulk option would be needed). I’m unsure if this would be possible but I’m throwing out the idea and then let’s hear what you guys say.

      (I guess I could do a search and replace in the xml files or create the correspondent post type temporarily, import and then do a search and replace in db but it would be so much nicer to have this built in)

      Thank you!

    • chzumbrunnen 3:14 pm on November 19, 2015 Permalink | Log in to Reply

      Wow, Ryan, that sounds great. When I understand you right my biggest question is already somehow fixed.
      Normally for an import the old site hast to be still reachable to get the images. Now with the new importer it should be possible to import images manually. But I’d like to go one step further and be able to enter the path to the images folder from where the images should be imported.

      • Ryan McCue 12:36 am on November 20, 2015 Permalink | Log in to Reply

        The tooling around this stuff will need some time to develop and mature, but I’d also love to have that eventually 🙂 Right now, should be technically possible, but just a pain to do so.

    • DoodleDogCody 3:56 pm on November 19, 2015 Permalink | Log in to Reply

      In regards to Media. I have noticed that with the current importer media items are not imported unless we select export all from the exporter. This seems a little odd to me. Media items should be imported if they exists in a post, page, or whatever post type that we export as media items are not thought of as a seperate psot type to most people. I know that’s how they are stored in the database but If I am building a new site for a client and need to export and import only their posts on the current site. I want to click export post and choose the date range of all. Then when I import I expect the images to import as well.

    • Ella Iseulde Van Dorpe 5:23 pm on November 19, 2015 Permalink | Log in to Reply


    • Samuel Wood (Otto) 12:33 am on November 20, 2015 Permalink | Log in to Reply

      Awesome. I’m gonna steal your XML code. 🙂

    • Michael Ecklund 8:02 pm on November 20, 2015 Permalink | Log in to Reply

      Thank you for taking the time to update the importer. It’s been begging for attention for quite some time.

    • RobbyMcCullough 7:42 pm on November 25, 2015 Permalink | Log in to Reply

      This is great news. Thanks, Ryan. Really excited to hear about this project.

    • Caleb Evans 1:06 am on November 29, 2015 Permalink | Log in to Reply

      Hey, @rmccue—very happy to hear that the Importer is getting some much-needed love. Do you have any plans to get the Importer merged into Core, or will it remain a separate plugin? I find it somewhat counterintuitive that WordPress can export its own content out of the box, but cannot import that exported content without a separate plugin. It’s not too big of a deal—just one less plugin to install for new installations—but it’s certainly something that would be nice to have, and it seems like the sensible thing to do.

    • pipdig 7:04 pm on November 29, 2015 Permalink | Log in to Reply

      Ryan, you have been added to my Christmas card list!

    • Caleb Evans 4:23 am on November 30, 2015 Permalink | Log in to Reply

      Hi, @rmccue. This all looks great, and it has me very excited. Out of curiosity, do you have any plans on trying to get this merged into Core? I find it strange that WordPress can export its own content, yet it cannot import that same exported content without a plugin. Thoughts?

      • Ryan McCue 4:27 am on November 30, 2015 Permalink | Log in to Reply

        No plans right now to integrate this into core. I think it makes more sense as a plugin personally. The export being built-in is fundamentally about your data being portable and free to access, whereas the importer doesn’t really affect data portability or freedom. Plus, having it as a plugin means we can improve it faster 🙂

        (That said, others may disagree, so might be something to discuss further!)

    • Tevya 7:19 pm on January 13, 2016 Permalink | Log in to Reply

      So what’s the status on this currently? Is it ready for general use? We’ve been having the old importer fail to bring over images and media, even though the box is checked. Hoping this can save us.

    • helmar 6:34 am on February 13, 2016 Permalink | Log in to Reply

      Sounds super promising. Hope to read more about this soon, even have something to work with. If financial support is needed, I’m in.

    • birdrockdesigns 9:39 am on March 9, 2016 Permalink | Log in to Reply

      I’m really, really hoping you will be successful with the repo.

  • Ryan McCue 6:30 am on October 28, 2015 Permalink
    Tags: , , ,   

    REST API: Welcome the Infrastructure to Core 

    Hi from the REST API team! We’re extremely excited to announce the API infrastructure has now been merged into core as of r34928 (plus a couple of fix up commits we won’t mention). Huzzah!

    Sincere thanks to every single one of the contributors, we wouldn’t be where we are today without you. It takes time and effort to produce great things, and it’s impossible to make things great without everyone helping. This has been a truly collaborative effort, and I wish I could do more than just give you props.

    (Important note: if you have a 2.0 beta already installed, you must upgrade to beta 5.)

    What’s included in 4.4?

    As mentioned in the merge proposal, the API comes in two parts: infrastructure and endpoints. In 4.4, the infrastructure is now available as part of core, while the endpoints continue to only be available in the plugin.

    You can think of the infrastructure as an “API construction kit”. WordPress 4.4 will make it possible for everyone to build RESTful APIs in a much easier fashion, which will benefit people building custom APIs for their site. The infrastructure handles the routing, argument handling, JSON serialisation/deserialisation, status codes, and all that other lovely REST stuff.

    For client authors, this doesn’t help you much right now. Hold tight, the team is working as fast as we can on the endpoints to get them ready for a future release. In the meantime, you can make sure sites you want to work with have the plugin installed, which isn’t a change from the current state.

    For plugin and theme authors, you can start building new APIs immediately using the infrastructure now in core. This can start replacing your existing custom admin-ajax endpoints or other bespoke code you already have.

    To authenticate with the API, only built-in cookie authentication is available out of the box right now. The OAuth 1 plugin will continue to work with WP 4.4 and the API plugin, as will the Basic Auth plugin for local development.

    It’s super easy to get started, and there’s even a guide available to kick-off. (Note: the WP_REST_Controller class is not included in WordPress 4.4.) This documentation will be migrated across to developer.wordpress.org soon.

    If you want to access any of the built-in data in WordPress without building it out yourself, you’ll need the endpoints as well. These will continue to be packaged in plugin form, and version 2.0 final will be released to accompany 4.4 before the end of this cycle.

    What if I’m using the API already?

    If you’re on version 2 of the API, you’ll need to update the API to beta 5 or later before updating to the latest version of core. This new version will use the new infrastructure in core if available, or fallback to a compatibility library otherwise.

    Important note: Earlier 2.0 betas (1 through 4) are incompatible with WP 4.4. Your site will fatal error if you don’t upgrade to beta 5 or later. You must upgrade to the latest API to run trunk and to run WP 4.4 when it’s released.

    If you’re on version 1 of the API, you won’t hit any fatal errors straight away, but endpoints will stop working with 4.4. We’re still planning on releasing a final version 1 release for compatibility, but now would be a great time to consider migrating forward to version 2. Apart from security releases, version 1 has ceased being actively maintained.

    Looking forward for the API

    Now that the API is past this first hurdle, it’s important to keep looking forward. Our immediate next step is to improve and polish the endpoints for phase two of our merge. There’s a lot of work still to be done here, so we’d love you to join us on GitHub.

    The infrastructure of the API will now be maintained via Trac, so new issues and patches should be sent there instead under the “REST API” component. Issues with endpoints should still be filed on GitHub. Don’t worry if you’re not sure; you can file issues on either Trac or GitHub, and they’ll be triaged into the correct place as needed. (It’s more important to make sure the issue is filed in the first place!)

    The team wants to keep pressing forward with the API and keep up our rate of progress, if not improve it even further, and we’d love your help. We still need help from content writers on our documentation, designers and developers on our authentication plugins, and developers on the endpoints. If you want to help, we can always use a hand, and we’d love to help get you started. We’re available in the #core-restapi room on Slack, and we’d love to see you at the weekly meeting at Monday 23:00 UTC 2015.

    We look forward to continuing to work on the API and getting these endpoints happening. Thanks again to everyone who got us here.

    (Then again, maybe a REST API is more of a Shelbyville idea…)

    • Ahmad Awais 9:20 am on October 28, 2015 Permalink | Log in to Reply

      It’s a new day 🙂 Looking forward to how REST API transforms WordPress as a platform and in turn the WWW.

    • Ahmad Awais 9:33 am on October 28, 2015 Permalink | Log in to Reply

      It’s a new day! Excited to see how REST API helps WordPress grow as an ultimate go-to platform for all the web development needs.

    • Manny Fleurmond 11:36 am on October 28, 2015 Permalink | Log in to Reply

      Are the models/collections/etc part of this or do those come later?

    • Stephane Daury (stephdau) 2:44 pm on October 28, 2015 Permalink | Log in to Reply

      Congrats on a monumental effort by the whole team.

    • Justin Sternberg 3:56 pm on October 28, 2015 Permalink | Log in to Reply


    • Mahesh Waghmare 3:57 pm on October 28, 2015 Permalink | Log in to Reply


    • karlazz 8:33 pm on October 28, 2015 Permalink | Log in to Reply


    • Jon Brown 11:25 pm on October 28, 2015 Permalink | Log in to Reply

      A whole new world awaits… thank you to the tireless devs working on this the last couple years!

    • Scott Bolinger 4:59 pm on October 29, 2015 Permalink | Log in to Reply

      Congrats guys! Is the wp-api 1.x plugin compatible with WordPress 4.4?

      • Aaron Jorbin 6:32 pm on October 29, 2015 Permalink | Log in to Reply

        As noted in the post:

        If you’re on version 1 of the API, you won’t hit any fatal errors straight away, but endpoints will stop working with 4.4. We’re still planning on releasing a final version 1 release for compatibility, but now would be a great time to consider migrating forward to version 2. Apart from security releases, version 1 has ceased being actively maintained.

    • Ian Dunn 11:02 pm on October 30, 2015 Permalink | Log in to Reply

      We’re still planning on releasing a final version 1 release for compatibility, but now would be a great time to consider migrating forward to version 2

      I’m still not clear on the upgrade path for sites running the 1.x plugin in production. I don’t really want to start running v2 on production until it’s stable, and I also don’t want to spend a lot of time upgrading my customizations until v2 is stable, in case anything changes.

      So, if I upgrade to 4.4 and the final 1.x plugin at the same time, will everything continue working? Or, do I need to wait until v2 is stable, then upgrade my customizations to work with it, and then install 4.4?

      • Joe Hoyle 1:22 am on November 14, 2015 Permalink | Log in to Reply

        The latest version of the 1.x plugin will continue to work with 4.4 – just make sure you update the plugin when you upgrade to 4.4.

        Right now, the 1.x plugin essentially overwrites the v2 infrastructure that is in core, so that does mean that you won’t be able to take advantage of the REST API infrastructure bundled in core if you are using v1. This will be resolved in a future update however.

    • suifengtec 4:25 pm on December 17, 2015 Permalink | Log in to Reply

      Good News!

compose new post
next post/next comment
previous post/previous comment
show/hide comments
go to top
go to login
show/hide help
shift + esc
Skip to toolbar