WordPress.org

Make WordPress Core

Updates from Ryan McCue Toggle Comment Threads | Keyboard Shortcuts

  • Ryan McCue 10:34 am on July 27, 2016 Permalink |
    Tags: , , http-api   

    HTTP API in 4.6 

    For WordPress 4.6, the HTTP API (wp_remote_request() and family) have undergone a large internal change. Rather than using a WordPress-specific HTTP library, WordPress now uses the open-source independent Requests library, developed by yours truly.

    Why Requests?

    The WP_HTTP library in previous releases has been primarily maintained by myself and @dd32, with my support time split between the two libraries. Both libraries are very similar, and code has been shared between them (when licensing permitted) in the past. Requests follows the same development philosophies as WordPress: developing for the masses with broad PHP support, and maintaining backwards compatibility.

    By switching to Requests, a library without any WordPress-specific dependencies, WordPress benefits from input from the wider PHP ecosystem and community

    Requests also has a huge number of unit tests, with test coverage at 92% of the codebase and increasing. It’s also used by other projects via the Composer ecosystem, including wp-cli’s HTTP functionality.

    What has changed?

    From your perspective as a developer, nothing should have changed visibly. You can and should continue to use WP’s HTTP functions the way you always have.

    (Note: For the 4.6 development cycle and beta 1, the HTTP functions returned an array-like object. Many plugins and themes in the real world are using direct is_array() checks, so it was decided to pull this functionality back a bit to be safe, see #37097.)

    Some new functionality has been introduced. In particular, the array returned from wp_remote_request() now includes a new http_response value, which contains a WP_HTTP_Response object (technically, WP_HTTP_Requests_Response). This shares functionality with WP_REST_Response objects introduced in WordPress 4.4, allowing common functionality to be developed for both APIs. In future releases, WordPress may introduce new WP_HTTP_Request objects as well, allowing common middleware to be used across both APIs.

    In addition, all of Requests’ features are now available in WordPress. This includes things like better HTTP standard support, case-insensitive headers, parallel HTTP requests, support for Internationalized Domain Names (like böcean901.ch), and many other internal improvements.

    Some new features are only available when using Requests::request() directly (such as parallel requests), however, these will be introduced into new WordPress-specific APIs in future releases (#37459). This release is focussing on switching internal implementation and remaining stable.

    For more background on the change, see #33055.

     
  • Ryan McCue 7:10 am on May 16, 2016 Permalink |
    Tags:   

    Rewrites Next Generation: Next Meeting 

    There was a bit of confusion with the last Rewrites meeting, so we never really got to kick off the conversation. Let’s give it another shot this week. The revised kick-off meeting for the Rewrites Next Generation project will hence be on May 18th 23:00 UTC.

    The agenda for this meeting is to discuss problems you see with the rewrites system, any particular use-cases you think should be considered, as well as agreeing on a broad approach for how we want to tackle the problems.

    (In case you missed it, you may want to read the introductory post for the project.)

    Hope to see you all there!

     
  • Ryan McCue 8:46 am on May 3, 2016 Permalink |
    Tags:   

    Proposal: Next Generation Rewrites 

    Hi everyone! Today I’d like to propose a new feature project for WordPress: Next Generation Rewrites. After proposing this at the last feature projects meeting, it’s time to flesh out the details of what this project would be.

    The aim of the project is to modernise the rewriting and routing system for the modern CMS and platform era of WordPress.

    (This project was previously proposed in a ticket on Trac, however the project is larger than a single Trac ticket and needs a larger discussion.)

    Overview

    If you’ve worked with custom development on WordPress, you’ve probably registered at least one rewrite rule. Rewrites are the key to what WordPress calls “pretty permalinks”, which are URLs designed for humans rather than the server. They look like /2016/04/19/, or /about/requirements/, rather than index.php?p=42.

    As a developer, you can register your own rewrite rules as well. For example, bbPress registers URLs like /forums/topic/jjj-is-awesome/ and WooCommerce has URLs like /product/gs3/. WordPress then “re-writes” these to the internal query variables (“query vars”), which are eventually passed into the main WP Query.

    The rewrite system was initially designed to implement pretty permalinks, but has been used and abused for so much more since then. With modern plugins and web apps built on top of WordPress, it’s not uncommon to have pages that don’t need a main query, or need a specialised system of querying. As an example of this, the REST API infrastructure added in WordPress 4.4 doesn’t use the main query, as many of the endpoints don’t require querying posts at all.

    While the usage of rewrites has developed, the internals of how URLs are handled internally hasn’t changed. The system is fundamentally still designed for the blog era of WordPress. It also predates a lot of the pieces of WordPress we take for granted, such as transients and object caching.

    Project Proposal

    It’s time to change the rewrite system. Rewrite rules should be changed from a layer on top of WP Query to a fully-fledged system all of their own. To do this, I’m proposing three initial steps:

    • Decouple rewrites from the query system
    • Refactor and rework the internals to make them testable
    • Rethink rewrite storage and remove flushing

    These steps would take place under the umbrella of a new feature project, Next Generation Rewrites. This feature project would coordinate the project and people working on it across the various parts of core it touches (primarily the Rewrite Rules component, but also potentially involving UI and UX discussion and changes; fixing the “visit the Permalinks page and save to fix your site” issue, for example). This would also coordinate the team across multiple release cycles as we slowly but surely make progress. It’s important to make progress while keeping rewrites stable, as they’re a critical part of WordPress that we cannot break.

    Decoupling Rewrites from Querying

    The first step I’m proposing is to decouple rewrites from querying. Right now, rewrites for anything except querying posts involves jumping through a few hoops. This includes registering temporary query vars, or using global state to track route matching. Separating these two components will make it easier to add non-query-based rewrites, as well as allowing more powerful query rewrites.

    Currently, rewrites are registered using code like the following:

    add_rewrite_rule( '/regex/', 'index.php?var=value' );

    These rewrite rules map a regular expression to a query string, which is later parsed into an array internally. You can achieve more complex routing via regular expressions by using a special syntax in the rewrite string:

    add_rewrite_rule( '/topic/(.+)', 'index.php?topic=$matches[1]' );

    Note that while this looks like a PHP variable, it’s actually a static string that gets replaced after the regular expression. This can lead to confusion with developers who are new to the syntax, and it also can make doing more complex rewrites hard. These typically involve temporary query vars, as well as quite a bit of processing.

    Instead, I want to introduce a new WP_Rewrite_Rule object to replace the current rewrite string. This object would contain a method called get_query_vars that receives the $matches variable and returns the query vars to set. This would look something like this:

    class MyRule extends WP_Rewrite_Rule {
        public function get_query_vars( $matches ) {
            return array(
                'post_type' => 'topic',
                'meta_query' => array(
                    array(
                        'key' => '_topic_name',
                        'value' => $matches[1],
                    ),
                ),
            );
        }
    }
    add_rewrite_rule( '/topic/(.+)', new MyRule() );

    (The exact format of the object is yet to be decided; we’ve also started discussing the possibility of passing a request object into this callback as well.)

    Using an object for this allows us to have multiple callbacks for different stages of the routing process, and starts to simplify some of the internal code in WP_Rewrite. For example, the routing code around pages that allows URLs like /parent/child/sub-child/ for child pages can be simplified and moved out of the main routing code. This also helps make the code more testable, which dovetails nicely with the third goal.

    Refactor rewrite internals

    Step 2 of changing rewrites is to make the rewrite system fully testable. Currently, a bunch of global state is mixed into the internals of rewrite matching, and the rewrite methods tend to be monolithic. @ericlewis has previously begun work on this in #18877, and this can be continued and rolled into the Next Generation Rewrites project.

    Ideally, this should happen at the same time or before the first step to allow easy checking of regressions. This is relatively boring work that won’t affect many developers, but it’s important we do it. The rewrite system is a critical part of WordPress, and it’s crucially important that it’s testable and verifiable.

    Rethink rewrite storage

    Once the first two steps are in place, we should begin reconsidering how rewrites are stored. Currently, rewrites are stored in the database in the rewrite_rules option. The option is less of an option (it’s not really configuration), and more of a caching technique. This is somewhat of a relic, as rewrites predate object caching and transients in WordPress, both of which are better techniques to handle caching in modern WordPress.

    Removing this option and changing it to use a proper caching subsystem in WordPress should fix multiple problems, including the need to flush rewrites. This should improve the UX for end users, as we should be able to remove the “links on your site don’t work, so go to the Permalinks page and save” trick (which only really requires visiting the page, not saving).

    The need to cache rewrites at all can also be reconsidered; many of the rewrite rules are simply entries in an array, and are not expensive to generate on-the-fly, while removing the caching so can make for an easier to use and more dynamic system. More expensive rules can be cached by the code generating the rule, rather than via the monolithic rewrite cache.

    This has the potential to improve the user experience and fix a class of issues, but it also has the potential to ruin performance, depending on how other code is using it. It’s important to tread carefully as we attempt to improve this, and ensure we remain compatible and performant. Introducing a two-tiered system is one approach we can consider, with the ability for plugins to opt-in to a newer, dynamic system to avoid problems with flushing.

    Why a Feature Project?

    I’m proposing Rewrites Next Generation as a feature project for several reasons. The concept and implementation are both large and complex, moreso than simply operating in an informal matter on Trac tickets. The impact of this project is also large, affecting many plugin and theme developers, as well as potential improvements to UX. Gaining feedback from all interested stakeholders is important, as this is a crucial part of WordPress.

    Getting Started

    The steps I’ve listed here are only a selection of the issues with rewrites. The rewrite system hasn’t been massively changed since its inception, so there are plenty of parts that could be changed to better suit the modern era of WordPress development. The issues here are simply the three most important that I’ve found, but I want to hear your feedback too.

    Let’s talk rewrites. I’m proposing a weekly meeting for Rewrites Next Generation, at Wednesday, 23:00 UTC with the first meeting at May 4th 23:00 UTC. If you’re interested in rewrites, or have had problems with them in the past, let’s talk and work out what we need to do to improve rewrites. (This is a non-permanent meeting to set the scope of the project; we’ll refine meeting frequency and timing at a later date.)

    After this initial discussion, we can settle on concrete goals and a timeline for the initial stages of the project. A proof-of-concept patch and ticket are available on Trac, however alternative approaches should be considered after this discussion. The short-term goal is to begin landing these improvements in trunk, with the goal of having our first changes in WordPress 4.6, which is a quick but achievable timeline.

    This project has the potential to make a large impact on developers and users, and it’s important that everyone has their say. I hope you’ll join me for the first meeting, and join in the fun of contributing to a key part of WordPress!

    Thanks for reading. <3

     
    • Omaar Osmaan 9:05 am on May 3, 2016 Permalink | Log in to Reply

      So excited for this!

    • Ryan Hellyer 9:28 am on May 3, 2016 Permalink | Log in to Reply

      A simpler system for adding new rewrite rules would be nice. The current system is unintuitive IMHO.

    • Liam Gladdy 9:47 am on May 3, 2016 Permalink | Log in to Reply

      Everything about this makes me happy! The initial meeting time doesn’t work well for me in the UK, but i’ll catch up the next day!

    • andreasnrb 11:21 am on May 3, 2016 Permalink | Log in to Reply

      Really interesting. The meeting time suggestion is really bad. Does not work very well for anyone in Europe really.

    • Takayuki Miyauchi 11:41 am on May 3, 2016 Permalink | Log in to Reply

      +1
      add_rewrite_rule( '/topic/(.+)', new MyRule() ); looks nice!

    • jagnew 1:15 pm on May 3, 2016 Permalink | Log in to Reply

      This would be great, I’ve actually been building a plugin to handle routing etc. based around what current PHP frameworks do these days. If its of any use (still writing the tests and docs for it): https://github.com/jasonagnew/WP-Router

      Example of using it:

      get('id') == 1)
              {
                  return 'No';
              }
      
              return $this->next($request);
          }
      }
      
      class Route_Test {
      
          protected $router;
      
          public function __construct()
          {
              global $wp_router;
      
              $this->router  = $wp_router;
      
              $this->register_routes();
          }
      
          public function register_protected_routes()
          {
              $this->router->get( array(
                  'as'   => 'getTest',
                  'uri'  => 'test/{id}',
                  'uses' => array( $this, 'get' )
              ) );
          }
      
          protected function register_routes()
          {
              $this->router->group( array(
                  'prefix' => '/protected',
                  'middlewares' => array( 'Middleware_One' ),
                  'uses' => array( $this, 'register_protected_routes' )
              ) );
      
              $this->router->post( array(
                  'as'   => 'postTest',
                  'uri'  => '/test/{id}',
                  'uses' => array( $this, 'post' ),
                  'prefix' => ''
              ) );
      
              $this->router->put( array(
                  'as'   => 'putTest',
                  'uri'  => '/test/{id}',
                  'uses' => array( $this, 'put' )
              ) );
          }
      
          public function get($id, WP_Request $request)
          {
              $all = $request->all();
      
              return new WP_JSON_Response($all);
          }
      
          public function post($id)
          {
              return 'POST: The ID is ' . $id;
          }
      
          public function put($id)
          {
              return 'PUT: The ID is ' . $id;
          }
      
      }
      
      new Route_Test();
      
    • Joost de Valk 2:22 pm on May 3, 2016 Permalink | Log in to Reply

      Love it. Would love to contribute but that meeting time is tough indeed.

      I’d want to add another thing to the mix: if we change rewrites, we also should consider creating a `get_current_url()` type function.

      I would also be very much in favor of caching URLs *very* aggressively, to the point of even giving them their own table. Right now, `get_permalink()` is one of the slowest things on many a page load…

    • Michael Ecklund 3:10 pm on May 3, 2016 Permalink | Log in to Reply

      Sounds interesting. Would definitely like to see improvement in this area for sure. Agreed with @joostdevalk.

    • Jacob Santos 5:07 pm on May 3, 2016 Permalink | Log in to Reply

      Composition over inheritance. It is easier to go to inheritance if composition is used first. Would help if used interfaces.

      I realize the least experienced don’t understand proper interface standards. I also realize there is a good argument for dynamic programming techniques.

      Since WordPress standardizes on neither, my preference would be proper OOP using interfaces to define a specification and Composition to tie the objects together.

      Whether the specification is defined as implicit with dynamic programming or explicit with interfaces, it would better serve then piling on the current OO mess that exists in WordPress currently.

    • Ryan McCue 12:41 am on May 4, 2016 Permalink | Log in to Reply

      Apologies on the meeting time for the Europeans, but there’s unfortunately no times that work well for everyone. Any earlier would push the boundary of whether I can make it (in Australia).

      I’ll consider having a secondary meeting for EU as well. 🙂

    • Ahmad Awais 2:57 am on May 4, 2016 Permalink | Log in to Reply

      Great stuff. Looking forward to it. Would love to contribute. WP most def needs a refresh to its routing sys.

    • schlessera 6:11 am on May 4, 2016 Permalink | Log in to Reply

      I agree with all of the issues you’re describing above. I cringe whenever I have to deal with rewrites, because in the end, it is an overly complex and fragile solution to a not-so-complex problem.

      I want to ask, though, if we are to go through the troubles of “rewriting” this part of the system and deal with all the BC headaches, why not go the whole nine yards? Instead of replacing the archaic special case system with a more modern special case system, why not create a modern general purpose system, and implement the existing rewrites as one special case of that general purpose system?

      You’ve probably guessed already that I’m talking about a full-blown routing system.

      If we define an interface (or an abstract base class if we absolutely must ignore best practices for the sake of “conformance”) called `WP_Route`, we can easily have one special case `WP_Rewrite_Rule_Route` that provides a completely BC way of implementing the current behaviour. But it allows us to have alternative routing behaviour that does not easily fit into the “rewriting” mechanism, too.

      So, the optimizations you’re talking about above are still a good idea, and should all be integrated into the `WP_Rewrite_Rule_Route`. But please don’t assume that we all just use “rewrites” for our routing purposes, when we have the opportunity to make an important change here. Assuming that all routing requests need to return query vars to pass to `WP_Query` is such an assumption. What if I want to have an URL that let’s me query custom external data? Why should this need to go through `WP_Query`?

      Most of what I’m talking about is already possible with the existing system, but it always feels dirty. I would love for WordPress to embrace a generalized approach (that can still default to special case implementations), so that I don’t constantly face situations where it feels like I’m working against the system.

    • schlessera 9:31 am on May 4, 2016 Permalink | Log in to Reply

      I thought some more about how a more generalized approach could look. I have to admit, though, that I’m not familiar with all of the intricacies and nuances that might come into play, so forgive me if anything I write here is just not possible.

      I imagine having WordPress pass the request to the routing system (let’s call it `WP_Router`). `WP_Router` will then do a minimum of parsing to be able to decide what specific routing action implementation it needs to call. Possible names of the interface would be `WP_Route` or `WP_Controller` (or anything along these lines). I’ll go with `WP_Controller` for the rest of the post to clearly differentiate it from `WP_Router`.

      The specific `WP_Controller` implementation is responsible for the main query. So, whatever `WP_Controller` you’re using, your widgets and navigation and so on should not be impacted.

      The `WP_Controller` does not return query vars or something in-between, it returns a completed “View” (the rendered content or archive, for example, or alternatively a standardized object).

      For the current default WP behaviour, we could have a `WP_Rewriting_Controller`, that collects the known rewriting rules, parses them for query vars, hands them over to `WP_Query` and returns the completed `View`.

      This would force us to have the individual elements that interact for such a request be decoupled from the overall WP lifecycle (which should mostly be the case, as far as I know).

      For any other `WP_Controller` implementation, you can freely choose what elements you need. You can, for example, completely forego `WP_Query`. The only contract that WP has with a `WP_Controller` is that it gets the request, and returns the response.

      It should be possible to build this entire system to be 100% BC, apart for the few plugins that directly mess with some data outside of the API. They will probably have plenty of time to adapt, though, as this is a bigger change anyway.

    • jeffmcneill 10:30 am on May 4, 2016 Permalink | Log in to Reply

      Not sure, but the whole issue of redirections might be something that this could include. The Redirection plugin recently was updated (after a few years) and it is a mainstay for managing the renaming of urls. Whenever there is a renaming of pages and reorganization of a site there are many of these redirection entries made to keep the 301 going properly. Not sure if a page-level thing like this should be dealt with at the same time as the much larger site-level redirection thing, but they do eventually lead to the same thing: a resolved url for page(s).

    • BenRacicot 4:10 pm on May 5, 2016 Permalink | Log in to Reply

      Awesome project! Would my ticket 34136 be applicable? https://core.trac.wordpress.org/ticket/34136 There seems to be a lot of devs looking for *much* better solutions / proper, core based solutions to CPT rewrites. Please see SO links within my trac 34136

  • Ryan McCue 9:23 pm on April 6, 2016 Permalink |
    Tags: , ,   

    REST API: Slashed Data in WordPress 4.4 and 4.5 

    Hi everyone. The REST API team recently discovered a bug with parameter parsing in the API infrastructure, part of WordPress 4.4. For those of you using the API infrastructure, you need to be aware of a bug fix we’re making with the API.

    The Problem

    The REST API has several types of parameters that it mixes together. These come from several sources including the request body as either JSON or URL-encoded form data ($_POST), query parameters ($_GET), the API route, and internally-set defaults. Unfortunately, due to an oversight on our behalf, these parameters can be inconsistently formatted.

    In WordPress, the superglobal request variables ($_POST and $_GET) are “slashed”; effectively, turning magic quotes on for everyone. This was originally built into PHP as a feature to help guard against SQL injection, but was later removed. Due to compatibility concerns, WP cannot change this behaviour for the superglobals. This only applies to the PHP superglobals, not to other sources of input like a JSON body or parameters in the URL. It additionally does not apply to form data on PUT or DELETE requests.

    Internally, some low-level WordPress functions expect slashed data. These functions internally call wp_unslash() on the data you pass in. This means input data from the superglobals can be passed in directly, but other data needs to be wrapped with a call to wp_slash().

    When the REST API gathers the data sources, it accidentally mixes slashed and unslashed sources. This results in inconsistent behaviour of parameters based on their source. For example, data passed as a JSON body is unslashed, whereas data passed via form data in the body is slashed (for POST requests).

    For example, the following two pieces of data are equivalent in the REST API:

    
    // JSON body:
    {"title": "Foo"}
    
    // Form-data ($_POST)
    title=Foo
    
    // Both result in:
    $request->get_param('title') === 'Foo';
    

    However, if the data contains slashes itself, this will be inconsistently passed to the callback:

    
    // JSON body:
    {"title": "Foo\Bar"}
    
    // Results in:
    $request->get_param('title') === 'Foo\Bar';
    
    // Form-data ($_POST) (%3D = "\")
    title=Foo%3DBar
    
    // Results in:
    $request->get_param('title') === 'Foo\\Bar';
    

    This means that callbacks need to understand where parameters come from in order to consistently handle them internally. Specifically:

    • Data passed in the query string ($_GET, $request->get_query_params()) is slashed
    • Data passed in the body as form-encoded ($_POST, $request->get_body_params()) is slashed for POST requests, and unslashed for PUT and DELETE requests.
    • Data passed in the body as JSON-encoded ($request->get_json_params()) is unslashed.
    • Data passed in the URL ($request->get_url_params()) is unslashed.
    • Data passed as a default ($request->get_default_params()) is unslashed.

    In addition, parameters set internally via $request->set_param() are unslashed. Unit and integration tests for API endpoints typically use these directly, so the majority of tested code (such as the WP REST API plugin) assumes parameters are unslashed.

    See the related Trac Ticket #36419 for more information.

    The Solution for WordPress 4.4 and 4.5

    We are regarding inconsistently-slashed data as a major bug, and are changing the API infrastructure to ensure unslashed data. This will ensure that data is consistent regardless of the source. Callbacks will now receive unslashed data only, and can rely on this regardless of the original data source or request method.

    If you are using functions that expect slashed data in your callback, you will need to slash your data before passing into these functions. Commonly used functions that expect slashed data are wp_insert_post, wp_update_post, update_post_meta, wp_insert_term, wp_insert_user, along with others. Before passing data into these functions, you must call wp_slash() on your data.

    The fix for this issue, will be included in the WordPress 4.5 release candidates and final release. Due to the severity of the bug, we are also backporting the fix to the next minor WordPress 4.4 update. This also ensures you can update your plugins can act consistently across all versions of the REST API.

    We understand that this may inadvertently break some plugins that are expecting slashed data. Right now, it’s not possible to consistently ensure that callbacks receive slashed data, so it is likely that these plugins will already break in some conditions.

    tl;dr: if you’re using wp_insert_* or *_post_meta in your REST API callback, you need to ensure you are calling wp_slash() on data you are passing in, regardless of source.

    We apologize for this bug existing in the first place. Slashed data is a problem that has plagued WordPress for a long time, and we’re not immune to getting caught by the issue ourselves.

     
    • Mike Schinkel 7:16 am on April 7, 2016 Permalink | Log in to Reply

      We apologize for this bug existing in the first place. Slashed data is a problem that has plagued WordPress for a long time, and we’re not immune to getting caught by the issue ourselves.

      Given that this issue created such a major bug, wouldn’t it be a good time to take backward compatible action to slowing start eliminating this as an issue?

      • Mike Schinkel 7:35 am on April 7, 2016 Permalink | Log in to Reply

        In case of interest in addressing this moving forward:

      • Ryan McCue 3:40 am on April 8, 2016 Permalink | Log in to Reply

        As it happens, this is part of my Secret Master Plan for Unslashing, and part of why I wanted to ensure we fixed this right now.

        My Secret Master Plan:

        • Take the generic parts of WP_REST_Request and make a new WP_HTTP_Request object
        • Ensure the Request object has unslashed data
        • Pass the Request around liberally to replace the superglobals (discussed somewhat in #36292)
        • Finally introduce internals that take unslashed data, and turn the existing functions into wrapper functions.
    • NateWr 7:54 am on April 7, 2016 Permalink | Log in to Reply

      Is this change in the latest WP 4.5 RC or do we need to get the latest commits to test our apps?

      • Marius (Clorith) 12:56 pm on April 7, 2016 Permalink | Log in to Reply

        It is not available in RC-1, that’s out at this time, as it was pushed in last evening/earlier today (depending on your time zones), the goal is to get RC-2 out some time this week, and it will include this change.

        If you are running trunk you’ll have access to it at this time, alternatively you can manually apply the patch to your own install if you wish to start testing it right away (the patch from #36419)

  • Ryan McCue 4:39 am on April 4, 2016 Permalink |
    Tags: ,   

    WP REST API: 2.0 Beta 13 & Roadmap 

    Hi folks! I’m here with another exciting update from the API team.

    Beta 13

    First off, we’re excited to announce 2.0 Beta 13 “yoink.adios\losers” is now available. Grab it from the plugins repo or GitHub while it’s hot. Here’s some of the key updates:

    • BREAKING CHANGE: Fix Content-Disposition header parsing. This technically breaks backwards compatibility to correctly match the header specification. (#2239)

    • BREAKING CHANGE: Use compact links for embedded responses if they are available. We now use CURIEs for sites on 4.5+, which look like wp:term (but canonicalise to the full URI relation). (#2412)

    • Updated JS client to the latest version. (#2403)

    There’s lots more changes in this release; check out the release notes or the commits for this release.

    Roadmap

    We’ve been thinking about how to tackle the API in the coming future. We want to do the most we can to ensure you can build sites with confidence.

    Along these lines, we’re going to release a 2.0 final version in the coming months. This will be a completely stable release with guaranteed backwards compatibility for the foreseeable future. This backwards compatibility ensures that your sites can remain up-to-date with minimal maintenance or issues with upgrading.

    We originally held the software in beta for a long period to ensure that breaking changes could be rolled in if deemed necessary to move the project forward. However, the majority of these breaks occurred at the start of the 2.0 lifecycle, and the API is mostly stable at this point. Keeping the ability to break compatibility benefits only us, whereas moving to a stable release benefits everyone.

    Moving forward, version 2.0 of the WP REST API will follow a normal project release cycle. We will have minor releases in the 2.x series as new features are added, and bugfix releases in the 2.0.x series.

    As for the core merge itself, we are not submitting a merge proposal of the core endpoints for WordPress 4.6. We believe endpoints for the main WordPress objects (posts, users, comments, terms, and taxonomies) are not enough to garner the support needed for the proposal to be accepted. Our hope is that with a stable version 2.0 release, we will attract our community members that have been waiting for the endpoints to be available in core, and submit a merge proposal for the WordPress 4.7 release cycle.

    In addition to attracting more developers within our community, we are also looking to get more contributors involved with the project. As noted in previous discussions, the four of us on the API team can’t keep pace with WordPress itself without help. We’re looking to get WordPress core component maintainers involved in their relevant components, as well as new developers from outside the project. Moving forward, the API team sees our role as advisory over the API itself, with the API treated as an integral part of the component rather than maintained by a separate team. We’re also going to continue to work on our feature plugins (metadata, site/multisite, menus/widgets, and authentication) in parallel, and are looking for help on these as well. (There’s also more news regarding authentication coming very soon.)

    If you’d like to get involved with the API, please let us know. You can comment here, ping us on Slack in the #core-restapi room, or via GitHub issues. We’re looking at spending significant time onboarding new users, so if you’d like to get involved, now’s the time! Our weekly meeting is at Monday 23:00 UTC

    Thanks for catching up with us, and have a wonderful day.

    With love,

    Ryan, Rachel, Daniel, and Joe

     
    • adamf321 4:19 pm on April 4, 2016 Permalink | Log in to Reply

      Hey guys,
      Firstly thanks for all the great work you’ve been producing on the API! I’m the CTO of a digital agency and we’re working on a new infrastructure which relies heavily on it. I would like for us to be involved in some part in the ongoing development of the API – so please keep me up to date with the onboarding process and I will ask who in our team is interested. We have dev and design resources. To give you an early heads-up, this month is pretty hectic, but from next month I think we can make some time to get started.

      Cheers,
      Adam

    • Mike Nelson 4:40 pm on April 4, 2016 Permalink | Log in to Reply

      +1 to 2.0 stable. If we really really really want to make breaking changes in the future it can always be on a 3.0 version. For now it’s nice having something stable.

      And I’m at least interested in helping out with the basic auth plugin

    • nathangraham 10:25 pm on April 14, 2016 Permalink | Log in to Reply

      We rely heavily on the REST API as well so thanks for all of your work. I’m happy to contribute as well.

  • Ryan McCue 12:35 am on December 15, 2015 Permalink  

    WP REST API: OAuth Plugin Security Update 

    Hi everyone. This is a quick update on the OAuth 1.0a Server plugin, available on GitHub.

    Versions of the OAuth plugin prior to this commit contain a security issue during the authorization flow, regarding signature and nonce checks. Due to the OAuth architecture, it is highly unlikely this can be used to compromise a site or client application; however due to an abundance of caution, we recommend all users update to 0.2.1 immediately. (Pull the latest changes from master.)

    Thanks to @bradyvercher for responsible disclosure of this issue via HackerOne.

     
    • AITpro 12:54 am on December 15, 2015 Permalink | Log in to Reply

      Wow! HackerOne is awesome! What an absolutely brilliant idea. I will definitely be signing up.

  • Ryan McCue 9:52 am on December 7, 2015 Permalink
    Tags: , ,   

    WP REST API: New Tools & OAuth Updates 

    Hi everyone! I’m here today with some special news for everyone, rather than a standard release announcement. I have three things to announce instead. 🙂

    Discovery Library for PHP

    A super cool thing that you might not know about with the API is that it’s entirely discoverable. We use a relatively simple process modelled after Atom/RSS feed discovery. All you need to do is check a site’s headers for a Link header:

    Link: <http://example.com/wp-json/>; rel="https://api.w.org/"

    This allows a tonne of possibilities, including renaming the API base, or even delegating API access to someone else. (For example, WordPress.com could use https://api.wordpress.com/{site_id}/ instead of per-site APIs.)

    However, this isn’t always the easiest process to use (even if you have an advanced degree in hypothetical topology). To simplify this, we now have a discovery library for PHP 5.3+.

    To use it, add wp-api/discovery to your Composer requirements manually or run composer require wp-api/discovery in your project directory. You can then use the WordPress\Discovery\discover() function to discover the API for a given URL. (Oh hey, namespaces!)

    The library also includes a demo that you can run using PHP’s built-in server, and we’re working on getting a public version of this up on wp-api.org so you don’t even need to install it.

    This should simplify the discovery process for everyone, and I’m hoping some wonderful folks in the community will help port this to other languages as well. (Hint hint.)

    New OAuth Server

    One of the weakest points of developing with the API right now is getting authentication working. For a long time, our OAuth server plugin has languished and fallen behind as we push forward with the API. No longer is this the case!

    Simply update to the latest master. This will also be available on WordPress.org very soon.

    This new version of the OAuth server has a bunch of changes, including:

    • Full admin UI, including application management and the ability to revoke tokens
    • Ability to delete applications or regenerate their secrets
    • Callback validation process overhaul, now supporting custom protocols
    • Overhauled internals

    In addition to this, I wanted to address one of the biggest problems with the OAuth process: documentation. The documentation for the OAuth server and process has been terrible in the past. Previously, I’d indicated to people that they should just go read the generic OAuth 1.0a documentation on the web and try with that, but as it turns out, there’s no good documentation on this. (Apologies to those of you who’ve struggled in the past.)

    Thankfully, this is now fixed, with a guide included and available on Gitbook. This is an initial version of the guide, and I imagine it’ll grow and improve further in the future.

    If there’s anything missing you’d like added or improved, file an issue on the plugin repo. (The guide’s source is also included with the plugin.)

    Demo API Client

    (pause for breath)

    Combining all these pieces, there’s now a demo API client in PHP that you can download and run to try all of this out. The demo client uses the discovery library to find your site, then connects with OAuth to display your user details. Again, you can run this locally using PHP’s built-in server, and we’re working on getting a version set up on wp-api.org too.

    Internally, this uses the wonderful OAuth 1 client library by The League of Extraordinary Packages. This repo also acts as a provider for that library, allowing you to use the library in your own code to handle OAuth.

    (Both the discovery library and the demo client are open source and MIT licensed, allowing you to use them in commercial projects should you so desire.)

    Hopefully you find these pieces useful. As always, your feedback on all of this is much appreciated; we’d also love to see discovery libraries and demo clients in other languages, if anyone wants to port them across. Thanks for being great.

     
    • jeffikus 10:41 am on December 7, 2015 Permalink | Log in to Reply

      This is awesome, the last piece I’ve been hoping would come out soon 🙂 Major kudos for this, quick question though, I see the demo API client is in PHP, any plans for a JS demo?

      • jeffikus 10:42 am on December 7, 2015 Permalink | Log in to Reply

        Oh gosh, I just re-read the last paragraph, ignore me 🙂 i’ll try get a JS version together and post it here 🙂

    • Justin Greer 1:37 pm on December 7, 2015 Permalink | Log in to Reply

      Super excited! I am looking forward to extensions like OpenID Connect and Discovery.

    • Ahmad Awais 2:19 pm on December 7, 2015 Permalink | Log in to Reply

      Awesome sauce! Shit, I wish I could take a holiday and read the source, oh wait, I can! Goes to read the source.

    • Collizo4sky 3:12 pm on December 7, 2015 Permalink | Log in to Reply

      Great job guys. Excited to see some PHP league package get some love in the WordPress community.
      Why reinvent the wheel when it isn’t broken 😀

      Keep up the good job Ryan.

    • Joseph Scott 3:17 pm on December 7, 2015 Permalink | Log in to Reply

      it’s entirely discoverable … including renaming the API base, or even delegating API access to someone else

      Historically this hasn’t worked out that well. I’d recommend shouting this super loud in every code example and article that deals with this API if you want to overcome the lazy momentum of clients assuming the URL will always be DOMAIN/wp-json/

      • Mike Nelson 5:52 pm on December 7, 2015 Permalink | Log in to Reply

        +1 to Joseph’s comment, especially if someone wants to run v1 of the API alongside v2, in which case they would need to have one of the APIs running on a different API base.

        And +1 to these advancements in general!

        • Ryan McCue 11:59 pm on December 7, 2015 Permalink | Log in to Reply

          Fingers crossed we’ll eventually have a 1.9 that has v1 endpoints on v2 infrastructure. 🙂

          • Mike Nelson 6:50 pm on December 8, 2015 Permalink | Log in to Reply

            huh ya that could be nice. I suppose it depends on how well folks start to adopt using v2. If lots of people stay on v1 then it might be nice

      • Ryan McCue 11:58 pm on December 7, 2015 Permalink | Log in to Reply

        > Historically this hasn’t worked out that well

        Agreed, though it has worked well for feed autodiscovery, so I’m optimistic. 🙂

  • Ryan McCue 1:27 am on November 18, 2015 Permalink
    Tags:   

    WordPress Importer Redux 

    Hi, I’m Ryan McCue. You may remember me from such projects as the REST API.

    I’m here today to talk about something a bit different: the WordPress Importer. The WordPress Importer is key to a tonne of different workflows, and is one of the most used plugins on the repo.

    Unfortunately, the Importer is also a bit unloved. After getting immensely frustrated at the Importer, I figured it was probably time we throw some attention at it. I’ve been working on fixing this with a new and improved Importer!

    If you’re interested in checking out this new version of the Importer, grab it from GitHub. It’s still some way from release, but the vast majority of functionality is already in place. The plan is to eventually replace the existing Importer with this new version.

    The key to these Importer improvements is rewriting the core processing, taking experience with the current Importer and building to fix those specific problems. This means fixing and improving a whole raft of problems:

    • Way less memory usage: Testing shows memory usage to import a 41MB WXR file is down from 132MB to 19MB (less than half the actual file size!). This means no more splitting files just to get them to import!
    • Faster parser: By using a streaming XML parser, we process data as we go, which is much more scalable than the current approach. Content can begin being imported as soon as the file is read, rather than waiting for pre-processing.
    • Resumable parsing: By storing more in the database instead of variables, we can quit and resume imports on-the-go.
    • Partial imports: Rethinking the deduplication approach allows better partial imports, such as when you’re updating a production site from staging.
    • Better CLI: Treating the CLI as a first-class citizen means a better experience for those doing imports on a daily basis, and better code quality and reusability.

    Curious as to how all of this is done? Read on!

    (More …)

     
    • pavot 1:36 am on November 18, 2015 Permalink | Log in to Reply

      I can’t express how happy I am about this progress!
      Thank you very much.

    • Jon Brown 1:54 am on November 18, 2015 Permalink | Log in to Reply

      Ditto. This is awesome. Having spent a lot of hours last weekend trying to debug a massive wp.com WXR/XML export (split in over 20 parts and failing on files 14-16)… this is so timely. Looking forward to testing it.

      Curious if the pull parser works with a folder full of xml files the way the WP importer did? Since as I said above, what .com dumped was a couple dozen xml files and I don’t have a convenient way to glue them back together.

      Huge thank you for attending to a long neglected corner of WordPress!

      • Ryan McCue 1:56 am on November 18, 2015 Permalink | Log in to Reply

        Curious if the pull parser works with a folder full of xml files the way the WP importer did?

        You should be able to work with split files by just importing them one after another. (You should also be able to import them in any order, theoretically.) This is something I haven’t put a lot of testing into yet though, so give it a shot and let me know!

        In the future, hopefully we’ll never need to split files again. 🙂

      • Brent Toderash 6:05 am on November 19, 2015 Permalink | Log in to Reply

        If you’re up for an adventure, on the command line you could do something like

        cat $(ls *.xml -t) > complete.xml

        to stitch the files back together in date order, then try importing the whole thing at once 🙂

    • Sakin Shrestha 2:12 am on November 18, 2015 Permalink | Log in to Reply

      Wow this is awesome… Thanks a lot 🙂

    • nicholas_io 2:13 am on November 18, 2015 Permalink | Log in to Reply

      This is Awesome! The WordPress importer as it currently stands is far from being good. I’ll be happy testing it out

    • Mike 2:20 am on November 18, 2015 Permalink | Log in to Reply

      Great stuff Ryan. Really nice performance improvements. I’ve often ran into memory limits during import.

      As an aside, has there been discussion around offline import, most specifically around media items? Such a possibility would allow for sites to be exported wholly together with all of their content items and either kept for archive or imported more easily. Such a site could have been mocked up on an unreachable development server.

      • Ryan McCue 2:24 am on November 18, 2015 Permalink | Log in to Reply

        Thanks Mike!

        We have an issue filed for that one. Deferring the actual images getting imported will get a few gains: offline imports, parallel downloads, or static content imports (i.e. image files in a zip).

        You can pass 'fetch_attachments' => false (also the default internally, but not in the CLI) into the importer class in __construct for this already. 🙂

    • Ryan Markel 2:25 am on November 18, 2015 Permalink | Log in to Reply

      Hey, Ryan.

      This is other Ryan. 🙂 We’ve been dealing with super-large imports on WordPress.com VIP for some time now and I’ve been dealing with them for a year-plus. I’d love to chat with you at some point about the kinds of things we have run into in these large imports and in other things that are edgier-case like site merges and other things that the import currently can struggle with.

      Some of what we are doing on WordPress.com is almost certainly not practical for movement to core, but we can probably provide some valuable insight on larger imports and problems that are encountered there.

      • Ryan McCue 2:49 am on November 18, 2015 Permalink | Log in to Reply

        Hey other Ryan! Let’s chat; one of the big use cases that lead to me rewriting this was huge VIP-level imports, so it’s definitely something I’m conscious of. I’d love to have your and VIP’s feedback 🙂

    • Ipstenu (Mika Epstein) 2:43 am on November 18, 2015 Permalink | Log in to Reply

      This will be a game changer for the many people who want to export from wordpress.com and import to self-hosted where they either can’t afford or aren’t able to use the migration package.

      You, sir, deserve all the pizza bites.

    • Matthew Eppelsheimer 3:15 am on November 18, 2015 Permalink | Log in to Reply

      This is so, so, great Ryan!

      Looking forward to testing this and seeing if our pet bugs in the 0.6 importer to fix “someday” still exist. I’ll bet not…

    • JakePT 3:38 am on November 18, 2015 Permalink | Log in to Reply

      My biggest frustration is the inability to import Posts only and import images at the same time. Images only come across if the entire site is exported. This is very annoying when trying to transfer a client’s blog between sites without bringing across menus and random plugin post types. I can understand the challenge since they’re only img tags and not necessarily attached to the post, but it’s definitely my biggest hangup with the importer.

    • bishless 4:59 am on November 18, 2015 Permalink | Log in to Reply

      This is great news! Thanks for your work, Ryan!

    • jeffmcneill 5:43 am on November 18, 2015 Permalink | Log in to Reply

      I’ve run into a ton of problems with importing, usually a memory or parsing issue. I’ve had to eject a lot of content from several site migrations because of this. That you are working on this critical tool for WordPress means you should be applauded, and lauded. Thank you!

    • DeBAAT 6:26 am on November 18, 2015 Permalink | Log in to Reply

      Reading the responses, you’ve definitely hit a soft spot!
      Thanks already for picking this challenge.

      I myself did have some issues with the importer lately as I was migrating some content from dev to prod.
      And this actually had to do with the GUID, so please take real good care to using this.
      I mean, a post developed on site x.dev typically has x.dev in the guid. When I try to import it into x.com, I would like to have an option whether or not to ‘translate’ the GUID from x.dev to x.com.

      The same goes for the GUID of an image. In some cases, the image may have been uploaded already in the x.com site before. So how do you discriminate between the two images?
      Hope you’ll get it working nicely.

      Reading your article, I thought you mentioned something about changing the exporter as well. I’m not sure however, so are there any plans to work on the exporter as well?

      • Ryan McCue 6:47 am on November 18, 2015 Permalink | Log in to Reply

        The main thing with the GUID is that it shouldn’t ever change. The new importer actually uses it more, so if you change the GUIDs, later imports will reimport the same data again. That said, @westonruter‘s pull request would let you write complex logic to map this back and forth, and I do know it’s a common use case. Happy to discuss further on a GH issue 🙂

        Specifically on images, there’s a flag in the options in the new importer called update_attachment_guids – the old importer always changes the GUID to the new filename, but this rewrite avoids that by default, as it makes deduplication on future imports impossible. (The reason the old importer did this is because originally the GUID contained the filename, so that’s occasionally used by older code to find the image URL.)

        There’s no real plans to change the exporter right now, but if it turns out we can improve import efficiency by changing data in exports (i.e. avoid post-processing by being careful about the element order), we’ll likely do that.

        Thanks for the feedback!

    • Ahmad Awais 6:38 am on November 18, 2015 Permalink | Log in to Reply

      I’ve had my fair share of WTF Moments™ with WordPress importer. So, believe me when I say this, I am glad someone is finally working on improving that. It’s a great news. Going to test it this weekend.

    • Sam Hotchkiss 6:51 am on November 18, 2015 Permalink | Log in to Reply

      You’re amazing, Ryan, well done.

    • Per Soderlind 6:53 am on November 18, 2015 Permalink | Log in to Reply

      Excellent Ryan, can’t wait to test it 🙂

    • Dreb Bits 7:30 am on November 18, 2015 Permalink | Log in to Reply

      Truly amazing! Will find time this weekend to test this baby! And thanks a lot initiating the movement to improve the WordPress importer 🙂

    • Ajay 7:52 am on November 18, 2015 Permalink | Log in to Reply

      This is awesome news. It’s been a long time coming. Are you planning to update the Exporter as well?

      • Ryan McCue 7:54 am on November 18, 2015 Permalink | Log in to Reply

        No major plans there yet, but if there’s improvements to be made there, I’ll be sure to take a look. 🙂

    • Morten Rand-Hendriksen 8:00 am on November 18, 2015 Permalink | Log in to Reply

      Thank you Ryan. This is a sorely needed upgrade.

    • connectr 8:22 am on November 18, 2015 Permalink | Log in to Reply

      Ooh, goodie! It always baffled me why such an important tool received so little love. The GSoC project looked hopeful, but then not much came from it. Super duper happy it’s back on the radar 🙂

    • Mario Peshev 8:37 am on November 18, 2015 Permalink | Log in to Reply

      Thanks for the great work Ryan – given the state of the original Importer for years now, that’s incredible news.

    • Omaar Osmaan 8:38 am on November 18, 2015 Permalink | Log in to Reply

      So excited about this- 😀

    • dimitris33 9:42 am on November 18, 2015 Permalink | Log in to Reply

      Great news Thanks!

    • capuderg 10:10 am on November 18, 2015 Permalink | Log in to Reply

      Great!!! So excited! Thanks Ryan for working on this!

    • Tran Ngoc Tuan Anh 10:36 am on November 18, 2015 Permalink | Log in to Reply

      This is a great news.

      I’m wondering about media import. Some themes use licenced images and don’t want to import, they usually are replaced by dummy image as a placeholder. Can we do that (maybe with a filter) with this new importer?

      Also, would it help in importing theme mods?

      • Ryan McCue 10:39 am on November 18, 2015 Permalink | Log in to Reply

        You can filter `wxr_importer.pre_process.post`, which gets passed the full post data, and go ahead and change anything in there. 🙂

        I’m not sure what the current state of theme mods is in the export data, so I can’t speak to that, but file an issue and I can take a look. 🙂

    • Primoz Cigler 10:42 am on November 18, 2015 Permalink | Log in to Reply

      Wow, incredible! Just awesome Ryan. It is clear from responses here that this was a huge pain for many people.

      Something came to my mind when I was reading this, I am not sure if it will be relevant, but still worth mentioning: would it be possible to utilize the REST API and switch to JSON instead? I believe it would make for much cleaner approach, as I was always struggling with any XML files, while JSON is so human-readable? I believe the import/export would be possible without the XML file at all, only import WP calling the export WP over the JSON calls.

      I would love to hear the feedback on my thinking and if that is something that should be considered in the future when migrating content over the web, as it sounds much leaner approach 🙂

      • Ryan McCue 10:48 am on November 18, 2015 Permalink | Log in to Reply

        As the lead developer of the API, I’m conscious of this. 😉

        There’s a number of reasons it’s better to stick with WXR/XML right now:

        • PHP has a streaming XML parser built-in, but it doesn’t have a streaming JSON parser out of the box. This means any streaming parsing would need to take place in userland (i.e. a library), which I suspect would cause performance to be worse.
        • WXR is compatible with RSS, which means the export format can be imported by a lot of other tools, not just WP. Switching to our own format might have benefits to us, but it would likely harm the wider ecosystem.
        • XML isn’t really a problem most of the time. It’s not the greatest format ever invented, but it’s reasonably readable, and there’s tonnes of tooling around it.

        With that said, if somebody wanted to try it, it’d be interesting to see at the least. 🙂 The XML parsing in this new version of the importer is contained in a few small methods, so you could probably swap that out for JSON reasonably easily.

        • Primoz Cigler 10:52 am on November 18, 2015 Permalink | Log in to Reply

          Gotcha, thank you for your swift reply!

          I don’t promise anything, but I might actually try to implement the REST API way. I will give you heads-up if I do.

    • sonjanyc 11:23 am on November 18, 2015 Permalink | Log in to Reply

      This is amazing, Ryan!! You will make so many people so very happy with this, me included!! 🙂

    • Alvaro Gois dos Santos 11:28 am on November 18, 2015 Permalink | Log in to Reply

      Wow. Awesome @rmccue, thanks!

      Let’s try that importer…

    • Damian 12:55 pm on November 18, 2015 Permalink | Log in to Reply

      Thanks Ryan, this is problem we all faced at some point. My concern is the exporter now. It will be also updated ? Currently how the exporter work is primitive. Eg: You can’t export featured images if you just export posts, you need to export the whole content.

      Or if you want to export just taxonomies becuase you have lots of them and you want to duplicate them in a new site.

      Thanks again!

      • Ryan McCue 1:35 pm on November 18, 2015 Permalink | Log in to Reply

        Right now, I don’t have any plans to work on the exporter (unless there’s performance gains to be had for the importer there), however there’s a few tickets already on that: #27048 and #17379 are the two most relevant here I think.

        At the end of the day, these things just need someone to sit down and spend some time on them. 🙂

    • Rich Tabor 1:32 pm on November 18, 2015 Permalink | Log in to Reply

      Really glad this is happening!

    • Josh Eaton 2:57 pm on November 18, 2015 Permalink | Log in to Reply

      Wait, so you’re saying I no longer need 8GB of RAM in my local VM just to import a WXR file? 🎉

    • Steven Gliebe 3:21 pm on November 18, 2015 Permalink | Log in to Reply

      This is music to my hears. Thank you for working on this!

    • Ben Doherty (Oomph, Inc) 5:33 pm on November 18, 2015 Permalink | Log in to Reply

      This is great, I’m very excited to see this development. I wonder if parallelization is also something that’s been considered, as PHP is just SLOW when walking through large files, and I found that parallelization can greatly improve overall import times, especially when munging XML files. My quick elevator pitch is that the importer spins off M processes, 1..N, each of which imports every M+Nth post from the file in parallel, and then the post-cleanup operation (translating URLs, IDs, etc) runs when all of the processes have completed.

      • Ryan McCue 11:55 am on November 19, 2015 Permalink | Log in to Reply

        Parallelisation is definitely something in my mind, but given the complex way that the object cache and database interact in concurrent requests (i.e. concurrent saves break basically everything in WP), not something I’ve put a huge amount of time into. 🙂

        I’d love to see someone have a go at it though! The new importer should let you build this as a plugin on top reasonably easily, so if you want to try it… 😉

    • pathartl 6:12 pm on November 18, 2015 Permalink | Log in to Reply

      A much needed update. I opened up #30887 a while ago when I first noticed the GUID issue, in case you wanted another ticket to close :). Great work as always.

    • Shmoo 2:26 am on November 19, 2015 Permalink | Log in to Reply

      Ryan should get a WordPress company sports car for doing this! 🚗 💨 vroom vroom..

      Can’t wait to test this.. Did you the WP Importer yesterday -10MB file #drama

    • lepardman 9:00 am on November 19, 2015 Permalink | Log in to Reply

      Awesome! Just a thought: it would be nice to have the ability to remap post types (maybe even taxonomies?) on import.

      Example: yesterday I was trying to import posts from an old site to a new one I’m working on. The site was so old that the developers used posts as a way to store the clients products and their actual news posts were just added as text to a huge list in the editor on a page. Some years after, a new developer added a new post type called news since post were “taken”. So I figured that I at least could do an export of the lastest news and the products. Then import them and switch post types, news as post and the products (old post type post) into a new post type called products. But that wasn’t possible, I just got and error that the post type news didn’t exist and the import failed.

      TL;DR: I couldn’t remap posts on import to other (nowadays more correct) post types. A nice way would be to be able to assign them to post type(s) just as you can change author (I guess a bulk option would be needed). I’m unsure if this would be possible but I’m throwing out the idea and then let’s hear what you guys say.

      (I guess I could do a search and replace in the xml files or create the correspondent post type temporarily, import and then do a search and replace in db but it would be so much nicer to have this built in)

      Thank you!

    • chzumbrunnen 3:14 pm on November 19, 2015 Permalink | Log in to Reply

      Wow, Ryan, that sounds great. When I understand you right my biggest question is already somehow fixed.
      Normally for an import the old site hast to be still reachable to get the images. Now with the new importer it should be possible to import images manually. But I’d like to go one step further and be able to enter the path to the images folder from where the images should be imported.

      • Ryan McCue 12:36 am on November 20, 2015 Permalink | Log in to Reply

        The tooling around this stuff will need some time to develop and mature, but I’d also love to have that eventually 🙂 Right now, should be technically possible, but just a pain to do so.

    • DoodleDogCody 3:56 pm on November 19, 2015 Permalink | Log in to Reply

      In regards to Media. I have noticed that with the current importer media items are not imported unless we select export all from the exporter. This seems a little odd to me. Media items should be imported if they exists in a post, page, or whatever post type that we export as media items are not thought of as a seperate psot type to most people. I know that’s how they are stored in the database but If I am building a new site for a client and need to export and import only their posts on the current site. I want to click export post and choose the date range of all. Then when I import I expect the images to import as well.

    • Ella Iseulde Van Dorpe 5:23 pm on November 19, 2015 Permalink | Log in to Reply

      ❤️

    • Samuel Wood (Otto) 12:33 am on November 20, 2015 Permalink | Log in to Reply

      Awesome. I’m gonna steal your XML code. 🙂

    • Michael Ecklund 8:02 pm on November 20, 2015 Permalink | Log in to Reply

      Thank you for taking the time to update the importer. It’s been begging for attention for quite some time.

    • RobbyMcCullough 7:42 pm on November 25, 2015 Permalink | Log in to Reply

      This is great news. Thanks, Ryan. Really excited to hear about this project.

    • Caleb Evans 1:06 am on November 29, 2015 Permalink | Log in to Reply

      Hey, @rmccue—very happy to hear that the Importer is getting some much-needed love. Do you have any plans to get the Importer merged into Core, or will it remain a separate plugin? I find it somewhat counterintuitive that WordPress can export its own content out of the box, but cannot import that exported content without a separate plugin. It’s not too big of a deal—just one less plugin to install for new installations—but it’s certainly something that would be nice to have, and it seems like the sensible thing to do.

    • pipdig 7:04 pm on November 29, 2015 Permalink | Log in to Reply

      Ryan, you have been added to my Christmas card list!

    • Caleb Evans 4:23 am on November 30, 2015 Permalink | Log in to Reply

      Hi, @rmccue. This all looks great, and it has me very excited. Out of curiosity, do you have any plans on trying to get this merged into Core? I find it strange that WordPress can export its own content, yet it cannot import that same exported content without a plugin. Thoughts?

      • Ryan McCue 4:27 am on November 30, 2015 Permalink | Log in to Reply

        No plans right now to integrate this into core. I think it makes more sense as a plugin personally. The export being built-in is fundamentally about your data being portable and free to access, whereas the importer doesn’t really affect data portability or freedom. Plus, having it as a plugin means we can improve it faster 🙂

        (That said, others may disagree, so might be something to discuss further!)

    • Tevya 7:19 pm on January 13, 2016 Permalink | Log in to Reply

      So what’s the status on this currently? Is it ready for general use? We’ve been having the old importer fail to bring over images and media, even though the box is checked. Hoping this can save us.

    • helmar 6:34 am on February 13, 2016 Permalink | Log in to Reply

      Sounds super promising. Hope to read more about this soon, even have something to work with. If financial support is needed, I’m in.

    • birdrockdesigns 9:39 am on March 9, 2016 Permalink | Log in to Reply

      I’m really, really hoping you will be successful with the repo.

  • Ryan McCue 6:30 am on October 28, 2015 Permalink
    Tags: , , ,   

    REST API: Welcome the Infrastructure to Core 

    Hi from the REST API team! We’re extremely excited to announce the API infrastructure has now been merged into core as of r34928 (plus a couple of fix up commits we won’t mention). Huzzah!

    Sincere thanks to every single one of the contributors, we wouldn’t be where we are today without you. It takes time and effort to produce great things, and it’s impossible to make things great without everyone helping. This has been a truly collaborative effort, and I wish I could do more than just give you props.

    (Important note: if you have a 2.0 beta already installed, you must upgrade to beta 5.)

    What’s included in 4.4?

    As mentioned in the merge proposal, the API comes in two parts: infrastructure and endpoints. In 4.4, the infrastructure is now available as part of core, while the endpoints continue to only be available in the plugin.

    You can think of the infrastructure as an “API construction kit”. WordPress 4.4 will make it possible for everyone to build RESTful APIs in a much easier fashion, which will benefit people building custom APIs for their site. The infrastructure handles the routing, argument handling, JSON serialisation/deserialisation, status codes, and all that other lovely REST stuff.

    For client authors, this doesn’t help you much right now. Hold tight, the team is working as fast as we can on the endpoints to get them ready for a future release. In the meantime, you can make sure sites you want to work with have the plugin installed, which isn’t a change from the current state.

    For plugin and theme authors, you can start building new APIs immediately using the infrastructure now in core. This can start replacing your existing custom admin-ajax endpoints or other bespoke code you already have.

    To authenticate with the API, only built-in cookie authentication is available out of the box right now. The OAuth 1 plugin will continue to work with WP 4.4 and the API plugin, as will the Basic Auth plugin for local development.

    It’s super easy to get started, and there’s even a guide available to kick-off. (Note: the WP_REST_Controller class is not included in WordPress 4.4.) This documentation will be migrated across to developer.wordpress.org soon.

    If you want to access any of the built-in data in WordPress without building it out yourself, you’ll need the endpoints as well. These will continue to be packaged in plugin form, and version 2.0 final will be released to accompany 4.4 before the end of this cycle.

    What if I’m using the API already?

    If you’re on version 2 of the API, you’ll need to update the API to beta 5 or later before updating to the latest version of core. This new version will use the new infrastructure in core if available, or fallback to a compatibility library otherwise.

    Important note: Earlier 2.0 betas (1 through 4) are incompatible with WP 4.4. Your site will fatal error if you don’t upgrade to beta 5 or later. You must upgrade to the latest API to run trunk and to run WP 4.4 when it’s released.

    If you’re on version 1 of the API, you won’t hit any fatal errors straight away, but endpoints will stop working with 4.4. We’re still planning on releasing a final version 1 release for compatibility, but now would be a great time to consider migrating forward to version 2. Apart from security releases, version 1 has ceased being actively maintained.

    Looking forward for the API

    Now that the API is past this first hurdle, it’s important to keep looking forward. Our immediate next step is to improve and polish the endpoints for phase two of our merge. There’s a lot of work still to be done here, so we’d love you to join us on GitHub.

    The infrastructure of the API will now be maintained via Trac, so new issues and patches should be sent there instead under the “REST API” component. Issues with endpoints should still be filed on GitHub. Don’t worry if you’re not sure; you can file issues on either Trac or GitHub, and they’ll be triaged into the correct place as needed. (It’s more important to make sure the issue is filed in the first place!)

    The team wants to keep pressing forward with the API and keep up our rate of progress, if not improve it even further, and we’d love your help. We still need help from content writers on our documentation, designers and developers on our authentication plugins, and developers on the endpoints. If you want to help, we can always use a hand, and we’d love to help get you started. We’re available in the #core-restapi room on Slack, and we’d love to see you at the weekly meeting at Monday 23:00 UTC 2015.

    We look forward to continuing to work on the API and getting these endpoints happening. Thanks again to everyone who got us here.

    (Then again, maybe a REST API is more of a Shelbyville idea…)

     
    • Ahmad Awais 9:20 am on October 28, 2015 Permalink | Log in to Reply

      It’s a new day 🙂 Looking forward to how REST API transforms WordPress as a platform and in turn the WWW.

    • Ahmad Awais 9:33 am on October 28, 2015 Permalink | Log in to Reply

      It’s a new day! Excited to see how REST API helps WordPress grow as an ultimate go-to platform for all the web development needs.

    • Manny Fleurmond 11:36 am on October 28, 2015 Permalink | Log in to Reply

      Are the models/collections/etc part of this or do those come later?

    • Stephane Daury (stephdau) 2:44 pm on October 28, 2015 Permalink | Log in to Reply

      Congrats on a monumental effort by the whole team.

    • Justin Sternberg 3:56 pm on October 28, 2015 Permalink | Log in to Reply

      🎉👏🍺

    • Mahesh Waghmare 3:57 pm on October 28, 2015 Permalink | Log in to Reply

      Great….!

    • karlazz 8:33 pm on October 28, 2015 Permalink | Log in to Reply

      Congratulations!

    • Jon Brown 11:25 pm on October 28, 2015 Permalink | Log in to Reply

      A whole new world awaits… thank you to the tireless devs working on this the last couple years!

    • Scott Bolinger 4:59 pm on October 29, 2015 Permalink | Log in to Reply

      Congrats guys! Is the wp-api 1.x plugin compatible with WordPress 4.4?

      • Aaron Jorbin 6:32 pm on October 29, 2015 Permalink | Log in to Reply

        As noted in the post:

        If you’re on version 1 of the API, you won’t hit any fatal errors straight away, but endpoints will stop working with 4.4. We’re still planning on releasing a final version 1 release for compatibility, but now would be a great time to consider migrating forward to version 2. Apart from security releases, version 1 has ceased being actively maintained.

    • Ian Dunn 11:02 pm on October 30, 2015 Permalink | Log in to Reply

      We’re still planning on releasing a final version 1 release for compatibility, but now would be a great time to consider migrating forward to version 2

      I’m still not clear on the upgrade path for sites running the 1.x plugin in production. I don’t really want to start running v2 on production until it’s stable, and I also don’t want to spend a lot of time upgrading my customizations until v2 is stable, in case anything changes.

      So, if I upgrade to 4.4 and the final 1.x plugin at the same time, will everything continue working? Or, do I need to wait until v2 is stable, then upgrade my customizations to work with it, and then install 4.4?

      • Joe Hoyle 1:22 am on November 14, 2015 Permalink | Log in to Reply

        The latest version of the 1.x plugin will continue to work with 4.4 – just make sure you update the plugin when you upgrade to 4.4.

        Right now, the 1.x plugin essentially overwrites the v2 infrastructure that is in core, so that does mean that you won’t be able to take advantage of the REST API infrastructure bundled in core if you are using v1. This will be resolved in a future update however.

    • suifengtec 4:25 pm on December 17, 2015 Permalink | Log in to Reply

      Good News!

  • Ryan McCue 1:03 am on September 21, 2015 Permalink
    Tags: , , , ,   

    WP REST API: Merge Proposal 

    Hello everyone! This is the post you’ve all been waiting for. 🙂

    We on the REST API team (myself, @rachelbaker, @joehoyle, @danielbachhuber, and newest member and core committer @pento) would like to propose merging the REST API into WordPress core. We’ve been working a while on this, and think it’s now ready to get your feedback.

    This is our first iteration of the proposal, and we’re actively looking for feedback. If you have thoughts on the project, or on this proposal, let us know! Only with your feedback can we make progress. 🙂

    What is the REST API?

    The REST API is a nice and easy way to get at your data in WordPress externally, whether that’s from JavaScript in a theme or plugin, mobile and desktop applications, or importing and exporting data. The API offers up all core data types (posts, terms comments, and users), plus support for meta and revisions; we’ve got plans to eventually have access to everything the admin and frontend have access to.

    The REST API differs from existing WordPress APIs in that it is explicitly designed from the ground up for modern mobile and browser usage, using the lightweight and widely-supported JSON data serialization format with a modern REST interface. Both of these are already familiar to most developers: JSON is a subset of JavaScript intended purely as a data interchange format, and REST is a set of best practices around HTTP. Both are supported natively by almost every programming language and platform.

    Why do we need a new API?

    WordPress already has external APIs: XML-RPC, designed for desktop clients; Atom and RSS feeds, designed for post syndication; and the venerable admin-ajax, designed for Ajax requests in the admin and frontend. These APIs all serve different purposes, but often have a great deal of overlap. In addition, these have all been stretched beyond their original intentions: XML-RPC now contains site management tools, RSS has been extended into the WXR export format, and admin-ajax is the catch-all of any sort of browser-server communication in plugins and themes.

    The REST API builds upon the heritage of these APIs to provide better support today for using these, as well as laying the groundwork for expanded use in the future.

    XML-RPC is the closest analogue to the REST API in terms of usage and capabilities. Originally designed back in 1998 to allow desktop clients to create and edit posts on blogs, WordPress has extended this with both other specifications (such as MetaWeblog) and with its own proprietary additions. Fundamentally, XML-RPC is built around Remote Procedure Calls (RPC), essentially a way of calling a function externally. It then uses XML to serialize the data for passing back and forth.

    Unfortunately, XML serialization can be problematic at times. XML has lots of power, but support for custom entities and DOCTYPEs can cause parsing problems and security attacks, including billion laughs, and XXE exploits. (Currently, WordPress has to disable parts of the XML parser and apply regular expression replacements to be able to parse these safely.) XML is also very verbose, and represents data in a way which doesn’t map easily to programmatic data structures. JSON on the other hand is both concise and well-represented in memory, as it’s based on JavaScript’s native syntax.

    The admin-ajax API is also very commonly used in WordPress, albeit typically only by plugins and themes. This is a very lightweight API that essentially acts as a mini-router. Typical usage of this API uses JSON, since it’s a browser-based API, but all usage is completely custom. A lot of the usage of this involves retrieving or updating posts on-the-fly, but due to its nature as simply a framework, these are done in completely different ways. This doesn’t lead itself to extensibility, and requires a lot of duplication every time developers want to get data in or out. We don’t want to replace all of admin-ajax though, since some use cases don’t map exactly: UI-related interactions or things like the Heartbeat API are great examples of this.

    The REST API can help to supplant these uses. Our aim is to eventually replace the XML-RPC API completely, to act as a secondary import/export format, and to replace most (but not all) uses of admin-ajax. The REST API offers an easier to use interface than the existing solutions.

    Why this project?

    We’ve been working on this project ever since the first WordPress Contributor Summit back in 2012. Since then, we’ve had lots of feedback from core developers, the community at large, and further beyond in the form of client developers. We believe that the REST API has an immense amount of experience behind it, and plenty of viewpoints have contributed to the project’s direction.

    The API has seen great usage in the community, from various mobile apps to large news sites. This usage has helped to battle-test and prove out the API. In the process, we’ve found plenty of bugs, and some security issues. Thanks to this feedback, the API is incredibly stable and secure. (The most recent security bugs we fixed were relatively minor bugs.)

    We also designed the API from the ground-up to be part of core, following a core-like mentality to our processes. The API is intended to be both a great feature and a base to build off in plugins. We undertook a significant refactoring and partial rewrite in version 2 to make this extensibility even better. This open process also means that most of the design decisions are documented publicly as we’ve engaged stakeholders to gauge feedback.

    We believe these pieces combined make this a fantastic feature for WordPress core, and we hope you all agree. 🙂

    What’s the plan?

    The plan we’re aiming for is a two part merge of the API. For the first stage, the infrastructure code would be merged into wp-includes and made available for plugins and themes. This is an internal API only, but offers an “API construction kit” for developers to use. For the second stage, the endpoints would be merged, and the API would be enabled for sites by default.

    This plan splits the API into two parts, infrastructure and endpoints:

    • Stage One: Infrastructure: The infrastructure is the code responsible for routing requests and handling the “meta” layer of the API, including JSON serialisation/deserialisation, linking, embedding, and REST best practices. This adds a simplified routing layer outside of WP’s rewrites, allowing non-query-var rewrites easily, acting as a base for building APIs inside WordPress.
    • Stage Two: Endpoints: These are where much of the complexity of the API lies, as they’re responsible for mapping data from the external JSON format to the internal data structures, and vice versa. The “business” logic of integrating with WordPress is almost entirely contained within the endpoints. These are the more complex part of the API, as they require using deep APIs in WordPress, and handling security and privacy concerns.

    With this plan, we would aim to have the infrastructure merged in 4.4, and the endpoints merged one release later in 4.5.

    The slow nature of this plan allows a longer review time on the API for core committers. It also gives extra time for reviewing the endpoints, since they would be delayed one release.

    Merging the infrastructure now would allow third-party code to begin using the API to build upon it, including migrating from existing custom code. It would also help to increase developer confidence in the API (as it represents a commitment by the project towards a REST API).

    In this plan, the first stage would not include any of the base controllers (such as the posts controller). This may limit the utility of the infrastructure for plugins and themes, however as the endpoints would be merged a cycle later, it’s expected that this wouldn’t have a huge impact.

    The infrastructure of the API is approximately 2700 lines of code (roughly a third of the API codebase), and the endpoints make up the remaining 5500 lines of code.

    What would happen after merge?

    After merging the REST API into core, we’d spend approximately two weeks partying and celebrating. 🙂

    Once we’re done with the parties, one major question would be how we manage the API in the future. The existing management and contribution process via GitHub has been extremely successful, as we’ve had 61 people’s pull requests merged into the API. Contribution via GitHub is especially useful for the API, as it’s a heavily developer-focussed project, and is aimed at external (non-WordPress) developers. On the other hand, all other contribution to WordPress is done via SVN and Trac, so integrating with this process is important for existing developers, as well as core’s general processes. We need to ensure the API is an integral part of core, not a separate project.

    Given the team’s experience with GitHub as well as Trac, we can bring the best of both worlds by helping integrate the two. This would also improve contribution for WordPress as a whole, and benefit the whole community. This will be especially important as we encourage more contributions from the wider community (client authors, for example). We think we can make good progress here, and we’d love to try and help improve the process. 🙂

    In addition to the project management, there are still further API projects we need to tackle. Authentication is the most important of these, as a huge focus on OAuth and similar would be needed to make the API more useful for external clients. Currently, we haven’t had enough time to spend on this as well as managing the API project, however the API is now reaching a finalised stage, so this should be able to improve quickly. Centralised authentication is a huge part of this, as the regular OAuth registration process is cumbersome for a distributed system like WordPress.

    Important note: We don’t believe authentication is required for the API merge, and we treat it as a separate project. The authentication is a completely separate system to the API itself. This is something we’d give more time to in the future, but we want to focus on shipping the API for now.

    Let’s go!

    This is our merge plan for the API, however it’s not finalised. If you’ve got comments, thoughts, suggestions, opinions, or words of encouragement, let us know. We’d love to know what you think. Thank you all, you’re wonderful, and stay golden.

     
    • Ahmad Awais 3:13 am on September 21, 2015 Permalink | Log in to Reply

      “After merging the REST API into core, we’d spend approximately two weeks partying and celebrating.” 🎉✨💥 Just like that 🙂 Hoping for REST API to get merged as soon as possible.

    • Jon Brown 3:37 am on September 21, 2015 Permalink | Log in to Reply

      One the one hand this can’t happen too soon or too fast… on the other hand I understand the slow and staged approach.

      I’ve never seen it happen but would it be feasible to merge it into core and flag it as “beta” for a release? I know that wouldn’t be “normal” but then this big of a merge hasn’t happened in a long while and I’m just wondering if something like that was considered?

      The GitHub issue is a big one and I’m glad you touched on it, but I’m entirely unclear what you’re actually proposing here going forward? Are API devs going to continue to have a way to contribute via GitHub post merge?

      • Ryan McCue 3:51 am on September 21, 2015 Permalink | Log in to Reply

        I’ve never seen it happen but would it be feasible to merge it into core and flag it as “beta” for a release?

        I believe the Heartbeat API was handled this way, but I can’t speak as to whether it’d be feasible here; that’s something for the core team to consider. 🙂

        Are API devs going to continue to have a way to contribute via GitHub post merge?

        The plan would be that the API will become an integral part of core, and that contributing to the API would be just like contributing to WordPress. However, we’d like to help people contribute to WP via GitHub, which would allow contributions to the API (or any other part of core) via GitHub. 🙂

        • borekb 7:02 am on September 21, 2015 Permalink | Log in to Reply

          > However, we’d like to help people contribute to WP via GitHub, which would allow contributions to the API (or any other part of core) via GitHub. 🙂

          Yes yes yes! That would be a nice by-product of the API project 🙂

      • Gary Pendergast 3:56 am on September 21, 2015 Permalink | Log in to Reply

        The GitHub issue is a big one and I’m glad you touched on it, but I’m entirely unclear what you’re actually proposing here going forward? Are API devs going to continue to have a way to contribute via GitHub post merge?

        To add to @rmccue‘s comments – the REST API team already have a well established workflow in GitHub. We’d like to experiment with combining their workflow with an (as yet undecided) method for syncing Pull Requests with Trac tickets.

        By experimenting with a small part of WordPress, we can hopefully create a model that could be expanded to work for the entire WordPress project.

    • kstover 3:47 am on September 21, 2015 Permalink | Log in to Reply

      +1

      We’re currently re-writing the moderately successful Ninja Forms plugin using primarily Backbone, and this would be amazing for our plugin. As you say in the proposal, we currently have to stretch admin-ajax.php to the limit.

      Hoping this happens!

    • kdpuvvadi 3:59 am on September 21, 2015 Permalink | Log in to Reply

      what happens to the releases on github, what pull requests and issues? will you handle those?

      • Ryan McCue 4:56 am on September 21, 2015 Permalink | Log in to Reply

        See comments above on how we want to handle GitHub integration. The existing issues on our tracker will either be migrated over, or closed out.

    • Ionel Roiban 4:31 am on September 21, 2015 Permalink | Log in to Reply

      Ggreat news. Where is the API documented?

    • markkaplun 5:08 am on September 21, 2015 Permalink | Log in to Reply

      +0.5 for inclusion, but the only reason is that is serves as a wrapper to Alax API which can speed up AJAX related development.

      OTOH I am still trying to find what does the RST API does that a seasoned developer can’t easily do by himself with the AJAX API.. It is not a replacement to XML-RPC as XML-RPC is a standard adopted by all major platforms and therefor do not required 3rd party (lts say flickr) an extra effort to support different platforms (another more popular usage is trackbacks). With the REST API, it is a wordpress specific thing and to be used by a 3rd party an extra development will be required.

      So the question is why do we need a 4rd way to do computer to computer communication, what is the use case that make the REST aPI shine, something that either impossible or is just so much better then doing it with other protocols? Right now caching of AJAX requests is the only thing I can come up with and that is the reason for the “include but the feature is a meh” above.

      • Ryan McCue 5:59 am on September 21, 2015 Permalink | Log in to Reply

        I am still trying to find what does the RST API does that a seasoned developer can’t easily do by himself with the AJAX API.

        The REST API is a much simpler API for interacting with, and doesn’t have a lot of the problems that XML-RPC does (although it has problems of its own). It’s typically much easier and faster to work with on almost every platform, and it’s a bunch less data that needs to be sent over the wire, which is especially important for mobile applications.

        The XML-RPC we have currently is actually significantly custom to WordPress: XML-RPC is just a standard for an over-the-wire API (similar to REST). The actual APIs for blogging are Blogger, MetaWeblog, Movable Type, and WordPress. (Plus the pingback API.) MetaWeblog is the only actually standard API, but offers very limited utility, which is why the others exist.

        That said, XML-RPC is going nowhere in the short term. It’s still very useful due to compatibility, although I wouldn’t recommend creating a new client using it, due to the complexity.

        a seasoned developer can’t easily do by himself

        Just to note: there are plenty of female developers, so “themselves” would be a better pronoun to use. 🙂

        • markkaplun 4:33 pm on September 21, 2015 Permalink | Log in to Reply

          I think you avoided the main question of use case in which the API shines. The problem with not having a use case is that without one it is impossible to compare if an full REST API is better then some other APIs, or maybe the main benefits can be achieved by some wrapper around the Ajax and rewrite APIs without the added bloat and increasing the attack surface.

          For example a simple use case is a scrolling news feed which refreshes every 10 minutes.With what I see in the API for each new post I will have to do 2 request, one getting the post and another getting the featured image. If I will code it by myself it will do it in one request. The difference between 1 and 2 requests can be the difference between shared hosting and VPS.
          Another variation of the same problem is that AFAICT post meta is not retturned as part of the post request,why is that?

          The point is that the API is huge and every piece of it should be justified IMHO before added to core. 90% (made up the number by myself!) of wordpress sites are not going to use this functionality in any way, at least not before the IOS and android apps will be changed to use it, so why should this bloat be added to core?

          as for male/female thing I can go into how ridiculous it is to expect people that english is not their mother tongue to use english correctly and be aware to all the small cultural assumption behind each word, but I am not PC Principal (people who do not watch south park should be ashamed of themselves)

          • Pete Nelson 6:32 pm on September 21, 2015 Permalink | Log in to Reply

            Using the rest_prepare_post filter, you can add featured image details to each post, thus preventing a round-trip to the server on the front-end. Could also be used for post meta.

            Here’s a link to the relevant slide from a talk I did at WordCamp DFW on extending the REST API: https://docs.google.com/presentation/d/1o4gJnEcq1vbDUsjZu_zRfh8D7crxzU45gTCquZOFODw/edit?pli=1#slide=id.ga423abea2_1_61

            • markkaplun 5:54 am on September 22, 2015 Permalink

              But this requires me to write a plugin that has to be installed server side, and them my JS is not compatible to other wordpress sites, so except for saving some coding what is the difference from now?

              I think your comment helped me to understand my own position. I fully on board with using the core code of the plugin in core in order to make it easier to deploy json based REST API, but I don’t think that the current endpoints are fully baked, it seems like they return too much internal information which no one should know about except for the admin (like list of post types) but also not sure that they give enough information for the most common use cases.

          • Matthew Eppelsheimer 11:42 pm on September 21, 2015 Permalink | Log in to Reply

            I think the key point here is that nearly anything developers can do with XML-RPC or admin-ajax APIs, they can do more easily with the JSON REST API. So it shines across the board, for many use cases.

            The iOS and Android apps can’t start to use this until it is in core. Core itself, and default themes, will also likely use the API in the future. So someday, most and perhaps even all WordPress installations will eventually use the API.

            • markkaplun 6:14 am on September 22, 2015 Permalink

              IOS and android apps will keep using XML-RPC as long as there will be sites that use WP versions 4.3 and below, which is years in the future, and even then why would they rewrite the code for no actual benefit? There is nothing wrong with XML-RPC except for being old and XML note being hype anymore.

              I am not even sure that there is a parity in capabilities of both protocols right now (doesn’t seem like there is an end point for options).

            • Ryan McCue 1:12 am on September 23, 2015 Permalink

              IOS and android apps will keep using XML-RPC as long as there will be sites that use WP versions 4.3 and below

              The iOS and Android apps only currently support 3.6 and 3.5 respectively, so they do have the ability to move faster. In addition, you can progressively support the API, with an XML-RPC fallback if you want.

          • Ryan McCue 1:08 am on September 23, 2015 Permalink | Log in to Reply

            One of the key benefits of the API is that it can remove the need to write anything at all on the server. In addition, data’s provided in a standard way across the wire, making it much easier for plugins to extend. Right now, custom bespoke APIs are unlikely to work across the board with plugins, or even with the edge cases in core. (For example, handling password protected posts is super painful right now.)

            With what I see in the API for each new post I will have to do 2 request, one getting the post and another getting the featured image.

            For the specific example here, try adding ?_embed to your request, and you’ll get back the linked resources as well. http://v2.wp-api.org/extending/linking/ has more detail on that.

            Another variation of the same problem is that AFAICT post meta is not retturned as part of the post request,why is that?

            Answered below; see https://github.com/WP-API/WP-API/issues/1425 for why the current situation is what it is, and what our plans are for improving it.

            The point is that the API is huge and every piece of it should be justified IMHO before added to core. 90% (made up the number by myself!) of wordpress sites are not going to use this functionality

            We can almost immediately begin removing custom Ajax handlers in the admin, such as the code that handles trashing comments. wp-admin/includes/ajax-actions.php might be able to disappear entirely (fingers crossed), and that single file is ~3000 lines just by itself.

            If we can replace all of this code with well-tested code, I think it’s already worth it.

      • Ipstenu (Mika Epstein) 1:32 pm on September 22, 2015 Permalink | Log in to Reply

        The xmlrpc approach is heavily targeted by ddos and brute force attacks, which makes it rather undesireable. From a host standpoint, we hate it 🙂

        https://blog.sucuri.net/2014/07/new-brute-force-attacks-exploiting-xmlrpc-in-wordpress.html Just as an example.

        While not all the issues go away with this, enough do that I welcome our new restful overlords.

    • Daniel Homer 7:03 am on September 21, 2015 Permalink | Log in to Reply

      This is fantastic news, thanks to everyone involved! A standardised API in core will simplify a lot of JS logic in plugins and will pave the way for more advanced web apps on top of WordPress.

      🎉

    • borekb 7:05 am on September 21, 2015 Permalink | Log in to Reply

      What will be the story for plugins that opt to use the API but also need to support 4.3 and below? Will there be a compat plugin for 4.3 and below? Or is that up to the plugin author to deal with that and for example ship parts of the API for older WP versions?

      • Ryan McCue 7:07 am on September 21, 2015 Permalink | Log in to Reply

        The plan is to turn the existing plugin into one that progressively enhances your WP install. It’ll include the infrastructure for sites that don’t have it, but will use core’s otherwise. Once the endpoints are merged, it’ll disable itself. 🙂

        I’d expect that we’ll aim to support this at least until the next major release after the endpoints are merged.

        • borekb 7:11 am on September 21, 2015 Permalink | Log in to Reply

          Cool, thanks for the answer.

        • xavivars 5:45 pm on September 23, 2015 Permalink | Log in to Reply

          That’s the answer I was dreaming!!! We’re working on a project right now where we’re building a site that would benefit a lot from having a powerfull rest-api like the one you guys have built, but my concern was that there was no way to fully use it until it was integrated.

          Seriously, you guys are awesome 🙂

      • Ipstenu (Mika Epstein) 1:43 pm on September 21, 2015 Permalink | Log in to Reply

        Keep in mind what the ‘compatible up to’ and ‘requires’ version actually means in plugins. If you say your plugin requires 4.4, then no one on 4.3 gets updates. In theory you could back port your 4.3 compat tags (say your version 2.5 is sans API while 2.6 is with the new API), and just update 2.5.1 and 2.6.1 for each user base.

    • Hugh Lashbrooke 8:25 am on September 21, 2015 Permalink | Log in to Reply

      > This adds a simplified routing layer outside of WP’s rewrites, allowing non-query-var rewrites easily.

      I never even thought of that as a possibility with this API and I think that excites me more than anything else about this project. It couldn’t be in core sooner 🙂

    • Erlend Sogge Heggen 10:28 am on September 21, 2015 Permalink | Log in to Reply

      First and foremost: CONGRATS! Major props to everyone involved.

      Just one question: What about all these overlapping technologies (XML-RPC, the admin-ajax etc.) that you mention. Is there a deprecation plan in place for the parts that the REST API will cover?

      • Ryan McCue 1:30 pm on September 21, 2015 Permalink | Log in to Reply

        There’s no deprecation plan for them, as they’re not complete replacements. We’ll see what happens in time though; if usage of another API were to drop significantly in favour of the REST API, maybe we could look at it in the future.

    • James DiGioia 12:11 pm on September 21, 2015 Permalink | Log in to Reply

      Very exciting! One question: Will the Backbone client be merged as part of the endpoints, the infrastructure, or as a separate step entirely?

      • Ryan McCue 1:30 pm on September 21, 2015 Permalink | Log in to Reply

        The Backbone client is linked to the endpoints, so it’d be merged with them if at all; we’ll discuss whether it should be as we get closer to merging them 🙂

    • Omaar Osmaan 12:14 pm on September 21, 2015 Permalink | Log in to Reply

      😀

    • Marcus 12:21 pm on September 21, 2015 Permalink | Log in to Reply

      I’m bummed that I haven’t had the chance yet to play with the API yet and give constructive feedback, so words of encouragement is all I can offer. I’ve been following this project closely and can’t wait for this to happen 🙂 this has so much potential for WP as a whole.

      Well done to the whole JSON API team, thanks for all your efforts. Go for it!

    • Daniele Mte90 Scasciafratte 12:43 pm on September 21, 2015 Permalink | Log in to Reply

      I’m in this state http://memecrunch.com/meme/3MCTZ/excited-picard/image.jpg?w=510&c=1
      Also today I’ve written a little article about to expand that api the post type with custom fields http://www.mte90.net/en/2015/09/extend-the-wordpress-rest-api-of-a-post-type-by-custom-fields/

    • Per Soderlind 4:42 pm on September 21, 2015 Permalink | Log in to Reply

      +1 🙂

    • mojaray2k 5:20 pm on September 21, 2015 Permalink | Log in to Reply

      I am so thrilled that this is happening. As a developer I am curious on how OAuth is going to be implimented. Right now it is very difficult to implement without some third party service. Or maybe I am not understanding how to implement it the WordPress way.

    • Aaron Jorbin 5:32 pm on September 21, 2015 Permalink | Log in to Reply

      Thanks for putting this together. A couple of questions that I have:

      1) What would happen to v1 of the API upon merge? Would a new version of it be released that relied upon the routing in core? Would it continue to function by itself? Would it break?

      2) Based on your statement ” On the other hand, all other contribution to WordPress is done via SVN” are you aware that there are git mirrors that all non-committers can use? 😀

      3) Is two weeks sufficient amount of time for 🎉?

      4) Has there been any discussion about marking any of the new classes or methods in the classes as final upon introduction?

      • Ryan McCue 11:13 pm on September 21, 2015 Permalink | Log in to Reply

        What would happen to v1 of the API upon merge?

        Our original plan was to have v1 become a shell that wraps the infrastructure in core to provide compatibility, and I suspect we’ll still do this. We’re yet to get the specifics of it down. 🙂

        are you aware that there are git mirrors that all non-committers can use?

        Using git vs managing contributions via GitHub is a different kettle of fish. Right now, I already use git for making patches, but the flow for contributing those changes back is a bit cumbersome. We can do better 🙂

        Is two weeks sufficient amount of time for 🎉?

        Maybe not, but real life commitments get in the way of any prolonged 🎉ing. 😬

        Has there been any discussion about marking any of the new classes or methods in the classes as final upon introduction?

        Our intention is to keep them all extensible for now, and they’re all designed for reuse. As for forwards-compatibility though, we’ll consider this as we “finalise” (😏) the classes for merging.

    • karlazz 8:17 pm on September 21, 2015 Permalink | Log in to Reply

      Hello,

      I used the API briefly and then rejected it for a few reasons:
      1) the Oauth1 is hard to use and not well supported with examples or libraries, why not use Oauth2 like everyone else?
      2) It was awkward to get at post meta information. I used a plugin.
      3) the requests returned too much information and my programs died with a memory fault. Due to the nature of my plans for the application, this was not acceptable so I wrote simpler more specific tools that did not return mountains of unnecessary fields and require my users to increase their PHP memory limit.

      1) There is a lack of community code available for using Oauth1, and the API authors didn’t provide any guidance for setting it up, just some information thrown over the shoulder as it were. Even the endpoints were hard to find in their literature, and not quite the same as the ones described as “standard”. I hope that will be improved, or that you will switch to Oauth2, like the rest of the world. Security is a very important aspect of this project and people need to be able to use the locks. It would be great to provide a description of how to set up and use curl calls to access the API security. Salesforce does a good job of this, as does Box.com.

      2) Couldn’t access postmeta information via the API unless Oauth1 is implemented. Why? Used work around.

      3) It would be great if I could designate which fields I need to have returned using POST data and json. Box.com does this, as do others. This would save on memory when returning large numbers of records. Maybe you have this already and I missed it?

      Thanks. Good luck. This is a great idea but I chose not to use it and would not recommend it for the above reasons.

      • Ryan McCue 11:19 pm on September 21, 2015 Permalink | Log in to Reply

        the Oauth1 is hard to use and not well supported with examples or libraries, why not use Oauth2 like everyone else?

        I totally agree, OAuth 1 is a pain. However, the reason that OAuth 2 is much nicer is that it relies on SSL to handle much of the message authentication. We can’t require SSL for WordPress core code, so OAuth 2 is unfortunately a non-starter there.

        The OAuth Working Group has been making slow progress towards removing this requirement, and we’ve been following that. If anything changes there, we’ll consider changing it.

        Couldn’t access postmeta information via the API unless Oauth1 is implemented. Why? Used work around.

        Working with postmeta requires a great deal of care, due to privacy and security concerns. As it turns out, there are lots of people who use the Custom Fields box to save private notes directly to post meta, so we cannot just expose all meta without considering.

        We’re looking at relaxing the restrictions here in the future: https://github.com/WP-API/WP-API/issues/1425

        It would be great if I could designate which fields I need to have returned using POST data and json.

        This is something that most people don’t need, and which also has the ability to play havoc with server-side caches. This is easily doable in a plugin however, and we’d love to see someone take up the work there.

        See https://github.com/WP-API/WP-API/issues/572 for more details.

        • pathartl 3:52 pm on September 24, 2015 Permalink | Log in to Reply

          Just showing my support for keeping post meta behind Oauth. At first I was skeptical, but it’s very easy to hook into the data via the `json_prepare_post` filter and add any data you want to the JSON response.

    • John Teague 8:26 pm on September 21, 2015 Permalink | Log in to Reply

      I can’t tell you how happy I am and how thankful I am to the entire REST API team. I have lobbied for the inclusion into core, and I know this will make WP stronger and better for the future. Kudos all around @ryan,@rachelbaker, contributors all. 🙌

    • Greg Ichneumon Brown 9:53 pm on September 21, 2015 Permalink | Log in to Reply

      The v2 docs mention 3 types of authentication: cookie auth, oath1, and basic auth. You mention though that Authentication is a separate project.

      Are some of these (all?) going to be part of the merge and when are they available (stage 2?). I assume many of the endpoints are not useful without some level of authentication (though I guess there are many GET endpoints that could work without auth).

      • Ryan McCue 11:20 pm on September 21, 2015 Permalink | Log in to Reply

        Cookie authentication is built-in to WordPress, so that’ll be there out of the box. Basic authentication is only intended as a development tool, so the intention is to never include this with WordPress. OAuth 1 is something we’ll consider in the future, either when merging the endpoints, or shortly afterwards.

    • Michael Beil 10:18 pm on September 21, 2015 Permalink | Log in to Reply

      This is such momentous news @ryan and @rachelbaker!

    • simonrcodrington 12:27 am on September 22, 2015 Permalink | Log in to Reply

      I’m all for the integration of the REST API into WordPress core. My concern is if this will produce any strange side effects that could affect my sites (I know this supports multiple types of security such as oAuth but it’s always on my mind)

      It’s been tested for a while but I still think a cautious approach to this would be better (with starting integration later at WP 4.5 and then moving from there). It’s one of the biggest introductions to core in a while so I’m happy with a slower pace 🙂

      • Ryan McCue 12:58 am on September 23, 2015 Permalink | Log in to Reply

        With this first phase, there won’t be any read or write capabilities added, only internal APIs. A future version will start integrating those as part of the second phase.

        We’ve also been working on this for ~2 years, and in the meantime we’ve fixed a few security bugs (mostly minor), so we’ve already had lots of eyes over the code. 🙂

    • Zach Russell 6:01 pm on September 22, 2015 Permalink | Log in to Reply

      I scanned through the comments and don’t think I found an answer there. I’m working on an app right now that uses the WP API. Will I have to make any changes to my code once the WP API is merged with core?

      • Ryan McCue 1:47 am on September 23, 2015 Permalink | Log in to Reply

        The plan is that the current v2 will become the core version with minimal changes. If you build off v2 now, you may need to change your code slightly, but changes will almost certainly be minor.

    • Praveen Chauhan 9:38 am on September 23, 2015 Permalink | Log in to Reply

      Great news !!! Congrats to the REST API team.

      Have few questions:

      1) How will REST API specific support queries/updates be handled?
      For a plugin, there can be many support queries, and there is a dedicated forum. So if some one has queries specifically related to the REST API, will you continue to use the REST API plugin’s forum or have you planned to shutdown that forum or plugin listing?
      More importantly, how will updates for the fixes be handled? I mean, if some one has issues specific to the REST API functionality which requires some fixes, will the users need to wait for the new release of WordPress or do you have an alternative plan for this?

      2) What will be the impact on WordPress’ performance?
      As a plugin, the functionality is optional for users. But after merging it into the core, its modules will be loaded along with WordPress. So what would be its impact on site load, performance etc?

    • pathartl 4:12 pm on September 24, 2015 Permalink | Log in to Reply

      +1 in full support. A question and comment:

      1. What will happen with the plugin (both v1 and v2) when it is activated on an install that has the API in core? We’ve written some read-only things for v1 and I’m concerned about having a major refactor later on down the road. No doubt it will eventually happen, but the concern lies in the first couple phases of integration.

      I’m glad the core committers will have to look at every aspect. I found the documentation lacking in places. Well not necessarily lacking, but not as well documented as most things in the codex. This is a big merge that is totally developer focused and should be hashed out really well.

      Thanks for the hard work! This is much needed

    • suifengtec 5:08 pm on October 9, 2015 Permalink | Log in to Reply

      It’s cool for the future of WordPress !

    • snstarosciak 1:21 pm on October 21, 2015 Permalink | Log in to Reply

      Nice! So here’s my question. How will this impact current sites using the WP API v2? Assuming building something like this into core will make API-creation plugins unecessary, I’m guessing that the “final” functionality will be available in 4.5 that is similar to how the WP API v2 works?

      Thankyou!

    • mmaumio 9:01 pm on November 5, 2015 Permalink | Log in to Reply

      That’s Awesome. It’ll surely save us from creating our API or using a third party plugin.
      +1 for this.

    • WP Plugin Dev dot com 12:41 pm on November 20, 2015 Permalink | Log in to Reply

      Two important questions arise:
      What is public in the API after the merge?
      Can I tell WordPress to don’t provide the API functionallity?

c
compose new post
j
next post/next comment
k
previous post/previous comment
r
reply
e
edit
o
show/hide comments
t
go to top
l
go to login
h
show/hide help
shift + esc
cancel
Skip to toolbar