Make WordPress Core

Updates from October, 2016 Toggle Comment Threads | Keyboard Shortcuts

  • K.Adam White 11:15 pm on October 13, 2016 Permalink |
    Tags: ,   

    Merge Proposal Discussion: REST API Content Endpoints 

    There are discussion meetings and office hours in #core-restapi at 2016-10-14 14:00UTC and 2016-10-14 19:00UTC on Friday the 14th. Our next team meeting is on 2016-10-17 14:00UTC. Please attend some of all of these, because

    We are meeting at 2016-10-18 01:00 UTC to make a decision on this merge proposal!

    To that end, the below discussion points will be updated regularly, please leave comments on this post or join the conversation in #core-restapi.

    Yesterday at the dev chat the API Team proposed the Content API Endpoints for merge in WordPress 4.7. There was popular support for this feature but as @jorbin and @helen noted that the lack of dissent suggested additional review is needed, so the API Team is continuing to seek thorough review & constructive criticism of the content endpoints, including the open questions previously shared on the week 7 and week 8 API team updates.

    The merge proposal also engendered follow-up discussion in the #core-restapi channel about the benefit content endpoints bring to core, whether having such endpoints built in is quantifiably more beneficial than having them as a plugin, whether moving development from a plugin to core would slow development, and whether the endpoints as-proposed have been sufficiently reviewed for security and performance. We attempt to capture those questions & concerns (and the perspectives on them) below.


    Have the content API endpoints been thoroughly reviewed for security?

    • The REST API plugin has been on HackerOne for over a year with paid bounties for bugs
    • @barry has begun a security review


    How does the API measure up against alternatives? Are there concerns about how the API could impact the servers to which it is deployed?

    • DeliciousBrains did a performance comparison with Admin AJAX and found the REST API to have a performance improvement (These tests have not yet been independently verified)
    • @mikeschroder notes in the comments that using the REST API in place of Admin-Ajax will also bring speed benefits by permitting many previously-uncacheable requests to be cached.

    User Feedback

    Are the content endpoints sufficiently well-tested & vetted by the community?

    • @matt questions whether feedback is coming too late in development for concerns to be acted upon
      • @rmccue notes that the v2 endpoints were created based on user feedback; REST API endpoints are being deployed by plugins and running on VIP, and the content endpoints have been in wide use across a variety of sites, leading to 90+ code contributors and far more developers’ support & feedback on the endpoints
    • @rmccue has also reached out to Phil Sturgeon for feedback and will follow up

    Do Content Endpoints Benefit Core Development?

    Will having these endpoints in core improve future core development, or solve any immediate problems?

    • @bradyvercher suggested that the content API endpoints would remove the need to write a variety of one-off ajax callbacks when developing future WordPress Core AJAX functionality
    • @westonruter notes that the customizer could dynamically create settings for posts and other kinds of content without having to wire up new admin-ajax handlers

    Will Merging Negatively Impact API Development?

    Will having to work through trac instead of GitHub cause development to atrophy?

    • @jjj argues that merging will slow development, because GitHub-hosted plugins are not bound to WordPress release cycles and have less overhead for features to be developed and deployed for testing. @jjj requested a plan for how the REST API will be developed going forward after the merge of these endpoints that would account for the added friction.
    • @krogsgard countered that core increases the visibility of a project like the content endpoints
      • The number of new contributors in this Slack discussion suggests that this merge proposal is bringing in new voices; whether this supports Brian’s point or not, the team is grateful for the breadth of perspectives being shared -Ed.
    • @rachelbaker suggested that the API endpoints are sufficiently inter-dependent with core WordPress code that maintaining the plugin separately amounts to maintaining a fork, and that such separated development is untenable long-term.
    • @matt hopes that a merge of these endpoints would slow release speed, but not development speed; @rmccue feels that development speed will stay the same or increase, and that tying releases to WordPress Core increases the stability and predictability of the API endpoints.
    • The versioning of the API endpoints supports forward compatibility

    Do Content Endpoints Belong on Every WordPress Site?

    What are the pros and cons to having every WordPress site have content API endpoints?

    • @rmccue suggests the API has network effects that can only be realized with a large install base. @krogsgard draws a comparison to RSS, the widespread availability of which enables everything from podcasting from WP to the use of apps like Feedly.
    • @matt suggests that the Atom API is a better analogue than RSS, which is an independent and pre-existing standard, and that network effects could be tested through inclusion in Jetpack
    • @joostdevalk notes that many plugins, like Yoast, have data tied to existing content such as posts and pages; either they expose the content through their own endpoints, or core does. If Core exposes content types through the API then plugins may build on top of that shared foundation, not independently reinvent the wheel. “if this doesn’t end up in core, we’ll start rolling our own API for stuff. Others will too. Interoperability won’t be there, for even the most basic stuff. I think this isn’t like RSS, I think this is much more like all of us using the same table space in MySQL.”
      • @shelob9 and @masonjames agree that merging the endpoints would create a consistent and reliable open “standard” that WordPress developers can use instead of continually reinventing how to read and edit post data over HTTP.
      • In response to the question “what prevents you from building on the endpoints in their plugin form,” @joostdevalk went on to note that plugin dependencies would make that a viable option, but that burden currently lies on the user. Plugin installation is not frictionless.
    • Can these endpoints be bundled? short takeaway: no
      • Woo bundled the API infrastructure before it was merged; doing so for content endpoints would require bundling prohibitively large amounts of endpoint code.
      • @nerrad worries that if plugins bundle different versions of the endpoints plugin, then those plugins may conflict if all bundled copies are not kept in sync.
        • @nerrad clarifies in the comments below that these worries also encompass the additional risk of conflicts when plugin authors each build their own versions of these content endpoints, instead of leveraging a shared standard: if two plugins each expose their own REST collection for posts, a developer working on a site with multiple such endpoints will need to decide which to extend, and will then have their extension tied to that specific plugin rather than to a core API.
    • @schrapel and @jorbin discussed that these content endpoints make it much easier to crawl a site, which also brings some potential performance concerns: no new content is exposed, but the process of aggregating it is easier and more easily automated.
    • In the  comments below @foliovision believes that merging the endpoints will be the best way to assert the level of back-compatibility that mid-size agencies need in order to confidently utilize the endpoints.

    Please leave thoughts, questions, concerns, comments & experience in the comments below. Thank you!

    Edited 2016-10-16 to include the below comments into the body of the post

    • FolioVision 11:27 pm on October 13, 2016 Permalink | Log in to Reply

      Great discussion summary Adam. As the leader of a mid-size development agency, the sooner these endpoints are in core, the sooner we can start to use the REST API. For smaller to medium size clients, trying to play pin the tail on a donkey on a moving train (i.e. different versions of the REST API plugin, different versions of core) is simply not economically or technologically viable.

      As long as the REST API team can commit to retaining some backward compatibility if the API endpoints must change significantly in a couple of major releases, I can’t see any benefit to keeping REST API separate from core. So much great work has gone into creating the additional REST potential, it would be a shame to see it remain bottled up and a niche product only for huge agencies and hosts like VIP.wordpress.com or WordPress.com.

    • Josh Pollock 11:39 pm on October 13, 2016 Permalink | Log in to Reply

      One thing I wanted to add to this is that the REST API adds standards. I and many others have written tons of custom systems for alternative ways to read and edit post data using admin-ajax, or using custom rewrite rules.

      The constant reinventing of the wheal is counter to the spirit of open-source. By agreeing to a standard, with tons of eyes on it, we can all do better with greater efficiency and security. This after all is why we use WordPress and other open-source tools, instead of making our own CMSes.

      • masonjames 12:53 pm on October 14, 2016 Permalink | Log in to Reply

        I want to affirm this point as well. Our agency looks to WordPress (core) to provide an opinion. We value the decisions and respect them (even when we disagree).

        Web development is an incredibly competitive space with pressures in open source and otherwise. Having content end points in core ensures consistency and reliability across the WP community.

    • rag-guay 7:00 am on October 14, 2016 Permalink | Log in to Reply

      I would like to be able to edit widgets from the desktop app. So slow over the net from Thailand!

      • K.Adam White 12:58 pm on October 14, 2016 Permalink | Log in to Reply

        That’s a good point and I agree we need a solution for it. On behalf of the API Team, support for editing widgets is on the roadmap but is not part of what’s being proposed for merge this cycle. It is definitely a common request, and is one of our priorities for 4.8 as we expand the site management capabilities of the API!

        You can see some early experiments from earlier this year towards this usage here: https://github.com/WP-API/wp-api-menus-widgets-endpoints

    • Darren Ethier (nerrad) 10:46 am on October 14, 2016 Permalink | Log in to Reply

      Great summary! A lot of great back and forth in that discussion thread that isn’t always easy to capture 🙂

      Just wanted to clarify that the comments I made were not only in regards to the possibility of conflicts if plugin authors bundled the plugin version of the REST API with their own plugins, but also, with the presence of the infrastructure already in WordPress core, there is the possibility of conflicts should plugin authors roll their own different versions of the content endpoints to support what they are doing.

      Imagine plugin author A has used to api infrastructure in WP core to build custom endpoints for his plugin and his users are asking if he can add some endpoints for their posts, pages and users as well, so he goes ahead and does that. In the meantime, plugin author B has users asking her to build custom endpoints for users that support her needs as well so she adds them. Then along comes User Sally who installs and activates both plugins. She contracts mobile app developer Jimmy to build an app for her that interacts with both plugins and her website. Jimmy starts running into conflicts between what “user” endpoint to use but he manages to get it working. Then Sally tells Jimmy that she wants to distribute this app for anybody using plugin A or plugin B and Jimmy starts thinking about building a custom user endpoint in a plugin that site owners will need to install to use the app to save having to worry in the mobile app about what user endpoint to utilize.

      There are definitely different technical solutions to the above example story, but the reality is, things would be MUCH simpler for all the actors in the story above if there was an accepted standard in WordPress core for the content endpoints. It prevents a lot of problems from occurring.

    • K.Adam White 1:00 pm on October 14, 2016 Permalink | Log in to Reply

      If I may make a request of the developer community: Share this with your colleagues and friends outside of WordPress. See what they like, and what they find rough; let us know what they think! We’ve been gathering outside input on this but for more is always better.

    • Mike Schroder 6:03 pm on October 14, 2016 Permalink | Log in to Reply

      I noted this in the #core chat this week, but from my perspective at a host, we’re excited that landing the REST API will have performance benefits — in particular, because it will mean that we can cache requests that could only go through `admin-ajax` previously (and were thus forced to be dynamic).

      This will help with scaling even if it’s found that performance for the actual time in PHP/MySQL is exactly the same.

      Gradually moving those requests (whether they be core related, or added with plugins) out of `admin-ajax` will also help SysEng/Support more easily spot which requests are problematic.

      • Matt Mullenweg 1:15 am on October 18, 2016 Permalink | Log in to Reply

        It probably won’t have any real-world performance benefit — the admin-ajax benchmark was kind of flawed, and you won’t be able to do HTTP-level caching unless you also support some quite-fancy invalidation.

    • Luke Cavanagh 9:49 pm on October 14, 2016 Permalink | Log in to Reply

      I only see this as positive thing to be in WP core.

    • Brian Richards 2:21 pm on October 17, 2016 Permalink | Log in to Reply

      I’m late to the party but I want to officially register my voice in favor of merging the content endpoints into core.

      As @joostdevalk mentioned in Slack (and others have discussed elsewhere), there are a lot of opportunities to include API-based interactions within a plugin that are stifled presently because of the dependency on the REST API as a plugin. While it does seem like a small thing to inform end-users “This plugin requires the WP REST API plugin” it’s still a burden they needn’t bear. Further, by relegating the REST API to a plugin it stifles the ability for API-based interactions across many sites. For instance, a plugin that creates a widget that shows the most recent comments across a number of sites that you don’t control (e.g. getting the latest comments from a variety make/core inside a personal dashboard … and having the ability to bookmark certain comments or posts into a “read later” list for later consumption).

      As @getsource mentions above there are some fantastic wins by moving these interactions that currently depend on admin-ajax.php into standard, cacheable API calls.

      The most convincing dissenting opinion that gave me reason to pause was @JJJ‘s post that merging would hinder development. I do believe that is true. However, I also believe @rachelbaker is correct in that maintaining parity within the plugin as a long-term stand-alone is implausible. Given the choice between a REST API that can adapt and change quickly at the expense of dying slowly from a thousand cuts and a REST API that changes slowly but maintains perfect core parity I will pick the latter, every time.

      TL;DR: I really want to see the REST API land in core because I want to build some fantastic utilities (for myself and others) that can communicate with many sites without necessarily requiring administrative access (the ability to install the REST API plugin) within those sites.

    • K.Adam White 2:30 pm on October 17, 2016 Permalink | Log in to Reply

      I don’t think we’ve noted yet that the endpoints here don’t just encompass the built-in types, but also the endpoints that extend them — `WP_REST_Controller` provides a strong base for implementing new endpoints, and a custom post type with `show_in_rest => true` will automatically pick up behavior extending that class without any further action from the developer than a single line in `register_post_type`.

      Aside from the benefits of augmenting a shared resource like the posts collection @joostdevalk mentions above, having a sharable code foundation for new custom classes has the potential to reduce a lot of endpoint class duplication and fragmentation across the plugin ecosystem.

  • K.Adam White 6:36 pm on October 12, 2016 Permalink |
    Tags: ,   

    REST API Team Update, 4.7 Week 8 

    Summary: Beta 15 has been released, there are open questions that would benefit from your feedback, and the Content API Endpoints and OAuth Server are being proposed for merge as distinct, separate enhancements to the existing WordPress REST API infrastructure.

    REST API v2 Beta 15 released

    The 15th beta release of the REST API content endpoints plugin was released on October 7. This release builds on top of the recent Beta 14 to…

    • Add support for Post Meta, Term Meta, User Meta and Comment Meta within their parent endpoints
    • Introduce a settings endpoint to allow key site setting values to be retrieved & modified using the API
    • Introduce query parameters to query for posts that are NOT IN one or many terms of specific taxonomies
    • Resolve bugs, including bad comparison logic when updating comments.

    Please try it out and report any outstanding issues; the REST API project gained its 90th code contributor this week and the team is deeply grateful for the energy and support of the broader WordPress community in testing out this merge-candidate plugin!

    New Questions & Discussion Items

    Items which have arisen through final ticket triage & review on which the team seeks feedback:

    1. Should the `filter` shim should be removed prior to merge? It is the majority position of the API team that `filter` be deprecated to dramatically improve the simplicity and consistency of API query functionality
    2. How should comments be handled for password-protected posts? Should the password be passed as a query parameter with the PUT/POST request, or is there a better option?
    3. Should the API match core’s logic when users with the `unfiltered_html` capability are creating or updating Posts or Comments?

    Meeting Notes

    At the weekly team meeting on October 10 the group reviewed open issues in the 2.0 milestone, which represents the candidate for our merge proposal shared last week.

    Meeting attendees agreed to review open issues and pull requests individually, and to reconvene on Tuesday at 1500UTC to ensure all priority tickets had an owner.

    At that meeting on October 11, the team reviewed the incoming feedback around the OAuth plugin (linked above). While the API team feels that having a built-in authentication solution provides a much-needed service, particularly to developers building mobile and desktop applications, the design and usability feedback we have received does indicate that the plugin needs more work.

    OAuth’s place in the Merge Proposal

    The API Team believes that the identified issues are resolvable, that the OAuth plugin is on track and that it should still be considered for merge in 4.7. However, after discussion within the team, input from @matt, and advice from @aaroncampbell and other core committers, we have edited our merge proposal to submit the Content API Endpoints and OAuth server as separate merge candidates. The API Team proposes both components for merge, but we submit the content endpoints for consideration independently of the OAuth1 server.

    Content Endpoints Without OAuth

    The Content API endpoints are stable, well-tested, and in wide production use across a variety of applications. Theme and plugin developers will benefit from having canonical, well-tested API endpoints in core, which may be used to query WordPress both from PHP code and from JavaScript applications running on the front-end or admin of WordPress. Sharing the endpoints for core data types enables increased consistency of what data is exposed and how it is persisted across different plugins, improving consistency and shortening development time by using . These themes and plugins have full read and write access to the API using the existing cookie & nonce authentication.

    Mobile and desktop applications can leverage these same endpoints in a read-only capacity to create a variety of powerful reader-oriented applications and tools that expand the capability of what WordPress can do today, such as a unified reader for Make WordPress blogs and other experiments hypothesized by @jorbin.

    Should OAuth 1 not be accepted for 4.7, secure write access for these external applications would still be only a plugin install away; and while having an OAuth server in core will provide a canonical approach for authenticating from remote applications, depending on the needs of a specific site or specific client application other authentication schemes may actually be preferable. Plugins exist for JWT Authentication and of course OAuth 2, and should OAuth 1 not be accepted for 4.7 these plugins may still be installed to enable an external application to opt-in to secure write access to your WordPress site.

    In Summary

    The API team submits for 4.7 merge consideration two enhancements to the REST API infrastructure: the Content API Endpoints for core WordPress datatypes, and an OAuth server which will reduce the setup time needed to securely interact with those endpoints from outside of WordPress. We believe these enhancements are each individually sufficiently tested and mature to meet the quality and security standards of WordPress Core, and each individually provides wide-reaching benefit to WordPress developers, and through them to the authors, readers & publishers of the web.

  • Matt Mullenweg 8:27 pm on May 12, 2016 Permalink

    New lead for 4.7: Helen Hou-Sandí 

    Due to some unexpected constraints on my time this year, I’m going to be stepping down as the 4.7 lead, and I’m happy to announce that Helen Hou-Sandí is stepping up to lead the release in my stead. I’ll still be behind the scenes providing whatever support is necessary, and I’m really looking forward to the release . You might remember Helen for her famous work on WordPress 4.0.

  • Adam Silverstein 4:58 pm on February 5, 2016 Permalink
    Tags: ,   

    REST API meeting summary, Feb 4 

    The core REST API team, members of the WordPress.com API team, and other interested developers met to discuss the REST API.  Full logs of the meeting are available on Slack.

    Existing endpoints

    The meeting opened with a discussion of the existing endpoints: what is and isn’t ready. @rmccue summarized:

    • The main focus has been on the 4 “core” objects in WordPress: posts, terms, comments, and users.
    • Posts
      • Password-protected posts are going to be ignored from the API until we have a good solution.
      • Meta
        • Post (and other) meta has been pushed out into a separate feature plugin led by @jeremyfelt. Discussion is on the github repo.
        • The main issue is that there’s no differentiation between meta fields. Meta could be added via the Custom Fields meta box, or by a plugin.
        • Enhancing register_meta() in core would allow opt-in to the meta REST API – requiring some core work. See the patch on #35658 for details.
        • There is no current way to explicitly register meta as public.
        • We need to support object meta (post, user, comment, term) and object types (custom post types, etc)
      • Media
        • Mostly complete (it’s a custom post type).
        • Some special handling around uploading media.
      • Autosaves and post previews
        • Tricky, but we think we have a solution for that moving forward.
        • Working on them in a separate feature plugin instead.
        • This will be an enhancement to the API in the future, and doesn’t block compatibility.
    • Terms
      • Work great.
    • Users
      • Works great; undergoing final review.
    • Comments
      • Works; custom comment types are harder until core better supports them.

    Merge proposal and discussion

    An extensive discussion ensued regarding the possibility and appropriateness of merging the endpoints into core. The discussion continued for over an hour; mainly covering whether we should merge the 4 core endpoints, or wait for a complete API before merging…  and what the definition of a complete API is.

    The REST API team’s proposal is that we merge the 4 core objects in the REST API with full read, create, update, delete support, then add more peripheral features when they’re ready. 

    @matt argued that we wait until the API supports everything you can do in wp-admin before merging.

    The discussion revolved around these proposals and what is best for the WordPress project. Would merging the existing well developed endpoints sooner help or hinder developer adoption, and further iteration/improvement? Would delaying the endpoint merge help or hinder progress?

    Here are some key comments from the discussion.

    • @rmccue  The approach to the API needs to change from “the API team creates all of the API” to “the API team controls the general shape of the API, and each team works with it”
    • @danielbachhuber Options / a Site Resource is more easily viable, as are Plugins and Themes. Widgets and Menus essentially need to be reinvented before they’ll work in a REST API
    • @jnylen It’s a question of when, and how much testing can be done first. Shipping an API that is read-only and incomplete in several key areas feels like a big mistake.
    • @kadamwhite we’re shipping an API with write capabilities but only cookie-based auth will be supported OOTB.
    • @matt  I would be pretty skeptical of merging a partial API into core…  no partial endpoints in core. let’s design a complete API, not half-do it.
    • @rmccue The API is specifically structured around progressive enhancement
    • @jorbin I think only supporting the four core object types allows for some amazing themes and amazing content editors to be created, but doesn’t allow for full wp-admin replacements. I’m ok with that.
    • @jnylen From what I’ve seen so far, I’m most concerned about shipping individual endpoints with missing features.
    • @jorbin I think the best way to pick up the pace is to get key things locked up and get the four core object types in core. This would help core develop API-first for those features
    • @codebykat On the WPCOM side, we’re committed to taking the plunge, but we’re not there yet which means things are untested.
    • @jnylen I think this needs more testing at large scale before shipping, including some of the more difficult and obscure features of WP.
    • @drew Shipping with full wp-admin replacement capability is unrealistic, imo. We need something flexible and stable that developers can use as solid jumping off point. All the rest can get separately iterated.
    • @mike I think that, realistically, to ship the API… we need an MVP, and I don’t think that defining our goalpost as “all of wp-admin” fits that criteria.
    • @matt Full wp-admin coverage is a firm goalpost, it gives us a super clear criteria for when it should be merged into core. is everything possible in wp-admin, possible through the API?
    • Matt Mullenweg 7:12 pm on February 5, 2016 Permalink | Log in to Reply

      We got on a bit of a rabbit hole on file editing in WP, and whether that should be in any API.

      I’m inclined toward no, but I think decisions about what we’re not going to support in the API should be made as deliberately as if we were removing the feature from WP itself. We should make that sort of decision explicitly and proactively, not allow things to die on the vine of existing in wp-admin but not the API.

      It’s also going to be difficult because there is often a big gap between what developers do and what people for whom WP is their entire interface do, which file editing accidentally illustrates very well. (What developer would trade their IDE or text editor for a form on a web page?)

      It might be easier to actually build out APIs for everything first and then choose whether to ship them or not than argue about whether they should be built in the first place, which is inevitably a realm of opinion, values, and sparse data.

      • Omaar Osmaan 7:47 pm on February 5, 2016 Permalink | Log in to Reply

        Absolutely- I like the idea of having API in core that allows to replace WP-Admin without loosing any feature it has. While so far I do have the itch of “get it in core faster”, but I really think your strategy is appropriately sound.

        It’s great to see the arguments- and for sure it all would make it only better in the end.

        My 2 cents.

    • Justin Sternberg 8:11 pm on February 5, 2016 Permalink | Log in to Reply

      Agree on iteration/MVP approach. Full wp-admin coverage, to me, is way overkill for initial merge to core. There is value in “paving the cow paths”… Let’s get the MVP working 100% and merged, and figure out what the next most valuable addition will be. Rinse and repeat.

    • K.Adam White 9:05 pm on February 5, 2016 Permalink | Log in to Reply

      The wp-admin parity argument caught me off-guard, which I believe stems from there being two basic types of use-case. Both types of usage are (and should be) valid:

      We (Bocoup, a non-WP-oriented company) have used the WP-API project to take advantage of the existing wp-admin editing interface, and has used the API solely to flow WordPress-authored data into or out of other applications. Without the API we would never have considered using WordPress for these projects: but the existence of a RESTful/non-XMLRPC interface made WP a big win for both us and our clients.

      Matt’s / Automattic’s experience is the complement to ours: Maintain the existing theme/plugin front-end, but put a custom admin interface behind it.

      The original “json-api” plugin was created by MoMA to flow WP data into a Ruby project; We’ve used the API to flow data into Node applications, and Java and C# clients for the WP-API are also under development. Our use-case is not new, and I believe it represents a significantly larger _potential_ target audience than any wp-admin re-skin does (at present). Daniel Bachhuber hypothesized recently that the way we’ll get WP to “100% of the web” isn’t by moving those sites to WordPress, but rather to allow WordPress to be integrated with existing platforms where appropriate.

      Waiting for API parity with wp-admin suits the Calypso use-case, but delaying core integration of API endpoints for basic WP data types will effectively block a significant group of potential adopters coming from external platforms. Right this second may not be the time, but unless we roll the API into core progressively (hopefully no later than 4.6/4.7) I think we are missing a tremendous opportunity.

      “ — Thank you to everyone involved with the project for being involved in this discussion.

      • Jeremy Clarke 5:33 pm on February 15, 2016 Permalink | Log in to Reply

        +1 Thoughtful and vital reply. I agree that getting the core types in is really valuable, and that seeing the situation from all angles (backend-simulation+frontend-simulation) leaves me feeling like waiting for full wp-admin support is very biased towards certain uses and not others.

        FWIW at Global Voices we have a third use-case that I think is really valuable: Communication between WordPress sites which each use wp-admin+themes in the normal way.

        We rebuilt our translation system, which lets separate WP sites translate each others content, to use the old 0.x version of the feature plugin. Our users create a draft and give it the URL of the post they will be translating, then the API negotiates the translation and delivers the post content which is inserted into the new draft. It works great and replaces a kludgy+insecure old trackback-like system we were using. We didn’t want to hitch ourselves to a plugin for such a vital tool, but the feature plugin felt safe and we knew it would be in core eventually.

        Fast forward and many versions later we’re still waiting, and are seeing multiple, incompatible versions of the plugin coming out. For now we’re still using legacy version and waiting for the endpoints to drop into core before refactoring all our work. In our case the 4 proposed endpoints would be perfect for our needs, so getting them into core ASAP would be a huge boon and make us feel a lot more confident. It would also be really nice to formalize the underlying concepts so any new endpoints we create are modeled after how core will be doing it.

        As with K.Adam infinite thanks to all those actively working on this!

    • Mike Nelson 4:44 am on February 6, 2016 Permalink | Log in to Reply

      > I would be pretty skeptical of merging a partial API into core… no partial endpoints in core. let’s design a complete API, not half-do it.

      Did @matt suggest that thinking it would be easier to iterate the API as a plugin than in core? Or what’s the motivation for this suggestion? I think lots of us give his suggestion a lot of weight, but because I wasn’t at the meeting I don’t understand it

      • Matt Mullenweg 9:30 pm on February 8, 2016 Permalink | Log in to Reply

        I’m 100% for continued iteration, and yes I think that would be easier as a plugin than in core.

        Everyone who wants the functionality today can have it in the plugin with just a few clicks, and we can update that with new functionality as much as often as we want, and if something goes wrong in only affects ~10k users. We know that most WP sites have 5 or 6 active plugins, with professionally-developed sites typically 2-3x that.

        In core we can only do major updates 3 times a year, at most, bundled with lots of other things and concerns that go into a major release, and those updates go out to tens of millions of sites. If we break something it’s in the mainstream tech news. What’s included or not depends on the release lead.

    • Philip Arthur Moore 11:03 am on February 6, 2016 Permalink | Log in to Reply

      I haven’t been involved in the development of this at all, but as a plugin/theme/custom web solutions developer I’d have to strongly agree with @matt with regard to setting full /wp-admin/ coverage as a firm goalpost before inclusion. What’s been keeping me from diving into the API and using it over what we have now is that I’m afraid of changes that may break things and not really sure what I can do with /wp-admin/ that I can’t do with the API. If there’s full parity then I can confidently jump into using the API; until then I feel largely like a spectator seeing how the development of the API plays out before overcommitting our resources to learning it and developing for it.

      • Jeremy Clarke 5:37 pm on February 15, 2016 Permalink | Log in to Reply

        This can swing both ways though. You don’t want to use the API if it’s still in flux, but a constantly-developed plugin will be in flux by default.

        For those of us already using it (the oxygen an API needs to grow and evolve) having a stable platform to work with (even if it’s limited) is really important. Getting the main endpoints into core and having them be treated with the usual obsessive back-compat would make our lives easier and increase the number of people willing to invest time and money into using it.

    • Mike Nelson 1:04 am on February 9, 2016 Permalink | Log in to Reply

      I didn’t think [putting the WP API scaffording in core would] increase the plugin usage, I thought it would increase plugins registering their own APIs, because they hadn’t before because of cross-plugin dependency see https://wordpress.slack.com/archives/core-restapi/p1454633785001643

      I don’t know about other plugins, but at least at Event Espresso we put our REST API, which depends on the WP API infrastructure, into our main plugin directly as a result of putting the WP API infrastructure into WP core 4.5. (See https://eventespresso.com/2016/01/rest-api-now-in-ee4-core/)

      We had some reservations though, thinking that maybe we would wait until the WP API endpoints were also in core (because that would be proof the WP API infrastructure would be more solid and less likely to break). It’s possible other plugins are having the same reservation?

      What’s more, the Event Espresso REST API is a little unique in that we’re using a LOT of custom tables, and our functionality is quite independent of WordPress’ posts etc, and so our REST API can exist quite happily without the WP API’s endpoints. I suspect most plugins’ APIs are much more dependent on WP API’s endpoints in order to work. Eg, there’s not much point in a YARPP REST API without WP API. Meaning, if WP API’s core endpoints are still in a plugin, their users STILL need to use the WP API plugin in order to use their plugin’s API… so just adding the WP API infrastructure isn’t much help to them: they need the endpoints too.

    • rclilly 7:18 pm on February 12, 2016 Permalink | Log in to Reply

      After reading this summary, as well as the entire discussion in Slack, I get the sense that, to a certain extent, people are talking past each other.

      @matt, I really don’t understand why you insist on ALL possible endpoints being completely ready before ANY endpoints can be merged into core.The four that are proposed cover a huge part of what many want the REST API for in the first place. I definitely think those four make for a minimum viable product, assuming they are, indeed, ready for prime time. The ability to completely replace the wp-admin goes way beyond an MVP for a REST API.

      My two cents: If there are endpoints ready for prime time, merge them. Let the others remain in the plugin.

  • Helen Hou-Sandi 7:56 pm on December 9, 2015 Permalink

    Welcome the 4.5 class of committers! 

    As announced in the State of the Word this year at WordCamp US by @matt, there are seven new committers to introduce.

    Many of you have seen Michael Arestad‘s (@michaelarestad) design and front-end development contributions over the last couple of years, notably with the redesign of Press This in WordPress 4.2. His numerous, high quality contributions are a welcome addition to core. I personally am looking forward to his work on markup and styling, having relied heavily on his judgment for quite some time now.

    WordPress 4.4 adds a new embed feature to WordPress, making it an oEmbed provider for the first time. Work on this new feature was done in a large part by Pascal Birchler (@swissspidy), who has been doing great work for the past few releases. Pascal’s clear communication and thorough support of the flow mindset are things we can all be inspired by.

    Rachel Baker (@rachelbaker) is the co-lead of the REST API, a Comments component maintainer, and a major contributor to WordPress 4.4. Her work has made it possible for sites around the world to utilize the REST API, making WordPress a great application platform. Look for more of these contributions as the REST API iterates within core.

    Likewise, Joe Hoyle (@joehoyle) is a major contributor to the REST API. As we prepare to commit the REST API endpoints in an upcoming WordPress release, there will be more and more to come from both him and Rachel.

    As a Media component maintainer and a long-time contributor across many components and features, Mike Schroder (@mikeschroder) helped shepherd the responsive images feature plugin into core for WordPress 4.4. He was also a backup release lead for WordPress 3.9.

    Throughout the WordPress admin interface, everywhere you look you’ll see the work of Mel Choyce (@melchoyce). Her design and experience contributions are long-standing and have benefited the entire ecosystem. As one of the maintainers of the Dashicons project, the icons you interact with daily are a big part of her contributions, as well as themes available in the WordPress.org Theme Directory.

    Eric Andrew Lewis (@ericlewis) has been contributing in various forms for many years, exploring lesser-known areas, documenting them, and challenging assumptions. Most recently, you may have seen his work as a Media component maintainer or with the shiny updates feature in WordPress 4.2.

    Additionally, Ella Van Dorpe (@iseulde), Konstantin Obenland (@obenland), Weston Ruter (@westonruter), Tammie Lister (@karmatosed), Andrea Fercia, (@afercia) and Ryan McCue (@rmccue [that’s one M, two C’s]) have all had their guest commit renewed.

    Please join me in welcoming this great set of new committers!

  • Helen Hou-Sandi 7:37 pm on December 9, 2015 Permalink

    Announcing the release leads for 2016 

    As announced during the State of the Word this year, we have a brand new selection of release leads for 2016.

    Mike Schroder
    Previously a co-lead for WordPress 3.9 and long time contributor, Mike Schroder (@mikeschroder) will kick off the year as the release lead for WordPress 4.5.

    Dominik Schilling
    Following WordPress 4.5, Dominik Schilling (@ocean90) will be the release lead for WordPress 4.6. Dominik has been a core committer for a couple of years now and was a backup release lead for WordPress 4.4.

    Matt Mullenweg
    Finally, closing out the year, Matt Mullenweg (@matt) will put on his release lead hat and lead WordPress 4.7. Matt previously led the WordPress 3.8 release.

    Each of these release leads need your help! Every release is made by hundreds of contributors over many months, not just by its release lead. Additionally, every release lead needs a backup lead or two to help ensure the release moves forward at a solid pace. These backup release leads get great training for the real deal, as they often become future release leads (see both Mike and Dominik above!).

    Are you interested in being a backup release lead? Just comment here to let Mike, Dominik, and Matt know.

  • Weston Ruter 2:00 pm on September 23, 2015 Permalink
    Tags: , ,   

    Outlining a possible roadmap for the Customizer 

    Planning for the future is a necessary and important part of the WordPress development process. As we consider the future of WordPress – both as a whole and individual features – we publish proposed roadmaps to encourage greater discussion and give insight into the core team’s thought process.

    The process of creating a roadmap is just as important as the vision behind it and the final roadmap itself. This process gives the entire community an opportunity to research and document history, define what specific items can be accomplished to bring us closer to the vision, and outlines how those tasks fit together within a possible timeframe.

    What follows is a potential roadmap for the Customize component. If you’re interested in the future of live preview in WordPress, now is the perfect time to get involved and leave your feedback.

    A couple of months ago, the WordPress lead developers met with the maintainers of the Customize component to discuss the future of live preview in WordPress. The goal of the chat was to come up with a potential roadmap for both the component and for how live preview can improve the user experience of WordPress for all users.

    The ultimate goal of live preview in WordPress is to create user trust and remove the “save and surprise” inherent in some of the backend features.

    After a lot of discussion, the group decided to target the following goals over the next two years:

    • Considerably improve performance.
    • Continue iterating on current live preview features to ensure they are solid and as easy-to-use as possible, including theme browsing and installation, menus, and widgets.
    • Experiment with new and different user interfaces. If we were creating live preview today, what would it look like? In what ways can we ease the feeling that you’re looking through a “porthole”?
    • Removal of the ambiguous mode. Currently, the Customizer is contained in a sidebar without the admin toolbar, but ideally there is the admin and the theme, and no in-between. One direction this may go is enabling “Customize” on the front end to immediately load the Customizer controls.
    • Experiment with a guided new user experience (NUX). Live preview lends itself to site setup. How can we improve the live preview experience and combine it with the NUX? Consider a “setup wizard” use case and ensure the flow has no dead ends, i.e. users can customize everything in one.

    Those overall goals for live preview in WordPress can be rewritten into some specific features that are in development or planned for the future of the Customize component. These include:

    • Transactions. This re-architecture of some of the Customizer internals improves compatibility with themes by loading the preview using a natural URL, and allows Ajax requests or even REST API requests to be previewed. It also allows the preview to be viewed independently of the Customizer, so changes can be shared for others to review. See #30937.
    • Selective refresh. Only a piece of the page will need to be refreshed when this backend feature is implemented. (Formerly known as “Partial Refresh”.) Currently, this is available for menus in the Customizer. This eliminates duplication of display between PHP and JS, keeping it DRY. See #27355.
    • Concurrency. Allows for “locking” settings using the Heartbeat API, improving the overall user experience by preventing users from overwriting each other’s changes. See #31436.
    • Revisions. Enables plugin developers to add features like draft, roll back, and scheduled changes (e.g. “change my background on January 1”). This builds upon transactions, as the setting changes are staged in a transaction, and this facilitates settings to be revisioned and for settings to be scheduled. See #28721, #31089.
    • Theme installation. Iterates on and completes the theme browsing experience.
    • Responsive preview. Iterates on the concept of live preview by giving users a better idea of what their site will look like on other devices. See #31195.
    • Bootstrapped Customizer. Lazy-load the Customizer into the current frontend view without having to leave the page. With selective refresh implemented, inline controls and frontend bootstrapping would be possible since full-page refreshes would no longer be required.
    • Improvements for both touch and small devices.

    Beyond those features, the group identified some specific changes that should be prioritized, in conjunction with the features planned:

    • The sliding animation between panels should feel more like “moving panels” (see: iOS).
    • Keyboard navigation should be consistent and clear.
    • Identify “dead ends” in the interface and remove them, when possible. For example, prior to menus in the Customizer, it was not possible to customize that aspect of your site’s design with the Customizer.

    The concepts surrounding live preview and the Customizer have been in development for a long time. Many of the concepts from Elastic Theme and the Visual CSS Editor have been incorporated over time. Over the next few years, experimentation with these concepts will likely take place in feature plugins. For example, this team may experiment with inline content editing, where it makes sense in the context of customizing a site. Another path for exploration is simple theme customization – e.g. change the header font, change the sidebar color, or change the width of the sidebar.

    As with all components and new features, we shouldn’t be afraid to experiment and fail and should continually push for new experiments and ideas, especially in the context of feature plugins. Further, some of the above experiments may not make it into core, but are meant as a general direction that live preview should take in WordPress.

    Taking these features together, below is a sequence outlining a possible roadmap for live preview and the Customize component in general, along with estimated targets. Please note that this is a proposed roadmap and is entirely dependent on contributor involvement. Additionally, many of these things will take place in a feature plugin prior to core inclusion.

    • Partial refresh. Performance Improvements. (Target: 4.4)
    • Responsive Preview. Transactions. (Target: 4.5)
    • Concurrency. Revisions. Theme Install. Beginning of NUX wizard. (Target: 4.6)
    • Focus on touch screen / small device improvements. (Target: 4.7)
    • Developer API improvements based on feedback from plugin developers. (Target: 4.8)
    • Improved UI after experiments in 2016. NUX “wizard mode.” (Target: 4.9)

    Live preview is one of the most critical features in WordPress as we continually combat “save and surprise.” The Customizer in its current form provides an improved user experience to WordPress users when customizing their site’s design. Each feature mentioned above is a continuation of the live preview concept, building and improving upon the Customizer.

    Everything above is just a proposal and we need your feedback to ensure it is the right direction. If you’re interested in any of the above, comment here with your feedback, or join the team in #core-customize.

    This post was a collaboration between @helen, @nacin, @mark, @celloexpressions, @samuelsidler, and yours truly.

    • Davide 'Folletto' Casali 4:46 pm on September 23, 2015 Permalink | Log in to Reply

      > NUX wizard

      I find a bit dangerous to think of it as a “wizard”, mostly because usually wizard UIs have specific connotations. We should think of it more as a “tutorial” like games do. Not separate UIs, but a guide through an existing UI that can be dismissed if you’re expert, or done partially, or followed all along. 🙂

      Other than that… this seems AWESOME. 🙂

      Performance is really the main thing there. That alone would reap huge benefits. Then you add Live Preview and well, boom! 🙂

    • Daniel Bachhuber 6:11 pm on September 23, 2015 Permalink | Log in to Reply

      We (Fusion) are very keen to adopt the Customizer as the UX for frontend management, and look forward to helping out with the feature plugins built as a part of the roadmap.

    • FolioVision 12:43 am on September 24, 2015 Permalink | Log in to Reply

      Weston, thanks for the comprehensive and forward thinking report.

      It would be great if the interface paradigm for Customizer would remain constant for the exterior world (i.e. agencies and clients) while a dev version gets tested by the more progressive among us.

      Perhaps a switch to throw in settings to enable experimental Customizer features? The switch could be retired when Customizer settles a bit.

      • Weston Ruter 4:05 am on September 24, 2015 Permalink | Log in to Reply

        @FolioVision yes, absolutely. This is why most of the new features will be developed in feature plugins so that they will remain opt-in until they are deemed solid for everyone.

    • devolute 4:05 pm on September 25, 2015 Permalink | Log in to Reply

      but ideally there is the admin and the theme, and no in-between.

      Here here! Does this mean that there will be a “front-end” mode, and a “dashboard” mode that fits into the current dashboard so that things can be altered without a preview being available?

      • Weston Ruter 4:16 pm on September 25, 2015 Permalink | Log in to Reply

        @devolute Basically the idea is that there would no longer be a separate “Customizer” URL (wp-admin/customize.php) that you go into. When on any frontend URL, clicking “Customize” would just slide out the Customizer sidebar pane in place. You’d still have the admin bar available and everything else that you would normally when on the frontend, except you’d also be able to make changes to the Customizer as you browse around. So no, this doesn’t have in mind making the Customizer controls available in the wp-admin without a corresponding preview.

        • devolute 4:19 pm on September 25, 2015 Permalink | Log in to Reply

          Ah, I understand your intention. I think my problem is that I’m using the customizer for things like phone numbers, addresses and other options when I perhaps should look at other solutions.

    • Ahmad Awais 3:38 pm on September 26, 2015 Permalink | Log in to Reply

      We at WPTie are slowly yet gradually adopting Customizer in all our themes and plugins (where needed). All in for customizer support in WordPress.

    • Alex Mangini 10:23 pm on September 28, 2015 Permalink | Log in to Reply

      On the “Bootstrapped Customizer” point, is this a move towards inline content editing and possibly even “page builder” type concepts?

      My biggest curiosity about the Customizer and what I really want to use it for as a developer is creating posts and pages without a 3rd party visual editor and without the drawback, as already stated, of guessing what they’ll look like after inputting content into specially built meta boxes or widgets.

      Thanks for doing these write-ups, as I build more and more WordPress plugins/products I find myself lurking around here to better educate myself and plan roadmaps for my own work.

      • Weston Ruter 10:45 pm on September 28, 2015 Permalink | Log in to Reply


        On the “Bootstrapped Customizer” point, is this a move towards inline content editing and possibly even “page builder” type concepts?

        This is already possible, actually. For an example of inline editing, see the Customize Inline Editing plugin. And page builders are also possible now by using widgets in the Customizer. I’m currently working on a site that uses a concept of query-specific widget areas to allow every URL to have custom sidebars configured, where the main content area is just another widget area: this allows drag-and-drop building of pages just with standard widgets and any custom widgets that you define. Widgets still have a lot of room for improvement in Core, but I’m very keen to see them move in the direction of “content blocks” which can be used for building page templates in the Customizer.

    • Nick Halsey 4:19 am on September 29, 2015 Permalink | Log in to Reply

      Thanks for getting this published Weston.

      A couple of additional comments on my end:

      • We need more help. The Customizer can do (and could do) a lot, but without more contributors (not just code), it’ll take time to see big improvements go through. I’d love to see more of an effort to embrace the Customizer throughout various other core projects, and to see more community involvement in Customizer design and development.
      • In the interest of experimentation and building on the benefits of live preview, I’m planning on looking at some ways to integrate things like page templates, post formats, and featured images (with cropping) into the Customizer. While there’s a good chance none of those things will make it into core, I’d like to encourage anyone with any ideas for the Customizer to try them out and offer them for discussion. We’re no where near the limit of what users could do with the Customizer and it’s an amazing framework to build additional features on top of.
      • While many of the topics in the post are fairly developer/backend oriented, it’s important to emphasize that there is still a significant amount of design and UI/UX work to be done, including looking at entirely different approached to the live preview UI concept. Don’t be shy to share your ideas if you have any – the technical side of things is evolving to support pretty much anything we may want to try.
    • pingram3541 3:59 am on November 24, 2015 Permalink | Log in to Reply

      A little late to the game but I just wanted to say thanks for pushing the Customizer forward. I do feel it is given the least amount of attention when it comes feature progression and I can’t figure out why. Yay for menu’s but how did menu’s make it on the plate before posts??? Seriously folks, this is a huge mechanism in which we are losing ground fast to many of the new front end content builders on the web and the Customizer is behind the times.

      @westonruter your example of the Customize Inline Editing is nice but seems only to serve a very narrow focus on ability as it still provides no capability of inline editing for actual post content/post meta. Don’t get me wrong, great working example but real world we need to break outside the box of wp_options, theme_mods…we need simpler access to post objects and other areas in WordPress.

      Again @westonruter looking at your code on the Customize Posts plugin I see that your way of getting around the lack of post object availability is by providing a drop down of all posts/pages of which to perform the query against which is genius. However, it doesn’t seem natural to have to select a page/post when you’re already viewing one in the Customizer already and when needed can navigate to other post/page where their post fields could be provided rather than choose it from a long drop down list, which could become unmanageable on some larger sites.

      So yes, I’m excited for the future and the BIG ONE thing I’d love to see is:

      Access to the current preview’s post object or at least a way to capture the post id for use with $customize_update and $customize_preview. Could be as simple as adding the post id to the $wp_customize object.

      • Weston Ruter 5:54 pm on November 25, 2015 Permalink | Log in to Reply

        @pingram3541 Thanks for the reply.

        I do feel it is given the least amount of attention when it comes feature progression and I can’t figure out why. Yay for menu’s but how did menu’s make it on the plate before posts???

        There are a limited number of people to contribute to each release. @carlhancock wishes that all admin management features could be added to the Customizer at one time instead of one piece at a time, but this is not practical for an open source volunteer effort with three releases a year. As we just saw with Automattic’s Calypso announcement from @matt, it took almost two years and dozens of contributors to re-develop the entire WordPress.com admin into a new JS-driven interface. We don’t have the time or the person power to do that. However, I think that interest is building, and with that, contributors will come.

        Compared to posts, it is relatively much simpler to build support for nav menus in the Customizer. They have a UI that fits in the Customizer pane, and the data model is pretty normalized. Posts, on the other hand, would need a full-on editor experience (inline to the preview) and can have an infinite number of variations to display (shortcodes, media embeds, etc). This is a hard problem to generalize for all themes, but it is something that @iseulde has worked hard on in the Front-end editor plugin. The Customize Posts plugin was developed as a prototype to show the architecture for how posts and postmeta can be managed in the Customizer so that the Front-end editor (or another such plugin) could leverage the Customizer for previewing and staging changes.

        your example of the Customize Inline Editing is nice but seems only to serve a very narrow focus on ability as it still provides no capability of inline editing for actual post content/post meta. Don’t get me wrong, great working example but real world we need to break outside the box of wp_options, theme_mods…we need simpler access to post objects and other areas in WordPress.

        As noted in the plugin’s description, Customize Inline Editing is an example (proof of concept), again to show how inline editing can be implemented in the Customizer. So its focus is intentionally narrow. The approach taken there can be implemented for managing post data inline. See also wp-customize-posts#8.

        on the Customize Posts plugin I see that your way of getting around the lack of post object availability is by providing a drop down of all posts/pages of which to perform the query against which is genius. However, it doesn’t seem natural to have to select a page/post when you’re already viewing one in the Customizer already and when needed can navigate to other post/page where their post fields could be provided rather than choose it from a long drop down list, which could become unmanageable on some larger sites.

        Actually, it’s not a dropdown of all posts/pages. It’s a list of the posts that currently appear in the preview. The dropdown’s options change as you navigate around the site in the preview, meaning that you should only be editing a post in the Customizer that is actually visible in the preview.

        So yes, I’m excited for the future and the BIG ONE thing I’d love to see is: Access to the current preview’s post object or at least a way to capture the post id for use with $customize_update and $customize_preview. Could be as simple as adding the post id to the $wp_customize object.

        The future is now. As noted above, Customize Posts does it to communicate the list of posts in the preview to the pane for the dropdown. Also @ericlewis has implemented this in wp-custom-css-per-post. I also have on my todo list to write a post on how to create a “metabox” in the Customizer for managing postmeta for the current singular post in the preview.

    • Philip Ingram 6:57 pm on November 25, 2015 Permalink | Log in to Reply

      Thanks for the quick and detailed response. I’d love to contribute as long as my foo is strong enough, let me know how I can help or if there is a more appropriate forum for continuing this discussion =)

      Additionally, regarding @ericlewis ‘s wp-custom-css-per-post, I already have a working live css editor using codemirror to provide live updating of the preview as well as updating the hidden Customizer input setting transport to postMessage and this works great allowing live preview before saving but also allowing the saving. This has greatly improved efficiency of my front end design workflow. However this currently is limited to my site-wide custom css via theme options. What brought me here in the first place was seeking out a way to GET and POST the current posts meta which I have a custom css per post option built into my themes. So maybe I can get with @ericlewis and we can figure this out. Looking at WP Customize Posts I see the query for $_POST[‘customized’] to iterate though and retrieve post id’s but when using the Kirki toolset (I’m sure the class needs something extra for this) it’s always an empty array upon initial load of the Cusotmizer and that is where I’m stuck currently. I just need the darned post id!

      You mention that posts are way more complex than menus and yes there’s no debate there, however in the time being, if there was a simple way to capture the current previewed post’s id or maybe a clearer example how to retrieve it, that opens the door for us to pioneer further without waiting for a more core-driven method of updating post data via the Customizer.

    • Philip Ingram 10:20 pm on November 25, 2015 Permalink | Log in to Reply

      @westonruter Yes! Thank you for this.

  • Robert Chapin 10:10 pm on September 2, 2015 Permalink
    Tags: , ,   

    Shortcode Roadmap Draft One (Proposal – Request for Comments) 

    This is the first draft of the Shortcode API Roadmap. It describes in broad terms a new feature set and migration that might take place across versions 4.4 through 4.7. This roadmap gives notice to plugin developers that significant changes in plugin design may be needed for compatibility with future versions of the Shortcode API. This roadmap also identifies steps taken to minimize the impact on plugin developers to allow most plugins to continue working with only small changes.

    The decision to create this roadmap arose from specific needs that are not met by the old code. Our old [ and ] delimiters were easily confused with the way those characters are commonly used in English quotations and sometimes even in URLs. The proposal to use [{{ and }}] instead should allow a better balance between being able to type in the shortcodes and avoiding confusion with any other input. With these more unique delimeters, we will be able to process registered shortcodes more efficiently because we will not have to search for them by name. Unregistered shortcodes will have more consistency because we can find them more accurately.

    Old style delimeters also gave no strong indication whether or not a closing tag was required. The proposal to use }$] as the delimeter of a shortcode with a following closing tag increases the efficiency of regular expressions, because the search for a closing tag will only happen as needed.

    Adding the new style of shortcode syntax provides an opportunity to make significant API changes. One of those major changes is to ensure that shortcodes are processed before paragraphs and before curly quotes. This will lead to greatly simplified code in related functions that currently must find and avoid shortcodes every time they run. We also have an opporunity to re-think the way shortcodes are filtered, and to give plugin authors more control over those filters when registering their shortcodes.

    4.4 – Introduce New Syntax

    A new syntax and documentation of how it works will be created in the 4.4 development cycle. Support for the new syntax will be introduced, allowing plugins to register for extra features. Core shortcodes will use the new syntax in all new posts. The old syntax will not change. Old posts will not be affected. Initial work on the syntax concept follows.

    Proposed New Syntax

    Self-Closing:  [{{shortcode}}]
    Attributes:  [{{shortcode  attr1="value1"  attr2='value2'  "value3"  'value4'  value5}}]
    Enclosing:  [{{shortcode}$] HTML [${shortcode}}]
    Multiple Enclosures:  [{{shortcode}$] HTML [${encl2}$] HTML [${encl3}$] HTML [${shortcode}}]
    Escaped Code:  [!{{shortcode}}]
    Stripped Unregistered Code:  [{{randomthing}}]
    Stripped Unregistered Enclosure:  [{{randomthing}$]  Content also stripped.  [${randomthing}}]
    Stripped Empty Code:  [{{ }}]
    Ignored Orphan:  [{{shortcode}$]
    Ignored Orphan:  [${shortcode}}]
    Ignored Orphan:  [${encl2}$]
    Ignored Context:  [{{shortcode <br> }}]
    Ignored Context:  <a href="[{{shortcode}}]">

    4.5 – Deprecate Old Syntax

    Starting in 4.5, plugins that register shortcodes without declaring support for new features will raise debugging errors to alert developers that support for the old shortcode syntax is ending.

    Old posts will continue to work normally while the old syntax is deprecated.

    4.6 – Upgrade Old Posts

    Old posts will be automatically converted to the new shortcode syntax. The Shortcode API will continue to provide deprecated support for old syntax so that there is no disruption during the conversion process.

    The API should be adequately abstracted so that old plugins are not affected by this conversion. However, as the new syntax will not support HTML inside of shortcode attributes, there is no guarantee that every shortcode will work the same way in 4.6 as in earlier versions.

    4.7 – End of Support for Old Syntax

    Old shortcodes will stop working in 4.7. Plugins that still produce the old shortcode syntax will be ignored by the Shortcode API.

    The upgrade to 4.7 will include a second pass of the conversion of old posts so that any old syntax that was added to posts during 4.6 will still get converted.

    In 4.6 or 4.7, if necessary, a filter could be added to automatically convert any old syntax still being produced by old plugins when new posts are created.

    • Daniel Bachhuber 10:15 pm on September 2, 2015 Permalink | Log in to Reply

      Old posts will be automatically converted to the new shortcode syntax. The Shortcode API will continue to provide deprecated support for old syntax so that there is no disruption during the conversion process.

      What about shortcodes stored in other fields?

      • Muhammad 10:18 pm on September 2, 2015 Permalink | Log in to Reply

        Hopefully, we’ll have helper functions to convert those into new format.

      • Robert Chapin 10:38 pm on September 2, 2015 Permalink | Log in to Reply

        Unless there are alternative ideas, I think plugins that do things in non-core ways will be expected to follow the roadmap with their own upgrades. If there are certain fields that are very commonly used in plugins, we should consider doing a core conversion of those, but with much caution.

      • Robert Chapin 11:10 pm on September 2, 2015 Permalink | Log in to Reply

        One alternative would be to leave the old syntax parsers available for plugin filters, while removing them from default filters.

    • AmirHelzer 10:16 pm on September 2, 2015 Permalink | Log in to Reply

      Thanks for the advanced notice on this huge upcoming change. Aren’t you worried that this process will basically break 99% of all existing WordPress sites? Almost all sites use shortcodes in one way or another (like image captions). doing automated updates to CONTENT is certainly possible, but also prone to corner cases and errors.

      It’s great to have a new API. Is it critically important to deprecate the old one?

      Also, for authors, it’s a lot nicer to have a simple [ ] syntax rather than a more complex sequence of characters.

      • Robert Chapin 10:44 pm on September 2, 2015 Permalink | Log in to Reply

        I think the conversion process will be seamless for most sites. This will mainly depend on corner cases such as plugins embedding shortcodes inside of HTML attributes.

    • Nick Haskins 10:18 pm on September 2, 2015 Permalink | Log in to Reply

      The advanced notice is great, but IMO this is absolutely ridiculous. Are there any benchmarks on how much more efficient it is having users type in more characters for a shortcode? Any benchmarks with how much more efficient core will be from having this change made? This is a HUGE change that will affect a VERY large subset of users.

      • chriscct7 10:40 pm on September 2, 2015 Permalink | Log in to Reply

        This is partially about efficiency. This has a lot to do with security, and basically making 4.2.3 impossible as possible to re-occur

        • Nick Haskins 10:45 pm on September 2, 2015 Permalink | Log in to Reply

          I’d love to see benchmarks on the regex processing of the current implementation against the proposed implementation. Re: security; ATM I’m failing to see how [{{this}}] is more secure than [this]?

        • kevinwhoffman 1:13 am on September 4, 2015 Permalink | Log in to Reply

          If the proposal has positive effects on security, it should mention them. A brief summary or a link to the issues with 4.2.3 would help inform that discussion. As of now the only focus seems to be on theoretical improvements to efficiency while complicating the process of using shortcodes.

      • FolioVision 11:31 pm on September 2, 2015 Permalink | Log in to Reply

        This is an absolutely hideous proposal with dreadful syntax. Is the goal of this WordPress project of ours not to bring software to the people? Syntax like this will be the beginning of a new and much better simple CMS which doesn’t do things in the ugliest, most complicated way possible.

        Code better database upgrades, test them properly before releasing and please stop trying to ruin the software for the end users (those who actually use the software and pay all the bills for us to build it).

        • Travis Northcutt 3:54 pm on September 3, 2015 Permalink | Log in to Reply

          “Syntax like this will be the beginning of a new and much better simple CMS which doesn’t do things in the ugliest, most complicated way possible.”

          The sky is falling!

    • Scott Taylor 10:18 pm on September 2, 2015 Permalink | Log in to Reply

      Let’s all note the “Draft One” portion of the title, and let’s all actively give feedback and participate in any decisions on the table here.

      • Nick Haskins 10:21 pm on September 2, 2015 Permalink | Log in to Reply

        My main concern is that this isn’t easier for authors, and that this will affect a very large portion of users and active sites. If I’m understanding correctly, is this being proposed because it’s more efficient for core?

        • Scott Taylor 10:23 pm on September 2, 2015 Permalink | Log in to Reply

          I have the same concerns, and hate the syntax. I also know that the current situation with shortcodes and content-parsing is potentially unstable and unsustainable.

        • Justin Sternberg 10:33 pm on September 2, 2015 Permalink | Log in to Reply

          I’m in agreement here with the concerns. This will break a lot of things because the short code API has been around so long. There’s just no way to account for all the hacks, workarounds, etc, and doing a migration effort on a WP update sounds like a mighty scary prospect.

          • leehodson 10:37 pm on September 2, 2015 Permalink | Log in to Reply

            Should be able to use the existing shortcode parser to recognize old format shortcodes then make the necessary changes to update them. It recognizes shortcodes efficiently enough at the moment.

            Can imagine breakage but as long as the updater shows the changes site admins ought to be able to determine where hands-on intervention is required.

          • FolioVision 11:33 pm on September 2, 2015 Permalink | Log in to Reply

            It seems a make work project. Are there not some important areas where we could make some real progress like integrated multilingual and better native galleries/media management (NOT reliant on external Automattic services)?

            The upgrade process here and cross-version incompatibilities are more than scary.

          • galbaras 11:44 pm on September 2, 2015 Permalink | Log in to Reply

            Backward compatibility should take into account site-specific customizations, i.e. declaring shortcodes in functions.php of a (child) theme. Most site owners use a “build and forget” approach, or at least neglect their sites for long periods. If a new version contains no support for old-style shortcode declarations, sites will break all over the web.

        • Solinx 11:09 pm on September 2, 2015 Permalink | Log in to Reply

          Yes. How am I going to explain anyone that this syntax is an improvement?

          What specifically are the reasons for not using {{shortcode}} and {{shortcode}} {{/shortcode}}? Or [[shortcode]] and [[shortcode]] [[/shortcode]]?

          Other than that the improvements to the API sound good.

          • Robert Chapin 11:18 pm on September 2, 2015 Permalink | Log in to Reply

            Lone curly braces are easily confused with PHP and other C-style code snippets. The latter square brace example is incompatible with the old syntax and would cause migration problems.

    • Andrew Ozz 10:20 pm on September 2, 2015 Permalink | Log in to Reply

      Was just about to say that this is a proposal. Scott beat me to it (as usual) 🙂

    • Pippin Williamson 10:22 pm on September 2, 2015 Permalink | Log in to Reply

      I would assume the answer is “yes”, but have there been benchmarks done that compare the differences in various syntaxes?

      Why [{{shortcode}}] and not {{shortcode}}?

      Most of the proposed syntax seems sane (though still curious on question above), but this one concerns me:

      [{{shortcode}$] HTML [${shortcode}}]

      For most users, that will be exceptionally unintuitive and we will almost certainly see a high error rate with users not being sure where and when a $ is used.

      • Nick Haskins 10:26 pm on September 2, 2015 Permalink | Log in to Reply

        Pippin I have to agree. If we absolutely have to move, how about something as simple as [{this}] HTML [${this}] . it’s easier to digest, and easier to remember that a $ ends a closing statement.

      • webaware 11:19 pm on September 2, 2015 Permalink | Log in to Reply

        The whole proposal sounds pretty horrible to me, but this suggestion by Pippin sounds moderately less ridiculous.

        Re: {{…}}, I would assume that’s because they are already used heavily in non-WP templating, especially JS templating.

        Essentially, this “fix” is coming rather too late in the piece and I think it’ll break more things than it solves. As a plugin writer heavily dependent on shortcodes, I can only see a greater number of support requests from broken shortcodes in my future if this goes ahead.

        Only Shortcake can save us now! 🙂

        • idealien 2:33 am on September 3, 2015 Permalink | Log in to Reply

          “Only Shortcake can save us now!”

          Seriously! If Shortcake were to be brought into core along with this (draft) proposed change – it would go a LONG way to mitigating the concerns many have raised around complexity.

          Without Shortcake:
          “Look – we took this feature you love and made it harder to use….But it’s safer / faster / better”

          With Shortcake:
          “Look – we made is hella-easy to use shortcodes without having to remember syntax type things. Oh, and that syntax is also getting harder b/c it needs to be safer / faster / better.”

          Car metaphor time… With Shortcake: You’re getting automatic transmission and manual is just a little bit harder to use. Without Shortcake: Manual transmission is now harder to use.

          Shortcake FTW!

      • Robert Chapin 11:33 pm on September 2, 2015 Permalink | Log in to Reply

        If we could get this to work…

        [{{shortcode}} HTML {{shortcode}}]

        Is that better? Or still confusing?

      • Mario Peshev 4:16 pm on September 3, 2015 Permalink | Log in to Reply

        I do agree with Pippin’s example with the dollar signs. Seems incredibly easy to mess up by users. I also noticed another syntax including an exclamation sign, which adds an additional layer of complexity for each and every user.

        +1 on the benchmark question. Given the several different supported syntax proposals, I’m curious to see the regex that is parsing those in a sensible and timely manner.

    • AmirHelzer 10:23 pm on September 2, 2015 Permalink | Log in to Reply

      Also, we deal a lot with shortcodes in Toolset plugins. I can’t see any evidence of any user confusing the [ ] characters for anything else than shortcodes.

      Please, besides the code changes, please also consider the huge amount of documentation update that such a change will require.

      Run a quick Google search for “WordPress shortcodes”. A few pages come from wordpress.org, but the majority are old pages, blog posts and tutorials. Nobody is ever going to update them.

      So, after making such a change, people will see two sets of documentation – with new syntax and with old syntax. You know that Google favours ‘old’ content, so the old tutorials that are never going to update will still explain to use [ ]. I don’t think that this will work out well.

      • chriscct7 10:34 pm on September 2, 2015 Permalink | Log in to Reply

        The team working on this is committed to the documentation updates that are necessary.

        • Pippin Williamson 10:35 pm on September 2, 2015 Permalink | Log in to Reply

          Core documentation isn’t an issue. It’s every plugin / theme developer in the world that also has to update their documentation.

          I don’t disagree with updating docs, just making it clear that’s the issue @AmirHelzer is raising.

          Also have to keep in mind the thousands (millions?) of blog posts and tutorials out there.

          • AmirHelzer 10:49 pm on September 2, 2015 Permalink | Log in to Reply

            Exactly. I’m sure that the core team would be quick to update documentation on wordpress.org. I’m talking about the many pages on other people’s sites that will likely never be updated.

    • Scott Fennell 10:25 pm on September 2, 2015 Permalink | Log in to Reply

      Thanks for your work on this!

      I’m trying to understand what I would need to do in order to prevent about 1,200 live blogs from breaking when this syntax is no longer supported:

      [my_shortcode before='<div class="my_class">' after='</div>']

      Andrew offered a helpful suggestion here:


      If I understand his suggestion correctly, I’d be on the hook to do a search and replace in the database, which will be quite scary and expensive for my company. We have literally thousands of posts on thousands of blogs.

      Please, let’s find a way to move forward with this that leaves the old format unscathed.

      • Scott Fennell 10:28 pm on September 2, 2015 Permalink | Log in to Reply

        Sorry, meant to provide the following code.


        Is there a reference somewhere for the comment syntax so I can stop posting gibberish?

      • Robert Chapin 11:49 pm on September 2, 2015 Permalink | Log in to Reply

        If the situation doesn’t support

        [{{my_shortcode class="my_class"}}]

        Then the alternative proposed in the roadmap is

        [{{my_shortcode}$] Optional Content [${before}$]<div class="my_class">[${after}$]</div>[${my_shortcode}}]

        We will also likely encode any HTML in attributes during a migration, so you would have the option of decoding that in your plugin.. security implications could be significant though.

        • Scott Fennell 4:10 pm on September 3, 2015 Permalink | Log in to Reply

          “We will also likely encode any HTML in attributes during a migration”

          Can you explain what you mean by migration, exactly? My imagination is running wild a bit with what you might mean by that. Do you mean to propose that, upon this update, the new version of WordPress will automatically fix my post_content across many gigabytes of posts, without me having to do anything to the posts?

          • Aaron D. Campbell 4:29 pm on September 3, 2015 Permalink | Log in to Reply

            That’s definitely the goal. Basically using the existing parser to pull shortcodes and replace them with the new syntax.

            • Scott Fennell 7:43 pm on September 3, 2015 Permalink

              Sorry for sounding clueless here but that would be … amazing. Is there a precedent for this level of work in a similar update? I’m having trouble imagining how this would be possible given a very large database like mine. Would it run as a cron job perhaps?

    • leehodson 10:30 pm on September 2, 2015 Permalink | Log in to Reply

      The new syntax is very confusing to the eye. Users would find it complicated. I feel we could make it less confusing by using back-to-back opening and closing brackets.

      For example

      Self Closing:



      []shortcode[] Content [/]shortcode[]

      Note the forward slash between the brackets of the closing tag on the enclosing shortcode.

      Is there a reason this wouldn’t work?

    • Peter Wilson 10:51 pm on September 2, 2015 Permalink | Log in to Reply

      I’d be happier with {{selfclosing}} and {{enclosing}} {{/enclosing}} or similar. Mixing the formats seems fraught.

      Upgrading old content is a necessary evil, backward compatibility on the deprecated format defeats the purpose somewhat. Seeing some benchmarking results would be great.

      As theme and plugin developers, we have been caught out by an ill-defined API and security problems in the past, a long term roadmap makes the best of a bad situation. I’d rather a two year defined process than a repeat of 4.3.2

      • webaware 12:46 am on September 3, 2015 Permalink | Log in to Reply

        [ignoring the double-moustache issue here…] but what signifies that the self-closing shortcode is self-closing? The problem is not merely one of mixed use, it’s the calling syntax that the regex needs to accommodate.

        • Peter Wilson 1:38 am on September 3, 2015 Permalink | Log in to Reply

          Okay, understood – that makes sense.

          {{selfclosing /}} and {{enclosing}} {{/enclosing}}, perhaps. That makes sense to me as a developer, so may be bad for a user.

          Could be double moustache, square, anything. Looking for the balance between performance and teaching.

        • Curtiss Grymala 3:44 pm on September 3, 2015 Permalink | Log in to Reply

          In HTML, what signifies that a self-closing tag is self-closing? Hint: it’s not the / at the end; that’s totally optional; in fact, even the closing HTML tag is optional in a lot of cases.

          For instance:

          <img src="/my-image.png" alt="">


          (a complete HTML page that is 100% valid HTML)

          are both perfectly valid HTML code samples, even if they do make those of us accustomed to XHTML syntax cringe.

          In a perfect world, there would be some way to indicate when registering the shortcode itself whether or not it’s supposed to be self-closing, but that’s not always possible (I believe there are plenty of use-cases where a single shortcode works one way when it’s self-closing, and works a slightly different way when it’s wrapped around content).

          That said, I would definitely support the idea of at least trying to indicate that in the shortcode registration (for instance, if my shortcode is only ever written to be a self-closing shortcode, let me indicate that when I register it; if it’s only ever intended to wrap around content, let me indicate that when using add_shortcode()).

          • webaware 11:09 am on September 4, 2015 Permalink | Log in to Reply

            Yes, and a parser for HTML is a pretty complex piece of code, much more significant than a handful of regular expressions. Ultimately, a true parser might be needed if we want to keep shortcodes the way they are. But perhaps that discussion is better held on the new post

    • Scott Fennell 10:52 pm on September 2, 2015 Permalink | Log in to Reply

      Othe use case for shortcodes I’m hoping you don’t break: Shortcodes in widget titles and widget text fields. This has been a gloriously simple and helpful solution for our agency, and many others I’d suspect.

      • williampatton 12:36 am on September 3, 2015 Permalink | Log in to Reply

        Shortcodes can be parsed anywhere by passing it through `do_shortcode()` which is what I expect your themes are doing to make use of them in widget titles and content.

        Your use case should be just fine so long as the shortcodes used obey the newer syntax 🙂

    • KalenJohnson 11:10 pm on September 2, 2015 Permalink | Log in to Reply

      I’m most likely horribly biased since I’ve started the discussion about [https://core.trac.wordpress.org/ticket/33472 templating engines], but aren’t those shortcodes looking an awful lot like templating engine syntax?

      I’m mostly confused as to why users apparently would be able to remember all those different syntax’s, yet theme and plugin developers wouldn’t, or rather don’t have built in functionality to.

      Also, I also feel rather strongly that there are MANY options now that are much preferable to shortcodes. Things like ACF, Pods, etc. They are not built in to WP core like shortcodes, but really, shouldn’t we focus on getting the Fields API set up correctly rather than continuing to pursue having complicated layouts in one single WYSIWYG editor (not so much WYSIWYG when it’s littered with shortcodes).

      • Jonathandejong 5:52 am on September 3, 2015 Permalink | Log in to Reply

        I’m with this guy ^ Complicated layouts in a single WYSIWYG field just feels wrong. Altho I recognize there’s a lot more to it than just the editor (do_shortcode(), widget fields etc.).

    • Aristeides Stathopoulos 11:33 pm on September 2, 2015 Permalink | Log in to Reply

      I love it!
      It was about time we use something more unique for shortcodes…

    • nick6352683 11:51 pm on September 2, 2015 Permalink | Log in to Reply

      Priorities core developers, priorities. With so many things to still fix and add to the core, you are going to spend time and energy on this, and for the lamest excuse for doing so? Who uses square brackets for quotations? Way to break millions of web sites, for nothing!

      • webaware 12:48 am on September 3, 2015 Permalink | Log in to Reply

        [I hate to pop your bubble but] many people use brackets for essentially parenthetical in-sentence comments. 🙂

    • Jon Brown 11:53 pm on September 2, 2015 Permalink | Log in to Reply

      Shortcode API, yay! This is an exciting proposal, but I’ve got to echo others that the proposed syntax looks like a disaster. I get it, but I can’t imagine users will. I get avoiding the sane syntax of handlebars, angular, C, etc… but do we really have to be _that_ different. I really hope there is a better way.

      On core updating old shortcodes. I’m sorry but no, just no. Don’t you dare go changing _my content_ on a WP update. Worse still is only planning on updating post_content and ignoring all the other places shortcodes show up. Custom Fields for one, but also in PHP files.

      I’d humbly suggest two significant change to this proposal (beyond finding better syntax).
      1. Don’t plan on actually dropping support for the old shortcodes. Do show deprecated notices, just don’t stop supporting the old syntax until WP 6.0. Of all the deprecated cruft I’d love to see dropped, this will never be one of them.
      2. Build a separate utility/plugin a la the WordPress XML importer that can update old shortcodes in a user controlled fashion. Bonus if it actually scans PHP files and identifies shortcodes there. This could then either bulk update, or step through shortcodes one by one for the super paranoid.

      Summary: Simpler Syntax, No real deprecation, Separate plugin for converting old shortcodes.

    • Alex Mills (Viper007Bond) 11:56 pm on September 2, 2015 Permalink | Log in to Reply

      Going to be… interesting migrating SyntaxHighlighter over to this. Maybe this is the motivation I need to finally finish my rewrite from scratch.

    • Peter Chester 12:44 am on September 3, 2015 Permalink | Log in to Reply

      I love clarity. I love easily parseable content. However, this seems like it’s destined to cause WordPress sites around the world to break. I’m sure we’ll see heaps of bare old shortcodes from this change.

      I’m not sure I understand why it’s a problem to check that a shortcode is actually a shortcode before trying to convert it. Why are we doing this?

      Also, I have a hard time teaching people the existing simple shortcode. I expect the error rate will be quite high for people with a more complicated new format.

      > Our old [ and ] delimiters were easily confused with the way those characters are commonly used in English quotations and sometimes even in URLs.

      What are some examples? I haven’t run into this at all in all my years of working with WordPress.

      • webaware 12:59 am on September 3, 2015 Permalink | Log in to Reply

        Very common to have brackets in quotes, to fill in implied references, like this:

        “I won’t be rushing out to get my daughters vaccinated [for cervical cancer], maybe that’s because I’m a cruel, callow, callous, heartless bastard but, look, I won’t be.” Tony Abbott, November 9th, 2006

        Also for excerpted quotes, like this:

        […] We shall go on to the end, we shall fight in France, we shall fight on the seas and oceans, we shall fight with growing confidence and growing strength in the air, we shall defend our Island, whatever the cost may be, we shall fight on the beaches, we shall fight on the landing grounds, we shall fight in the fields and in the streets, we shall fight in the hills; we shall never surrender [… continued]

        For URLs, consider that PHP only handles query parameters as arrays when the parameter name has [] or [key], and some URLs pass such parameters.

    • Greg Ross 12:47 am on September 3, 2015 Permalink | Log in to Reply

      I agree with many of the above comments about the syntax, it’s cumbersome and confusing.

      HTML seems to get by just fine with a single character for it’s delimiters and I see no reason a shortcode can’t as well. Perhaps square brackets are too common, fine let’s change to something else (curly brackets seem reasonable).

      And don’t reinvent the wheel, HTML already has a standard for self-closing and encapsulated tags ( and ) use the same format for shortcodes ( {shortcode /} and {shortcode} {/shortcode}).

      This would also allow both shortcode API’s to run side by side as plugin authors and content creators updated to the new format. Perhaps even make the new syntax a feature plugin for the next few releases until it is merged in to core and then move the old API out to a plugin for several more releases.

      The timeline seems far to aggressive, shortcodes were added way back in WP2.5 and how you want to yank out the old format in just three point releases? The amount of code and content that will require updating is significant.

      My final thought is about the assertion in the proposal that shortcodes *should* only do certain things (like not pass in html as attributes). Keep shortcodes flexible, don’t artificially limit them to certain things. While it may take more to support that flexibility, it would seem to have paid off greatly in the past looking at all the things people have done with them.

      • Ben Huson 4:11 pm on September 3, 2015 Permalink | Log in to Reply

        Completely agree on not re-inventing the wheel for self-closing and encapsulated tags. Stick with the same convention as HTML using the trailing/leading slash within the syntax.

    • Justin Tadlock 1:00 am on September 3, 2015 Permalink | Log in to Reply

      On syntax:

      Users know shortcodes. WP wasn’t the first system to have them. Anyone remember the old BBCodes?

      Open bracket. Type short tag. Close bracket. Simple. Intuitive. Hard to break if you’re not doing anything crazy. Let’s make sure this simplicity is at the core of any discussion on the topic.

      I’d pretty much put the [{{shortcode}}] syntax into the non-starter category. I could get behind {{shortcode}} if we must change syntax, but mixing and matching brackets is destined to fail.

      On updating:

      One thing to watch out for when auto-updating old shortcode syntax in the post content is those of us who output code examples, especially those of us who write in Markdown and don’t need to convert our [ and ].

    • Maxime Jobin 1:04 am on September 3, 2015 Permalink | Log in to Reply

      My main problem is understanding what real problem does it solve ?

      The proposed syntax is really confusing. Isn’t there a way to simply “improve” the actual syntax instead of changing it like this ?

      Also, wouldn’t it a good idea to abstract the concept in the editor. That would mean displaying a visual representation of the shortcode. I would see that as Facebook highlights someone you tag in a status/comment.

      But, that would mean “forcing” developers to specify expected/accepted attributes.

      Aren’t we ready for something more powerful than writing shortcodes ? For devs, it’s perfect. But for non-tech, it’s hard to understand and use. Changing that would mean a lot of confusion.

    • Justin Busa 1:07 am on September 3, 2015 Permalink | Log in to Reply

      If custom fields aren’t going to be migrated to the new syntax, then it would be nice if there was some sort of filter that would allow plugin developers to migrate fields themselves.

      Also, in terms of sites breaking, it would be nice if there was some sort of constant that would enable the old API. Reason being, lets say a site has shortcodes defined in the theme or is using do_shortcode in theme files. Enabling the old API would instantly fix the site while allowing the developer time to update the theme’s code.

    • Ipstenu (Mika Epstein) 1:18 am on September 3, 2015 Permalink | Log in to Reply

      My major concerns have been raised already but I’ll stress if from a different angle.

      If we learned anything from the shortcode change to secure them, we cannot keep breaking users sites. I know that we’re talking about a long term plan to possibly remove shortcodes as we know them by 4.7, but even with all the notification in the world, we will break sites.

      We have 3 releases of WP a year. 4.7 is 18 months away at the outside. That’s a lot of information to get out there really fast and really clearly, AND a lot of code to fix.

      We have to consider:

      • All the plugins and themes on the planet we will break (because we will, they won’t read or test). We have to degrade them as gracefully as humanly possibly. Continuing to say “Well the developers were notified and should have updated” now that we’re as big as we are is not sustainable.
      • All the very (legitimately) angry end users who are broken because they didn’t upgrade plugins and themes (or the themes/plugins didn’t get updated). People were rightly angry last time. It’s the end users, not the developers, who will be hardest hit by this change.
      • Communicating clearly to the users that it’s now {{gallery}}. That’s going to be very hard. Incredibly hard. Updating their old posts (keeping in mind Justin’s Markdown caveat and those who use them as an aside – I know I know) is easier than making sure everyone knows what to do. At best we can keep tabs on the ones built into WP and perhaps use the logic we have in the visual editor NOW to convert them, but we have to figure out how to make sure everyone knows. This is nothing like the move of Menus to customizer. That was confusing, but the users could see what happened. This is a legit change, your old way is no longer going to work. That is huge.
      • The number of users who have premium themes and plugins that do not get update alerts. These people are simply not going to know they need to update and this is not their fault. We should never be breaking them if there’s possibly any alternative.
      • Users will be upgraded by their hosts vis-a-vis one-click installs and managed hosting so they will have up to date WP and out of date plugins/themes. So yes, many users will be on 4.7 and then a theme from 2014. It sucks, it’s the reality, we know it’s the reality, we cannot stick our heads in the sand.
      • Plugins that are already using {{template}} tags in their code. Yeah, I’ve seen it. Most of them use it for search/replace within their own code, but we’ll want to make sure we check for everyone in the repo who might be doing it on their own.
      • Gabe Shackle 1:55 am on September 3, 2015 Permalink | Log in to Reply

        This sums up my feelings perfectly: “we cannot keep breaking users sites”

        Ending support for the current style of shortcode syntax within 18 months will absolutely break sites and frustrate end-users, site owners, and developers in a huge way.

        The best way to make developers stop using a platform is to force them to repeatedly have to retrain their clients and refactor their code.

    • Jean-Francois Arseneault 1:19 am on September 3, 2015 Permalink | Log in to Reply

      Clients, you know, the small business owers, who we train to use their site, and the 3-5 shortcodes they might need, already have a hard time understanding these “square brackets”.

      The draft I read seems to me like the start of “pleasing the machine, and not the human”

    • Luke Carbis 1:20 am on September 3, 2015 Permalink | Log in to Reply

      I’m concerned about client-brain-backward-compatibility. Any changes to the shortcode syntax should still work with my clients brain.

      For example, I provide my client with a site, and teach them to use [button] to insert a button. We finish our contract, and I am no longer in touch with them.

      First the client updates WordPress to 4.4. Nothing changes. Then 4.5. They’re in a production environment, so they don’t see the warnings. Then 4.6. They don’t happen review old posts / pages, but if they do, they don’t realise that their shortcodes have changed.

      Then the client updates to WordPress 4.7, but unlike me, they don’t read the Make WordPress Core blog. They don’t know about the changes. They don’t know that what they’ve learned has become deprecated and now unsupported.

      The client publishes a new page, and they put the button in the way they’ve been taught, which has always worked. But now, it doesn’t work. It’s broken. They call me.

      The client is now angry because a WordPress update cost them money, and confused because the new syntax is much harder to understand.

      I recognise that there are always changes to WordPress, and that the nature of incremental releases is that a user must learn to deal with change. But as far as I know, these changes have always been easy enough for a reasonable user to intuitively navigate. i.e. changes to the menu design, or updates to the nav menu UI.

      I have three questions:

      1. Has there ever been a change to WordPress which could not be intuitively learned by a reasonable user?

      2. Is it okay to introduce unintuitive changes to WordPress?

      3. How can we make this change intuitive to a reasonable user?

      • Luke Carbis 1:29 am on September 3, 2015 Permalink | Log in to Reply

        Here’s a suggestion to answer my third question.

        • When the user enters a shortcode (with either the [simple] or the [{{new}}] format), check if the shortcode exists.
        • If it does, replace the text shortcode with a GUI placeholder element
        • Also, behind the scenes (i.e. in the HTML editor), replace the [simple] shortcode with the [{{new}}] format

        The user experience doesn’t get worse – it gets better. My client enters a shortcode as they always have. They see a visual indicator to show that it was recognised and accepted. Behind the scenes, the new format is saved.

        • Weston Ruter 5:52 am on September 3, 2015 Permalink | Log in to Reply

          Yeah, if this new syntax goes in then GUI placeholders (TinyMCE views) are going to have to be the answer, which also seems like Shortcake.

    • webaware 1:30 am on September 3, 2015 Permalink | Log in to Reply

      Upon some further thought, I think that really there are two problems that are presented:

      1. brackets are already commonly used elsewhere, e.g. quotations, URLs with array parameters

      I see this as being a problem only when developers register shortcodes for real words. Using a sensible prefix for shortcodes prevents this. Unfortunately, there’s been a push for shortcode portability which directly undermines that, e.g. by recommending shortcodes like [product …] for products. [and of course, that prevents combining plugins with similar intent on the same site, but whatever…]

      Has it actually been a problem thus far? I imagine that anyone writing “blah-de-blah [product name] de-blah” will quickly discover that they should use parentheses instead if they don’t want to trigger the shop plugin they’ve installed. Content editors learn. They already have stuff they need to learn, and this is a very simple one that they learn pretty damned quickly anyhow.

      Is it necessary to strip “unregistered shortcodes” when we can just assume that they are part of normal sentence structure? So some websites will display the odd shortcode when its code [e.g. plugin] isn’t active; so what?

      2. self-closing shortcodes don’t look like they are self-closing until they don’t have a closing shortcode

      … which is easily resolved by demanding that they close with /] similarly to XML and HTML.

      To me, this is THE performance problem, and it is easily resolved by not allowing self-closing shortcodes without a terminating /].

      Now, bear in mind also that cleaning up old code might require changing content in:

      • wp_posts.post_content
      • wp_posts.post_excerpt (some plugins / themes use it for rich text)
      • wp_postmeta.meta_value
      • wp_usermeta.meta_value
      • custom tables
      • code (e.g. directly calling `do_shortcode(‘[product …]’);`

      And it has to take into account all sorts of edge cases.

      There’s only so much auto-upgrading that can be achieved. The changes proposed WILL BREAK WEBSITES. Even simply requiring self-closing shortcodes to have /] WILL BREAK WEBSITES. All I can see coming from this proposal is pain. “There will be a great gnashing of teeth” etc. 🙁

    • Jean-Francois Arseneault 1:47 am on September 3, 2015 Permalink | Log in to Reply

      Pardon the pink-colored glasses for a second, but why does it seem that our only choice moving forward, to improve the state of shortcodes, is to change *how* shortcodes work, and not *if* there isn’t a better / newer way to accomplish the goal of inserting content into a page ?

      In the spirit of seing the forest from the trees, why are we still trying to desperately manage mixed-source content using text-based shortcodes ?

      Why are we still using a paradigm where the main content area (if talking about posts) is still an area where we can add content, but also perform layout decisions *within* the content? Where’s the separation of concerns in this way of doing things?

      I think Page Builders are the start of a better solution for managing all these different contents within a page, but perhaps forcing plugin authors to expose their functionality into native widgets and ‘page builder elements’, vs letting them add shortcodes anywhere to render content might be an approach that is easier to grasp, from a UI perspective, for *end-users*.

      Sure, developers, integrators… we can handle everything you throw at us, but in the end, the sites are built for end users, for real-life applications and business purposes. End users are the ones that matter.

      Simplicity always wins.

      • Peter Wilson 2:19 am on September 3, 2015 Permalink | Log in to Reply

        The Shortcake feature plugin covers providing a UI for shortcodes.

      • Chris Van Patten 2:30 am on September 3, 2015 Permalink | Log in to Reply

        Although I’m generally in favour of this proposal, I think this merits some consideration too. Is there potentially a way to eliminate shortcodes — as in a manually entered bit of pseudo-tag-like code — altogether? I think Shortcake is an interesting approach, but it still depends on the shortcode API in the backend. What if shortcodes were deprecated, and replaced with an entirely new concept for inserting dynamic “chunks” of content that doesn’t fall back on parsing through plaintext?

        Replacing shortcodes is probably a much-longer term play, but it does mean that instead of patching the old they can be replaced with something newer, more flexible, and more in tune with the direction core has been moving already.

      • John Teague 1:53 pm on September 4, 2015 Permalink | Log in to Reply

        I totally agree. We need to be thinking of an intuitive replacement for shortcodes altogether. That provides incentives for developers and users to make the change.

        I really appreciate the hard work by @Robert Chapman and others putting a lot of effort into solving the increasingly unsustainable shortcode system. Sadly, I think the proposal in its current form, will confuse already error prone users, and likely break a lot of sites.

    • Chris Van Patten 2:36 am on September 3, 2015 Permalink | Log in to Reply

      As a supplement to my other comment (which was a nested reply), I’m generally in favour of this proposal. I think there’s merit in discussing completely new alternatives to shortcodes, but that’s a semi-separate issue.

      Like many others, I’m not sold on the suggested syntax, but it’s obviously still early days.

      One other thought: why not make escaped code the default, and make unescaped the exception with its own special syntax? Are there backcompat reasons that wouldn’t work? Seems like (potentially) an easy win for security.

    • Andrew Nacin 3:03 am on September 3, 2015 Permalink | Log in to Reply

      Thanks for your feedback, everyone. Developers, truly, honestly, really: Thank you for paying attention. I often feel that requests for comments usually fall on deaf ears, and I know I am not the only committer who feels that way. This has been one of the most substantial and productive threads on make/core I’ve seen, and it’s encouraging to see.

      Here are some thoughts on the proposal:

      • This was clearly labeled a “draft”. I am glad many of you noticed that this was not a decree, but a call for feedback. (And thank you again for obliging.)
      • The vision of shortcodes should be something like “it should be easy and intuitive for non-technical authors to embed and enrich their posts” (I have cribbed this from @matt).
      • This vision does not actually mean that shortcodes are the solution. I actually have a strong dislike for shortcodes. I think they are a terrible user experience, and that even something like Shortcake is still only a marginal improvement.
      • Some changes to shortcode parsing is absolutely necessary in order to keep WordPress secure. Yes, this is about security, not just general sanity.
      • The proposed syntax significantly clashes with the proposed vision. And given all of your feedback, we’re clearly going to have to go back to the drawing board. Please note that we still need to do something, but maybe we can think further outside the box.
      • And for goodness sake, PLEASE STOP RELYING ON HTML IN SHORTCODE ATTRIBUTES. This also does not align with the outlined vision, either, and it’s what largely precipitates proposals like these.
      • Thank you @miqrogroove for your incredibly hard work on this. Honestly, folks, many of you simply cannot see how much work Robert has poured into shortcodes over the last year, as it takes place in private security discussions. He’s been a tremendous asset to the project.

      I’m looking forward to future make/core threads getting this much high quality feedback. Now if only it can also carry over to more Trac tickets. 😀

      • Solinx 9:21 am on September 3, 2015 Permalink | Log in to Reply


        “I’m looking forward to future make/core threads getting this much high quality feedback. Now if only it can also carry over to more Trac tickets.”

        I think this could be helped by making Trac more accessible and by providing a better overview of what is currently going on.

        1. Trac
        When I first heard of Trac I had to get there through google. The “Get Involved” page didn’t help at all. Why not add a link to both the tickets dashboard and the ‘next release’ tickets list at https://make.wordpress.org?

        And why not link to related tickets, github repo’s, slack groups, etc. at the bottom of these blog posts? That makes it a whole lot easier to get to the right places to get involved.

        2. Progress overview
        It is difficult to keep track of all that is going on. The blogs provide updates, but not an overview. Perhaps you could introduce something such as: http://dev.modern.ie/platform/status/adownloadattribute/

        Also here you could link to the related tickets, github, etc.

      • Nick Haskins 12:11 pm on September 3, 2015 Permalink | Log in to Reply

        “Now if only it can also carry over to more Trac tickets.”

        The problem with Trac, for me at least, is that it’s not part of my daily workflow. I’m a bit intimidated by Trac, not to mention making patches. I take part in conversations on Github all day, because that’s how we manage code and issues for my job. I suspect that if these conversations were on Github, with PR’s being allowed there would be a lot more participation. After all, I don’t know any developers outside of WordPress who use SVN in their workflow.

        • NateWr 2:08 pm on September 3, 2015 Permalink | Log in to Reply

          Thanks for outlining this response @nacin. I’d second thinking outside the box. I suspect a lot of the problematic uses of shortcodes stem from people trying to plug gaps in other kinds of functionality (layout, class assignment, conditionals). Maybe solving some of those problems would take a bit of heat off of shortcodes, and make them a smaller attack vector. Or maybe not. Can’t say I really understand the security implications of shortcodes.

        • John Teague 2:33 pm on September 3, 2015 Permalink | Log in to Reply

          Totally agree with you on this Nick. Think how moving to Git would invite solid developers to submit great fixes and features? It’s not that a lot of us can’t use Trac/SVN. It’s just who the hell wants to.

          • Helen Hou-Sandi 4:06 pm on September 3, 2015 Permalink | Log in to Reply

            I imagine that a lot of the workflow folks think of centers around GitHub and pull requests as opposed to Git itself, but just a note that there is a proper Git mirror that many people patch against, including a number of committers. https://make.wordpress.org/core/2014/01/15/git-mirrors-for-wordpress/

            • KalenJohnson 4:49 pm on September 3, 2015 Permalink

              Yes that’s very true, it is Github itself which has become such a boon for developers to be able to quickly and easily find issues, discuss them, and create pull requests. Trac seems very antiquated by comparison, and submitting patch files is not something the rest of the open source web development world does that I’ve seen.

              I can understand how large the move to Github would be for WP, but don’t underestimate that it could really remove WP from the silo it’s in and open up to better quality issues, more issue discussion (as Nacin is looking for), and possibly more patches as people are much more familiar and willing to make pull requests.

            • J.D. Grimes 6:15 pm on September 3, 2015 Permalink

              Yes, the workflow on Trac is needlessly complex. I’d +1 moving to GitHub, or something where we can have PRs instead of patches.

            • John Teague 1:58 pm on September 4, 2015 Permalink

              My bad. I meant github.

      • dlouwe 1:07 am on September 4, 2015 Permalink | Log in to Reply

        “This vision does not actually mean that shortcodes are the solution. I actually have a strong dislike for shortcodes. I think they are a terrible user experience, and that even something like Shortcake is still only a marginal improvement.”

        Very much this. Users shouldn’t have to learn syntax to make use of shortcode-like functionality. They can already use the Visual editor to insert and manipulate HTML without ever knowing what HTML is, and that should be a clear goal for whatever shortcodes end up being in the future. It doesn’t much matter what the new syntax looks like if the only people who are going to be editing it directly are people who have business doing so. As a developer, I don’t mind jumping through some extra hoops to make sure my users have an easy and intuitive experience!

      • raamdev 3:23 am on September 4, 2015 Permalink | Log in to Reply

        > The vision of shortcodes should be something like “it should be easy and intuitive for non-technical authors to embed and enrich their posts”

        I agree. That makes this a good time to redesign shortcodes from scratch, to rethink how we would create something that provides the same flexibility and power of the existing Shortcode API while also more accurately fulfilling the goal of being something that is “easy and intuitive for non-technical authors to embed and enrich their posts”.

        The ideas that came to mind while reading your comment were: Do shortcodes need to be something that is entirely configurable from within the post editor? What if they were instead something that pointed to a configuration, a configuration stored in the database that allowed a plugin or theme to determine what should be returned? (I’m thinking something like `[{{shortcode id=”plugin_slug_123″}}]`, i.e., something like a shortcode with a single attribute.)

        Then a new API could provide plugin and theme developers with a way of building a UI that would allow users to configure ‘shortcodes’ (the configuration being stored in the database, e.g., as `plugin_slug_123`), similar to the way the Plugin API allows plugins to register meta boxes for post types.

        My feeling is that creating something that is both easy and intuitive for non-technical users will require getting rid of syntax as much as possible and replacing that with a UI.

        Building a whole new feature to replace the Shortcode API would also make it easier to transition away from the current Shortcode API: Eventually WordPress Core could disable the old Shortcode API by default (allowing site owners to re-enable it, perhaps using an MU Plugin, with the knowledge that it’s no longer recommended) while plugin and theme developers could update their code to support the new feature (their motivation to do so will be that future versions of WordPress won’t have the old Shortcode API enabled).

    • markkaplun 5:48 am on September 3, 2015 Permalink | Log in to Reply

      [might not be needed after so many comments but…] I want to also voice my opinion against the proposed. I can see how software developer *might* be happy with this kind of syntax but this is very far from freely written text. The need to balance the number of braces is a no go, it is just too easy for my cat to erase one of them and making me waste an hour trying to figure out what is wrong because it will not look obvious. I actually don’t feel too strongly about the syntax of the complex variants, as they require some kind of programming state of mind, but the self closing one should be extremely simple.
      In addition, The use of the $ sign for anything in this context is ridiculous. In free written text it indicates some reference to money or value and the reason it used in regexp is because it is used in free text but rarely in programming.

      1. Whatever is the final syntax, syntax highlighting for shortcodes should be part of both the tinymce and the “text” editors. This will make it easier to identify shortcode and find broken ones.

      2. Since this will be such a big chagne, might as well fix [the lack of] permission checks. On a multi author site everyone can use all shortcodes. There is no reason why a contributor should be able to insert a contact form via a short code. Sure the editor is supposed to review and remove it, but there is no point in giving the ability in the first place.

    • umchal 6:21 am on September 3, 2015 Permalink | Log in to Reply

      The biggest problem of shortcodes from my view is that they remain in the posts even after the plugin which uses the shortcode gets uninstalled.

      I suggest using the HTML comment syntax.

      • HTML

      This way, users do not have to concern about removing them form the post. I suggested this a while back (https://wordpress.org/ideas/topic/commentcode-alternative-to-shortcode).

    • Florian Girardey 7:04 am on September 3, 2015 Permalink | Log in to Reply

      How can you possibly want to break a lot of websites with 4.7 with end of support of the current shortcode syntax and absolutely rejecting the idea of using a recent version of PHP…

      I like the idea of improvement and this roadmap is a good idea but i don’t understand the logic behind the WordPress team right now.

      • BrashRebel 12:43 pm on September 3, 2015 Permalink | Log in to Reply

        If you want to understand, read the whole discussion. In particular @nacin‘s comment https://make.wordpress.org/core/2015/09/02/shortcode-roadmap-draft-one/#comment-27484

        • Florian Girardey 5:51 pm on September 3, 2015 Permalink | Log in to Reply

          I understand why this roadmap is necessary and i absolutely agree.

          But the arguments advanced for such a changement are closely the same that a PHP version update, i don’t understand how the WordPress team can accept this roadmap but also refuse a PHP version update. That’s all

          • Ipstenu (Mika Epstein) 7:18 pm on September 3, 2015 Permalink | Log in to Reply

            While I understand the comparison, it’s a logical fallacy.

            A minimum requirement is just that. A minimum. WP’s minimum is PHP 5.2.4 and MySQL 5.0.15. We still say mod_rewrite is optional. Supporting older versions doesn’t make the codebase insecure in and of itself, and there is no external dependency update that can fix the current security issues with shortcodes.

            The point of this post is to ensure, when we do fix shortcodes, we do it in as graceful and as minimally intrusive a way as humanly possible. To not repeat the shortcode sins of the past. Your concern, while well meaning and valid, has a marked tendency to detract from that.

    • Cristian Antohe 7:10 am on September 3, 2015 Permalink | Log in to Reply

      We’re using Mustache template tags in both our plugins and I can tell you {{tag}} is in no way simpler then [tag]. Once you start adding {{!tag}} or {{^tag}} HTML {{/tag}} things start to go wrong fast. Users simply have a hard time understanding how that works.

      Start to mix up template brakes like [{{tag}}] and that’s just stupid. It’s un-intuitive and will give a lot of headaches to ALL plugin developers who also actively support their plugins.

      • Cristian Antohe 7:19 am on September 3, 2015 Permalink | Log in to Reply

        With these more unique delimeters, we will be able to process registered shortcodes more efficiently because we will not have to search for them by name. Unregistered shortcodes will have more consistency because we can find them more accurately.

        The new syntax is making it harder for users to use shortcodes for the sake of developers who have a hard time working around the constraints that come with using [tag].

    • leemon 7:25 am on September 3, 2015 Permalink | Log in to Reply

      Sorry to say this, but that new syntax is hideous. If this goes forward as it stands, I predict major breakages, lol

    • Mike Schinkel 8:08 am on September 3, 2015 Permalink | Log in to Reply

      Reviewed all the comments and did not see anyone suggest what @cfc suggested on @miqrogroove‘s blog post which just use Custom HTML Elements:


      They seem perfectly suited for this:


      What would they look like?

      Old Shortcode: [foo]bar[/foo]

      New Shortcode: <wp-foo>bar</foo>

      One I saw that idea by @cfc everything else seemed like painful overkill. Besides, shortcodes are very closely aligned with the goal of the custom HTML element proposal with shortcodes being server-side and custom HTML element being client side.

      • Benny 5:56 pm on September 3, 2015 Permalink | Log in to Reply

        I like the idea of custom HTML elements / web components replacing shortcodes. But that still means we can’t phase out the old shortcode syntax because it would break so many sites. How would you suggest to solve that problem? Maybe by adding you’re “You are doing it wrong” notices (for ever)?

      • Michael Davis 10:53 pm on September 3, 2015 Permalink | Log in to Reply

        I hope this gets more discussion. Particularly, what would be potential shortcomings of this method compared to what the new syntax would bring.

      • Knut Sparhell 1:18 am on September 4, 2015 Permalink | Log in to Reply

        I like this idea of custom HTML elements. Registered elements gets executed either server-side or client-side. An advantage is that it’s quite obvious that such elements can’t be used in attribute values, and attributes can’t contain such elements.

        What has to be solved is the slow deprecation of the old shortcode syntax, and if old shortcodes should be replaced upon upgrade, at some point.

        I also look to the more tag and page tags, that are HTML comment tags. When shortcodes was first introduced I was surprised the HTML comment tag was not considered.

        Custom HTML elements seem very future proof, outright elegant, even if the draft never becomes a recommendation or they will never be supported by browsers (for the client side types).

      • webaware 11:09 pm on September 4, 2015 Permalink | Log in to Reply

        Much love. Was thinking about how NextGEN Gallery does its galleries these days (as a decorated image tag it parses out), but this sounds even better.

      • Braad 12:57 am on September 5, 2015 Permalink | Log in to Reply

        +1 for web components. Great idea Mike!

    • chrishoward 8:24 am on September 3, 2015 Permalink | Log in to Reply

      Sorry, I see these new syntax changes as ridiculous.

      The current syntax is untidy and cumbersome enough, and now you want to throw all these braces in it? We should be making shortcodes easier to use, not harder.

      Have you forgotten the user totally?

      These are great for coders. We love “mustaches” and parameters!

      But I pity the user having to key in a multiple-enclosured shortcode.

      Personally, I believe shortcodes should be invisible to the user, just as HTML is. Show the code in text mode, sure, but not in WYSIWYG. In WYSIWYG mode shortcodes should only be entered via a button and form, and then represented in full if possible (like the gallery shortcode) or by a clickable icon.

      For day-to-day users, this new syntax is a big step backwards.

    • asarosenberg 8:45 am on September 3, 2015 Permalink | Log in to Reply

      I agree with previous comments that this suggestion is very bad for users. People can understand shortcodes if they are somewhat familiar with HTML or online forums but the suggested syntax is unreadable to anyone who is not a programmer.

      An easier way to solve the problem with wanting to use brackets for other things is to allow people to escape brackets for example by ignoring brackets prefaced by backslash, star or whatever char that makes sense. If you want WordPress to be a CMS and not just a blog tool, shortcodes are more important than brackets in text.

      As for deprecation and conversion consider that there are shortcodes in meta fields. Many themes use page templates with multiple wp_editors and shortcodes. To avoid massive sudden breakage the old format should work for at least a year or two after the changes have been implemented and announced.

    • Arunas Liuiza 9:07 am on September 3, 2015 Permalink | Log in to Reply

      While I do get the technical reasoning behind this proposal, but I am really worried changes like that will soon make WordPress only usable for people with PhD in Computer Science. Seriously, I think something like 简shortcode简 would be easier to use and understand than the proposed syntax for the average WordPress *user*.

    • andreasnrb 9:27 am on September 3, 2015 Permalink | Log in to Reply

      The first thing I think that should be done before even touching or discussing syntax is to clearly define the purpose of shortcodes and come up with appropriate use cases.
      And just to ignore my first statement. Why not go in other direction and introduce custom HTML tags instead? You register your tag, what attributes it accepts with sanitation callbacks.
      <wp-button href=””>Click me</wp-button>
      Even makes it possible to integrate with HTML5 custom tags later on.

      • andreasnrb 9:31 am on September 3, 2015 Permalink | Log in to Reply

        Yes I know it encodes when written in the HTML window. So it should be combined with a insert button in the menu.

    • HasinHayder 11:56 am on September 3, 2015 Permalink | Log in to Reply

      Now you need a Ph.D in CS to use WordPress 🙁

    • FolioVision 12:11 pm on September 3, 2015 Permalink | Log in to Reply

      I see that doing this will open new possibilities, but I think the basic shortcodes should stay without change:

      Self-Closing: [shortcode]
      Attributes: [shortcode attr1=”value1″ attr2=’value2′ “value3” ‘value4’ value5]
      Enclosing: [shortcode] HTML [/shortcode]
      Escaped Code: [[shortcode]]

      If you want to add more features (like nested shortcodes here), I think yout can do it without breaking too much stuff:

      Multiple Enclosures (this one should work if they could look for closing tag for [shortcode] – a regex won’t suffice though): [shortcode] HTML [encl2] HTML [encl3] HTML [/shortcode]

      I’m not sure about the rest though, but these special cases could probably use the new syntax:

      Stripped Unregistered Code: [{{randomthing}}]
      Stripped Unregistered Enclosure: [{{randomthing}$] Content also stripped. [${randomthing}}]
      Stripped Empty Code: [{{ }}]
      Ignored Orphan: [{{shortcode}$]
      Ignored Orphan: [${shortcode}}]
      Ignored Orphan: [${encl2}$]
      Ignored Context: [{{shortcode
      Ignored Context:

      Do you see any issue with keeping the basic shortcodes unchanged?

    • jakeparis 12:46 pm on September 3, 2015 Permalink | Log in to Reply

      I think that the new syntax is difficult and burdensome to the average user. I would suggest:

      a) Make at least the basic “Self-Closing” shortcode operate without the brackers (just as it does now).
      b) The “Enclosing” style should close with a slash, not a $. There is not precedent for using a $ as a closer in any programming language that I know of — therefore that makes this difficult to remember. The slash it a much better choice.

    • J.D. Grimes 1:26 pm on September 3, 2015 Permalink | Log in to Reply

      I do not know how serious the security implications might be, but perhaps we could avoid removing the old syntax at all, and just disable it on new sites, like we’ve done with other deprecated features.

      I think the real lesson here though, is that shortcodes don’t need to be fixed, they need to be removed. We would still have to figure out how to maintain some amount of backward compatibility for old sites, but it might be more productive to work on a broader roadmap that would actually make shortcodes obsolete. Others, including @nacin, have hinted at this. Is a syntax overhaul really necessary as an intermediate step? Is it even plausible as an intermediate step? What if instead we approached it like this:

      1. Bring Shortcake (or some other “shortcode UI”) into core. This would still use the shortcode API, but would provide a UI for it so the user never has to directly manipulate the shortcode string.
      2. Start using an alternate, secure method of indicating the insertion points of the different content bits when this UI is used.
      3. Disable the shortcode API on new sites.
      4. Let the old sites get slowly broken and hacked over time, or break them ourselves all at once with an update.

    • JakePT 2:28 pm on September 3, 2015 Permalink | Log in to Reply

      I’ve got a couple of things I want to say. Firstly, I really need clarification on what these security issues with Shortcodes are. I read the post on the 4.2.3 change and it didn’t really help. Maybe it’s because it’s happening in an area of WordPress development that I just haven’t stepped into, but I’m having trouble understanding what contexts Shortcodes are a problem in. Isn’t their output defined by the plugin? Why isn’t the security problem with the *plugin*? Aren’t authors the only ones who can use them anyway? What could they do with them that’s so bad? I’m having trouble accepting such a drastic proposal when it doesn’t come with a clear rationale.

      But my biggest problem by *far* is that this would be so so so much easier to swallow if there was *any vision at all* for the future of the content editor. A single TinyMCE field just doesn’t cut it anymore. What happened to the content blocks idea that came up around 3.8/3.9? WordPress needs to address the problem of editing and creating more complex layouts, not breaking functionality that is only being abused because of this lack of vision and progress.

      Shortcodes suck, but don’t make them suck more without something better to replace them.

    • Stefano Aglietti 3:02 pm on September 3, 2015 Permalink | Log in to Reply

      I don’t know if someone already rised the question, but apart that making shortcode delimiter like the ones propose make user in a complicated situation.

      Maybe english people don’ìt know that { and } are not present in alla keyboards directly as the [ and ] are, in Italian keyboard [ and ] are generated using ALT-GR+è or ALT-GR++ and keyboards show ALT-GR basic sign like €@## and [ ].

      To get { and } you net to type ALT-GR+SHIFT+è and ALT-GR+SHIFT++. Most people don’t know about this combination (i think only who write code does) and the parentesys are not written on any key.

      This change would be a a really complication for italian users and i supect a lot of non english keyboard suffer from the same problem. Please when you think to change something like this for end users, use ALWAYS the KISS Method, [{{ is not KISS at alli suggest adding after the square bracket somthig that is awailable on alla keyboards like ! or ? or % or & but for sure nothing is really hide in system and that 99% of people ignore that existes or how to get it leke curly braces

    • Curtiss Grymala 3:04 pm on September 3, 2015 Permalink | Log in to Reply

      First of all, let me say “Thank you” for beginning the process of thinking about this issue.

      As we begin to move forward, has anyone thought about trying to identify where & how shortcodes are currently used, then trying to come up with the best solution for each of those situations? Maybe a shortcode isn’t the best solution for all of the situations in which it’s being used; it’s just the only viable solution we’ve had up until now.

      The bottom-line is, we are still going to need placeholders of some sort within WordPress. There are perfectly reasonable uses of placeholders, regardless of the syntax they use.

      Some are within HTML tags, such as <div class=”[my-custom-class extras=’one-third first’]”> or <img src=”[some-img-url]” alt=”This is my awesome image”/> or <a href=”[my-home-page]” title=”Return to [my-site-name]”>. There are perfectly valid & viable reasons to need to use some sort of placeholder inside of HTML (even if it’s not necessarily inside of the Visual Editor). If we stop allowing these sorts of things, then we’re going to end up with more plugins implementing silly things like [custom-html tag="a" href="{{home-page}}" title="Return to {{site-name}}"]Go Home[/custom-html] which will introduce potentially even bigger security issues and more usability issues for the users.

      Some stand on their own, such as We currently have [number-of-users] users on the site.

      Some output HTML, such as .

      Some are, IMHO, poor attempts to get around security restrictions in the editor, such as [iframe] (there are many plugins that implement this, by basically allowing the editor to enter an HTML tag in normal syntax, but replacing the < and > with square brackets).

      Some need to be nested in order to work properly, such as something like:
      [accordion-title]My accordion item title[/accordion-title]
      [accordion-content]Some content to appear within the accordion item[/accordion-content]

      Some are also completely related to formatting the content, attempting to “fix” the problem of only having one content area, and trying to keep editors from needing to enter HTML on their own, such as the various implementations of column shortcodes (this is a very similar reason that implementations like the accordion shortcodes above exist, as well).

      There are many more perfectly valid & useful implementations of the current shortcode API.

      Each of these may have a completely different and much better solution than the others.

      Finally, as WordPress has evolved very, very far from a simple blogging platform, we need to stop thinking that post titles and post content are the only features that people are using.

      There are plenty of places where shortcodes might appear within the Options table, possibly even inside of serialized data. For instance, shortcodes could, potentially be used within widgets or theme options.

      Shortcodes could absolutely be used, within plain-text or serialized data, within the postmeta table (there are a ton of places that custom fields on a post could include shortcodes) or possibly even the usermeta table.

      The problem I see with the original discussion we had after 4.2.3, and the problem I see with some of the comments on this post, are that there still seem to be people saying “You shouldn’t have used shortcodes to do that; that was stupid, so it doesn’t matter if we break them.”

      The thing I appreciate about this original post, regardless of the fact that the proposed solutions are certainly more complicated than they should be, is that the core team is back to at least trying to solve the issue rather than continuing to feel like it’s okay to break sites just because they disagree with the use-case.

      Again, though, I want to thank @miqrogroove for bringing this to the table and beginning the process of trying to move forward.

    • Stanislav Khromov 5:01 pm on September 3, 2015 Permalink | Log in to Reply

      I’d like to echo the dismay most people feel with this proposed roadmap, and also stress that this can cause a fragmentation in the WP ecosystem.

      Consider that it’s easier than ever to not keep WordPress up to date with the latest version – security patches are backported all the way back to 3.7 and minor security fixes get applied automatically.

      By introducing big, “scary” changes that people feel uneasy about (which the comments prove include even seasoned developers), many will just not update. Consider the kind of fragmentation Python faced after they introduced syntax changes in version 3. Most people just didn’t want or need these changes. I feel the same way about this proposed roadmap.

    • Grant Palin 5:07 pm on September 3, 2015 Permalink | Log in to Reply

      One other gripe I have with the current shortcode syntax is that it sometimes confuses Markdown. Particularly when using within posts authored using WP-Markdown plugin. There’s a workaround, but still.

    • Jacob Santos 8:59 pm on September 3, 2015 Permalink | Log in to Reply

      Why was shortcodes added and what problem did they attempt to solve? What was their original purpose?

      The reason it is a bad idea is that it neglects the reason shortcodes were created and why they have the syntax they do. The history for the most part, I believe, was based in part on BBCode. If Handlebars existed and was more popular, then who knows.

      You will always have a security issue. Tough, but true. Introducing a wholly obtuse syntax will not change that and will make it more difficult for users and developers. Most likely you will just see an influx of plugins that put back the same behavior or those who move away from shortcodes altogether. It will also be a slap in the face once another security flaw is discovered.

      Unless you wish to remove the security issue by pushing people away from using shortcodes completely or creating their own solutions of various quality for security.

      The timeline is far too short. If you go back as far as 3.7, then you must follow that pattern. Deprecate it and update it if you can, but you must support the current until 2018 or 2 years after 4.4 is released.

      • Jacob Santos 9:06 pm on September 3, 2015 Permalink | Log in to Reply

        Lest my argument be misconstrued; I am saying that any syntax change is a bad idea. Solve the security problems with the current or work around them. The bed is made, you have to deal with it.

        Biggest problem I have is that it looks like you are creating a new template engine. There are already those that exist. It would be better if you used them as several have already been audited by security professionals or could be audited by security professionals. They also have a community and those that support it outside of WordPress.

        If you are going to introduce a template engine to WordPress posts, which in and of itself is charge to be warily, you are better off going with something developed by professionals.

        Shortcodes were supposed to be an easy replacement for plugins without a lot of features. When did it become the end all, be all to implement a feature?

        If anything, the RFC should formally suggest where shortcodes should and should not be used, deprecate those usages and at a later date, remove support for them. If advanced templating is required, move to an actual established templating engine. Oh my god, I can’t believe I’m writing this.

        I must be in shock, because the horror of this is just beyond comprehension. I can’t imagine the discussions or the problems that lead to, “hey, why don’t we create a template engine for posts?”

        I have to conclude that this is an early or late April Fool’s joke.

    • chriscct7 2:44 am on September 4, 2015 Permalink | Log in to Reply

      As an update, the team working on the Shortcodes Roadmap, has been working on a new draft proposal which will be posted to make at some point in the near future, which is significantly different than Draft 1. Also, there will be a weekly meeting that will be announced during the same post.

      • John Teague 2:10 pm on September 4, 2015 Permalink | Log in to Reply

        Thank you Chris. And to everyone working to find a solution on this. Hope people realize how much time and effort has to go into this.

    • Ustimenko Alexander 4:45 am on September 8, 2015 Permalink | Log in to Reply

      Users must not pay for developers weakness. If you want something stron-n-cool, then incorporate twig syntax here, but this all is for DEVELOPERS an dnot for users. This syntax is super-extra-bad-thing for end user.

    • renzms 11:57 am on September 17, 2015 Permalink | Log in to Reply

      I’d also like to join in to say that this is more of a step sideways rather than forward/backward with this proposed roadmap.

      Whether a seasoned developer, or just a regular user with no coding experience, these shortcodes aren’t in any way user or developer friendly.

      I understand this is a draft, and a new proposal will be made, especially after all the sentiment seen here. So thanks for the warning, and not changing anything right away as drastic as this. I know this proposed change is due to a security issue, but it causes more problems down the road that developers/users just might not use WordPress anymore to begin with. I do hope a simpler and more unified solution is found.

    • Tom Belknap 8:08 pm on December 15, 2015 Permalink | Log in to Reply

      This isn’t quite such a draft when you consider that 4.4 is already here and already, it’s messing up my shortcodes. Specifically the nested ones.

      This might be the worst proposal since the rightfully-maligned “Settings API,” an innovation the devs needed to ditch in favour of the Customizer. How error-prone is this syntax?

      `[{{shortcode}$] HTML [${encl2}$] HTML [${encl3}$] HTML [${shortcode}}]`

      I thought this double curly brace business must have meant that you were calling out the things we were supposed to change. Then I saw you were serious about it.

  • Simon Wheatley 4:43 pm on June 29, 2015 Permalink
    Tags: , ,   

    WordCamp Europe 2015 – Multilingual Discussion 


    Drupal Multilingual Overview

    Christian López Espínola is a Drupal contributor to the multi-language features in that project, and gave us this overview of the multi-language features in Drupal 7 and upcoming in Drupal 8:

    In Drupal we have different modules bundled with core. The language module was added in Drupal 7, which gives support for assigning languages to content. Drupal already had Interface translation support as WordPress does.

    Drupal 8 adds content translation, with a UI.

    At first the problem was that there are two different strategies for doing this. Drupal 7 gave support for translating a node by creating copies; having multiple posts for one piece of content. This made it hard to select the content required for display in case other modules were not aware of or not interested in multilingual support. Moved to translations to the field level, so stored in the equivalent of post meta. One content node now contains all the translations, so fields attached to the node have translations of the title, content, etc, into all languages if those fields are translatable, this way content is not duplicated (i.e. fields are not duplicated between nodes if they are not translatable).

    Q: How does Drupal connect languages into groups of translations of one piece of content.
    A: If one post is a translation of another, there is a field which links it. D7 two nodes. D8 has one node, and translated content is stored in fields (with language attribute).

    Q: What issues did the adding of content language cause?
    A: At some level we had two different posts, one for each language. So if your plugin doesn’t consider internationalisation, then this causes issues because you are considering translations different content, and mix languages in the UI. For example if we want to rate a post from a rating plugin, we may want the rating to be “shared” between all translations of that content.

    Some Approaches to Attributing Language to Content

    We looked at just a few multi-lingual plugins, to see how they addressed the issue of storing language content.

    <caveat>This is by no means an exhaustive list, and only reflects the solutions that the people in the discussion have had experience with 🙂 Please feel free to add other examples or corrections in the comments.</caveat>

    • WPML table structures – to attempt to synthesise this link: each translated content object is stored as a post, and a WPML database table links the content into groups and specifies the language of each content object.
    • Babble puts each the translation for each content object into a custom post type or custom taxonomy, e.g. `post`, `post_fr_fr`, `post_pt_pt`, and uses a taxonomy to group translated content objects together, so you can say “this post is a translation of this other post”. A disadvantage of this approach is that translated content does not have an expected post type, e.g. post, but instead a Babble translation post type, e.g. `post_pt_pt`.
    • Multilingual Press stores content from each language in a separate site in a WordPress multisite. There is a database table which links translated content across sites.
    • Polylang does not create any additional database tables at all. It creates 4 taxonomies: `language`(to hold all the languages you configured), `term_language` (the terms in each language ex. EN: Uncategorized, DE: Allgemein), `term_translations`(connects the translated terms in each languages) `post_translations` (connects translated posts). The plugins seems to be fairly lightweight compared to others and works well with many additional plugins too.


    There are lots of varied issues which multi-language, translation, and/or localisation plugins and projects seek to solve. WordPress core should not provide a translation or localisation UI and/or workflow, we should continue to rely on the plugin space to address different user scenarios.

    We do believe that there are some things which core could provide which would facilitate translation in the ecosystem for this type of plugin.

    Proposal one: core could provide a minimal way to mark content (e.g. posts, terms) as a particular language.

    • In the simplest case, a single language site, all posts would be implicitly assumed to be in the selected front end language for that site.
    • When a translation/localisation plugin is added, the plugin has the duty to set the language for each piece of content (post, term, etc).
    • If this shipped, it would be, by design, “invisible functionality”, and an example plugin would be useful.
    • How would this affect the WordPress exporter and the importer? The translation/localisation plugin would have the duty to add any UI to the importer/exporter, and core would need to provide the necessary hooks, etc.
    • Should we consider special locales like “no language” and “unknown language” (Drupal does this)? Perhaps core specifies these “locales” as a standard, but doesn’t use it.
    • This might be implemented as an additional column on the `wp_posts`and `wp_terms`tables, with associated post and taxonomy API additions and enhancements, which is available for plugins to use.

    Proposal two: core could provide a method or standard for translating strings stored in content objects like widgets

    In some contexts it is hard for a translation or localisation plugin to know what requires translation, e.g. in widgets when the data is stored in a blob in the database. It would be useful if core provided a pattern for others to follow to mark particular strings of text as translatable.

    Taking the example of a plugin providing a text widget, with user editable title and body fields, this plugin could follow the same standard to make these strings available to translation plugins. A possible implementation might be a filter or set of filters to pass the string for translation, and perhaps also the nature of the string to give a hint for the translation UI required, e.g. “rich text” (perhaps the translation plugin would provide a TinyMCE instance), “plain text” (perhaps a simple text area), etc.

    Other things discussed:

    • Setting the admin area language differently to the front end language, including showing the admin bar in the admin area language – being addressed in #26511
    • Supporting variants on locale, e.g. Portuguese informal, as these cannot be defined within the ISO standard currently – being addressed in #28303
    • dejudicibus 4:57 pm on June 29, 2015 Permalink | Log in to Reply

      You forgot to mention QTranslate X, one of the best fre plugin for multi-language sites.

      • jennybeaumont 5:01 pm on June 29, 2015 Permalink | Log in to Reply

        As Simon states above, this is not an exhaustive list, but simply the plugins that were discussed by (because used by) those who attended this meeting.

      • Simon Wheatley 5:11 pm on June 29, 2015 Permalink | Log in to Reply

        If you have knowledge of how QTranslate X stores and manages translations, please do add those details here.

        • MSTannu 7:51 am on June 30, 2015 Permalink | Log in to Reply

          The qTranslate family of plugins store multilingual content in the same database field, wrapped in special markup to differentiate each language. While this seems like extra work to pull content in all languages every time, in practise it’s considerably faster than queries with increased complexity to link different fields / tables to language information stored elsewhere.

        • Mike Manger 9:17 am on June 30, 2015 Permalink | Log in to Reply

          It stores all the language content in the post content areas with wrappers for each language (e.g. [:en]Title[:fr]Titre[:]). It has hooks to filter the content and return the requested language. There is no interface for widget support (although you can manually use the tags).

          Not a very clean implementation but I do like the tab interface for changing language.

        • Ihor Vorotnov 3:37 pm on October 13, 2015 Permalink | Log in to Reply

          I use qTranslate X on some projects, as well as Polylang, WPML and Multilingual Press. The qTX stores all traslations in the same fields, with a special markup. It has pros and cons. However, the biggest con for me always was its incompatibility with other plugins. Currently, I’m trying to make it work with ACF Pro and bbPress. Damn, it’s a pain in the a$$. Possible, but still lots of work and debug.

    • Pascal Birchler 5:12 pm on June 29, 2015 Permalink | Log in to Reply

      As this is a very complex topic where we need to consider lots of details and exceptions, I suggest starting a working group with weekly meetings etc. I’m very willing to help out.

      • Nick Weisser 10:59 am on June 30, 2015 Permalink | Log in to Reply


      • Simon Wheatley 10:11 am on July 1, 2015 Permalink | Log in to Reply

        @swissspidy This sounds good to me. I am free in the evenings on Monday and Thursday, EU hours, or during the day, EU time, most days. No time is good for all the world, but evening might be better?

        • Silvan Hagen 12:34 pm on July 2, 2015 Permalink | Log in to Reply

          Same here, could do Monday or usually during the day in the afternoon. Would love to put this into a working group.

          • Simon Wheatley 11:48 am on July 8, 2015 Permalink | Log in to Reply

            Sounds like Monday afternoon or evening, EU time, is a good option. I’m going to suggest we start the week after next (20th July), purely because I want to be there but I’m away next Monday. 🙂

            I’ll see what Slack channel we can find for these weekly conversations, and report back.

    • wycks 5:19 pm on June 29, 2015 Permalink | Log in to Reply

      You should really engage the folks over at WPML and Multilingual Press (Toscho), once you get into the details lots of little problems crop up, for example this has been bounced around for years and is a big pain for multi-lingual sites: https://core.trac.wordpress.org/ticket/18962

      At the end of the day if WP can help define relationships between content (taxonomy, posts, etc) it would go a long way to facilitate the already available plugins and other non translation plugins being able to work with complex content.

      • Francesco Di Candia 5:23 pm on June 29, 2015 Permalink | Log in to Reply


      • Misamee 6:17 am on June 30, 2015 Permalink | Log in to Reply

        @wycks, WPML is already engaged.
        Unfortunately, due to flight schedules, we couldn’t manage to attend the contributors day, but the day before we did share already some thoughts with Simon.

      • Frank Bueltge 12:22 pm on June 30, 2015 Permalink | Log in to Reply

        We (Inpsyde Company an the MLP team, include @ocean90 ) will help. We discuss the topic very often, also on contributor days. Our MultilingualPress is a result, dev and and maintain since 2007. But always with focus solid, stable, fast, no changes on core or content tables. I’m willing to help.
        Import for me is, that the WP core should provide the foundation/data, plugins should provide the details/UI.

      • Simon Wheatley 10:22 am on July 1, 2015 Permalink | Log in to Reply

        @wycks Thanks for the ticket reference.

        Everyone is welcome to contribute. 🙂 As @bueltge says, Multilingual Press are involved. Robert (@nullbytes), who is a colleague of @toscho, was in the discussions at WCEU. The second proposal builds on a conversation with @amirhelzer and @sciamannikoo from WPML on Saturday at WCEU, and it was good to get their input.

        Regarding relationships and groupings: core already supports grouping posts and terms via taxonomies (e.g. to group all the translations of one post together), and this seems (to me) to be sufficient to the requirement here. Further, the taxonomy roadmap (update post) may offer a different way of linking posts together in the future, providing another route plugins could take advantage of.

        /cc @francescodicandia @paddyullrich

    • Michel - xiligroup dev 6:13 pm on June 29, 2015 Permalink | Log in to Reply

      Just to contribute in this multilingual context (well known and studied since 2009), before I will translate them in english, please find below two links to short recent articles.

      As data-designer, the very first thing to consider is to choose a monosite (standalone) WP install and a multisite (network) install. Secondly the main thing, in each type of install, is to consider standards so each developer can decide what type of plugin (easy to use for newbies or flexible full customizable for developer). In previous WordCamp in Paris and Lyon, I have done some analysis… and solution is not unique…



    • Matt Mullenweg 7:42 pm on June 29, 2015 Permalink | Log in to Reply

      Both proposals seem to assume that core should do something and this is no longer plugin territory. I haven’t seen anything yet that makes the case definitively for that.

      • nikolov.tmw 10:46 pm on June 29, 2015 Permalink | Log in to Reply

        Well, I guess there’s also the possibility of creating a working group that would be given the task of coming up with a proposal and then standard for how plugins should handle the content to language/locale association. This can be created as a feature plugin and if there’s an obvious benefit it could land in core in the future.
        In any case, having a standard way of creating that would be a great benefit in terms of plugins being compatible with each other and switching from one to the other would be a painless transition.

        It’d definitely be interesting to see if we can figure out how many sites actually use more than one language(although there are also those sites that don’t use a plugin, but instead just create separate sites for each locale).

      • Caspar 4:32 am on June 30, 2015 Permalink | Log in to Reply

        Both proposals do in fact assume this should continue to be plugin territory and core should do something about it in order to provide basic means of production.

        • Frank Klein 1:09 pm on June 30, 2015 Permalink | Log in to Reply

          That’s my understanding from reading the write up as well. It’s about Core being conscious of the existence of multilingual plugins, and accommodating them as much as possible.

      • Pascal Birchler 1:26 pm on July 1, 2015 Permalink | Log in to Reply

        Both proposals do in fact assume this should continue to be plugin territory and core should do something about it in order to provide basic means of production.

        The same was done with the REST API for example, where lots of new functions were introduced in WordPress to better support it. And core definitely needs better support for multilingual content.

      • Silvan Hagen 12:47 pm on July 2, 2015 Permalink | Log in to Reply

        In my opinion core should at least be aware of the language the content is written in and therefore provide some form of support towards storing the language of any type of content within core (all post types, taxonomies, widgets, user meta could be plugin stuff as it’s handled differently with different multilingual solutions).

        The problem with multilingual stuff in general is that not only does it add a huge layer of complexity to the solution, but also to the people using it (editors for example). For example Typo3 has multilingual baked into core and sadly it’s tied to a main language tree, so let’s say you have English as your main language and French as a secondary language, you can’t create content that is only available in French as languages are tied to the tree of pages for the main language.

        So for WordPress it’s clear that if we get some basic support in core, the means for translating the content and how to do it needs to stay plugin territory. For example I know sites where they display comments from all languages on translated posts and use basic machine translation for it. Others have a singular discussion stream per language and post. User meta for example is dealt with differently, if your site has a content team per language, the author bio doesn’t need translation and will be filled in the language the specific author is writing for, but if an author gets translations for their posts, a plugin needs to handle translations for the user meta.

        Here are some thing that could use basic support in core to make it easier:

        • Portability and content migration
        • Best practices and easy transition between multilingual plugins
        • Simple API in core to allow plugins to be aware of multilingual content

        We are far from a solution for this, that’s why I’m in favour of forming a working group for the topic as I think this is something that needs a lot of testing and experiments before a bullet proof minimal solution would be possible at all.

      • Simon Wheatley 3:02 pm on July 3, 2015 Permalink | Log in to Reply

        @matt Neither proposal intends to move the problem out of plugin territory; we are proposing that core provides some additional or enhanced APIs, and that core leads with some new hook standards or patterns. Plugins would still provide the UI and workflow, and would address different use cases. Any additions to core would be designed to make it easier for plugin authors (and ultimately better for users). We would obviously want to ensure that any changes do not break existing solutions, other plugins, or user sites, as is The WordPress Way.

        In relation to the first proposal, for core to provide a way to mark a post as in a particular locale: to my mind, all the approaches used by plugins described so far in this post and the comments have drawbacks. There is work required to integrate with other plugins. There are additional queries or query complexity. For me, ideally, adding multiple languages to a WordPress install should have little performance impact, and should not cause the need for workarounds with other plugins; currently this is not the case.

        • Matt Mullenweg 5:22 pm on July 3, 2015 Permalink | Log in to Reply

          But at least with a pure-plugin (no core changes) approach the added complexity and overhead of queries is only borne by sites running that plugin, not every user regardless of whether they’re multilingual or not.

          • Frank Klein 5:32 pm on July 5, 2015 Permalink | Log in to Reply

            So far I’ve seen no precise architecture proposal, or even code.

            Yet we’re dismissing the proposal outright for some vague fear that this might be too complex or not performant enough?

          • Benny 3:48 am on July 15, 2015 Permalink | Log in to Reply

            I guess for most of us living in a country where it’s natural to have sites with multilingual content it’s a no brainer that the content language should be a core field of any database table that stores content.

            On the other hand I can understand that someone who only has to deal with single language sites is not eager to see core to become multilingual when that means more complexity/overhead for common queries. But that same argument is also true for us who have to deal with the daily fact of sites being naturally multilingual, the sites of our clients, just like non-multilingual sites, should also function with a minimum of complexity/overhead.

            Maybe this difference in point of view can be overcome by introducing an core option or wp-config setting ‘IS_MULTILINGUAL’ and an related API conditional $is_multilingual.

            Core queries would then only execute the somewhat more expensive multilingual variant of a query (i.e. with an added WHERE language = %s clause/clauses) of any core query when the $is_multilingual query is set to true.

            To make these ml queries as fast as possible (i.e. no extra JOINs) we should concider to add a language field to any relevant table. This would bring the fastest possible solutions to both single language sites AND multilingual sites!

            I know you are not keen on db schema changes but I think the language column on relevant tables makes perfect sense.

            The user interfaces side of this could stay in the plug-in territory.

      • FolioVision 1:43 pm on July 9, 2015 Permalink | Log in to Reply

        Matt, may I remind you about the socio-economic phenomenon called Globalisation? I know Americans tend to hang out in monolingual environments but here in Europe almost every website needs to be bilingual at least. A bog-standard example would be a site like Czech computer vendor Alza.com which comes in Slovak, Czech and English. Our European issues with multilingual extend to Canada (French, English), China (Cantonese, Mandarin), much of South America (Spanish, Portugese for sites which traverse more than one territory). English speaking monolingual territories are a tiny minority (USA, England). Among other large nations, Russia is a similar monolingual environment but even Ukraine needs two languages on most websites. France, Germany and Holland most often want their native language with an English option.

        Leaving multilingual in plugin territory means there is no standardised solution to accelerate multilingual development. Every time one takes on a multilingual site one is reinventing the wheel. I’ll grant you Google’s games going back and with with ranking and demoting content on subdomains and Google’s confusion about multilingual articles on the same domain do not help at all with standardisation. Still many of the SEO bonuses and penalties for different languages on one site have been retired now.

        Plugins should be ENHANCING multilingual, not building it from scratch. At the very least, WordPress.org should add all the key building blocks. I would suggest you talk to the three or four top multilingual plugin authors, whether WPML and Polylang and ask them what additional syntax they would like to see in core to make their plugins faster, lighter and more reliable.

        Again just improving core support for multilingual is still a second best solution. Basic, reliable multilingual should really be in core.

        • Ihor Vorotnov 4:20 pm on October 13, 2015 Permalink | Log in to Reply

          +1 on this from Ukraine. I’ve built many multilingual sites for clients from Ukraine, Russia, Europe and even US (some US regions in fact have 2nd working language, spanish). Used Polylang, WPML, qTranslate X, Multilingual Press, tried and tested ALL available solutions (xili, Bogo etc), even built custom lightweight solution for one of the projects from scratch.

          I keep saying “there’s life outside US”. Because it is.

    • Alvaro Gois dos Santos 2:40 pm on July 2, 2015 Permalink | Log in to Reply

      First of all, let me congratulate @simonwheatley and all who were involved in the discussions we had in Seville regarding this subject.

      I second the notion that giving core the ability to identify content language could eventually be useful to all multilingual plugins and, hopefully, to make the switch between them easier. In a way, it would give the infrastructure, plugins would do the rest, UI included. It would also account for different approaches and I guess all the work that has been put into this solutions wouldn’t be lost.

      Another discussion is about locales and variations. Several locales have variations that, for now, WordPress doesn’t recognise. This makes it hard (if not impossible) for the common user to change his language variation from the default to a different one, even if it’s included in GlotPress. For instance: in Portugal (pt_PT) we have three versions, default (formal), informal and the Ortographic Reform version (not yet in GlotPress). We’ve tried to make it easy for the user with our PT Variants plugin, but I’m certain there could/should be a more efficient approach.

      • Simon Wheatley 11:03 pm on July 3, 2015 Permalink | Log in to Reply

        @alvarogois I believe the locale variants, e.g. Portuguese formal/informal, is being looked at actively in ticket #28303 🙂

        • Alvaro Gois dos Santos 11:07 am on July 6, 2015 Permalink | Log in to Reply

          Thank you @simonwheatley, and thank you @ocean90 for being so active on this. I was not aware that we were so close 😉

        • Alvaro Gois dos Santos 4:27 pm on July 7, 2015 Permalink | Log in to Reply

          Hi again @simonwheatley: I’m having this idea about variants and managing variants translation through GlotPress, I’m not really sure where I should post it or if it has something to do with this discussion.

          Anyway, my idea (regarding GlotPress) is this: some variants are very close to the default language, only some of the strings actually differ, at least in Portuguese. I’m thinking of the formal vs. informal but also the orthographic differences like our own pre and post Orthographic Reform. There should be a way, in GlotPress, to link variants to the default translation. This would allow to have only one translation flow, and changes to be simultaneous in all variants. Now we have different flows, which makes it inefficient.

    • Vadym Volos 3:01 pm on July 7, 2015 Permalink | Log in to Reply

      I did not understand what the problem is?
      See how it is implemented on this engine: opencart.com

      You can take as a basis the plugin Polylang.

      • FolioVision 8:33 pm on July 14, 2015 Permalink | Log in to Reply

        I’m with you Vadym. Polylang would be a very good start. It’s lightweight clean code, the best of the current solutions.

        • Ihor Vorotnov 4:54 pm on October 13, 2015 Permalink | Log in to Reply

          I love Polylang, use it for most of the projects, however, it’s not the only way to go. It’s good, fast, but not perfect. And to make it better, we really need some Core changes. Ask Polylang’s author, @chouby, check WordPress.org Polylang support forum threads and some GitHub repos, related to Polylang, like this and this.

    • Vadym Volos 3:17 pm on July 7, 2015 Permalink | Log in to Reply

      When you do language support.
      Should work plugin “Page Builder by SiteOrigin” (https://wordpress.org/plugins/siteorigin-panels/)
      This ingenious plug! It helps make any page.

      Then I wrote about it: https://wordpress.org/support/topic/page-builder-by-siteorigin-only-works-for-one-language?replies=16

      • Ihor Vorotnov 4:58 pm on October 13, 2015 Permalink | Log in to Reply

        Having some basic API in WordPress Core, then multilingual plugins built over this shared API, will allow other plugin developers (including Page Builder) to add ML support easily. This whole discussion is about establishing some standard and following it by other plugins too.

    • Benjamin Lupu 10:28 am on August 10, 2015 Permalink | Log in to Reply

      First thank you for addressing this painful issue. I am still surprised that WordPress doesn’t provide a minimal multilingual support. I am working with since 2008 and at every project, it’s a problem. I also hear complains at every WordCamp (and not only French). There’s a huge need for that! IMO it could be a huge competitive advantage for WordPress providing a way its users to expand their businesses and publish globally. It would be also a selling point for enterprises. This is one of the first question asked when selling WordPress to an enterprises. I totally agree with what have been said earlier in this thread: we have to go further than an API and provide a minimal ready to use UI. It’s also a good idea

      • Benjamin Lupu 10:32 am on August 10, 2015 Permalink | Log in to Reply

        Shot too fast… So it’s also a good idea to be able to activate the multilingual in the wp-config settings. Anyway, correctly addressing the multilingual issue in WordPress would be a huge improvement and a sign that WordPress is really going global.

    • starise 1:00 am on September 8, 2015 Permalink | Log in to Reply

      Multisite Language Switcher is the best plugin for multilanguage so far.

      • Ihor Vorotnov 5:00 pm on October 13, 2015 Permalink | Log in to Reply

        Why it is? Can you list it’s pros from your point of view / experience?

        • starise 11:29 pm on October 26, 2015 Permalink | Log in to Reply

          I use multisite language switcher in production.
          Pros: fast and compatible, easy to use, good apis
          Cons: need the use of multisite.

          I have to say that i like the approach of “sublanguages” plugin too, in particular because it doesn’t need multisites. It uses most of core functionalities and marks post_type field as “translation_{ID}” to identify translations.

          Unfortunately this kind of “all-in-one” approach for my experience, can cause compatibility issues, where multisites is much more safe for production imho.

    • Vadym Volos 1:14 pm on September 23, 2015 Permalink | Log in to Reply

      Hi guys!
      What news?
      What progress?

  • Pascal Birchler 6:28 pm on October 15, 2014 Permalink
    Tags: ,   

    WordPress Core Weekly 

    Hi everyone!

    It’s time for WordPress Core Weekly (formerly known as Last Week in WordPress Core) again! There have been suggestions to change the name of these summary posts to stop confusions about being the last week of WordPress.

    This updates covers all commits since last week up to today, October 15th. There has been much progress this week with lots of improvements to the WP_*_Query classes, fixes in the admin and of course the introduction of the Twenty Fifteen theme! The complete summary:


    • Add missing labels to category filter dropdowns. [29870] [29871] #29921
    • Differentiate between invalid and missing admin emails when adding a new site [29877] #17890
    • Multisite: Do not send a welcome notification when noconfirmation has been flagged [29880] #16235
    • Admin menu: [29898] #29806
      • Fix pinning after resizing the window.
      • Merge the two DOM ready callbacks in common.js
      • Fix the submenus position adjustment on focus.


    • TinyMCE: fix the ‘wpgallery’ plugin to use a placeholder for galleries when either the ‘wpview’ plugin or wp.mce is not loaded. [29883] #28756
    • Quicktags: move focusing the editor after inserting content to the end of the code blocks. [29884] #29944
    • Editor-expand: reset the editor height after the window is resized. [29886] #29952


    • Change instances of “Theme Customizer” to just “Customizer”, as the Customizer isn’t necessarily theme-specific. [29903] #29947
    • Only POST dirty settings to preview to improve performance. [29905] #29983
    • Don’t trigger a change event if two unchanged object values are equal, second pass. [29907] #26061

    Bundled Themes

    • Add an alt attribute with the site title for header images linked to the home page. [29842] #15926

    Twenty Fifteen


    • Add aria-describedby attributes to comment_form(). [29846] #24148

    External Scripts

    • Update jQuery UI to 1.11.1. [29847] #29833
      • Rename all files, remove the jquery.ui. prefix. Add old files to $_old_files.
      • Add and use non-minified files in /src.
      • Add grunt task to minify jQuery UI files.
      • (Non-minified files will not be shipped.)

    Language Packs

    • Language packs: Remove translations when deleting a theme or a plugin. [29856] #29860


    • Handle deficiencies in PHP’s parse_url in older versions of PHP (<5.4.7) in WP_HTTP::make_absolute_url(). [29851] [29850] [29861] #28001, #29886
      • Correctly handle url’s containing url’s in WP_HTTP::make_absolute_url().
      • Correctly support Schemeless URLs in WP_HTTP::make_absolute_url() by respecting the ‘host’ field if present in the relative url.
    • New remove() method and some unit tests for the WP_Error class. @29854 #28092
    • Return an error when adding a term to a non-existent parent. [29867] #29614


    • Use the primary meta_query clause when parsing orderby in WP_Query. [29855] #16814
    • Introduce support for nested queries in WP_Meta_Query. [29887] #29642
    • Use only LEFT JOINs when a meta_query contains a NOT EXISTS clause. [29890] #29062
    • Introduce support for nested queries in WP_Tax_Query. [29891] #29718 #29738
    • Support EXISTS and NOT EXISTS in WP_Tax_Query. [29896] #29181
    • Support nested tax query syntax in redirect_canonical(). [29901] #29738
    • Avoid redundant table joins in WP_Tax_Query. [29902] #18105

    Thanks to @rianrietveld, @tschutter, @netweb, @joedolson, @bramd, @Fab1en, @ocean90, @stephenharris, @boonebgorges, @georgestephanis, @jesin, @afercia, @miqrogroove, @avryl, @mboynes, @transom, @DrewAPicture, @johnrom, @matt, @iandstewart, @iamtakashi, @obenland, @cainm, @kristastevens, @karmatosed, @chellycat, @lancewillett, @kwight, @davidakennedy, @otto42, @jakub.tyrcha, @studionashvegas, @tareq1988, @westonruter for their core contributions!

    Revisions covered: [29842] to [29907]. For the complete list of commits to trunk, check out the log on Trac.

    Interested in joining in? Write or test a patch for 4.1.

compose new post
next post/next comment
previous post/previous comment
show/hide comments
go to top
go to login
show/hide help
shift + esc
Skip to toolbar