Make WordPress Core

Updates from January, 2017 Toggle Comment Threads | Keyboard Shortcuts

  • Matias Ventura 4:52 pm on January 17, 2017 Permalink |

    Editor Technical Overview 

    As we start looking at the editor from a technical perspective it’s important we identify the main obstacles and requirements we face before we start conjecturing solutions. As @matt wrote before, the editor focus aims to make writing rich posts effortless. This has taken the path of treating a post as being composed of distinct pieces of content called blocks. These pieces should be easy to insert and manipulate, providing rich and contextual interfaces to interact with as you craft a post.

    So how do we go about turning this into a reality? Content in WordPress is, fundamentally, HTML-augmented text; that is to say, it has no inherent data structure. This has been a very important aspect of WordPress and a force for the open web—it speaks to the sense of ownership and freedom WordPress gives you, since it’s always easy to get the full content of your publications—yet the lack of structure gets in the way of the goal to treat content as composed from individual pieces. (This reality also became an issue for the development of post formats a few years ago, but I digress.)

    It’s relatively easy to add structure, but it’s not trivial to do so in a way that doesn’t harm data integrity, portability, and the cohesiveness of the post_content. So let’s lay out a first requirement:

    ① Shape of the Data: Portable Text

    To ensure we keep a key component of WordPress’ strength intact, the source of truth for the content should remain in post_content, where the bulk of the post data needs to be present in a way that is accessible and portable, while still providing additional structure on top of HTML semantics for our editing tools. Data needs to be praised and respected. This additional structure would hopefully be invisible to the document’s display context, as it ensures the rendered content is viewable in situations that may not be aware of blocks at all. (Think of email clients, RSS, older editor versions, mobile editors, database dumps, etc.)

    Storing things separately means post_content becomes either a pointer or duplicated data, which fragments the source of truth since they can get out of sync easily. (A few content block plugins do this by storing structured data in postmeta and pure data in post_content.) On the other hand, storing things together means structure can become gibberish if it’s not formatted properly before display.

    How can we then offer users a great experience when creating or manipulating content without sacrificing the spirit of integrity and data reliability that is expected from WordPress? Good representations of the data would also make it easier to develop robust collaboration tools in the future by allowing us to lock things in a more fine grained way. I believe this is important to figure out soon to allow us to prototype quickly, so I’ll follow up with an initial proposal by the end.

    Honouring HTML leads to a second requirement:

    ② Simplicity and Semantics of Representation

    Unless we are improving the semantics of the document we should minimize what markup we add to identify a block; for example, avoid adding extra DOM elements or attributes, both for simplicity and standards sake. WordPress has been a champion of web standards, and we should not venture away from this quality. How can we add structure in a way that remains invisible to the output (as meaningful content as possible) but gives the necessary hooks to infer a structured view for editing purposes?

    While we discuss how to structure the content to include invisible demarcation of blocks, the aspect of their nature leads to a third requirement:

    ③ Static & Dynamic Blocks

    Blocks can be either static or dynamic. That is to say, some blocks can be stored with all the necessary aspects needed for their display, while others need to be processed before generating their final output (shortcodes, embeds, widgets, etc). This distinction is important because the most common two blocks people will naturally use are text and images. We should not break their clarity as we treat them as blocks.

     # Static
    Here’s some text.

    # Dynamic
    [text id=123] // Pulls "Here's some text" from somewhere.

    This conceptual separation of blocks is useful for designing our project, yet they generate abstract complexity which users should not be exposed to, leading to a fourth requirement:

    ④ Consistent Experience

    One of the biggest benefits of blocks is that composing a post becomes more intuitive and reliable. Everything is inserted under the same assumptions; discovering what can be inserted is a natural part of inserting content. To the user all blocks behave in a consistent and familiar way, even if they provide tailored UIs for their controls.

    The user should also be able to edit a post in a different system (mobile, REST, older version of core, apps like Mars Edit, etc), even if they lose the advantages of block editing. That’s another reason why post_content as source of truth matters, compared to storing a JSON structure in postmeta. Having things in two places means they can get out of sync depending on what tool you used to edit.

    These nuances of data, UI, and display lead to a final and more general requirement, which is understanding the system we’ll be crafting:

    ⑤ The System

    The editor experience has three fundamental aspects to its system: the UI used to manipulate a block; the demarcation of the block; the rendered output of the block. These are all separate concerns, from the tools we craft to edit a post, to the document syntax that holds the data structure, to the way the final output is generated to be displayed as HTML. (With static blocks that last aspect may be of no significant concern since the document doesn’t care about the presence of static blocks, it just displays them.)

    Picturing these concerns as connected but fundamentally separate would help us figure out the best design and technical solutions for each stage, while avoiding us taking aggressive moves by coupling expectations. For instance, JavaScript is a natural technology to look at when it comes to the ability to manipulate and interact with content blocks, yet it may not be the best at all when it comes to rendering the final post to a viewer. Avoiding painting the whole flow under the same light should allow us to focus efforts, because we don’t have to change everything.

    ❶ Coda

    As a final coda, and following @joen‘s design exploration, let’s keep in mind that our first goal should be to set up a reliable foundation to allow us to iterate quickly and test assumptions. I propose we focus initially on a few static blocks (text, image with and without caption, quote) to limit scope of the project.

    In which ways can we fulfil ① and ② from the above requirements?

    Shortcodes fall short in that they are not invisible, they are opaque, not standing to the scrutiny of semantics, and are also painful to parse. Alternatives could be data-* attributes in the HTML elements or custom elements (paving the way for web components, perhaps), yet we need to be careful with adding cruft.

    One other possibility is to look at HTML comments as a way to provide explicit demarcation to post_content without affecting the node structure of the document for something that is inherently spurious to the HTML semantics. WordPress already uses comments for the more tag (<!--more-->) and splitting content into pages in a way that has proven to be quite robust.

    It could look something like this:

    <!-- wp:image -->
      <img src="/">
      <figcaption>A picture is worth a thousand words</figcaption>
    <!-- /wp -->

    There are drawbacks, benefits, and implications with each that we should discuss separately. Are there any other possible solutions?

    Please, join us in #core-editor if you are interested. We’re holding weekly meetings in Slack, Wednesdays at 19:00 CET.

    • Mike 5:33 pm on January 17, 2017 Permalink | Log in to Reply

      There is a trend towards the use of more attributes and elements. The editor needs to allow for this or for others to create a plugin to do this.

      • programmin 10:07 pm on January 17, 2017 Permalink | Log in to Reply

        How does it not already? Are you referring to how KSES strips out data-attribute=”attributes” in html?

    • Davide 'Folletto' Casali 5:34 pm on January 17, 2017 Permalink | Log in to Reply

      Thanks for outlining this so crisply. Stating all the requirements clearly is something often overlooked, and I feel it’s a great starting point. I’m sure there are going to be things to be expanded in the following weeks, but seeing everything together in this way is excellent.

      Agreed on all the pieces. Let’s the discussion move forward! 🙂

    • FolioVision 7:09 pm on January 17, 2017 Permalink | Log in to Reply

      Wow, that’s quite an essay on the content editor. We’ve been doing this for a long time and know the lay of the land. We even built a WYSIWYG editor we can’t give up (we tried for two years). First in praise of the WordPress editing experience: the Media Library has grown to be very powerful and easy to use. It’s awesome. We were delighted to retire the original image editing module we were using last year and fully integrate the Media Library.

      Still what are the issues with WordPress content editing which drive us to maintain a separate WYSWIYG editor?

      Hint, it’s not the money, thousands of hours have gone into this purely GPL and non-commercial product. It’s not boredom. We have too much to do in three lifetimes and very short weekends already.

      It’s not shortcodes. Shortcodes are great and reliable ways to easily and visibly add content to posts. As soon as you start trying to hide the pieces, reliability falls through the cracks. It’s why lawyers still use WordPerfect.

      Why were we unable to give up our own FCK WYSIWYG editor? Code and javascript mangling. The WordPress TinyMCE editor does not reliably maintain paragraphs and linebreaks. It’s a bit WYSYMightNotGet. When creating complex nested tables and code examples our editor never fails (don’t thank us, thank FCK in Poland, we just maintain a solid WordPress port).

      What is missing with the shortcodes and code inside a fairly utilitarian textbox is true WYSIWYG. The only way to preview a post properly is to open it up with “View Post” and then reload the page to see any changes. This is slow and clunky.

      We’ve done what we can to improve WYSIWYG by in our editor by allowing with just the tiniest modification of your theme CSS real WYSIWYG. This only covers the CSS though and not the javascript or the short codes.

      What would make the WordPress editing experience amazing would be a live preview tab which turns the shortcodes and media into their visuals.

      We’ve looked long and hard at alternative edit experiences (updating FCK to CK was very hard and there aren’t really any world changing benefits: CKeditor is roughly the same experience albeit, better behaved on mobile). When we looked around at an editor to build for WordPress we settled on redaktor.js. Andrew Oz is very sceptical about Redaktor’s suitability for WordPress (licensing issues aside, just speaking technically):

      I like it too. However it’s far from being ready for anything more than
      comments/bbPress editor. Been following several jQuery based editors,
      all have little more than the default browser contentEditable functionality.

      This one is definitely more polished than most but still lacks a lot of
      things. For example: no plugins, no way to extend it other than hack
      into it, very minimal APIs, try copying and pasting something from
      another browser tab, then look at the HTML. And a big one: no undo/redo
      for all actions?

      I don’t know… Probably some will as it is nice. However considering
      the amount of work needed, not sure if that would be enough. Roughly it
      would take a good team of several full-time developers at least a year
      to bring it closer to TinyMCE.

      Nearly all JS editors get off the ground fast by adding UI to the
      browsers built-in contentEditable functionality. Once there, most try to
      tackle few more tasks and stop. The next step requires a lot of work:
      creating APIs, dealing with tons of browser quirks, etc.

      Currently there are several other editors at approximately the same
      stage. The problem is that a lot of the functionality for editing HTML
      should be in the browsers. It can be handles much better there than
      trying to extend the limited contentEditable mode with JS.

      So unless the browser vendors decide to sit down and make and implement
      a nice specification for editing of HTML, don’t see how this situation
      would change.

      We are a long way from 2012. Redaktor is awesome visually, very fast, beautiful preview, lightweight fast editing tools for media. In short everything we’d like the WordPress editing experience to be. In my opinion, TinyMCE will never have that shiny and fast and brilliant feel. It’s in neither the personality or the technology of the software.

      Alas Redaktor is not open source. A platform license would be $199. I’d happily pay it myself for WordPress but it cannot be as it’s against our community GPL principles.

      Perhaps there is something else out there which comes close and does have an open source license or at least a split license like CKeditor and FCKedit?

      That’s the holy grail. The same editing capabilities in the editor, legacy shortcodes but fast responsive and with **immediate full preview**.

      What we should avoid at all costs would be to deprecate shortcodes and create endless compatibility issues. Such a decision would cripple the platform and leave most of our shared library of powerful and obscure plugin obsolete.

      • chatmandesign 4:51 pm on January 18, 2017 Permalink | Log in to Reply

        I’m a big fan of CKeditor. It’s a popular enough option for this I have to assume the WordPress team already has reasons they haven’t adopted it so far, but it really seems like a perfect fit here:

        1) Available under GPL license.

        2) Includes built-in support for managing blocks of content.

        3) Extensible architecture supports plugins.

        4) Robust, battle-tested, actively maintained—and used by a host of major companies and projects, so it will likely stay that way.

        5) As an added bonus, it supports inline editing that could be integrated into the Customizer or some form of front end editor.

    • buzztone 11:45 pm on January 17, 2017 Permalink | Log in to Reply

      Really helpful overview – I look forward to following development of this project.

    • Joy 12:03 am on January 18, 2017 Permalink | Log in to Reply

      I agree with @FolioVision that shortcodes should remain intact, and would even go so far as to say that they should be the basis of content blocks. Why isn’t there a list of registered shortcodes available when editing a post? Why can’t the editor show the shortcode as a block with the parameters editable? Why wouldn’t the preview call do_shortcode() to show what that block looks like? And why can’t widgets be handled the same way, as part of the content?

      • Xavi Ivars 12:15 am on January 18, 2017 Permalink | Log in to Reply

        And that’s what (more or less….) Shortcake already does as a plug in: shortcodes can register UI to define how they should be edited, appear rendered in the WYSIWYG editor and in plain old-style text in the text editor

    • Weston Ruter 1:35 am on January 18, 2017 Permalink | Log in to Reply

      Thanks for the great writeup. I see a lot of alignments between this and the solutions being prototyped in the JS Widgets feature plugin. The plugin specifically aligns in terms of “the UI used to manipulate a block” and “the rendered output of the block”, leaving “the demarcation of the block” to encoding data via structured HTML/Microdata/comments. Likewise in:

      JavaScript is a natural technology to look at when it comes to the ability to manipulate and interact with content blocks, yet it may not be the best at all when it comes to rendering the final post to a viewer.

      The JS Widgets plugin specifically focuses on the use of JavaScript for managing the block’s attributes (widget instance) whereas by default PHP is used to render the block to HTML.

      WordPress needs to have a standard interface for how to define a block which can represent its state, present a UI for manipulating its state, and render its state onto a template. The widget provides such an interface, and with modernization and iteration I think the widget can serve as a standard interface for defining a block which can be used both in post content as well as in widget sidebars and elsewhere.

      Also, a general note that the JS Widgets plugin is focused on improving the WP_Widget API itself. It is not concerned with widget areas (sidebars) and thus is not focused on how widgets are managed in sidebars on the widgets admin page or via the widgets panel in the customizer. Rather, the plugin is about iterating on fundamental widget building block and allowing it as an interface to be leveraged and re-used in other contexts, such as to manage content blocks in the editor (as prototyped for the Post Elements in Shortcake). It’s also not concerned with the storage of widgets, as in the case of Shortcake post elements the instance data is stored as shortcode attributes, though the instance data could be stored as data-* attributes, inside of HTML comments, in HTML5 Microdata, and so on.

      • Matias Ventura 4:36 pm on January 18, 2017 Permalink | Log in to Reply

        Thanks! The stages I see us progressing from would be:

        • Define structure of post_content document that includes representation for both static and dynamic blocks.
        • Parsing of document on the fly to infer more manageable structure (JSON representation).
        • Implementation of general UI layer around manipulation and insertion of blocks.
        • The system to generate final output of dynamic blocks.

        Those last two could converge in a single API, but I think we’ll see if that’s the best path once we get closer to prototyping it.

    • Weston Ruter 1:47 am on January 18, 2017 Permalink | Log in to Reply

      Also, circling back to Twenty Seventeen and its “front page sections” feature (subset of #37974). There’s a lot of overlap between that proposed feature and nav menu items, as well as with widgets and blocks in the editor. I think it is key that we define a common interface that can be used in all of these contexts. The nav menu item in core could be re-implemented using a specific kind of such block, and the nav menu should be a block container just like a widget sidebar is a block container and the post content is a block container, though each may have restrictions on the kinds of blocks they allow. They should all re-use the same underlying block API which is a thing that has a set of attributes, a UI for managing them, and a method for rendering them onto a template.

    • dmccan 2:38 pm on January 18, 2017 Permalink | Log in to Reply

      Thank you for laying out the challenges and requirements. A couple of things came to mind:

      • While I think it is a good idea to limit the scope in order to set up a reliable foundation, there is value in identifying the types of ‘blocks’ that we might want to support in the future. Such a backdrop would be good before starting to iterate so that we know that an architecture is capable.
      • From a user perspective, the #1 shortcoming of the editor is the lack of true, easily editable columns. There are true (not shortcode) column solutions available for WordPress.
      • There are two dimensions: layout and content. From a layout perspective, ‘blocks’ are content that fits into rows and columns. In the requirements discussion, perhaps layout needs to be addressed.
      • In the Coda section you mention data attributes and HTML comments for demarcating blocks, what about divs with reserved class names? That is standards compliant. Bootstrap got a lot of mileage out of that approach.
      • Matias Ventura 4:24 pm on January 18, 2017 Permalink | Log in to Reply

        While I think it is a good idea to limit the scope in order to set up a reliable foundation, there is value in identifying the types of ‘blocks’ that we might want to support in the future.

        Definitely! Starting small should not stop us from considering if our solutions are a good fit for more complex blocks.

        I purposely left out layout considerations at this stage (including columns) because they are a bit ortogonal to the basic requirements here. It’s probably worth focusing on how to address layout in a separate instance, starting with the design goals for it. There are questions like «Do we have layout blocks (column, box) that hold child blocks?» or «Do you setup layout placeholder that you fill in or is layout a result of disposition of blocks?» that have their own complexities.

        > what about divs with reserved class names?

        This is partially questioned in ②. We should avoid creating spurious DOM nodes for the sake of representing “blocks”, unless the semantics of the document are improved by doing that. The requirement is about separating the demarcation of where a block begins from the markup of the block.

        The demarcation should be simple and consistent to allow easy parsing, but it should not pollute markup. For instance, the best representation of many blocks are native HTML elements and wrapping them in divs with classes would worsen the document representation (a paragraph, a blockquote, etc).

        Thanks for the thoughtful reply.

    • jhned 3:04 pm on January 18, 2017 Permalink | Log in to Reply

      Here’s how I currently put “blocks” in my page content: I use Advanced Custom Fields Pro to set up flexible content layouts, that are generated and appended to the post content using the `the_content` filter. The problem with that is that I end up with tons of post meta boxes on complicated pages, which slows editing down.

      I think the best way for us to achieve this idea is to go full front-end editor. Reason 1: tinyMCE is notorious for stripping out markup. Reason 2: editing on the front-end gives users an accurate view of how the “blocks” are going to look. Reason 3: the UI for each block could be focused on the data, and then silently insert any markup necessary in the background.

      In addition to your points, @matias, I would also suggest that we ought to create an API for developers to define custom blocks, much like how we have the Shortcode API.

    • davidperez 3:45 pm on January 18, 2017 Permalink | Log in to Reply

      Why don’t go to allow insert widgets inside content? Shortcodes are hard for users. With blocks of widgets are more intuitive and the user can set all fields needed for th block easily….

    • Mark Root-Wiley 4:29 pm on January 18, 2017 Permalink | Log in to Reply

      Just want to say that this is great and makes me feel much better about some of my concerns about the future of WordPress. Getting Portability and Semantic listed as top priorities is awesome! I also think that if the editor is truly going to get overhauled, following the WCAG Authoring Tool Accessibility Guidelines from the start should be a priority. I assume that as with most accessibility work, tacking it on afterwards will almost certainly be a nightmare while including it up front won’t just make the tool accessible but more usable.

    • chatmandesign 4:32 pm on January 18, 2017 Permalink | Log in to Reply

      It seems to me that associating editable blocks with chunks of editor content without mangling the “pure data” is essentially about how to best store metadata for those chunks. I think there are two good options:

      1) HTML data attributes. Storing metadata related to chunks of HTML is literally why they exist, so it seems like a pretty good option. The benefits here are that we’d be leveraging an existing standard, instead of creating something new where it may not be needed; it keeps the metadata with the data it belongs to; and is easily extensible without risking conflicts (providing name spacing is used). The only real downside, as you noted, is that it adds some cruft to the markup—however, a post_content filter could be added to remove editor-related data attributes in display contexts.

      2) Store editable block metadata in postmeta, associated with HTML elements by ID. Storing metadata about posts is the whole purpose of the postmeta table, after all. The benefits here are largely similar: we’d be leveraging an existing WordPress system, rather than trying to create something totally new, and it’s easily extensible with name spacing. The big difference is that the metadata is stored completely outside the content itself; I think whether that’s good or bad overall may be debatable, but one definite positive is that it adds no cruft to the markup (beyond IDs, which is pretty minimal), and no extra filtering step is necessary.

      I think the big question then is: how tightly coupled should the block metadata be to the blocks themselves? The answer depends how we’re going to use it. The implication in this post is that the post_content should be raw, valid HTML that’s ready for output without extra steps to generate content from it (unlike with the existing shortcode system), and I’m inclined to agree with that for static blocks. In that case the metadata is _only_ relevant to the editor, so I don’t see any benefit to keeping the metadata stored in the HTML itself. The HTML only needs to store the final result.

      If we want to support truly dynamic blocks that pull in current data from elsewhere without requiring a manual post re-save, there are more factors to consider. In that case, the data can’t really be stored in the post. These would require some processing to display, with the block metadata potentially pointing to an external data source. It’s not too much of a stretch to imagine that sensitive data like API keys might get stored in the block metadata by some plugins—I don’t think that would be best practice, but I could easily see it happening—which makes me think that in the case of dynamic blocks, storing the metadata in the HTML could increase the risk of unintended data leaks.

    • James Nylen 6:08 pm on January 18, 2017 Permalink | Log in to Reply

      Instead of `` I think the closing tag should be `` for clarity.

      Do we anticipate a need for these custom “tags” to be nested? This is probably something we should decide up front and either plan for, or prevent.

      • James Nylen 6:11 pm on January 18, 2017 Permalink | Log in to Reply

        WP ate my HTML comments… for clarity. Instead of !-- /wp -- it should be !-- /wp:image --.

      • Mel Choyce 5:26 pm on January 19, 2017 Permalink | Log in to Reply

        I think many of them will have to be nest-able, especially once we get into the customization and layout side of things.

    • Greg Ichneumon Brown 12:50 am on January 20, 2017 Permalink | Log in to Reply

      I think one other thing to add to the list is speed on the front end. We’ve seen many cases where just outputting the final rendered content (especially on APIs) means processing the post content with very large regexp logic (especially the shortcode logic) that then can make returning a dozen posts from an api take many seconds. Most of this was all CPU time. We’ve looked at caching and other options on WP.com, but they’ve been very complicated.

      In summary: performance is a feature, particularly of our APIs and it may be good to start testing that early on.

  • Jeffrey Paul 5:30 am on January 4, 2017 Permalink |
    Tags: , , ,   

    Dev Chat Summary: December 21st (4.7.1 week 2) 

    This post summarizes the dev chat meeting from December 21st (Slack archive).

    4.7 Retrospective

    • Reviewing comments on 4.7 Retrospective post on Make/Core
    • We will go through comments and discuss if there are changes to our process that we should recommend
    • Goal is not to second-guess decisions that were made, the goal is to figure out if the process can be improved in future releases
    • Things to start doing:
      • “We failed at getting the field guide and email to plugin dev out early enough. We have aimed to have that out around beta 2 and usually end up getting it out around RC the last few releases. This time it came out the day before (field guide) and day after the release (email).”
        • Coming up with some documentation and ensuring that it’s not just owned by one person is a good way to improve it
        • We should also ensure it is included in the release checklist
      • “The posts explaining new features and changes are helpful, but I think that we need more of them. I follow the trac feed, and sometimes I know that as a plugin developer a particular ticket is important for me to take note of, but it can be difficult to unravel exactly what the final decision was just based on the changesets. An example of something that is going on right now is the a11y team’s work on removing excessive content from headings on admin screens. Often API changes get documented and UI changes don’t, but I’m a perfectionist and I like to stay up to date on the latest design/a11y evolutions as well. I can usually figure out what changes I might want to make in my plugins based on the changes in core, but I’m sure that often most plugin developers don’t even know that there was a change, if they don’t read every ticket.”
        • Request here is to have more Dev Notes and explanations about what is changing
        • It would help to spread the load of writing Dev Notes a bit, sometimes they are time consuming, especially if you’re not much for writing this sort of contextual summary
        • Some components generate a component summary dev note which is a good practice
        • Should we also maybe reach out to the docs team to see if they want/can help?
        • Anyone have ideas for how we get people involved with drafting the note even if they aren’t leading developers on a feature/component?
        • We do list out every change in the “this week in core posts” (shout out to the team that works on those!) already, so there isn’t anything that goes unannounced
        • The solution suggested is more posts, but the problem appears to be that people aren’t seeing changes that they think might affect them
        • Getting to Trac and subscribing to whatever feed is a little hidden. Even Slack notifications is hidden. A more public place would be good.
        • The solution could be to push people to the Week in Core posts, they already list every change categorized by components
        • Someone willing to lead a discussion (likely on make/core) on how to improve this?
        • Action Item: wait and see how lack of timed release cycles plays out
      • “We need to collectively review the “feature plugin merge guidelines” listed here. While development in plugins has become less prominent, most of the bigger projects merging into core in 4.7 (I would exclude the REST API since that’s less user-facing) skipped many of the steps here. A lot of the points don’t apply to most projects – to the point that the checklist is often forgotten entirely. But there remains a need for better quality control and an updated checklist or recommended merge considerations for larger projects should be created accordingly. These written guidelines can better inform merge decisions and assess readiness.”
        • Can we reasonably make full test coverage (covering basic use cases at least) a requirement there?
        • This may be null as feature plugins may not play a significant, or any, role in the future
        • More “wait and see how new process/focus shakes out”
        • Action Item: No more feature plugins
      • “On a related note, clearer procedures about backing out merged features are needed. Particularly if a feature goes through an extensive process to demonstrate readiness and is approved for merge, input on removing the feature during beta/RC should be solicited publicly via make/core posts and scheduled meetings, similarly to the initial merge decision. Otherwise, the decision to remove a feature can seem to ignore the value of the opinions that went into making the decision initially and may not be fully informed about the broader implications with respect to related aspects of a component. If work on a feature seems to stagnate as bugs accumulate during beta and a lead is considering pulling it due to lack of attention, contributors working on the feature should be notified so that they can address the issues or recommend removing the feature based on availability. Perhaps putting more trust in the feature teams and component maintainers that are most intimately familiar with a given project could help ensure that decisions are more broadly considered.”
        • Still a question of who really owns final decision/veto power; @matt as product owner likely
        • Whomever is leading the release has final decision. That’s why they’re a lead. That much should be clear.
        • Action Item: continue to communicate changes clearly and early
        • Release leads and core leads need to be trusted to prioritize based on goals for the release
        • When somebody is unable to solicit feedback, we need to have honest conversations about why this is happening
      • “Add automated acceptance testing for the user flows. If we add these for new features added, we can ensure they work across browsers. And in future releases, these tests can ensure that we don’t break existing flows. Run tests on BrowserStack. See #34693.”
        • Any volunteers to help work on this?
        • Anyone think automated acceptance testing for user flows is a bad idea?
        • It could be difficult to maintain
        • This is done at Automattic: https://github.com/Automattic/wp-e2e-tests
        • Action Item: keep exploring in the ticket
      • “more focus on Trac and tickets, every committer should try to follow Trac on a daily basis. Not to know 100% of the details of each ticket but at least to get a sense of what’s going on. Also, the number of tickets on Trac is increasing more and more, there’s the need of some serious ‘Trac Gardening'”
        • A big ask for every committer following Trac on a daily basis
        • Especially since the vast majority of committers aren’t being donated anywhere near full time (and a large number are 100% volunteer)
        • This is why there’s component maintainers, so that we don’t overburden each person
        • Trac Gardening is something anyone is welcome to do, you don’t need to be a committer
        • Trac Gardening would benefit from some mentorship to be more effective
        • If there could be some mentoring for this – an initial joint meeting to get people started might we get some more interest?
        • We could benefit from improved workflows for managing the resources we do have and to reduce the uncertainty in gardening/contributing in non-code ways
        • Trac Gardening can be a thankless task to a novice who comments on tickets, asks for dev-feedback and then nothing further happens for months. Perhaps the dev-feedback tag needs watching more rather than all tickets.
        • Action Item: generalize 4.7 Bug Scrubs page “to run a bug scrub, announce it here, talk to these people, look at this report in Trac, then ping people on the tickets listed”
    • Things to continue doing:
      • “The combination of a Git startup phase and Slack is excellent. At least for the Twenty Seventeen theme.”
        • GitHub likely helps get new contributors involved, but not sure they stick around
        • GitHub is easier to follow along, post mockups, get feedback, review code
        • GitHub better with searching, labelling, organizing, looking at PRs, realtime updates, making branches and then submitting PRs from branches, plus its app
        • Our current code review process is sub-optimal because there are no workflows to support it (e.g., line-by-line comments on changes)
        • It would be good to separate what is the project management tool vs. version control method
        • GitHub is sub-optimal when iterating on PRs. In Trac, you can make minor changes to a patch and upload it to a ticket. In Github, depending on permissions patch iteration is not straightforward.
      • “Weekly design meetings.”
      • “On the upside, having clear deadlines for when enhancements need to be merged into core is very helpful for prioritizing time and focussing resources. I hope we will continue some form product calendar in the spirit of “deadlines aren’t arbitrary,” even if they take a different rhythm.”
      • “increase the effort to involve different teams in collaborating on features development, where different skills and knowledge are needed, of course.”
    • @jorbin to work on a summary of what was discussed here and post it on Make/Core

    General Announcements

    • Uncertain if anyone is planning on running a core dev chat next week (or any weeks going forward), so watch for agendas on Make/Core or other notifications in #core
  • K.Adam White 11:15 pm on October 13, 2016 Permalink |
    Tags: ,   

    Merge Proposal Discussion: REST API Content Endpoints 

    There are discussion meetings and office hours in #core-restapi at 2016-10-14 14:00UTC and 2016-10-14 19:00UTC on Friday the 14th. Our next team meeting is on 2016-10-17 14:00UTC. Please attend some of all of these, because

    We are meeting at 2016-10-18 01:00 UTC to make a decision on this merge proposal!

    To that end, the below discussion points will be updated regularly, please leave comments on this post or join the conversation in #core-restapi.

    Yesterday at the dev chat the API Team proposed the Content API Endpoints for merge in WordPress 4.7. There was popular support for this feature but as @jorbin and @helen noted that the lack of dissent suggested additional review is needed, so the API Team is continuing to seek thorough review & constructive criticism of the content endpoints, including the open questions previously shared on the week 7 and week 8 API team updates.

    The merge proposal also engendered follow-up discussion in the #core-restapi channel about the benefit content endpoints bring to core, whether having such endpoints built in is quantifiably more beneficial than having them as a plugin, whether moving development from a plugin to core would slow development, and whether the endpoints as-proposed have been sufficiently reviewed for security and performance. We attempt to capture those questions & concerns (and the perspectives on them) below.


    Have the content API endpoints been thoroughly reviewed for security?

    • The REST API plugin has been on HackerOne for over a year with paid bounties for bugs
    • @barry has begun a security review


    How does the API measure up against alternatives? Are there concerns about how the API could impact the servers to which it is deployed?

    • DeliciousBrains did a performance comparison with Admin AJAX and found the REST API to have a performance improvement (These tests have not yet been independently verified)
    • @mikeschroder notes in the comments that using the REST API in place of Admin-Ajax will also bring speed benefits by permitting many previously-uncacheable requests to be cached.

    User Feedback

    Are the content endpoints sufficiently well-tested & vetted by the community?

    • @matt questions whether feedback is coming too late in development for concerns to be acted upon
      • @rmccue notes that the v2 endpoints were created based on user feedback; REST API endpoints are being deployed by plugins and running on VIP, and the content endpoints have been in wide use across a variety of sites, leading to 90+ code contributors and far more developers’ support & feedback on the endpoints
    • @rmccue has also reached out to Phil Sturgeon for feedback and will follow up

    Do Content Endpoints Benefit Core Development?

    Will having these endpoints in core improve future core development, or solve any immediate problems?

    • @bradyvercher suggested that the content API endpoints would remove the need to write a variety of one-off ajax callbacks when developing future WordPress Core AJAX functionality
    • @westonruter notes that the customizer could dynamically create settings for posts and other kinds of content without having to wire up new admin-ajax handlers

    Will Merging Negatively Impact API Development?

    Will having to work through trac instead of GitHub cause development to atrophy?

    • @jjj argues that merging will slow development, because GitHub-hosted plugins are not bound to WordPress release cycles and have less overhead for features to be developed and deployed for testing. @jjj requested a plan for how the REST API will be developed going forward after the merge of these endpoints that would account for the added friction.
    • @krogsgard countered that core increases the visibility of a project like the content endpoints
      • The number of new contributors in this Slack discussion suggests that this merge proposal is bringing in new voices; whether this supports Brian’s point or not, the team is grateful for the breadth of perspectives being shared -Ed.
    • @rachelbaker suggested that the API endpoints are sufficiently inter-dependent with core WordPress code that maintaining the plugin separately amounts to maintaining a fork, and that such separated development is untenable long-term.
    • @matt hopes that a merge of these endpoints would slow release speed, but not development speed; @rmccue feels that development speed will stay the same or increase, and that tying releases to WordPress Core increases the stability and predictability of the API endpoints.
    • The versioning of the API endpoints supports forward compatibility

    Do Content Endpoints Belong on Every WordPress Site?

    What are the pros and cons to having every WordPress site have content API endpoints?

    • @rmccue suggests the API has network effects that can only be realized with a large install base. @krogsgard draws a comparison to RSS, the widespread availability of which enables everything from podcasting from WP to the use of apps like Feedly.
    • @matt suggests that the Atom API is a better analogue than RSS, which is an independent and pre-existing standard, and that network effects could be tested through inclusion in Jetpack
    • @joostdevalk notes that many plugins, like Yoast, have data tied to existing content such as posts and pages; either they expose the content through their own endpoints, or core does. If Core exposes content types through the API then plugins may build on top of that shared foundation, not independently reinvent the wheel. “if this doesn’t end up in core, we’ll start rolling our own API for stuff. Others will too. Interoperability won’t be there, for even the most basic stuff. I think this isn’t like RSS, I think this is much more like all of us using the same table space in MySQL.”
      • @shelob9 and @masonjames agree that merging the endpoints would create a consistent and reliable open “standard” that WordPress developers can use instead of continually reinventing how to read and edit post data over HTTP.
      • In response to the question “what prevents you from building on the endpoints in their plugin form,” @joostdevalk went on to note that plugin dependencies would make that a viable option, but that burden currently lies on the user. Plugin installation is not frictionless.
    • Can these endpoints be bundled? short takeaway: no
      • Woo bundled the API infrastructure before it was merged; doing so for content endpoints would require bundling prohibitively large amounts of endpoint code.
      • @nerrad worries that if plugins bundle different versions of the endpoints plugin, then those plugins may conflict if all bundled copies are not kept in sync.
        • @nerrad clarifies in the comments below that these worries also encompass the additional risk of conflicts when plugin authors each build their own versions of these content endpoints, instead of leveraging a shared standard: if two plugins each expose their own REST collection for posts, a developer working on a site with multiple such endpoints will need to decide which to extend, and will then have their extension tied to that specific plugin rather than to a core API.
    • @schrapel and @jorbin discussed that these content endpoints make it much easier to crawl a site, which also brings some potential performance concerns: no new content is exposed, but the process of aggregating it is easier and more easily automated.
    • In the  comments below @foliovision believes that merging the endpoints will be the best way to assert the level of back-compatibility that mid-size agencies need in order to confidently utilize the endpoints.

    Please leave thoughts, questions, concerns, comments & experience in the comments below. Thank you!

    Edited 2016-10-16 to include the below comments into the body of the post

    • FolioVision 11:27 pm on October 13, 2016 Permalink | Log in to Reply

      Great discussion summary Adam. As the leader of a mid-size development agency, the sooner these endpoints are in core, the sooner we can start to use the REST API. For smaller to medium size clients, trying to play pin the tail on a donkey on a moving train (i.e. different versions of the REST API plugin, different versions of core) is simply not economically or technologically viable.

      As long as the REST API team can commit to retaining some backward compatibility if the API endpoints must change significantly in a couple of major releases, I can’t see any benefit to keeping REST API separate from core. So much great work has gone into creating the additional REST potential, it would be a shame to see it remain bottled up and a niche product only for huge agencies and hosts like VIP.wordpress.com or WordPress.com.

    • Josh Pollock 11:39 pm on October 13, 2016 Permalink | Log in to Reply

      One thing I wanted to add to this is that the REST API adds standards. I and many others have written tons of custom systems for alternative ways to read and edit post data using admin-ajax, or using custom rewrite rules.

      The constant reinventing of the wheal is counter to the spirit of open-source. By agreeing to a standard, with tons of eyes on it, we can all do better with greater efficiency and security. This after all is why we use WordPress and other open-source tools, instead of making our own CMSes.

      • masonjames 12:53 pm on October 14, 2016 Permalink | Log in to Reply

        I want to affirm this point as well. Our agency looks to WordPress (core) to provide an opinion. We value the decisions and respect them (even when we disagree).

        Web development is an incredibly competitive space with pressures in open source and otherwise. Having content end points in core ensures consistency and reliability across the WP community.

    • rag-guay 7:00 am on October 14, 2016 Permalink | Log in to Reply

      I would like to be able to edit widgets from the desktop app. So slow over the net from Thailand!

      • K.Adam White 12:58 pm on October 14, 2016 Permalink | Log in to Reply

        That’s a good point and I agree we need a solution for it. On behalf of the API Team, support for editing widgets is on the roadmap but is not part of what’s being proposed for merge this cycle. It is definitely a common request, and is one of our priorities for 4.8 as we expand the site management capabilities of the API!

        You can see some early experiments from earlier this year towards this usage here: https://github.com/WP-API/wp-api-menus-widgets-endpoints

    • Darren Ethier (nerrad) 10:46 am on October 14, 2016 Permalink | Log in to Reply

      Great summary! A lot of great back and forth in that discussion thread that isn’t always easy to capture 🙂

      Just wanted to clarify that the comments I made were not only in regards to the possibility of conflicts if plugin authors bundled the plugin version of the REST API with their own plugins, but also, with the presence of the infrastructure already in WordPress core, there is the possibility of conflicts should plugin authors roll their own different versions of the content endpoints to support what they are doing.

      Imagine plugin author A has used to api infrastructure in WP core to build custom endpoints for his plugin and his users are asking if he can add some endpoints for their posts, pages and users as well, so he goes ahead and does that. In the meantime, plugin author B has users asking her to build custom endpoints for users that support her needs as well so she adds them. Then along comes User Sally who installs and activates both plugins. She contracts mobile app developer Jimmy to build an app for her that interacts with both plugins and her website. Jimmy starts running into conflicts between what “user” endpoint to use but he manages to get it working. Then Sally tells Jimmy that she wants to distribute this app for anybody using plugin A or plugin B and Jimmy starts thinking about building a custom user endpoint in a plugin that site owners will need to install to use the app to save having to worry in the mobile app about what user endpoint to utilize.

      There are definitely different technical solutions to the above example story, but the reality is, things would be MUCH simpler for all the actors in the story above if there was an accepted standard in WordPress core for the content endpoints. It prevents a lot of problems from occurring.

    • K.Adam White 1:00 pm on October 14, 2016 Permalink | Log in to Reply

      If I may make a request of the developer community: Share this with your colleagues and friends outside of WordPress. See what they like, and what they find rough; let us know what they think! We’ve been gathering outside input on this but for more is always better.

    • Mike Schroder 6:03 pm on October 14, 2016 Permalink | Log in to Reply

      I noted this in the #core chat this week, but from my perspective at a host, we’re excited that landing the REST API will have performance benefits — in particular, because it will mean that we can cache requests that could only go through `admin-ajax` previously (and were thus forced to be dynamic).

      This will help with scaling even if it’s found that performance for the actual time in PHP/MySQL is exactly the same.

      Gradually moving those requests (whether they be core related, or added with plugins) out of `admin-ajax` will also help SysEng/Support more easily spot which requests are problematic.

      • Matt Mullenweg 1:15 am on October 18, 2016 Permalink | Log in to Reply

        It probably won’t have any real-world performance benefit — the admin-ajax benchmark was kind of flawed, and you won’t be able to do HTTP-level caching unless you also support some quite-fancy invalidation.

    • Luke Cavanagh 9:49 pm on October 14, 2016 Permalink | Log in to Reply

      I only see this as positive thing to be in WP core.

    • Brian Richards 2:21 pm on October 17, 2016 Permalink | Log in to Reply

      I’m late to the party but I want to officially register my voice in favor of merging the content endpoints into core.

      As @joostdevalk mentioned in Slack (and others have discussed elsewhere), there are a lot of opportunities to include API-based interactions within a plugin that are stifled presently because of the dependency on the REST API as a plugin. While it does seem like a small thing to inform end-users “This plugin requires the WP REST API plugin” it’s still a burden they needn’t bear. Further, by relegating the REST API to a plugin it stifles the ability for API-based interactions across many sites. For instance, a plugin that creates a widget that shows the most recent comments across a number of sites that you don’t control (e.g. getting the latest comments from a variety make/core inside a personal dashboard … and having the ability to bookmark certain comments or posts into a “read later” list for later consumption).

      As @getsource mentions above there are some fantastic wins by moving these interactions that currently depend on admin-ajax.php into standard, cacheable API calls.

      The most convincing dissenting opinion that gave me reason to pause was @JJJ‘s post that merging would hinder development. I do believe that is true. However, I also believe @rachelbaker is correct in that maintaining parity within the plugin as a long-term stand-alone is implausible. Given the choice between a REST API that can adapt and change quickly at the expense of dying slowly from a thousand cuts and a REST API that changes slowly but maintains perfect core parity I will pick the latter, every time.

      TL;DR: I really want to see the REST API land in core because I want to build some fantastic utilities (for myself and others) that can communicate with many sites without necessarily requiring administrative access (the ability to install the REST API plugin) within those sites.

    • K.Adam White 2:30 pm on October 17, 2016 Permalink | Log in to Reply

      I don’t think we’ve noted yet that the endpoints here don’t just encompass the built-in types, but also the endpoints that extend them — `WP_REST_Controller` provides a strong base for implementing new endpoints, and a custom post type with `show_in_rest => true` will automatically pick up behavior extending that class without any further action from the developer than a single line in `register_post_type`.

      Aside from the benefits of augmenting a shared resource like the posts collection @joostdevalk mentions above, having a sharable code foundation for new custom classes has the potential to reduce a lot of endpoint class duplication and fragmentation across the plugin ecosystem.

  • K.Adam White 6:36 pm on October 12, 2016 Permalink |
    Tags: ,   

    REST API Team Update, 4.7 Week 8 

    Summary: Beta 15 has been released, there are open questions that would benefit from your feedback, and the Content API Endpoints and OAuth Server are being proposed for merge as distinct, separate enhancements to the existing WordPress REST API infrastructure.

    REST API v2 Beta 15 released

    The 15th beta release of the REST API content endpoints plugin was released on October 7. This release builds on top of the recent Beta 14 to…

    • Add support for Post Meta, Term Meta, User Meta and Comment Meta within their parent endpoints
    • Introduce a settings endpoint to allow key site setting values to be retrieved & modified using the API
    • Introduce query parameters to query for posts that are NOT IN one or many terms of specific taxonomies
    • Resolve bugs, including bad comparison logic when updating comments.

    Please try it out and report any outstanding issues; the REST API project gained its 90th code contributor this week and the team is deeply grateful for the energy and support of the broader WordPress community in testing out this merge-candidate plugin!

    New Questions & Discussion Items

    Items which have arisen through final ticket triage & review on which the team seeks feedback:

    1. Should the `filter` shim should be removed prior to merge? It is the majority position of the API team that `filter` be deprecated to dramatically improve the simplicity and consistency of API query functionality
    2. How should comments be handled for password-protected posts? Should the password be passed as a query parameter with the PUT/POST request, or is there a better option?
    3. Should the API match core’s logic when users with the `unfiltered_html` capability are creating or updating Posts or Comments?

    Meeting Notes

    At the weekly team meeting on October 10 the group reviewed open issues in the 2.0 milestone, which represents the candidate for our merge proposal shared last week.

    Meeting attendees agreed to review open issues and pull requests individually, and to reconvene on Tuesday at 1500UTC to ensure all priority tickets had an owner.

    At that meeting on October 11, the team reviewed the incoming feedback around the OAuth plugin (linked above). While the API team feels that having a built-in authentication solution provides a much-needed service, particularly to developers building mobile and desktop applications, the design and usability feedback we have received does indicate that the plugin needs more work.

    OAuth’s place in the Merge Proposal

    The API Team believes that the identified issues are resolvable, that the OAuth plugin is on track and that it should still be considered for merge in 4.7. However, after discussion within the team, input from @matt, and advice from @aaroncampbell and other core committers, we have edited our merge proposal to submit the Content API Endpoints and OAuth server as separate merge candidates. The API Team proposes both components for merge, but we submit the content endpoints for consideration independently of the OAuth1 server.

    Content Endpoints Without OAuth

    The Content API endpoints are stable, well-tested, and in wide production use across a variety of applications. Theme and plugin developers will benefit from having canonical, well-tested API endpoints in core, which may be used to query WordPress both from PHP code and from JavaScript applications running on the front-end or admin of WordPress. Sharing the endpoints for core data types enables increased consistency of what data is exposed and how it is persisted across different plugins, improving consistency and shortening development time by using . These themes and plugins have full read and write access to the API using the existing cookie & nonce authentication.

    Mobile and desktop applications can leverage these same endpoints in a read-only capacity to create a variety of powerful reader-oriented applications and tools that expand the capability of what WordPress can do today, such as a unified reader for Make WordPress blogs and other experiments hypothesized by @jorbin.

    Should OAuth 1 not be accepted for 4.7, secure write access for these external applications would still be only a plugin install away; and while having an OAuth server in core will provide a canonical approach for authenticating from remote applications, depending on the needs of a specific site or specific client application other authentication schemes may actually be preferable. Plugins exist for JWT Authentication and of course OAuth 2, and should OAuth 1 not be accepted for 4.7 these plugins may still be installed to enable an external application to opt-in to secure write access to your WordPress site.

    In Summary

    The API team submits for 4.7 merge consideration two enhancements to the REST API infrastructure: the Content API Endpoints for core WordPress datatypes, and an OAuth server which will reduce the setup time needed to securely interact with those endpoints from outside of WordPress. We believe these enhancements are each individually sufficiently tested and mature to meet the quality and security standards of WordPress Core, and each individually provides wide-reaching benefit to WordPress developers, and through them to the authors, readers & publishers of the web.

  • Matt Mullenweg 8:27 pm on May 12, 2016 Permalink

    New lead for 4.7: Helen Hou-Sandí 

    Due to some unexpected constraints on my time this year, I’m going to be stepping down as the 4.7 lead, and I’m happy to announce that Helen Hou-Sandí is stepping up to lead the release in my stead. I’ll still be behind the scenes providing whatever support is necessary, and I’m really looking forward to the release . You might remember Helen for her famous work on WordPress 4.0.

  • Adam Silverstein 4:58 pm on February 5, 2016 Permalink
    Tags: ,   

    REST API meeting summary, Feb 4 

    The core REST API team, members of the WordPress.com API team, and other interested developers met to discuss the REST API.  Full logs of the meeting are available on Slack.

    Existing endpoints

    The meeting opened with a discussion of the existing endpoints: what is and isn’t ready. @rmccue summarized:

    • The main focus has been on the 4 “core” objects in WordPress: posts, terms, comments, and users.
    • Posts
      • Password-protected posts are going to be ignored from the API until we have a good solution.
      • Meta
        • Post (and other) meta has been pushed out into a separate feature plugin led by @jeremyfelt. Discussion is on the github repo.
        • The main issue is that there’s no differentiation between meta fields. Meta could be added via the Custom Fields meta box, or by a plugin.
        • Enhancing register_meta() in core would allow opt-in to the meta REST API – requiring some core work. See the patch on #35658 for details.
        • There is no current way to explicitly register meta as public.
        • We need to support object meta (post, user, comment, term) and object types (custom post types, etc)
      • Media
        • Mostly complete (it’s a custom post type).
        • Some special handling around uploading media.
      • Autosaves and post previews
        • Tricky, but we think we have a solution for that moving forward.
        • Working on them in a separate feature plugin instead.
        • This will be an enhancement to the API in the future, and doesn’t block compatibility.
    • Terms
      • Work great.
    • Users
      • Works great; undergoing final review.
    • Comments
      • Works; custom comment types are harder until core better supports them.

    Merge proposal and discussion

    An extensive discussion ensued regarding the possibility and appropriateness of merging the endpoints into core. The discussion continued for over an hour; mainly covering whether we should merge the 4 core endpoints, or wait for a complete API before merging…  and what the definition of a complete API is.

    The REST API team’s proposal is that we merge the 4 core objects in the REST API with full read, create, update, delete support, then add more peripheral features when they’re ready. 

    @matt argued that we wait until the API supports everything you can do in wp-admin before merging.

    The discussion revolved around these proposals and what is best for the WordPress project. Would merging the existing well developed endpoints sooner help or hinder developer adoption, and further iteration/improvement? Would delaying the endpoint merge help or hinder progress?

    Here are some key comments from the discussion.

    • @rmccue  The approach to the API needs to change from “the API team creates all of the API” to “the API team controls the general shape of the API, and each team works with it”
    • @danielbachhuber Options / a Site Resource is more easily viable, as are Plugins and Themes. Widgets and Menus essentially need to be reinvented before they’ll work in a REST API
    • @jnylen It’s a question of when, and how much testing can be done first. Shipping an API that is read-only and incomplete in several key areas feels like a big mistake.
    • @kadamwhite we’re shipping an API with write capabilities but only cookie-based auth will be supported OOTB.
    • @matt  I would be pretty skeptical of merging a partial API into core…  no partial endpoints in core. let’s design a complete API, not half-do it.
    • @rmccue The API is specifically structured around progressive enhancement
    • @jorbin I think only supporting the four core object types allows for some amazing themes and amazing content editors to be created, but doesn’t allow for full wp-admin replacements. I’m ok with that.
    • @jnylen From what I’ve seen so far, I’m most concerned about shipping individual endpoints with missing features.
    • @jorbin I think the best way to pick up the pace is to get key things locked up and get the four core object types in core. This would help core develop API-first for those features
    • @codebykat On the WPCOM side, we’re committed to taking the plunge, but we’re not there yet which means things are untested.
    • @jnylen I think this needs more testing at large scale before shipping, including some of the more difficult and obscure features of WP.
    • @drew Shipping with full wp-admin replacement capability is unrealistic, imo. We need something flexible and stable that developers can use as solid jumping off point. All the rest can get separately iterated.
    • @mike I think that, realistically, to ship the API… we need an MVP, and I don’t think that defining our goalpost as “all of wp-admin” fits that criteria.
    • @matt Full wp-admin coverage is a firm goalpost, it gives us a super clear criteria for when it should be merged into core. is everything possible in wp-admin, possible through the API?
    • Matt Mullenweg 7:12 pm on February 5, 2016 Permalink | Log in to Reply

      We got on a bit of a rabbit hole on file editing in WP, and whether that should be in any API.

      I’m inclined toward no, but I think decisions about what we’re not going to support in the API should be made as deliberately as if we were removing the feature from WP itself. We should make that sort of decision explicitly and proactively, not allow things to die on the vine of existing in wp-admin but not the API.

      It’s also going to be difficult because there is often a big gap between what developers do and what people for whom WP is their entire interface do, which file editing accidentally illustrates very well. (What developer would trade their IDE or text editor for a form on a web page?)

      It might be easier to actually build out APIs for everything first and then choose whether to ship them or not than argue about whether they should be built in the first place, which is inevitably a realm of opinion, values, and sparse data.

      • Omaar Osmaan 7:47 pm on February 5, 2016 Permalink | Log in to Reply

        Absolutely- I like the idea of having API in core that allows to replace WP-Admin without loosing any feature it has. While so far I do have the itch of “get it in core faster”, but I really think your strategy is appropriately sound.

        It’s great to see the arguments- and for sure it all would make it only better in the end.

        My 2 cents.

    • Justin Sternberg 8:11 pm on February 5, 2016 Permalink | Log in to Reply

      Agree on iteration/MVP approach. Full wp-admin coverage, to me, is way overkill for initial merge to core. There is value in “paving the cow paths”… Let’s get the MVP working 100% and merged, and figure out what the next most valuable addition will be. Rinse and repeat.

    • K.Adam White 9:05 pm on February 5, 2016 Permalink | Log in to Reply

      The wp-admin parity argument caught me off-guard, which I believe stems from there being two basic types of use-case. Both types of usage are (and should be) valid:

      We (Bocoup, a non-WP-oriented company) have used the WP-API project to take advantage of the existing wp-admin editing interface, and has used the API solely to flow WordPress-authored data into or out of other applications. Without the API we would never have considered using WordPress for these projects: but the existence of a RESTful/non-XMLRPC interface made WP a big win for both us and our clients.

      Matt’s / Automattic’s experience is the complement to ours: Maintain the existing theme/plugin front-end, but put a custom admin interface behind it.

      The original “json-api” plugin was created by MoMA to flow WP data into a Ruby project; We’ve used the API to flow data into Node applications, and Java and C# clients for the WP-API are also under development. Our use-case is not new, and I believe it represents a significantly larger _potential_ target audience than any wp-admin re-skin does (at present). Daniel Bachhuber hypothesized recently that the way we’ll get WP to “100% of the web” isn’t by moving those sites to WordPress, but rather to allow WordPress to be integrated with existing platforms where appropriate.

      Waiting for API parity with wp-admin suits the Calypso use-case, but delaying core integration of API endpoints for basic WP data types will effectively block a significant group of potential adopters coming from external platforms. Right this second may not be the time, but unless we roll the API into core progressively (hopefully no later than 4.6/4.7) I think we are missing a tremendous opportunity.

      “ — Thank you to everyone involved with the project for being involved in this discussion.

      • Jeremy Clarke 5:33 pm on February 15, 2016 Permalink | Log in to Reply

        +1 Thoughtful and vital reply. I agree that getting the core types in is really valuable, and that seeing the situation from all angles (backend-simulation+frontend-simulation) leaves me feeling like waiting for full wp-admin support is very biased towards certain uses and not others.

        FWIW at Global Voices we have a third use-case that I think is really valuable: Communication between WordPress sites which each use wp-admin+themes in the normal way.

        We rebuilt our translation system, which lets separate WP sites translate each others content, to use the old 0.x version of the feature plugin. Our users create a draft and give it the URL of the post they will be translating, then the API negotiates the translation and delivers the post content which is inserted into the new draft. It works great and replaces a kludgy+insecure old trackback-like system we were using. We didn’t want to hitch ourselves to a plugin for such a vital tool, but the feature plugin felt safe and we knew it would be in core eventually.

        Fast forward and many versions later we’re still waiting, and are seeing multiple, incompatible versions of the plugin coming out. For now we’re still using legacy version and waiting for the endpoints to drop into core before refactoring all our work. In our case the 4 proposed endpoints would be perfect for our needs, so getting them into core ASAP would be a huge boon and make us feel a lot more confident. It would also be really nice to formalize the underlying concepts so any new endpoints we create are modeled after how core will be doing it.

        As with K.Adam infinite thanks to all those actively working on this!

    • Mike Nelson 4:44 am on February 6, 2016 Permalink | Log in to Reply

      > I would be pretty skeptical of merging a partial API into core… no partial endpoints in core. let’s design a complete API, not half-do it.

      Did @matt suggest that thinking it would be easier to iterate the API as a plugin than in core? Or what’s the motivation for this suggestion? I think lots of us give his suggestion a lot of weight, but because I wasn’t at the meeting I don’t understand it

      • Matt Mullenweg 9:30 pm on February 8, 2016 Permalink | Log in to Reply

        I’m 100% for continued iteration, and yes I think that would be easier as a plugin than in core.

        Everyone who wants the functionality today can have it in the plugin with just a few clicks, and we can update that with new functionality as much as often as we want, and if something goes wrong in only affects ~10k users. We know that most WP sites have 5 or 6 active plugins, with professionally-developed sites typically 2-3x that.

        In core we can only do major updates 3 times a year, at most, bundled with lots of other things and concerns that go into a major release, and those updates go out to tens of millions of sites. If we break something it’s in the mainstream tech news. What’s included or not depends on the release lead.

    • Philip Arthur Moore 11:03 am on February 6, 2016 Permalink | Log in to Reply

      I haven’t been involved in the development of this at all, but as a plugin/theme/custom web solutions developer I’d have to strongly agree with @matt with regard to setting full /wp-admin/ coverage as a firm goalpost before inclusion. What’s been keeping me from diving into the API and using it over what we have now is that I’m afraid of changes that may break things and not really sure what I can do with /wp-admin/ that I can’t do with the API. If there’s full parity then I can confidently jump into using the API; until then I feel largely like a spectator seeing how the development of the API plays out before overcommitting our resources to learning it and developing for it.

      • Jeremy Clarke 5:37 pm on February 15, 2016 Permalink | Log in to Reply

        This can swing both ways though. You don’t want to use the API if it’s still in flux, but a constantly-developed plugin will be in flux by default.

        For those of us already using it (the oxygen an API needs to grow and evolve) having a stable platform to work with (even if it’s limited) is really important. Getting the main endpoints into core and having them be treated with the usual obsessive back-compat would make our lives easier and increase the number of people willing to invest time and money into using it.

    • Mike Nelson 1:04 am on February 9, 2016 Permalink | Log in to Reply

      I didn’t think [putting the WP API scaffording in core would] increase the plugin usage, I thought it would increase plugins registering their own APIs, because they hadn’t before because of cross-plugin dependency see https://wordpress.slack.com/archives/core-restapi/p1454633785001643

      I don’t know about other plugins, but at least at Event Espresso we put our REST API, which depends on the WP API infrastructure, into our main plugin directly as a result of putting the WP API infrastructure into WP core 4.5. (See https://eventespresso.com/2016/01/rest-api-now-in-ee4-core/)

      We had some reservations though, thinking that maybe we would wait until the WP API endpoints were also in core (because that would be proof the WP API infrastructure would be more solid and less likely to break). It’s possible other plugins are having the same reservation?

      What’s more, the Event Espresso REST API is a little unique in that we’re using a LOT of custom tables, and our functionality is quite independent of WordPress’ posts etc, and so our REST API can exist quite happily without the WP API’s endpoints. I suspect most plugins’ APIs are much more dependent on WP API’s endpoints in order to work. Eg, there’s not much point in a YARPP REST API without WP API. Meaning, if WP API’s core endpoints are still in a plugin, their users STILL need to use the WP API plugin in order to use their plugin’s API… so just adding the WP API infrastructure isn’t much help to them: they need the endpoints too.

    • rclilly 7:18 pm on February 12, 2016 Permalink | Log in to Reply

      After reading this summary, as well as the entire discussion in Slack, I get the sense that, to a certain extent, people are talking past each other.

      @matt, I really don’t understand why you insist on ALL possible endpoints being completely ready before ANY endpoints can be merged into core.The four that are proposed cover a huge part of what many want the REST API for in the first place. I definitely think those four make for a minimum viable product, assuming they are, indeed, ready for prime time. The ability to completely replace the wp-admin goes way beyond an MVP for a REST API.

      My two cents: If there are endpoints ready for prime time, merge them. Let the others remain in the plugin.

  • Helen Hou-Sandi 7:56 pm on December 9, 2015 Permalink

    Welcome the 4.5 class of committers! 

    As announced in the State of the Word this year at WordCamp US by @matt, there are seven new committers to introduce.

    Many of you have seen Michael Arestad‘s (@michaelarestad) design and front-end development contributions over the last couple of years, notably with the redesign of Press This in WordPress 4.2. His numerous, high quality contributions are a welcome addition to core. I personally am looking forward to his work on markup and styling, having relied heavily on his judgment for quite some time now.

    WordPress 4.4 adds a new embed feature to WordPress, making it an oEmbed provider for the first time. Work on this new feature was done in a large part by Pascal Birchler (@swissspidy), who has been doing great work for the past few releases. Pascal’s clear communication and thorough support of the flow mindset are things we can all be inspired by.

    Rachel Baker (@rachelbaker) is the co-lead of the REST API, a Comments component maintainer, and a major contributor to WordPress 4.4. Her work has made it possible for sites around the world to utilize the REST API, making WordPress a great application platform. Look for more of these contributions as the REST API iterates within core.

    Likewise, Joe Hoyle (@joehoyle) is a major contributor to the REST API. As we prepare to commit the REST API endpoints in an upcoming WordPress release, there will be more and more to come from both him and Rachel.

    As a Media component maintainer and a long-time contributor across many components and features, Mike Schroder (@mikeschroder) helped shepherd the responsive images feature plugin into core for WordPress 4.4. He was also a backup release lead for WordPress 3.9.

    Throughout the WordPress admin interface, everywhere you look you’ll see the work of Mel Choyce (@melchoyce). Her design and experience contributions are long-standing and have benefited the entire ecosystem. As one of the maintainers of the Dashicons project, the icons you interact with daily are a big part of her contributions, as well as themes available in the WordPress.org Theme Directory.

    Eric Andrew Lewis (@ericlewis) has been contributing in various forms for many years, exploring lesser-known areas, documenting them, and challenging assumptions. Most recently, you may have seen his work as a Media component maintainer or with the shiny updates feature in WordPress 4.2.

    Additionally, Ella Van Dorpe (@iseulde), Konstantin Obenland (@obenland), Weston Ruter (@westonruter), Tammie Lister (@karmatosed), Andrea Fercia, (@afercia) and Ryan McCue (@rmccue [that’s one M, two C’s]) have all had their guest commit renewed.

    Please join me in welcoming this great set of new committers!

  • Helen Hou-Sandi 7:37 pm on December 9, 2015 Permalink

    Announcing the release leads for 2016 

    As announced during the State of the Word this year, we have a brand new selection of release leads for 2016.

    Mike Schroder
    Previously a co-lead for WordPress 3.9 and long time contributor, Mike Schroder (@mikeschroder) will kick off the year as the release lead for WordPress 4.5.

    Dominik Schilling
    Following WordPress 4.5, Dominik Schilling (@ocean90) will be the release lead for WordPress 4.6. Dominik has been a core committer for a couple of years now and was a backup release lead for WordPress 4.4.

    Matt Mullenweg
    Finally, closing out the year, Matt Mullenweg (@matt) will put on his release lead hat and lead WordPress 4.7. Matt previously led the WordPress 3.8 release.

    Each of these release leads need your help! Every release is made by hundreds of contributors over many months, not just by its release lead. Additionally, every release lead needs a backup lead or two to help ensure the release moves forward at a solid pace. These backup release leads get great training for the real deal, as they often become future release leads (see both Mike and Dominik above!).

    Are you interested in being a backup release lead? Just comment here to let Mike, Dominik, and Matt know.

  • Weston Ruter 2:00 pm on September 23, 2015 Permalink
    Tags: , ,   

    Outlining a possible roadmap for the Customizer 

    Planning for the future is a necessary and important part of the WordPress development process. As we consider the future of WordPress – both as a whole and individual features – we publish proposed roadmaps to encourage greater discussion and give insight into the core team’s thought process.

    The process of creating a roadmap is just as important as the vision behind it and the final roadmap itself. This process gives the entire community an opportunity to research and document history, define what specific items can be accomplished to bring us closer to the vision, and outlines how those tasks fit together within a possible timeframe.

    What follows is a potential roadmap for the Customize component. If you’re interested in the future of live preview in WordPress, now is the perfect time to get involved and leave your feedback.

    A couple of months ago, the WordPress lead developers met with the maintainers of the Customize component to discuss the future of live preview in WordPress. The goal of the chat was to come up with a potential roadmap for both the component and for how live preview can improve the user experience of WordPress for all users.

    The ultimate goal of live preview in WordPress is to create user trust and remove the “save and surprise” inherent in some of the backend features.

    After a lot of discussion, the group decided to target the following goals over the next two years:

    • Considerably improve performance.
    • Continue iterating on current live preview features to ensure they are solid and as easy-to-use as possible, including theme browsing and installation, menus, and widgets.
    • Experiment with new and different user interfaces. If we were creating live preview today, what would it look like? In what ways can we ease the feeling that you’re looking through a “porthole”?
    • Removal of the ambiguous mode. Currently, the Customizer is contained in a sidebar without the admin toolbar, but ideally there is the admin and the theme, and no in-between. One direction this may go is enabling “Customize” on the front end to immediately load the Customizer controls.
    • Experiment with a guided new user experience (NUX). Live preview lends itself to site setup. How can we improve the live preview experience and combine it with the NUX? Consider a “setup wizard” use case and ensure the flow has no dead ends, i.e. users can customize everything in one.

    Those overall goals for live preview in WordPress can be rewritten into some specific features that are in development or planned for the future of the Customize component. These include:

    • Transactions. This re-architecture of some of the Customizer internals improves compatibility with themes by loading the preview using a natural URL, and allows Ajax requests or even REST API requests to be previewed. It also allows the preview to be viewed independently of the Customizer, so changes can be shared for others to review. See #30937.
    • Selective refresh. Only a piece of the page will need to be refreshed when this backend feature is implemented. (Formerly known as “Partial Refresh”.) Currently, this is available for menus in the Customizer. This eliminates duplication of display between PHP and JS, keeping it DRY. See #27355.
    • Concurrency. Allows for “locking” settings using the Heartbeat API, improving the overall user experience by preventing users from overwriting each other’s changes. See #31436.
    • Revisions. Enables plugin developers to add features like draft, roll back, and scheduled changes (e.g. “change my background on January 1”). This builds upon transactions, as the setting changes are staged in a transaction, and this facilitates settings to be revisioned and for settings to be scheduled. See #28721, #31089.
    • Theme installation. Iterates on and completes the theme browsing experience.
    • Responsive preview. Iterates on the concept of live preview by giving users a better idea of what their site will look like on other devices. See #31195.
    • Bootstrapped Customizer. Lazy-load the Customizer into the current frontend view without having to leave the page. With selective refresh implemented, inline controls and frontend bootstrapping would be possible since full-page refreshes would no longer be required.
    • Improvements for both touch and small devices.

    Beyond those features, the group identified some specific changes that should be prioritized, in conjunction with the features planned:

    • The sliding animation between panels should feel more like “moving panels” (see: iOS).
    • Keyboard navigation should be consistent and clear.
    • Identify “dead ends” in the interface and remove them, when possible. For example, prior to menus in the Customizer, it was not possible to customize that aspect of your site’s design with the Customizer.

    The concepts surrounding live preview and the Customizer have been in development for a long time. Many of the concepts from Elastic Theme and the Visual CSS Editor have been incorporated over time. Over the next few years, experimentation with these concepts will likely take place in feature plugins. For example, this team may experiment with inline content editing, where it makes sense in the context of customizing a site. Another path for exploration is simple theme customization – e.g. change the header font, change the sidebar color, or change the width of the sidebar.

    As with all components and new features, we shouldn’t be afraid to experiment and fail and should continually push for new experiments and ideas, especially in the context of feature plugins. Further, some of the above experiments may not make it into core, but are meant as a general direction that live preview should take in WordPress.

    Taking these features together, below is a sequence outlining a possible roadmap for live preview and the Customize component in general, along with estimated targets. Please note that this is a proposed roadmap and is entirely dependent on contributor involvement. Additionally, many of these things will take place in a feature plugin prior to core inclusion.

    • Partial refresh. Performance Improvements. (Target: 4.4)
    • Responsive Preview. Transactions. (Target: 4.5)
    • Concurrency. Revisions. Theme Install. Beginning of NUX wizard. (Target: 4.6)
    • Focus on touch screen / small device improvements. (Target: 4.7)
    • Developer API improvements based on feedback from plugin developers. (Target: 4.8)
    • Improved UI after experiments in 2016. NUX “wizard mode.” (Target: 4.9)

    Live preview is one of the most critical features in WordPress as we continually combat “save and surprise.” The Customizer in its current form provides an improved user experience to WordPress users when customizing their site’s design. Each feature mentioned above is a continuation of the live preview concept, building and improving upon the Customizer.

    Everything above is just a proposal and we need your feedback to ensure it is the right direction. If you’re interested in any of the above, comment here with your feedback, or join the team in #core-customize.

    This post was a collaboration between @helen, @nacin, @mark, @celloexpressions, @samuelsidler, and yours truly.

    • Davide 'Folletto' Casali 4:46 pm on September 23, 2015 Permalink | Log in to Reply

      > NUX wizard

      I find a bit dangerous to think of it as a “wizard”, mostly because usually wizard UIs have specific connotations. We should think of it more as a “tutorial” like games do. Not separate UIs, but a guide through an existing UI that can be dismissed if you’re expert, or done partially, or followed all along. 🙂

      Other than that… this seems AWESOME. 🙂

      Performance is really the main thing there. That alone would reap huge benefits. Then you add Live Preview and well, boom! 🙂

    • Daniel Bachhuber 6:11 pm on September 23, 2015 Permalink | Log in to Reply

      We (Fusion) are very keen to adopt the Customizer as the UX for frontend management, and look forward to helping out with the feature plugins built as a part of the roadmap.

    • FolioVision 12:43 am on September 24, 2015 Permalink | Log in to Reply

      Weston, thanks for the comprehensive and forward thinking report.

      It would be great if the interface paradigm for Customizer would remain constant for the exterior world (i.e. agencies and clients) while a dev version gets tested by the more progressive among us.

      Perhaps a switch to throw in settings to enable experimental Customizer features? The switch could be retired when Customizer settles a bit.

      • Weston Ruter 4:05 am on September 24, 2015 Permalink | Log in to Reply

        @FolioVision yes, absolutely. This is why most of the new features will be developed in feature plugins so that they will remain opt-in until they are deemed solid for everyone.

    • devolute 4:05 pm on September 25, 2015 Permalink | Log in to Reply

      but ideally there is the admin and the theme, and no in-between.

      Here here! Does this mean that there will be a “front-end” mode, and a “dashboard” mode that fits into the current dashboard so that things can be altered without a preview being available?

      • Weston Ruter 4:16 pm on September 25, 2015 Permalink | Log in to Reply

        @devolute Basically the idea is that there would no longer be a separate “Customizer” URL (wp-admin/customize.php) that you go into. When on any frontend URL, clicking “Customize” would just slide out the Customizer sidebar pane in place. You’d still have the admin bar available and everything else that you would normally when on the frontend, except you’d also be able to make changes to the Customizer as you browse around. So no, this doesn’t have in mind making the Customizer controls available in the wp-admin without a corresponding preview.

        • devolute 4:19 pm on September 25, 2015 Permalink | Log in to Reply

          Ah, I understand your intention. I think my problem is that I’m using the customizer for things like phone numbers, addresses and other options when I perhaps should look at other solutions.

    • Ahmad Awais 3:38 pm on September 26, 2015 Permalink | Log in to Reply

      We at WPTie are slowly yet gradually adopting Customizer in all our themes and plugins (where needed). All in for customizer support in WordPress.

    • Alex Mangini 10:23 pm on September 28, 2015 Permalink | Log in to Reply

      On the “Bootstrapped Customizer” point, is this a move towards inline content editing and possibly even “page builder” type concepts?

      My biggest curiosity about the Customizer and what I really want to use it for as a developer is creating posts and pages without a 3rd party visual editor and without the drawback, as already stated, of guessing what they’ll look like after inputting content into specially built meta boxes or widgets.

      Thanks for doing these write-ups, as I build more and more WordPress plugins/products I find myself lurking around here to better educate myself and plan roadmaps for my own work.

      • Weston Ruter 10:45 pm on September 28, 2015 Permalink | Log in to Reply


        On the “Bootstrapped Customizer” point, is this a move towards inline content editing and possibly even “page builder” type concepts?

        This is already possible, actually. For an example of inline editing, see the Customize Inline Editing plugin. And page builders are also possible now by using widgets in the Customizer. I’m currently working on a site that uses a concept of query-specific widget areas to allow every URL to have custom sidebars configured, where the main content area is just another widget area: this allows drag-and-drop building of pages just with standard widgets and any custom widgets that you define. Widgets still have a lot of room for improvement in Core, but I’m very keen to see them move in the direction of “content blocks” which can be used for building page templates in the Customizer.

    • Nick Halsey 4:19 am on September 29, 2015 Permalink | Log in to Reply

      Thanks for getting this published Weston.

      A couple of additional comments on my end:

      • We need more help. The Customizer can do (and could do) a lot, but without more contributors (not just code), it’ll take time to see big improvements go through. I’d love to see more of an effort to embrace the Customizer throughout various other core projects, and to see more community involvement in Customizer design and development.
      • In the interest of experimentation and building on the benefits of live preview, I’m planning on looking at some ways to integrate things like page templates, post formats, and featured images (with cropping) into the Customizer. While there’s a good chance none of those things will make it into core, I’d like to encourage anyone with any ideas for the Customizer to try them out and offer them for discussion. We’re no where near the limit of what users could do with the Customizer and it’s an amazing framework to build additional features on top of.
      • While many of the topics in the post are fairly developer/backend oriented, it’s important to emphasize that there is still a significant amount of design and UI/UX work to be done, including looking at entirely different approached to the live preview UI concept. Don’t be shy to share your ideas if you have any – the technical side of things is evolving to support pretty much anything we may want to try.
    • pingram3541 3:59 am on November 24, 2015 Permalink | Log in to Reply

      A little late to the game but I just wanted to say thanks for pushing the Customizer forward. I do feel it is given the least amount of attention when it comes feature progression and I can’t figure out why. Yay for menu’s but how did menu’s make it on the plate before posts??? Seriously folks, this is a huge mechanism in which we are losing ground fast to many of the new front end content builders on the web and the Customizer is behind the times.

      @westonruter your example of the Customize Inline Editing is nice but seems only to serve a very narrow focus on ability as it still provides no capability of inline editing for actual post content/post meta. Don’t get me wrong, great working example but real world we need to break outside the box of wp_options, theme_mods…we need simpler access to post objects and other areas in WordPress.

      Again @westonruter looking at your code on the Customize Posts plugin I see that your way of getting around the lack of post object availability is by providing a drop down of all posts/pages of which to perform the query against which is genius. However, it doesn’t seem natural to have to select a page/post when you’re already viewing one in the Customizer already and when needed can navigate to other post/page where their post fields could be provided rather than choose it from a long drop down list, which could become unmanageable on some larger sites.

      So yes, I’m excited for the future and the BIG ONE thing I’d love to see is:

      Access to the current preview’s post object or at least a way to capture the post id for use with $customize_update and $customize_preview. Could be as simple as adding the post id to the $wp_customize object.

      • Weston Ruter 5:54 pm on November 25, 2015 Permalink | Log in to Reply

        @pingram3541 Thanks for the reply.

        I do feel it is given the least amount of attention when it comes feature progression and I can’t figure out why. Yay for menu’s but how did menu’s make it on the plate before posts???

        There are a limited number of people to contribute to each release. @carlhancock wishes that all admin management features could be added to the Customizer at one time instead of one piece at a time, but this is not practical for an open source volunteer effort with three releases a year. As we just saw with Automattic’s Calypso announcement from @matt, it took almost two years and dozens of contributors to re-develop the entire WordPress.com admin into a new JS-driven interface. We don’t have the time or the person power to do that. However, I think that interest is building, and with that, contributors will come.

        Compared to posts, it is relatively much simpler to build support for nav menus in the Customizer. They have a UI that fits in the Customizer pane, and the data model is pretty normalized. Posts, on the other hand, would need a full-on editor experience (inline to the preview) and can have an infinite number of variations to display (shortcodes, media embeds, etc). This is a hard problem to generalize for all themes, but it is something that @iseulde has worked hard on in the Front-end editor plugin. The Customize Posts plugin was developed as a prototype to show the architecture for how posts and postmeta can be managed in the Customizer so that the Front-end editor (or another such plugin) could leverage the Customizer for previewing and staging changes.

        your example of the Customize Inline Editing is nice but seems only to serve a very narrow focus on ability as it still provides no capability of inline editing for actual post content/post meta. Don’t get me wrong, great working example but real world we need to break outside the box of wp_options, theme_mods…we need simpler access to post objects and other areas in WordPress.

        As noted in the plugin’s description, Customize Inline Editing is an example (proof of concept), again to show how inline editing can be implemented in the Customizer. So its focus is intentionally narrow. The approach taken there can be implemented for managing post data inline. See also wp-customize-posts#8.

        on the Customize Posts plugin I see that your way of getting around the lack of post object availability is by providing a drop down of all posts/pages of which to perform the query against which is genius. However, it doesn’t seem natural to have to select a page/post when you’re already viewing one in the Customizer already and when needed can navigate to other post/page where their post fields could be provided rather than choose it from a long drop down list, which could become unmanageable on some larger sites.

        Actually, it’s not a dropdown of all posts/pages. It’s a list of the posts that currently appear in the preview. The dropdown’s options change as you navigate around the site in the preview, meaning that you should only be editing a post in the Customizer that is actually visible in the preview.

        So yes, I’m excited for the future and the BIG ONE thing I’d love to see is: Access to the current preview’s post object or at least a way to capture the post id for use with $customize_update and $customize_preview. Could be as simple as adding the post id to the $wp_customize object.

        The future is now. As noted above, Customize Posts does it to communicate the list of posts in the preview to the pane for the dropdown. Also @ericlewis has implemented this in wp-custom-css-per-post. I also have on my todo list to write a post on how to create a “metabox” in the Customizer for managing postmeta for the current singular post in the preview.

    • Philip Ingram 6:57 pm on November 25, 2015 Permalink | Log in to Reply

      Thanks for the quick and detailed response. I’d love to contribute as long as my foo is strong enough, let me know how I can help or if there is a more appropriate forum for continuing this discussion =)

      Additionally, regarding @ericlewis ‘s wp-custom-css-per-post, I already have a working live css editor using codemirror to provide live updating of the preview as well as updating the hidden Customizer input setting transport to postMessage and this works great allowing live preview before saving but also allowing the saving. This has greatly improved efficiency of my front end design workflow. However this currently is limited to my site-wide custom css via theme options. What brought me here in the first place was seeking out a way to GET and POST the current posts meta which I have a custom css per post option built into my themes. So maybe I can get with @ericlewis and we can figure this out. Looking at WP Customize Posts I see the query for $_POST[‘customized’] to iterate though and retrieve post id’s but when using the Kirki toolset (I’m sure the class needs something extra for this) it’s always an empty array upon initial load of the Cusotmizer and that is where I’m stuck currently. I just need the darned post id!

      You mention that posts are way more complex than menus and yes there’s no debate there, however in the time being, if there was a simple way to capture the current previewed post’s id or maybe a clearer example how to retrieve it, that opens the door for us to pioneer further without waiting for a more core-driven method of updating post data via the Customizer.

    • Philip Ingram 10:20 pm on November 25, 2015 Permalink | Log in to Reply

      @westonruter Yes! Thank you for this.

  • Robert Chapin 10:10 pm on September 2, 2015 Permalink
    Tags: , ,   

    Shortcode Roadmap Draft One (Proposal – Request for Comments) 

    This is the first draft of the Shortcode API Roadmap. It describes in broad terms a new feature set and migration that might take place across versions 4.4 through 4.7. This roadmap gives notice to plugin developers that significant changes in plugin design may be needed for compatibility with future versions of the Shortcode API. This roadmap also identifies steps taken to minimize the impact on plugin developers to allow most plugins to continue working with only small changes.

    The decision to create this roadmap arose from specific needs that are not met by the old code. Our old [ and ] delimiters were easily confused with the way those characters are commonly used in English quotations and sometimes even in URLs. The proposal to use [{{ and }}] instead should allow a better balance between being able to type in the shortcodes and avoiding confusion with any other input. With these more unique delimeters, we will be able to process registered shortcodes more efficiently because we will not have to search for them by name. Unregistered shortcodes will have more consistency because we can find them more accurately.

    Old style delimeters also gave no strong indication whether or not a closing tag was required. The proposal to use }$] as the delimeter of a shortcode with a following closing tag increases the efficiency of regular expressions, because the search for a closing tag will only happen as needed.

    Adding the new style of shortcode syntax provides an opportunity to make significant API changes. One of those major changes is to ensure that shortcodes are processed before paragraphs and before curly quotes. This will lead to greatly simplified code in related functions that currently must find and avoid shortcodes every time they run. We also have an opporunity to re-think the way shortcodes are filtered, and to give plugin authors more control over those filters when registering their shortcodes.

    4.4 – Introduce New Syntax

    A new syntax and documentation of how it works will be created in the 4.4 development cycle. Support for the new syntax will be introduced, allowing plugins to register for extra features. Core shortcodes will use the new syntax in all new posts. The old syntax will not change. Old posts will not be affected. Initial work on the syntax concept follows.

    Proposed New Syntax

    Self-Closing:  [{{shortcode}}]
    Attributes:  [{{shortcode  attr1="value1"  attr2='value2'  "value3"  'value4'  value5}}]
    Enclosing:  [{{shortcode}$] HTML [${shortcode}}]
    Multiple Enclosures:  [{{shortcode}$] HTML [${encl2}$] HTML [${encl3}$] HTML [${shortcode}}]
    Escaped Code:  [!{{shortcode}}]
    Stripped Unregistered Code:  [{{randomthing}}]
    Stripped Unregistered Enclosure:  [{{randomthing}$]  Content also stripped.  [${randomthing}}]
    Stripped Empty Code:  [{{ }}]
    Ignored Orphan:  [{{shortcode}$]
    Ignored Orphan:  [${shortcode}}]
    Ignored Orphan:  [${encl2}$]
    Ignored Context:  [{{shortcode <br> }}]
    Ignored Context:  <a href="[{{shortcode}}]">

    4.5 – Deprecate Old Syntax

    Starting in 4.5, plugins that register shortcodes without declaring support for new features will raise debugging errors to alert developers that support for the old shortcode syntax is ending.

    Old posts will continue to work normally while the old syntax is deprecated.

    4.6 – Upgrade Old Posts

    Old posts will be automatically converted to the new shortcode syntax. The Shortcode API will continue to provide deprecated support for old syntax so that there is no disruption during the conversion process.

    The API should be adequately abstracted so that old plugins are not affected by this conversion. However, as the new syntax will not support HTML inside of shortcode attributes, there is no guarantee that every shortcode will work the same way in 4.6 as in earlier versions.

    4.7 – End of Support for Old Syntax

    Old shortcodes will stop working in 4.7. Plugins that still produce the old shortcode syntax will be ignored by the Shortcode API.

    The upgrade to 4.7 will include a second pass of the conversion of old posts so that any old syntax that was added to posts during 4.6 will still get converted.

    In 4.6 or 4.7, if necessary, a filter could be added to automatically convert any old syntax still being produced by old plugins when new posts are created.

    • Daniel Bachhuber 10:15 pm on September 2, 2015 Permalink | Log in to Reply

      Old posts will be automatically converted to the new shortcode syntax. The Shortcode API will continue to provide deprecated support for old syntax so that there is no disruption during the conversion process.

      What about shortcodes stored in other fields?

      • Muhammad 10:18 pm on September 2, 2015 Permalink | Log in to Reply

        Hopefully, we’ll have helper functions to convert those into new format.

      • Robert Chapin 10:38 pm on September 2, 2015 Permalink | Log in to Reply

        Unless there are alternative ideas, I think plugins that do things in non-core ways will be expected to follow the roadmap with their own upgrades. If there are certain fields that are very commonly used in plugins, we should consider doing a core conversion of those, but with much caution.

      • Robert Chapin 11:10 pm on September 2, 2015 Permalink | Log in to Reply

        One alternative would be to leave the old syntax parsers available for plugin filters, while removing them from default filters.

    • AmirHelzer 10:16 pm on September 2, 2015 Permalink | Log in to Reply

      Thanks for the advanced notice on this huge upcoming change. Aren’t you worried that this process will basically break 99% of all existing WordPress sites? Almost all sites use shortcodes in one way or another (like image captions). doing automated updates to CONTENT is certainly possible, but also prone to corner cases and errors.

      It’s great to have a new API. Is it critically important to deprecate the old one?

      Also, for authors, it’s a lot nicer to have a simple [ ] syntax rather than a more complex sequence of characters.

      • Robert Chapin 10:44 pm on September 2, 2015 Permalink | Log in to Reply

        I think the conversion process will be seamless for most sites. This will mainly depend on corner cases such as plugins embedding shortcodes inside of HTML attributes.

    • Nick Haskins 10:18 pm on September 2, 2015 Permalink | Log in to Reply

      The advanced notice is great, but IMO this is absolutely ridiculous. Are there any benchmarks on how much more efficient it is having users type in more characters for a shortcode? Any benchmarks with how much more efficient core will be from having this change made? This is a HUGE change that will affect a VERY large subset of users.

      • chriscct7 10:40 pm on September 2, 2015 Permalink | Log in to Reply

        This is partially about efficiency. This has a lot to do with security, and basically making 4.2.3 impossible as possible to re-occur

        • Nick Haskins 10:45 pm on September 2, 2015 Permalink | Log in to Reply

          I’d love to see benchmarks on the regex processing of the current implementation against the proposed implementation. Re: security; ATM I’m failing to see how [{{this}}] is more secure than [this]?

        • kevinwhoffman 1:13 am on September 4, 2015 Permalink | Log in to Reply

          If the proposal has positive effects on security, it should mention them. A brief summary or a link to the issues with 4.2.3 would help inform that discussion. As of now the only focus seems to be on theoretical improvements to efficiency while complicating the process of using shortcodes.

      • FolioVision 11:31 pm on September 2, 2015 Permalink | Log in to Reply

        This is an absolutely hideous proposal with dreadful syntax. Is the goal of this WordPress project of ours not to bring software to the people? Syntax like this will be the beginning of a new and much better simple CMS which doesn’t do things in the ugliest, most complicated way possible.

        Code better database upgrades, test them properly before releasing and please stop trying to ruin the software for the end users (those who actually use the software and pay all the bills for us to build it).

        • Travis Northcutt 3:54 pm on September 3, 2015 Permalink | Log in to Reply

          “Syntax like this will be the beginning of a new and much better simple CMS which doesn’t do things in the ugliest, most complicated way possible.”

          The sky is falling!

    • Scott Taylor 10:18 pm on September 2, 2015 Permalink | Log in to Reply

      Let’s all note the “Draft One” portion of the title, and let’s all actively give feedback and participate in any decisions on the table here.

      • Nick Haskins 10:21 pm on September 2, 2015 Permalink | Log in to Reply

        My main concern is that this isn’t easier for authors, and that this will affect a very large portion of users and active sites. If I’m understanding correctly, is this being proposed because it’s more efficient for core?

        • Scott Taylor 10:23 pm on September 2, 2015 Permalink | Log in to Reply

          I have the same concerns, and hate the syntax. I also know that the current situation with shortcodes and content-parsing is potentially unstable and unsustainable.

        • Justin Sternberg 10:33 pm on September 2, 2015 Permalink | Log in to Reply

          I’m in agreement here with the concerns. This will break a lot of things because the short code API has been around so long. There’s just no way to account for all the hacks, workarounds, etc, and doing a migration effort on a WP update sounds like a mighty scary prospect.

          • leehodson 10:37 pm on September 2, 2015 Permalink | Log in to Reply

            Should be able to use the existing shortcode parser to recognize old format shortcodes then make the necessary changes to update them. It recognizes shortcodes efficiently enough at the moment.

            Can imagine breakage but as long as the updater shows the changes site admins ought to be able to determine where hands-on intervention is required.

          • FolioVision 11:33 pm on September 2, 2015 Permalink | Log in to Reply

            It seems a make work project. Are there not some important areas where we could make some real progress like integrated multilingual and better native galleries/media management (NOT reliant on external Automattic services)?

            The upgrade process here and cross-version incompatibilities are more than scary.

          • galbaras 11:44 pm on September 2, 2015 Permalink | Log in to Reply

            Backward compatibility should take into account site-specific customizations, i.e. declaring shortcodes in functions.php of a (child) theme. Most site owners use a “build and forget” approach, or at least neglect their sites for long periods. If a new version contains no support for old-style shortcode declarations, sites will break all over the web.

        • Solinx 11:09 pm on September 2, 2015 Permalink | Log in to Reply

          Yes. How am I going to explain anyone that this syntax is an improvement?

          What specifically are the reasons for not using {{shortcode}} and {{shortcode}} {{/shortcode}}? Or [[shortcode]] and [[shortcode]] [[/shortcode]]?

          Other than that the improvements to the API sound good.

          • Robert Chapin 11:18 pm on September 2, 2015 Permalink | Log in to Reply

            Lone curly braces are easily confused with PHP and other C-style code snippets. The latter square brace example is incompatible with the old syntax and would cause migration problems.

    • Andrew Ozz 10:20 pm on September 2, 2015 Permalink | Log in to Reply

      Was just about to say that this is a proposal. Scott beat me to it (as usual) 🙂

    • Pippin Williamson 10:22 pm on September 2, 2015 Permalink | Log in to Reply

      I would assume the answer is “yes”, but have there been benchmarks done that compare the differences in various syntaxes?

      Why [{{shortcode}}] and not {{shortcode}}?

      Most of the proposed syntax seems sane (though still curious on question above), but this one concerns me:

      [{{shortcode}$] HTML [${shortcode}}]

      For most users, that will be exceptionally unintuitive and we will almost certainly see a high error rate with users not being sure where and when a $ is used.

      • Nick Haskins 10:26 pm on September 2, 2015 Permalink | Log in to Reply

        Pippin I have to agree. If we absolutely have to move, how about something as simple as [{this}] HTML [${this}] . it’s easier to digest, and easier to remember that a $ ends a closing statement.

      • webaware 11:19 pm on September 2, 2015 Permalink | Log in to Reply

        The whole proposal sounds pretty horrible to me, but this suggestion by Pippin sounds moderately less ridiculous.

        Re: {{…}}, I would assume that’s because they are already used heavily in non-WP templating, especially JS templating.

        Essentially, this “fix” is coming rather too late in the piece and I think it’ll break more things than it solves. As a plugin writer heavily dependent on shortcodes, I can only see a greater number of support requests from broken shortcodes in my future if this goes ahead.

        Only Shortcake can save us now! 🙂

        • idealien 2:33 am on September 3, 2015 Permalink | Log in to Reply

          “Only Shortcake can save us now!”

          Seriously! If Shortcake were to be brought into core along with this (draft) proposed change – it would go a LONG way to mitigating the concerns many have raised around complexity.

          Without Shortcake:
          “Look – we took this feature you love and made it harder to use….But it’s safer / faster / better”

          With Shortcake:
          “Look – we made is hella-easy to use shortcodes without having to remember syntax type things. Oh, and that syntax is also getting harder b/c it needs to be safer / faster / better.”

          Car metaphor time… With Shortcake: You’re getting automatic transmission and manual is just a little bit harder to use. Without Shortcake: Manual transmission is now harder to use.

          Shortcake FTW!

      • Robert Chapin 11:33 pm on September 2, 2015 Permalink | Log in to Reply

        If we could get this to work…

        [{{shortcode}} HTML {{shortcode}}]

        Is that better? Or still confusing?

      • Mario Peshev 4:16 pm on September 3, 2015 Permalink | Log in to Reply

        I do agree with Pippin’s example with the dollar signs. Seems incredibly easy to mess up by users. I also noticed another syntax including an exclamation sign, which adds an additional layer of complexity for each and every user.

        +1 on the benchmark question. Given the several different supported syntax proposals, I’m curious to see the regex that is parsing those in a sensible and timely manner.

    • AmirHelzer 10:23 pm on September 2, 2015 Permalink | Log in to Reply

      Also, we deal a lot with shortcodes in Toolset plugins. I can’t see any evidence of any user confusing the [ ] characters for anything else than shortcodes.

      Please, besides the code changes, please also consider the huge amount of documentation update that such a change will require.

      Run a quick Google search for “WordPress shortcodes”. A few pages come from wordpress.org, but the majority are old pages, blog posts and tutorials. Nobody is ever going to update them.

      So, after making such a change, people will see two sets of documentation – with new syntax and with old syntax. You know that Google favours ‘old’ content, so the old tutorials that are never going to update will still explain to use [ ]. I don’t think that this will work out well.

      • chriscct7 10:34 pm on September 2, 2015 Permalink | Log in to Reply

        The team working on this is committed to the documentation updates that are necessary.

        • Pippin Williamson 10:35 pm on September 2, 2015 Permalink | Log in to Reply

          Core documentation isn’t an issue. It’s every plugin / theme developer in the world that also has to update their documentation.

          I don’t disagree with updating docs, just making it clear that’s the issue @AmirHelzer is raising.

          Also have to keep in mind the thousands (millions?) of blog posts and tutorials out there.

          • AmirHelzer 10:49 pm on September 2, 2015 Permalink | Log in to Reply

            Exactly. I’m sure that the core team would be quick to update documentation on wordpress.org. I’m talking about the many pages on other people’s sites that will likely never be updated.

    • Scott Fennell 10:25 pm on September 2, 2015 Permalink | Log in to Reply

      Thanks for your work on this!

      I’m trying to understand what I would need to do in order to prevent about 1,200 live blogs from breaking when this syntax is no longer supported:

      [my_shortcode before='<div class="my_class">' after='</div>']

      Andrew offered a helpful suggestion here:


      If I understand his suggestion correctly, I’d be on the hook to do a search and replace in the database, which will be quite scary and expensive for my company. We have literally thousands of posts on thousands of blogs.

      Please, let’s find a way to move forward with this that leaves the old format unscathed.

      • Scott Fennell 10:28 pm on September 2, 2015 Permalink | Log in to Reply

        Sorry, meant to provide the following code.


        Is there a reference somewhere for the comment syntax so I can stop posting gibberish?

      • Robert Chapin 11:49 pm on September 2, 2015 Permalink | Log in to Reply

        If the situation doesn’t support

        [{{my_shortcode class="my_class"}}]

        Then the alternative proposed in the roadmap is

        [{{my_shortcode}$] Optional Content [${before}$]<div class="my_class">[${after}$]</div>[${my_shortcode}}]

        We will also likely encode any HTML in attributes during a migration, so you would have the option of decoding that in your plugin.. security implications could be significant though.

        • Scott Fennell 4:10 pm on September 3, 2015 Permalink | Log in to Reply

          “We will also likely encode any HTML in attributes during a migration”

          Can you explain what you mean by migration, exactly? My imagination is running wild a bit with what you might mean by that. Do you mean to propose that, upon this update, the new version of WordPress will automatically fix my post_content across many gigabytes of posts, without me having to do anything to the posts?

          • Aaron D. Campbell 4:29 pm on September 3, 2015 Permalink | Log in to Reply

            That’s definitely the goal. Basically using the existing parser to pull shortcodes and replace them with the new syntax.

            • Scott Fennell 7:43 pm on September 3, 2015 Permalink

              Sorry for sounding clueless here but that would be … amazing. Is there a precedent for this level of work in a similar update? I’m having trouble imagining how this would be possible given a very large database like mine. Would it run as a cron job perhaps?

    • leehodson 10:30 pm on September 2, 2015 Permalink | Log in to Reply

      The new syntax is very confusing to the eye. Users would find it complicated. I feel we could make it less confusing by using back-to-back opening and closing brackets.

      For example

      Self Closing:



      []shortcode[] Content [/]shortcode[]

      Note the forward slash between the brackets of the closing tag on the enclosing shortcode.

      Is there a reason this wouldn’t work?

    • Peter Wilson 10:51 pm on September 2, 2015 Permalink | Log in to Reply

      I’d be happier with {{selfclosing}} and {{enclosing}} {{/enclosing}} or similar. Mixing the formats seems fraught.

      Upgrading old content is a necessary evil, backward compatibility on the deprecated format defeats the purpose somewhat. Seeing some benchmarking results would be great.

      As theme and plugin developers, we have been caught out by an ill-defined API and security problems in the past, a long term roadmap makes the best of a bad situation. I’d rather a two year defined process than a repeat of 4.3.2

      • webaware 12:46 am on September 3, 2015 Permalink | Log in to Reply

        [ignoring the double-moustache issue here…] but what signifies that the self-closing shortcode is self-closing? The problem is not merely one of mixed use, it’s the calling syntax that the regex needs to accommodate.

        • Peter Wilson 1:38 am on September 3, 2015 Permalink | Log in to Reply

          Okay, understood – that makes sense.

          {{selfclosing /}} and {{enclosing}} {{/enclosing}}, perhaps. That makes sense to me as a developer, so may be bad for a user.

          Could be double moustache, square, anything. Looking for the balance between performance and teaching.

        • Curtiss Grymala 3:44 pm on September 3, 2015 Permalink | Log in to Reply

          In HTML, what signifies that a self-closing tag is self-closing? Hint: it’s not the / at the end; that’s totally optional; in fact, even the closing HTML tag is optional in a lot of cases.

          For instance:

          <img src="/my-image.png" alt="">


          (a complete HTML page that is 100% valid HTML)

          are both perfectly valid HTML code samples, even if they do make those of us accustomed to XHTML syntax cringe.

          In a perfect world, there would be some way to indicate when registering the shortcode itself whether or not it’s supposed to be self-closing, but that’s not always possible (I believe there are plenty of use-cases where a single shortcode works one way when it’s self-closing, and works a slightly different way when it’s wrapped around content).

          That said, I would definitely support the idea of at least trying to indicate that in the shortcode registration (for instance, if my shortcode is only ever written to be a self-closing shortcode, let me indicate that when I register it; if it’s only ever intended to wrap around content, let me indicate that when using add_shortcode()).

          • webaware 11:09 am on September 4, 2015 Permalink | Log in to Reply

            Yes, and a parser for HTML is a pretty complex piece of code, much more significant than a handful of regular expressions. Ultimately, a true parser might be needed if we want to keep shortcodes the way they are. But perhaps that discussion is better held on the new post

    • Scott Fennell 10:52 pm on September 2, 2015 Permalink | Log in to Reply

      Othe use case for shortcodes I’m hoping you don’t break: Shortcodes in widget titles and widget text fields. This has been a gloriously simple and helpful solution for our agency, and many others I’d suspect.

      • williampatton 12:36 am on September 3, 2015 Permalink | Log in to Reply

        Shortcodes can be parsed anywhere by passing it through `do_shortcode()` which is what I expect your themes are doing to make use of them in widget titles and content.

        Your use case should be just fine so long as the shortcodes used obey the newer syntax 🙂

    • KalenJohnson 11:10 pm on September 2, 2015 Permalink | Log in to Reply

      I’m most likely horribly biased since I’ve started the discussion about [https://core.trac.wordpress.org/ticket/33472 templating engines], but aren’t those shortcodes looking an awful lot like templating engine syntax?

      I’m mostly confused as to why users apparently would be able to remember all those different syntax’s, yet theme and plugin developers wouldn’t, or rather don’t have built in functionality to.

      Also, I also feel rather strongly that there are MANY options now that are much preferable to shortcodes. Things like ACF, Pods, etc. They are not built in to WP core like shortcodes, but really, shouldn’t we focus on getting the Fields API set up correctly rather than continuing to pursue having complicated layouts in one single WYSIWYG editor (not so much WYSIWYG when it’s littered with shortcodes).

      • Jonathandejong 5:52 am on September 3, 2015 Permalink | Log in to Reply

        I’m with this guy ^ Complicated layouts in a single WYSIWYG field just feels wrong. Altho I recognize there’s a lot more to it than just the editor (do_shortcode(), widget fields etc.).

    • Aristeides Stathopoulos 11:33 pm on September 2, 2015 Permalink | Log in to Reply

      I love it!
      It was about time we use something more unique for shortcodes…

    • nick6352683 11:51 pm on September 2, 2015 Permalink | Log in to Reply

      Priorities core developers, priorities. With so many things to still fix and add to the core, you are going to spend time and energy on this, and for the lamest excuse for doing so? Who uses square brackets for quotations? Way to break millions of web sites, for nothing!

      • webaware 12:48 am on September 3, 2015 Permalink | Log in to Reply

        [I hate to pop your bubble but] many people use brackets for essentially parenthetical in-sentence comments. 🙂

    • Jon Brown 11:53 pm on September 2, 2015 Permalink | Log in to Reply

      Shortcode API, yay! This is an exciting proposal, but I’ve got to echo others that the proposed syntax looks like a disaster. I get it, but I can’t imagine users will. I get avoiding the sane syntax of handlebars, angular, C, etc… but do we really have to be _that_ different. I really hope there is a better way.

      On core updating old shortcodes. I’m sorry but no, just no. Don’t you dare go changing _my content_ on a WP update. Worse still is only planning on updating post_content and ignoring all the other places shortcodes show up. Custom Fields for one, but also in PHP files.

      I’d humbly suggest two significant change to this proposal (beyond finding better syntax).
      1. Don’t plan on actually dropping support for the old shortcodes. Do show deprecated notices, just don’t stop supporting the old syntax until WP 6.0. Of all the deprecated cruft I’d love to see dropped, this will never be one of them.
      2. Build a separate utility/plugin a la the WordPress XML importer that can update old shortcodes in a user controlled fashion. Bonus if it actually scans PHP files and identifies shortcodes there. This could then either bulk update, or step through shortcodes one by one for the super paranoid.

      Summary: Simpler Syntax, No real deprecation, Separate plugin for converting old shortcodes.

    • Alex Mills (Viper007Bond) 11:56 pm on September 2, 2015 Permalink | Log in to Reply

      Going to be… interesting migrating SyntaxHighlighter over to this. Maybe this is the motivation I need to finally finish my rewrite from scratch.

    • Peter Chester 12:44 am on September 3, 2015 Permalink | Log in to Reply

      I love clarity. I love easily parseable content. However, this seems like it’s destined to cause WordPress sites around the world to break. I’m sure we’ll see heaps of bare old shortcodes from this change.

      I’m not sure I understand why it’s a problem to check that a shortcode is actually a shortcode before trying to convert it. Why are we doing this?

      Also, I have a hard time teaching people the existing simple shortcode. I expect the error rate will be quite high for people with a more complicated new format.

      > Our old [ and ] delimiters were easily confused with the way those characters are commonly used in English quotations and sometimes even in URLs.

      What are some examples? I haven’t run into this at all in all my years of working with WordPress.

      • webaware 12:59 am on September 3, 2015 Permalink | Log in to Reply

        Very common to have brackets in quotes, to fill in implied references, like this:

        “I won’t be rushing out to get my daughters vaccinated [for cervical cancer], maybe that’s because I’m a cruel, callow, callous, heartless bastard but, look, I won’t be.” Tony Abbott, November 9th, 2006

        Also for excerpted quotes, like this:

        […] We shall go on to the end, we shall fight in France, we shall fight on the seas and oceans, we shall fight with growing confidence and growing strength in the air, we shall defend our Island, whatever the cost may be, we shall fight on the beaches, we shall fight on the landing grounds, we shall fight in the fields and in the streets, we shall fight in the hills; we shall never surrender [… continued]

        For URLs, consider that PHP only handles query parameters as arrays when the parameter name has [] or [key], and some URLs pass such parameters.

    • Greg Ross 12:47 am on September 3, 2015 Permalink | Log in to Reply

      I agree with many of the above comments about the syntax, it’s cumbersome and confusing.

      HTML seems to get by just fine with a single character for it’s delimiters and I see no reason a shortcode can’t as well. Perhaps square brackets are too common, fine let’s change to something else (curly brackets seem reasonable).

      And don’t reinvent the wheel, HTML already has a standard for self-closing and encapsulated tags ( and ) use the same format for shortcodes ( {shortcode /} and {shortcode} {/shortcode}).

      This would also allow both shortcode API’s to run side by side as plugin authors and content creators updated to the new format. Perhaps even make the new syntax a feature plugin for the next few releases until it is merged in to core and then move the old API out to a plugin for several more releases.

      The timeline seems far to aggressive, shortcodes were added way back in WP2.5 and how you want to yank out the old format in just three point releases? The amount of code and content that will require updating is significant.

      My final thought is about the assertion in the proposal that shortcodes *should* only do certain things (like not pass in html as attributes). Keep shortcodes flexible, don’t artificially limit them to certain things. While it may take more to support that flexibility, it would seem to have paid off greatly in the past looking at all the things people have done with them.

      • Ben Huson 4:11 pm on September 3, 2015 Permalink | Log in to Reply

        Completely agree on not re-inventing the wheel for self-closing and encapsulated tags. Stick with the same convention as HTML using the trailing/leading slash within the syntax.

    • Justin Tadlock 1:00 am on September 3, 2015 Permalink | Log in to Reply

      On syntax:

      Users know shortcodes. WP wasn’t the first system to have them. Anyone remember the old BBCodes?

      Open bracket. Type short tag. Close bracket. Simple. Intuitive. Hard to break if you’re not doing anything crazy. Let’s make sure this simplicity is at the core of any discussion on the topic.

      I’d pretty much put the [{{shortcode}}] syntax into the non-starter category. I could get behind {{shortcode}} if we must change syntax, but mixing and matching brackets is destined to fail.

      On updating:

      One thing to watch out for when auto-updating old shortcode syntax in the post content is those of us who output code examples, especially those of us who write in Markdown and don’t need to convert our [ and ].

    • Maxime Jobin 1:04 am on September 3, 2015 Permalink | Log in to Reply

      My main problem is understanding what real problem does it solve ?

      The proposed syntax is really confusing. Isn’t there a way to simply “improve” the actual syntax instead of changing it like this ?

      Also, wouldn’t it a good idea to abstract the concept in the editor. That would mean displaying a visual representation of the shortcode. I would see that as Facebook highlights someone you tag in a status/comment.

      But, that would mean “forcing” developers to specify expected/accepted attributes.

      Aren’t we ready for something more powerful than writing shortcodes ? For devs, it’s perfect. But for non-tech, it’s hard to understand and use. Changing that would mean a lot of confusion.

    • Justin Busa 1:07 am on September 3, 2015 Permalink | Log in to Reply

      If custom fields aren’t going to be migrated to the new syntax, then it would be nice if there was some sort of filter that would allow plugin developers to migrate fields themselves.

      Also, in terms of sites breaking, it would be nice if there was some sort of constant that would enable the old API. Reason being, lets say a site has shortcodes defined in the theme or is using do_shortcode in theme files. Enabling the old API would instantly fix the site while allowing the developer time to update the theme’s code.

    • Ipstenu (Mika Epstein) 1:18 am on September 3, 2015 Permalink | Log in to Reply

      My major concerns have been raised already but I’ll stress if from a different angle.

      If we learned anything from the shortcode change to secure them, we cannot keep breaking users sites. I know that we’re talking about a long term plan to possibly remove shortcodes as we know them by 4.7, but even with all the notification in the world, we will break sites.

      We have 3 releases of WP a year. 4.7 is 18 months away at the outside. That’s a lot of information to get out there really fast and really clearly, AND a lot of code to fix.

      We have to consider:

      • All the plugins and themes on the planet we will break (because we will, they won’t read or test). We have to degrade them as gracefully as humanly possibly. Continuing to say “Well the developers were notified and should have updated” now that we’re as big as we are is not sustainable.
      • All the very (legitimately) angry end users who are broken because they didn’t upgrade plugins and themes (or the themes/plugins didn’t get updated). People were rightly angry last time. It’s the end users, not the developers, who will be hardest hit by this change.
      • Communicating clearly to the users that it’s now {{gallery}}. That’s going to be very hard. Incredibly hard. Updating their old posts (keeping in mind Justin’s Markdown caveat and those who use them as an aside – I know I know) is easier than making sure everyone knows what to do. At best we can keep tabs on the ones built into WP and perhaps use the logic we have in the visual editor NOW to convert them, but we have to figure out how to make sure everyone knows. This is nothing like the move of Menus to customizer. That was confusing, but the users could see what happened. This is a legit change, your old way is no longer going to work. That is huge.
      • The number of users who have premium themes and plugins that do not get update alerts. These people are simply not going to know they need to update and this is not their fault. We should never be breaking them if there’s possibly any alternative.
      • Users will be upgraded by their hosts vis-a-vis one-click installs and managed hosting so they will have up to date WP and out of date plugins/themes. So yes, many users will be on 4.7 and then a theme from 2014. It sucks, it’s the reality, we know it’s the reality, we cannot stick our heads in the sand.
      • Plugins that are already using {{template}} tags in their code. Yeah, I’ve seen it. Most of them use it for search/replace within their own code, but we’ll want to make sure we check for everyone in the repo who might be doing it on their own.
      • Gabe Shackle 1:55 am on September 3, 2015 Permalink | Log in to Reply

        This sums up my feelings perfectly: “we cannot keep breaking users sites”

        Ending support for the current style of shortcode syntax within 18 months will absolutely break sites and frustrate end-users, site owners, and developers in a huge way.

        The best way to make developers stop using a platform is to force them to repeatedly have to retrain their clients and refactor their code.

    • Jean-Francois Arseneault 1:19 am on September 3, 2015 Permalink | Log in to Reply

      Clients, you know, the small business owers, who we train to use their site, and the 3-5 shortcodes they might need, already have a hard time understanding these “square brackets”.

      The draft I read seems to me like the start of “pleasing the machine, and not the human”

    • Luke Carbis 1:20 am on September 3, 2015 Permalink | Log in to Reply

      I’m concerned about client-brain-backward-compatibility. Any changes to the shortcode syntax should still work with my clients brain.

      For example, I provide my client with a site, and teach them to use [button] to insert a button. We finish our contract, and I am no longer in touch with them.

      First the client updates WordPress to 4.4. Nothing changes. Then 4.5. They’re in a production environment, so they don’t see the warnings. Then 4.6. They don’t happen review old posts / pages, but if they do, they don’t realise that their shortcodes have changed.

      Then the client updates to WordPress 4.7, but unlike me, they don’t read the Make WordPress Core blog. They don’t know about the changes. They don’t know that what they’ve learned has become deprecated and now unsupported.

      The client publishes a new page, and they put the button in the way they’ve been taught, which has always worked. But now, it doesn’t work. It’s broken. They call me.

      The client is now angry because a WordPress update cost them money, and confused because the new syntax is much harder to understand.

      I recognise that there are always changes to WordPress, and that the nature of incremental releases is that a user must learn to deal with change. But as far as I know, these changes have always been easy enough for a reasonable user to intuitively navigate. i.e. changes to the menu design, or updates to the nav menu UI.

      I have three questions:

      1. Has there ever been a change to WordPress which could not be intuitively learned by a reasonable user?

      2. Is it okay to introduce unintuitive changes to WordPress?

      3. How can we make this change intuitive to a reasonable user?

      • Luke Carbis 1:29 am on September 3, 2015 Permalink | Log in to Reply

        Here’s a suggestion to answer my third question.

        • When the user enters a shortcode (with either the [simple] or the [{{new}}] format), check if the shortcode exists.
        • If it does, replace the text shortcode with a GUI placeholder element
        • Also, behind the scenes (i.e. in the HTML editor), replace the [simple] shortcode with the [{{new}}] format

        The user experience doesn’t get worse – it gets better. My client enters a shortcode as they always have. They see a visual indicator to show that it was recognised and accepted. Behind the scenes, the new format is saved.

        • Weston Ruter 5:52 am on September 3, 2015 Permalink | Log in to Reply

          Yeah, if this new syntax goes in then GUI placeholders (TinyMCE views) are going to have to be the answer, which also seems like Shortcake.

    • webaware 1:30 am on September 3, 2015 Permalink | Log in to Reply

      Upon some further thought, I think that really there are two problems that are presented:

      1. brackets are already commonly used elsewhere, e.g. quotations, URLs with array parameters

      I see this as being a problem only when developers register shortcodes for real words. Using a sensible prefix for shortcodes prevents this. Unfortunately, there’s been a push for shortcode portability which directly undermines that, e.g. by recommending shortcodes like [product …] for products. [and of course, that prevents combining plugins with similar intent on the same site, but whatever…]

      Has it actually been a problem thus far? I imagine that anyone writing “blah-de-blah [product name] de-blah” will quickly discover that they should use parentheses instead if they don’t want to trigger the shop plugin they’ve installed. Content editors learn. They already have stuff they need to learn, and this is a very simple one that they learn pretty damned quickly anyhow.

      Is it necessary to strip “unregistered shortcodes” when we can just assume that they are part of normal sentence structure? So some websites will display the odd shortcode when its code [e.g. plugin] isn’t active; so what?

      2. self-closing shortcodes don’t look like they are self-closing until they don’t have a closing shortcode

      … which is easily resolved by demanding that they close with /] similarly to XML and HTML.

      To me, this is THE performance problem, and it is easily resolved by not allowing self-closing shortcodes without a terminating /].

      Now, bear in mind also that cleaning up old code might require changing content in:

      • wp_posts.post_content
      • wp_posts.post_excerpt (some plugins / themes use it for rich text)
      • wp_postmeta.meta_value
      • wp_usermeta.meta_value
      • custom tables
      • code (e.g. directly calling `do_shortcode(‘[product …]’);`

      And it has to take into account all sorts of edge cases.

      There’s only so much auto-upgrading that can be achieved. The changes proposed WILL BREAK WEBSITES. Even simply requiring self-closing shortcodes to have /] WILL BREAK WEBSITES. All I can see coming from this proposal is pain. “There will be a great gnashing of teeth” etc. 🙁

    • Jean-Francois Arseneault 1:47 am on September 3, 2015 Permalink | Log in to Reply

      Pardon the pink-colored glasses for a second, but why does it seem that our only choice moving forward, to improve the state of shortcodes, is to change *how* shortcodes work, and not *if* there isn’t a better / newer way to accomplish the goal of inserting content into a page ?

      In the spirit of seing the forest from the trees, why are we still trying to desperately manage mixed-source content using text-based shortcodes ?

      Why are we still using a paradigm where the main content area (if talking about posts) is still an area where we can add content, but also perform layout decisions *within* the content? Where’s the separation of concerns in this way of doing things?

      I think Page Builders are the start of a better solution for managing all these different contents within a page, but perhaps forcing plugin authors to expose their functionality into native widgets and ‘page builder elements’, vs letting them add shortcodes anywhere to render content might be an approach that is easier to grasp, from a UI perspective, for *end-users*.

      Sure, developers, integrators… we can handle everything you throw at us, but in the end, the sites are built for end users, for real-life applications and business purposes. End users are the ones that matter.

      Simplicity always wins.

      • Peter Wilson 2:19 am on September 3, 2015 Permalink | Log in to Reply

        The Shortcake feature plugin covers providing a UI for shortcodes.

      • Chris Van Patten 2:30 am on September 3, 2015 Permalink | Log in to Reply

        Although I’m generally in favour of this proposal, I think this merits some consideration too. Is there potentially a way to eliminate shortcodes — as in a manually entered bit of pseudo-tag-like code — altogether? I think Shortcake is an interesting approach, but it still depends on the shortcode API in the backend. What if shortcodes were deprecated, and replaced with an entirely new concept for inserting dynamic “chunks” of content that doesn’t fall back on parsing through plaintext?

        Replacing shortcodes is probably a much-longer term play, but it does mean that instead of patching the old they can be replaced with something newer, more flexible, and more in tune with the direction core has been moving already.

      • John Teague 1:53 pm on September 4, 2015 Permalink | Log in to Reply

        I totally agree. We need to be thinking of an intuitive replacement for shortcodes altogether. That provides incentives for developers and users to make the change.

        I really appreciate the hard work by @Robert Chapman and others putting a lot of effort into solving the increasingly unsustainable shortcode system. Sadly, I think the proposal in its current form, will confuse already error prone users, and likely break a lot of sites.

    • Chris Van Patten 2:36 am on September 3, 2015 Permalink | Log in to Reply

      As a supplement to my other comment (which was a nested reply), I’m generally in favour of this proposal. I think there’s merit in discussing completely new alternatives to shortcodes, but that’s a semi-separate issue.

      Like many others, I’m not sold on the suggested syntax, but it’s obviously still early days.

      One other thought: why not make escaped code the default, and make unescaped the exception with its own special syntax? Are there backcompat reasons that wouldn’t work? Seems like (potentially) an easy win for security.

    • Andrew Nacin 3:03 am on September 3, 2015 Permalink | Log in to Reply

      Thanks for your feedback, everyone. Developers, truly, honestly, really: Thank you for paying attention. I often feel that requests for comments usually fall on deaf ears, and I know I am not the only committer who feels that way. This has been one of the most substantial and productive threads on make/core I’ve seen, and it’s encouraging to see.

      Here are some thoughts on the proposal:

      • This was clearly labeled a “draft”. I am glad many of you noticed that this was not a decree, but a call for feedback. (And thank you again for obliging.)
      • The vision of shortcodes should be something like “it should be easy and intuitive for non-technical authors to embed and enrich their posts” (I have cribbed this from @matt).
      • This vision does not actually mean that shortcodes are the solution. I actually have a strong dislike for shortcodes. I think they are a terrible user experience, and that even something like Shortcake is still only a marginal improvement.
      • Some changes to shortcode parsing is absolutely necessary in order to keep WordPress secure. Yes, this is about security, not just general sanity.
      • The proposed syntax significantly clashes with the proposed vision. And given all of your feedback, we’re clearly going to have to go back to the drawing board. Please note that we still need to do something, but maybe we can think further outside the box.
      • And for goodness sake, PLEASE STOP RELYING ON HTML IN SHORTCODE ATTRIBUTES. This also does not align with the outlined vision, either, and it’s what largely precipitates proposals like these.
      • Thank you @miqrogroove for your incredibly hard work on this. Honestly, folks, many of you simply cannot see how much work Robert has poured into shortcodes over the last year, as it takes place in private security discussions. He’s been a tremendous asset to the project.

      I’m looking forward to future make/core threads getting this much high quality feedback. Now if only it can also carry over to more Trac tickets. 😀

      • Solinx 9:21 am on September 3, 2015 Permalink | Log in to Reply


        “I’m looking forward to future make/core threads getting this much high quality feedback. Now if only it can also carry over to more Trac tickets.”

        I think this could be helped by making Trac more accessible and by providing a better overview of what is currently going on.

        1. Trac
        When I first heard of Trac I had to get there through google. The “Get Involved” page didn’t help at all. Why not add a link to both the tickets dashboard and the ‘next release’ tickets list at https://make.wordpress.org?

        And why not link to related tickets, github repo’s, slack groups, etc. at the bottom of these blog posts? That makes it a whole lot easier to get to the right places to get involved.

        2. Progress overview
        It is difficult to keep track of all that is going on. The blogs provide updates, but not an overview. Perhaps you could introduce something such as: http://dev.modern.ie/platform/status/adownloadattribute/

        Also here you could link to the related tickets, github, etc.

      • Nick Haskins 12:11 pm on September 3, 2015 Permalink | Log in to Reply

        “Now if only it can also carry over to more Trac tickets.”

        The problem with Trac, for me at least, is that it’s not part of my daily workflow. I’m a bit intimidated by Trac, not to mention making patches. I take part in conversations on Github all day, because that’s how we manage code and issues for my job. I suspect that if these conversations were on Github, with PR’s being allowed there would be a lot more participation. After all, I don’t know any developers outside of WordPress who use SVN in their workflow.

        • NateWr 2:08 pm on September 3, 2015 Permalink | Log in to Reply

          Thanks for outlining this response @nacin. I’d second thinking outside the box. I suspect a lot of the problematic uses of shortcodes stem from people trying to plug gaps in other kinds of functionality (layout, class assignment, conditionals). Maybe solving some of those problems would take a bit of heat off of shortcodes, and make them a smaller attack vector. Or maybe not. Can’t say I really understand the security implications of shortcodes.

        • John Teague 2:33 pm on September 3, 2015 Permalink | Log in to Reply

          Totally agree with you on this Nick. Think how moving to Git would invite solid developers to submit great fixes and features? It’s not that a lot of us can’t use Trac/SVN. It’s just who the hell wants to.

          • Helen Hou-Sandi 4:06 pm on September 3, 2015 Permalink | Log in to Reply

            I imagine that a lot of the workflow folks think of centers around GitHub and pull requests as opposed to Git itself, but just a note that there is a proper Git mirror that many people patch against, including a number of committers. https://make.wordpress.org/core/2014/01/15/git-mirrors-for-wordpress/

            • KalenJohnson 4:49 pm on September 3, 2015 Permalink

              Yes that’s very true, it is Github itself which has become such a boon for developers to be able to quickly and easily find issues, discuss them, and create pull requests. Trac seems very antiquated by comparison, and submitting patch files is not something the rest of the open source web development world does that I’ve seen.

              I can understand how large the move to Github would be for WP, but don’t underestimate that it could really remove WP from the silo it’s in and open up to better quality issues, more issue discussion (as Nacin is looking for), and possibly more patches as people are much more familiar and willing to make pull requests.

            • J.D. Grimes 6:15 pm on September 3, 2015 Permalink

              Yes, the workflow on Trac is needlessly complex. I’d +1 moving to GitHub, or something where we can have PRs instead of patches.

            • John Teague 1:58 pm on September 4, 2015 Permalink

              My bad. I meant github.

      • dlouwe 1:07 am on September 4, 2015 Permalink | Log in to Reply

        “This vision does not actually mean that shortcodes are the solution. I actually have a strong dislike for shortcodes. I think they are a terrible user experience, and that even something like Shortcake is still only a marginal improvement.”

        Very much this. Users shouldn’t have to learn syntax to make use of shortcode-like functionality. They can already use the Visual editor to insert and manipulate HTML without ever knowing what HTML is, and that should be a clear goal for whatever shortcodes end up being in the future. It doesn’t much matter what the new syntax looks like if the only people who are going to be editing it directly are people who have business doing so. As a developer, I don’t mind jumping through some extra hoops to make sure my users have an easy and intuitive experience!

      • raamdev 3:23 am on September 4, 2015 Permalink | Log in to Reply

        > The vision of shortcodes should be something like “it should be easy and intuitive for non-technical authors to embed and enrich their posts”

        I agree. That makes this a good time to redesign shortcodes from scratch, to rethink how we would create something that provides the same flexibility and power of the existing Shortcode API while also more accurately fulfilling the goal of being something that is “easy and intuitive for non-technical authors to embed and enrich their posts”.

        The ideas that came to mind while reading your comment were: Do shortcodes need to be something that is entirely configurable from within the post editor? What if they were instead something that pointed to a configuration, a configuration stored in the database that allowed a plugin or theme to determine what should be returned? (I’m thinking something like `[{{shortcode id=”plugin_slug_123″}}]`, i.e., something like a shortcode with a single attribute.)

        Then a new API could provide plugin and theme developers with a way of building a UI that would allow users to configure ‘shortcodes’ (the configuration being stored in the database, e.g., as `plugin_slug_123`), similar to the way the Plugin API allows plugins to register meta boxes for post types.

        My feeling is that creating something that is both easy and intuitive for non-technical users will require getting rid of syntax as much as possible and replacing that with a UI.

        Building a whole new feature to replace the Shortcode API would also make it easier to transition away from the current Shortcode API: Eventually WordPress Core could disable the old Shortcode API by default (allowing site owners to re-enable it, perhaps using an MU Plugin, with the knowledge that it’s no longer recommended) while plugin and theme developers could update their code to support the new feature (their motivation to do so will be that future versions of WordPress won’t have the old Shortcode API enabled).

    • markkaplun 5:48 am on September 3, 2015 Permalink | Log in to Reply

      [might not be needed after so many comments but…] I want to also voice my opinion against the proposed. I can see how software developer *might* be happy with this kind of syntax but this is very far from freely written text. The need to balance the number of braces is a no go, it is just too easy for my cat to erase one of them and making me waste an hour trying to figure out what is wrong because it will not look obvious. I actually don’t feel too strongly about the syntax of the complex variants, as they require some kind of programming state of mind, but the self closing one should be extremely simple.
      In addition, The use of the $ sign for anything in this context is ridiculous. In free written text it indicates some reference to money or value and the reason it used in regexp is because it is used in free text but rarely in programming.

      1. Whatever is the final syntax, syntax highlighting for shortcodes should be part of both the tinymce and the “text” editors. This will make it easier to identify shortcode and find broken ones.

      2. Since this will be such a big chagne, might as well fix [the lack of] permission checks. On a multi author site everyone can use all shortcodes. There is no reason why a contributor should be able to insert a contact form via a short code. Sure the editor is supposed to review and remove it, but there is no point in giving the ability in the first place.

    • umchal 6:21 am on September 3, 2015 Permalink | Log in to Reply

      The biggest problem of shortcodes from my view is that they remain in the posts even after the plugin which uses the shortcode gets uninstalled.

      I suggest using the HTML comment syntax.

      • HTML

      This way, users do not have to concern about removing them form the post. I suggested this a while back (https://wordpress.org/ideas/topic/commentcode-alternative-to-shortcode).

    • Florian Girardey 7:04 am on September 3, 2015 Permalink | Log in to Reply

      How can you possibly want to break a lot of websites with 4.7 with end of support of the current shortcode syntax and absolutely rejecting the idea of using a recent version of PHP…

      I like the idea of improvement and this roadmap is a good idea but i don’t understand the logic behind the WordPress team right now.

      • BrashRebel 12:43 pm on September 3, 2015 Permalink | Log in to Reply

        If you want to understand, read the whole discussion. In particular @nacin‘s comment https://make.wordpress.org/core/2015/09/02/shortcode-roadmap-draft-one/#comment-27484

        • Florian Girardey 5:51 pm on September 3, 2015 Permalink | Log in to Reply

          I understand why this roadmap is necessary and i absolutely agree.

          But the arguments advanced for such a changement are closely the same that a PHP version update, i don’t understand how the WordPress team can accept this roadmap but also refuse a PHP version update. That’s all

          • Ipstenu (Mika Epstein) 7:18 pm on September 3, 2015 Permalink | Log in to Reply

            While I understand the comparison, it’s a logical fallacy.

            A minimum requirement is just that. A minimum. WP’s minimum is PHP 5.2.4 and MySQL 5.0.15. We still say mod_rewrite is optional. Supporting older versions doesn’t make the codebase insecure in and of itself, and there is no external dependency update that can fix the current security issues with shortcodes.

            The point of this post is to ensure, when we do fix shortcodes, we do it in as graceful and as minimally intrusive a way as humanly possible. To not repeat the shortcode sins of the past. Your concern, while well meaning and valid, has a marked tendency to detract from that.

    • Cristian Antohe 7:10 am on September 3, 2015 Permalink | Log in to Reply

      We’re using Mustache template tags in both our plugins and I can tell you {{tag}} is in no way simpler then [tag]. Once you start adding {{!tag}} or {{^tag}} HTML {{/tag}} things start to go wrong fast. Users simply have a hard time understanding how that works.

      Start to mix up template brakes like [{{tag}}] and that’s just stupid. It’s un-intuitive and will give a lot of headaches to ALL plugin developers who also actively support their plugins.

      • Cristian Antohe 7:19 am on September 3, 2015 Permalink | Log in to Reply

        With these more unique delimeters, we will be able to process registered shortcodes more efficiently because we will not have to search for them by name. Unregistered shortcodes will have more consistency because we can find them more accurately.

        The new syntax is making it harder for users to use shortcodes for the sake of developers who have a hard time working around the constraints that come with using [tag].

    • leemon 7:25 am on September 3, 2015 Permalink | Log in to Reply

      Sorry to say this, but that new syntax is hideous. If this goes forward as it stands, I predict major breakages, lol

    • Mike Schinkel 8:08 am on September 3, 2015 Permalink | Log in to Reply

      Reviewed all the comments and did not see anyone suggest what @cfc suggested on @miqrogroove‘s blog post which just use Custom HTML Elements:


      They seem perfectly suited for this:


      What would they look like?

      Old Shortcode: [foo]bar[/foo]

      New Shortcode: <wp-foo>bar</foo>

      One I saw that idea by @cfc everything else seemed like painful overkill. Besides, shortcodes are very closely aligned with the goal of the custom HTML element proposal with shortcodes being server-side and custom HTML element being client side.

      • Benny 5:56 pm on September 3, 2015 Permalink | Log in to Reply

        I like the idea of custom HTML elements / web components replacing shortcodes. But that still means we can’t phase out the old shortcode syntax because it would break so many sites. How would you suggest to solve that problem? Maybe by adding you’re “You are doing it wrong” notices (for ever)?

      • Michael Davis 10:53 pm on September 3, 2015 Permalink | Log in to Reply

        I hope this gets more discussion. Particularly, what would be potential shortcomings of this method compared to what the new syntax would bring.

      • Knut Sparhell 1:18 am on September 4, 2015 Permalink | Log in to Reply

        I like this idea of custom HTML elements. Registered elements gets executed either server-side or client-side. An advantage is that it’s quite obvious that such elements can’t be used in attribute values, and attributes can’t contain such elements.

        What has to be solved is the slow deprecation of the old shortcode syntax, and if old shortcodes should be replaced upon upgrade, at some point.

        I also look to the more tag and page tags, that are HTML comment tags. When shortcodes was first introduced I was surprised the HTML comment tag was not considered.

        Custom HTML elements seem very future proof, outright elegant, even if the draft never becomes a recommendation or they will never be supported by browsers (for the client side types).

      • webaware 11:09 pm on September 4, 2015 Permalink | Log in to Reply

        Much love. Was thinking about how NextGEN Gallery does its galleries these days (as a decorated image tag it parses out), but this sounds even better.

      • Braad 12:57 am on September 5, 2015 Permalink | Log in to Reply

        +1 for web components. Great idea Mike!

    • chrishoward 8:24 am on September 3, 2015 Permalink | Log in to Reply

      Sorry, I see these new syntax changes as ridiculous.

      The current syntax is untidy and cumbersome enough, and now you want to throw all these braces in it? We should be making shortcodes easier to use, not harder.

      Have you forgotten the user totally?

      These are great for coders. We love “mustaches” and parameters!

      But I pity the user having to key in a multiple-enclosured shortcode.

      Personally, I believe shortcodes should be invisible to the user, just as HTML is. Show the code in text mode, sure, but not in WYSIWYG. In WYSIWYG mode shortcodes should only be entered via a button and form, and then represented in full if possible (like the gallery shortcode) or by a clickable icon.

      For day-to-day users, this new syntax is a big step backwards.

    • asarosenberg 8:45 am on September 3, 2015 Permalink | Log in to Reply

      I agree with previous comments that this suggestion is very bad for users. People can understand shortcodes if they are somewhat familiar with HTML or online forums but the suggested syntax is unreadable to anyone who is not a programmer.

      An easier way to solve the problem with wanting to use brackets for other things is to allow people to escape brackets for example by ignoring brackets prefaced by backslash, star or whatever char that makes sense. If you want WordPress to be a CMS and not just a blog tool, shortcodes are more important than brackets in text.

      As for deprecation and conversion consider that there are shortcodes in meta fields. Many themes use page templates with multiple wp_editors and shortcodes. To avoid massive sudden breakage the old format should work for at least a year or two after the changes have been implemented and announced.

    • Arunas Liuiza 9:07 am on September 3, 2015 Permalink | Log in to Reply

      While I do get the technical reasoning behind this proposal, but I am really worried changes like that will soon make WordPress only usable for people with PhD in Computer Science. Seriously, I think something like 简shortcode简 would be easier to use and understand than the proposed syntax for the average WordPress *user*.

    • andreasnrb 9:27 am on September 3, 2015 Permalink | Log in to Reply

      The first thing I think that should be done before even touching or discussing syntax is to clearly define the purpose of shortcodes and come up with appropriate use cases.
      And just to ignore my first statement. Why not go in other direction and introduce custom HTML tags instead? You register your tag, what attributes it accepts with sanitation callbacks.
      <wp-button href=””>Click me</wp-button>
      Even makes it possible to integrate with HTML5 custom tags later on.

      • andreasnrb 9:31 am on September 3, 2015 Permalink | Log in to Reply

        Yes I know it encodes when written in the HTML window. So it should be combined with a insert button in the menu.

    • HasinHayder 11:56 am on September 3, 2015 Permalink | Log in to Reply

      Now you need a Ph.D in CS to use WordPress 🙁

    • FolioVision 12:11 pm on September 3, 2015 Permalink | Log in to Reply

      I see that doing this will open new possibilities, but I think the basic shortcodes should stay without change:

      Self-Closing: [shortcode]
      Attributes: [shortcode attr1=”value1″ attr2=’value2′ “value3” ‘value4’ value5]
      Enclosing: [shortcode] HTML [/shortcode]
      Escaped Code: [[shortcode]]

      If you want to add more features (like nested shortcodes here), I think yout can do it without breaking too much stuff:

      Multiple Enclosures (this one should work if they could look for closing tag for [shortcode] – a regex won’t suffice though): [shortcode] HTML [encl2] HTML [encl3] HTML [/shortcode]

      I’m not sure about the rest though, but these special cases could probably use the new syntax:

      Stripped Unregistered Code: [{{randomthing}}]
      Stripped Unregistered Enclosure: [{{randomthing}$] Content also stripped. [${randomthing}}]
      Stripped Empty Code: [{{ }}]
      Ignored Orphan: [{{shortcode}$]
      Ignored Orphan: [${shortcode}}]
      Ignored Orphan: [${encl2}$]
      Ignored Context: [{{shortcode
      Ignored Context:

      Do you see any issue with keeping the basic shortcodes unchanged?

    • jakeparis 12:46 pm on September 3, 2015 Permalink | Log in to Reply

      I think that the new syntax is difficult and burdensome to the average user. I would suggest:

      a) Make at least the basic “Self-Closing” shortcode operate without the brackers (just as it does now).
      b) The “Enclosing” style should close with a slash, not a $. There is not precedent for using a $ as a closer in any programming language that I know of — therefore that makes this difficult to remember. The slash it a much better choice.

    • J.D. Grimes 1:26 pm on September 3, 2015 Permalink | Log in to Reply

      I do not know how serious the security implications might be, but perhaps we could avoid removing the old syntax at all, and just disable it on new sites, like we’ve done with other deprecated features.

      I think the real lesson here though, is that shortcodes don’t need to be fixed, they need to be removed. We would still have to figure out how to maintain some amount of backward compatibility for old sites, but it might be more productive to work on a broader roadmap that would actually make shortcodes obsolete. Others, including @nacin, have hinted at this. Is a syntax overhaul really necessary as an intermediate step? Is it even plausible as an intermediate step? What if instead we approached it like this:

      1. Bring Shortcake (or some other “shortcode UI”) into core. This would still use the shortcode API, but would provide a UI for it so the user never has to directly manipulate the shortcode string.
      2. Start using an alternate, secure method of indicating the insertion points of the different content bits when this UI is used.
      3. Disable the shortcode API on new sites.
      4. Let the old sites get slowly broken and hacked over time, or break them ourselves all at once with an update.

    • JakePT 2:28 pm on September 3, 2015 Permalink | Log in to Reply

      I’ve got a couple of things I want to say. Firstly, I really need clarification on what these security issues with Shortcodes are. I read the post on the 4.2.3 change and it didn’t really help. Maybe it’s because it’s happening in an area of WordPress development that I just haven’t stepped into, but I’m having trouble understanding what contexts Shortcodes are a problem in. Isn’t their output defined by the plugin? Why isn’t the security problem with the *plugin*? Aren’t authors the only ones who can use them anyway? What could they do with them that’s so bad? I’m having trouble accepting such a drastic proposal when it doesn’t come with a clear rationale.

      But my biggest problem by *far* is that this would be so so so much easier to swallow if there was *any vision at all* for the future of the content editor. A single TinyMCE field just doesn’t cut it anymore. What happened to the content blocks idea that came up around 3.8/3.9? WordPress needs to address the problem of editing and creating more complex layouts, not breaking functionality that is only being abused because of this lack of vision and progress.

      Shortcodes suck, but don’t make them suck more without something better to replace them.

    • Stefano Aglietti 3:02 pm on September 3, 2015 Permalink | Log in to Reply

      I don’t know if someone already rised the question, but apart that making shortcode delimiter like the ones propose make user in a complicated situation.

      Maybe english people don’ìt know that { and } are not present in alla keyboards directly as the [ and ] are, in Italian keyboard [ and ] are generated using ALT-GR+è or ALT-GR++ and keyboards show ALT-GR basic sign like €@## and [ ].

      To get { and } you net to type ALT-GR+SHIFT+è and ALT-GR+SHIFT++. Most people don’t know about this combination (i think only who write code does) and the parentesys are not written on any key.

      This change would be a a really complication for italian users and i supect a lot of non english keyboard suffer from the same problem. Please when you think to change something like this for end users, use ALWAYS the KISS Method, [{{ is not KISS at alli suggest adding after the square bracket somthig that is awailable on alla keyboards like ! or ? or % or & but for sure nothing is really hide in system and that 99% of people ignore that existes or how to get it leke curly braces

    • Curtiss Grymala 3:04 pm on September 3, 2015 Permalink | Log in to Reply

      First of all, let me say “Thank you” for beginning the process of thinking about this issue.

      As we begin to move forward, has anyone thought about trying to identify where & how shortcodes are currently used, then trying to come up with the best solution for each of those situations? Maybe a shortcode isn’t the best solution for all of the situations in which it’s being used; it’s just the only viable solution we’ve had up until now.

      The bottom-line is, we are still going to need placeholders of some sort within WordPress. There are perfectly reasonable uses of placeholders, regardless of the syntax they use.

      Some are within HTML tags, such as <div class=”[my-custom-class extras=’one-third first’]”> or <img src=”[some-img-url]” alt=”This is my awesome image”/> or <a href=”[my-home-page]” title=”Return to [my-site-name]”>. There are perfectly valid & viable reasons to need to use some sort of placeholder inside of HTML (even if it’s not necessarily inside of the Visual Editor). If we stop allowing these sorts of things, then we’re going to end up with more plugins implementing silly things like [custom-html tag="a" href="{{home-page}}" title="Return to {{site-name}}"]Go Home[/custom-html] which will introduce potentially even bigger security issues and more usability issues for the users.

      Some stand on their own, such as We currently have [number-of-users] users on the site.

      Some output HTML, such as .

      Some are, IMHO, poor attempts to get around security restrictions in the editor, such as [iframe] (there are many plugins that implement this, by basically allowing the editor to enter an HTML tag in normal syntax, but replacing the < and > with square brackets).

      Some need to be nested in order to work properly, such as something like:
      [accordion-title]My accordion item title[/accordion-title]
      [accordion-content]Some content to appear within the accordion item[/accordion-content]

      Some are also completely related to formatting the content, attempting to “fix” the problem of only having one content area, and trying to keep editors from needing to enter HTML on their own, such as the various implementations of column shortcodes (this is a very similar reason that implementations like the accordion shortcodes above exist, as well).

      There are many more perfectly valid & useful implementations of the current shortcode API.

      Each of these may have a completely different and much better solution than the others.

      Finally, as WordPress has evolved very, very far from a simple blogging platform, we need to stop thinking that post titles and post content are the only features that people are using.

      There are plenty of places where shortcodes might appear within the Options table, possibly even inside of serialized data. For instance, shortcodes could, potentially be used within widgets or theme options.

      Shortcodes could absolutely be used, within plain-text or serialized data, within the postmeta table (there are a ton of places that custom fields on a post could include shortcodes) or possibly even the usermeta table.

      The problem I see with the original discussion we had after 4.2.3, and the problem I see with some of the comments on this post, are that there still seem to be people saying “You shouldn’t have used shortcodes to do that; that was stupid, so it doesn’t matter if we break them.”

      The thing I appreciate about this original post, regardless of the fact that the proposed solutions are certainly more complicated than they should be, is that the core team is back to at least trying to solve the issue rather than continuing to feel like it’s okay to break sites just because they disagree with the use-case.

      Again, though, I want to thank @miqrogroove for bringing this to the table and beginning the process of trying to move forward.

    • Stanislav Khromov 5:01 pm on September 3, 2015 Permalink | Log in to Reply

      I’d like to echo the dismay most people feel with this proposed roadmap, and also stress that this can cause a fragmentation in the WP ecosystem.

      Consider that it’s easier than ever to not keep WordPress up to date with the latest version – security patches are backported all the way back to 3.7 and minor security fixes get applied automatically.

      By introducing big, “scary” changes that people feel uneasy about (which the comments prove include even seasoned developers), many will just not update. Consider the kind of fragmentation Python faced after they introduced syntax changes in version 3. Most people just didn’t want or need these changes. I feel the same way about this proposed roadmap.

    • Grant Palin 5:07 pm on September 3, 2015 Permalink | Log in to Reply

      One other gripe I have with the current shortcode syntax is that it sometimes confuses Markdown. Particularly when using within posts authored using WP-Markdown plugin. There’s a workaround, but still.

    • Jacob Santos 8:59 pm on September 3, 2015 Permalink | Log in to Reply

      Why was shortcodes added and what problem did they attempt to solve? What was their original purpose?

      The reason it is a bad idea is that it neglects the reason shortcodes were created and why they have the syntax they do. The history for the most part, I believe, was based in part on BBCode. If Handlebars existed and was more popular, then who knows.

      You will always have a security issue. Tough, but true. Introducing a wholly obtuse syntax will not change that and will make it more difficult for users and developers. Most likely you will just see an influx of plugins that put back the same behavior or those who move away from shortcodes altogether. It will also be a slap in the face once another security flaw is discovered.

      Unless you wish to remove the security issue by pushing people away from using shortcodes completely or creating their own solutions of various quality for security.

      The timeline is far too short. If you go back as far as 3.7, then you must follow that pattern. Deprecate it and update it if you can, but you must support the current until 2018 or 2 years after 4.4 is released.

      • Jacob Santos 9:06 pm on September 3, 2015 Permalink | Log in to Reply

        Lest my argument be misconstrued; I am saying that any syntax change is a bad idea. Solve the security problems with the current or work around them. The bed is made, you have to deal with it.

        Biggest problem I have is that it looks like you are creating a new template engine. There are already those that exist. It would be better if you used them as several have already been audited by security professionals or could be audited by security professionals. They also have a community and those that support it outside of WordPress.

        If you are going to introduce a template engine to WordPress posts, which in and of itself is charge to be warily, you are better off going with something developed by professionals.

        Shortcodes were supposed to be an easy replacement for plugins without a lot of features. When did it become the end all, be all to implement a feature?

        If anything, the RFC should formally suggest where shortcodes should and should not be used, deprecate those usages and at a later date, remove support for them. If advanced templating is required, move to an actual established templating engine. Oh my god, I can’t believe I’m writing this.

        I must be in shock, because the horror of this is just beyond comprehension. I can’t imagine the discussions or the problems that lead to, “hey, why don’t we create a template engine for posts?”

        I have to conclude that this is an early or late April Fool’s joke.

    • chriscct7 2:44 am on September 4, 2015 Permalink | Log in to Reply

      As an update, the team working on the Shortcodes Roadmap, has been working on a new draft proposal which will be posted to make at some point in the near future, which is significantly different than Draft 1. Also, there will be a weekly meeting that will be announced during the same post.

      • John Teague 2:10 pm on September 4, 2015 Permalink | Log in to Reply

        Thank you Chris. And to everyone working to find a solution on this. Hope people realize how much time and effort has to go into this.

    • Ustimenko Alexander 4:45 am on September 8, 2015 Permalink | Log in to Reply

      Users must not pay for developers weakness. If you want something stron-n-cool, then incorporate twig syntax here, but this all is for DEVELOPERS an dnot for users. This syntax is super-extra-bad-thing for end user.

    • renzms 11:57 am on September 17, 2015 Permalink | Log in to Reply

      I’d also like to join in to say that this is more of a step sideways rather than forward/backward with this proposed roadmap.

      Whether a seasoned developer, or just a regular user with no coding experience, these shortcodes aren’t in any way user or developer friendly.

      I understand this is a draft, and a new proposal will be made, especially after all the sentiment seen here. So thanks for the warning, and not changing anything right away as drastic as this. I know this proposed change is due to a security issue, but it causes more problems down the road that developers/users just might not use WordPress anymore to begin with. I do hope a simpler and more unified solution is found.

    • Tom Belknap 8:08 pm on December 15, 2015 Permalink | Log in to Reply

      This isn’t quite such a draft when you consider that 4.4 is already here and already, it’s messing up my shortcodes. Specifically the nested ones.

      This might be the worst proposal since the rightfully-maligned “Settings API,” an innovation the devs needed to ditch in favour of the Customizer. How error-prone is this syntax?

      `[{{shortcode}$] HTML [${encl2}$] HTML [${encl3}$] HTML [${shortcode}}]`

      I thought this double curly brace business must have meant that you were calling out the things we were supposed to change. Then I saw you were serious about it.

compose new post
next post/next comment
previous post/previous comment
show/hide comments
go to top
go to login
show/hide help
shift + esc
Skip to toolbar