WordPress.org

Make WordPress Core

Tagged: shortcodes Toggle Comment Threads | Keyboard Shortcuts

  • Daniel Bachhuber 9:30 pm on November 2, 2015 Permalink |
    Tags: , , , , shortcodes,   

    Shortcake (Shortcode UI) chat summary – November 2nd, 2015 

    Present: @danielbachhuber, @goldenapples, @matth_eu

    Logs: https://wordpress.slack.com/archives/feature-shortcode/p1446494424000273

    • We released Shortcake v0.6.0. Read through the full release notes.
    • Weekly meetings are on hold until January. Between now and then, we’ll be thinking about what we need to do to put forth a core proposal. @matth_eu might put together sketches.
    • We missed the boat on getting a Shortcake representative to the community summit, and are researching ways to helicopter @goldenapples to said community summit boat.

    Next chat: sometime in January 2016

     
  • Andrew Ozz 4:31 pm on October 14, 2015 Permalink |
    Tags: , , shortcodes   

    Shortcodes roadmap — clarifications 

    As mentioned in the initial shortcodes roadmap post, the main purpose of this roadmap is to find the best way for improving the shortcodes API and moving it forward. Currently it is slow, fragile, and attempts to handle a lot of edge cases. For this, the most important part is:

    • No shortcodes in HTML tags attributes.
    • No HTML in shortcodes attributes.

    There are a few things that deserve to be clarified. Simple shortcodes are great. They are easy to understand and be typed directly by the users. Example: [gallery].

    Unfortunately many plugins add complex shortcodes with many attributes and often with nested shortcodes. These are a nightmare for the users. They are not intended be typed directly and can be edited by some sort of UI. Using shortcodes to store this type of data in post_content is not a good idea. Since there is a UI for entering and editing, it would be better to use a simple shortcode to “hold the place”, and save all the data in post meta.

    Many of these complex shortcodes also include HTML tags in their attributes. To keep that functionality, the second roadmap draft proposed an extended syntax that allows the shortcodes “content” (the text wrapped by [shortcode] and [/shortcode]) to be additionally separated by delimiters. That would allow for shortcode attributes to contain HTML tags that are stored in the shortcode content.

    These delimiters are not intended to be typed directly by the users. They are intended for the plugins that have shortcode editing UI and cannot function without storing HTML in shortcode attributes.

    At first look this makes the syntax needlessly complex. However after looking at how complex shortcodes are used now, it is relatively the same: these shortcodes cannot be typed directly and are useless without some sort of UI.

    There have been questions about line breaks in shortcode content. It is possible to add support for this. However it will benefit only a very small amount of users. Since shortcodes “live” in HTML context, and line breaks are ignored there, typing in the Text editor and switching to the Visual editor will remove all line breaks. Typing in the Visual editor will add paragraph tags. So only users that never use the Visual editor and have to type long, complex shortcodes will see some benefit.

    The Shortcode API Roadmap meeting is in #feature-shortcode today at 17z, which is 2015-10-14 1700.

     
    • FolioVision 5:39 pm on October 16, 2015 Permalink | Log in to Reply

      This is a great summary Andrew.

      I’d say we should keep the simplest shortcode syntax possible even if it means that some plugins can no longer freestyle with shortcodes. John Godley wrote software years ago called Sniplets which is meant to allow developers to call PHP inside posts.

      Horses for courses. Shortcodes should not be Sniplets but relatively simple syntax. Certainly without html elements inside.

      Why not deprecate complex shortcodes now with removal in 4.6? That gives us developers lots of time to comply and adapt without making WordPress core software more complex (worse).

    • Jon Brown 11:07 pm on October 18, 2015 Permalink | Log in to Reply

      This is a great summary.

      I’ve mentioned many time over the years, without anyone seeming to think it was the great idea I do that there should be some sort of “smart image/media object”. My reasoning is I’ve long wanted a way in which images could be inserted in posts such that processing could occur at the time of page render. The major use case for me being wanting to insert an image that could call in media metadata (copyright, etc..) and have the meta data updatableble in the DB and then automagically update on page render, rather than hard coded in a short code (ie. gallery/caption). It seems however that this sort of things might also be useful for responsive images (RICG) that could literally point to a single database entry, but later at the time of page render be updated according the current theme settings, etc…

      Perhaps the way forward is an alternative “smart/advanced shortcode”. One which only inserted and manipulated through an advanced UI. The standard current short code could than be kept simple, with the few plugins that currently need advanced functionality moving forward.

      At the same time, perhaps this would be of help to RICG and others?

    • maxwelton 12:57 am on October 20, 2015 Permalink | Log in to Reply

      My initial thought (as someone who writes a lot of “one off” plugins to suit particular clients, some of which have long lists of configurable options per post) is wouldn’t it be nice if a user could:

      • Use a button next to the media gallery button to open a dialog where they can choose from a “nice names” list of registered shortcodes, and insert one of their choice
      • If that inserted shortcode has required parameters, a box similar to the “edit image” box opens automatically. This box is essentially a metabox containing fields defined by the developer in the same way a regular metabox would be added to a post, obviously on a special hook or filter…
      • User fills in their choices and clicks ok.
      • In the visual editor, shortcodes with parameters get an edit icon within their shortcode (or the entire shortcode is “hot” or an icon, and not discretely editable), which opens the parameters box again.
      • In the text editor, each shortcode with parameters will be shown with some sort of ID string matching the auto-generated ID (“gallery-7” or whatever) WP created when the shortcode was inserted. Using this ID allows users to find said metabox in a section below the editor (hidden in visual mode?), or perhaps the shortcode selection dialog can show both a list of available shortcodes and a list of those embedded in-page; clicking on the in-page ones opens the parameters dialog for that shortcode. Clicking on “delete” in the parameters box removes the shortcode from the editor (easy enough to say…)

      This would make the editor a lot cleaner and allow for complex parameters to be passed to the shortcode, including HTML markup, images via gallery dialogs, etc., etc.

      The shortcode handling function is passed the custom ID WP generated, which the function uses to retrieve the parameters from postmeta (I’m imaging core saving the contents of the parameters metabox to a single custom field with the same ID referenced above, as a serialized array).

      Obvious questions include “what happens when you delete a shortcode with opening and closing codes AND nested shortcodes?” and “what happens when a user in text mode alters a shortcode manually and it no longer meets the criteria for being “savable” (ie, either a shortcode which has no required parameters or one whose internal ID matches a metabox with those values filled in).

      Anyway, I might try this approach for my next complex plugin, if time allows, just to see if its workable.

    • Tom Belknap 7:46 pm on December 15, 2015 Permalink | Log in to Reply

      Just wanting a clarification: are we looking to end the practice of nesting shortcodes? Because I’ve found shortcodes to be helpful with creating blocks of post_content HTML without using actual HTML in the Editor screen. For example, I use custom shortcodes to create Zurb Foundation grid blocks within content.

      I’d hate to have to lose such functionalty. It allows me to create much more robust layouts simply. No, as Ozz points out, they’re definitely not for end users. But given how much of the Internet is powered by WordPress, I’d hate to think that we’re limiting ourselves unnecessarily?

  • Robert Chapin 12:42 am on October 8, 2015 Permalink
    Tags: , , shortcodes   

    Shortcode Roadmap Draft Three 

    This is the third draft of the Shortcode API Roadmap. It describes in broad terms the changes that might take place across versions 4.4 through 4.6. This roadmap gives notice to plugin developers that significant changes in plugin design may be needed for compatibility with future versions of the Shortcode API. This roadmap also identifies steps taken to minimize the impact on plugin developers to allow most plugins to continue working with only small changes.

    This roadmap is based on decisions made at meetings, feedback received on previous posts, and consultation with the core developers.

    Our need for a roadmap arose from specific problems in the old code.  There are performance problems in parsing shortcodes, and we need to fix those problems with backward compatibility in mind.  Recent security patches illustrated the problem of not being proactive about security hardening.  Bloat in content filters is another big problem that complicates efforts to correct problems.

    Please comment on this new draft.  We will have another meeting Wednesday at 17Z, which is 2015-10-14 1700.

    4.4 – New Restriction on Shortcode Names

    There is only one item on the 4.4 Milestone. It helps us move toward our goal of security hardening without breaking websites.  Names of registered shortcodes will be slightly restricted by disallowing angle braces.  It should be possible to implement this change immediately with no impact on existing content or plugins.

    The < and > characters will be no longer allowed in the $tag parameter of add_shortcode(). Starting in 4.4, attempting to register an invalid shortcode name will result in a helpful error message.

    These characters will be forbidden in shortcode names: [ ] < > / &

    Also forbidden are the “non-printing characters” including spaces, tabs, new lines, and other control sequences.

    (More …)

     
    • Scott Fennell 1:43 am on October 8, 2015 Permalink | Log in to Reply

      Excellent progress here. Thank you very much for your work and for considering community feedback on this!

      Very relieved to see the bit about allowing an opt-in for shortcodes in html attributes.

      One question. You mentioned,

      “continued core support for HTML in shortcode attributes is likely to evolve through the automatic update system when unplanned future patches are released”

      As the owner of many many instances of HTML in shortcode attributes, I’m not quite sure what to make of this. Are you proposing that my HTML in shortcode atts might break on an automatic update?

      • Robert Chapin 1:53 am on October 8, 2015 Permalink | Log in to Reply

        Yes, some sites were already affected in the 4.2.3 and 4.3.1 updates. There is some concern that this could happen again. It would have been ideal to get more HTML separation but I think we are running out of ideas for that.

        • J.D. Grimes 12:40 pm on October 8, 2015 Permalink | Log in to Reply

          What was wrong with the enclosures syntax? Too complex?

          P.S.—Thank you for your continued work on this!

          • J.D. Grimes 12:56 pm on October 8, 2015 Permalink | Log in to Reply

            Or, why wasn’t HTML in shortcode atts a per-shortcode opt-in feature? But maybe nothing would really be gained by that…

            (Sorry, I didn’t attend the meeting, so I don’t know if this was discussed, and I don’t see a link to the chat in the post.)

          • Robert Chapin 1:38 pm on October 8, 2015 Permalink | Log in to Reply

            https://wordpress.slack.com/archives/core/p1444251538001280

            There was no traction at the core meeting. Mostly negative feedback and no alternatives were proposed. One suggestion was to add closing tags for enclosures, but if I write out an example of that it looks much worse than the original idea. The shortcode would become two or three times longer, and I can’t imagine that would get traction either. So without a syntax that has a chance to see beta, the enclosures idea is out of the roadmap.

            • J.D. Grimes 2:27 pm on October 8, 2015 Permalink

              OK. It sounds like what we really need is an alternative to shortcodes. I don’t mean syntax, I mean the whole concept. I don’t think anyone is going to be happy with the proposal that we just keep breaking things without notice. What we need here is not a shortcode roadmap. Not even Shortcode UI. What we need here is a make-shortcodes-obsolete roadmap.

              Here’s a really wild idea. Now that we have the oEmbed plugin in core, what if in future we replaced shortcodes with the embed feature—except we’d be embedding content from within the site. So we create our content for the shortcode and it is identified by some URL. Then we just paste that URL into our post where we want that content to appear, and viola.

              Of course, that may be completely crazy. But I think we need to begin looking beyond shortcodes for the ultimate solution.

    • Ipstenu (Mika Epstein) 3:29 pm on October 8, 2015 Permalink | Log in to Reply

      > continued core support for HTML in shortcode attributes is likely to evolve through the automatic update system when unplanned future patches are released

      To paraphrase what Helen said in the meeting, the issues with our enclosures of [shortcode:foo=bar] is that we were now actually encouraging people to do the thing we didn’t want them to do in the first place.

      Summarizing what I know we all know, but it’s best sometimes to put this plainly… Using HTML within shortcodes is something WordPress never intended to be a thing, it causes issues, and trying to support it has gotten us into this nightmare we’re at now.

      The options we have boil down to this:

      1) Stop supporting it and break it willy nilly – This sucks because now users, who may have no idea WHY this broke, are stuck with dead sites.

      2) Find a janky workaround to support something we don’t want to support – This sucks because there’s just no way to find and contact and force every developer to update their plugins and core is left holding a bag they don’t want.

      I can argue both cases. Neither is tenable. In both cases, though, communication seems like what we need to do. As much as I don’t want to break users’ sites, no matter what we choose we will be doing this to some extent, and it may be best to do option 1.

      By WordPress 5.0 (I’m throwing a dart at the wall here, this isn’t gospel!), you can’t use HTML in shortcode attributes. Any time between now and then, it might break due to security updates.

      • Scott Fennell 8:43 pm on October 8, 2015 Permalink | Log in to Reply

        “This sucks because there’s just no way to find and contact and force every developer to update their plugins”

        I really appreciate your voice on this @Ipstenu, but there’s a point I’ve been trying to make that I feel like is getting lost. Updating a plugin is pretty easy. What’s hard is updating the post/page content that uses that plugin. That’s the real headache. I know you mentioned the idea of doing a regex search. My database is many gigabytes in size, so this would be a very cumbersome operation. What if I had clients on many databases?

        I’m confused and disappointed that the momentum on this thread seems to be to stop supporting HTML in shortcode atts (HISA) on a security update. Good morning, many, many broken sites. I’ve never known WordPress to do that on such a scale. There has to be something we can do to ease this transition. What if…

        4.5 – Core keeps supporting HISA when rendering content on the front end, but when a field is saved (any text field that might contain a shortcode, to include text widgets and user bio), angle brackets in the shortcode atts are escaped. As a plugin author, I’ll have had time to update my plugins to handle escaped angle brackets however I so choose.

        4.6 – Core ships with an API to execute the update_post(), update_user(), update_option() actions on all fields on all blogs in the network. Agency people like me would be expected to test and execute this step.

        4.7 – Stop supporting HISA.

        • Ipstenu (Mika Epstein) 8:53 pm on October 8, 2015 Permalink | Log in to Reply

          Good morning, many, many broken sites. I’ve never known WordPress to do that on such a scale.

          Sure you do. 4.2.3 – We’re trying to avoid that as much as possible.

          Updating the plugin is easy. In a perfect world, sure that’s true. But getting it updated is not.

          1) Developers have to KNOW you need to. And frankly, the developers who will be most impacted by this change will not. Yes, that’s their fault for not keeping up with things. But the people who get nailed by the break is the users. We need to avoid that if humanly possibly.

          2) Users have to update. And lets be frank, they won’t.

          So what’s worse? Telling everyone “You will be broken by 5.0” or “You might be broken at any time in the future, if we have to edit the shortcode parser, because your plugin was letting you do things wrong” or “You have to edit all your posts in order to match the new parser we’re going to use, which we don’t want to support anyway and we think it’s a bad idea”

          No matter how we look at this, we’re going to be asking users to make a major change.

          If we change shortcode structure, they have to edit all their posts. If we stop supporting it, they have to edit all their posts.

          The only way to ‘not’ break anyone or require anyone to make a change is to never change the shortcodes as we know them today.

          And the thing is, that’s literally where my brilliance fails me. I don’t want to impact the user, but I don’t see a way around it except keeping things as they are, which we shouldn’t for a variety of reasons.

          We dug ourselves into a hole years ago 🙁

          • Scott Fennell 9:04 pm on October 8, 2015 Permalink | Log in to Reply

            “Sure you do. 4.2.3 – We’re trying to avoid that as much as possible.”
            I disagree. Even I think the use cases broken in 4.2.3 were wacky 😀 . In all seriousness, I think the the use cases being threatened in this thread are far, far more popular (though I don’t have data to back that up).

            “Developers have to KNOW you need to. And frankly, the developers who will be most impacted by this change will not.”
            But instead of just impacting developers, this thread threatens to harm end users who would have to be trained to update their shortcodes.

            “So what’s worse? … [ three rough options ] …”
            I’m a bit of a child at the grown-up table here, but I dunno, what about the roadmap I outlined in my previous comment?

            • Ipstenu (Mika Epstein) 10:56 pm on October 8, 2015 Permalink

              I think people using HTML in shortcodes and some of the crazy things they do with layouts is wacky 🙂 It’s a perception thing. But what happened in 4.2.3 is a smaller scale of things. It’s the canary in the mine for me. I look at that and I know how bad this can get as we scale up.

              Now… This part is where we’re arguing the same thing 🙂

              Core keeps supporting HISA when rendering content on the front end, but when a field is saved (any text field that might contain a shortcode, to include text widgets and user bio), angle brackets in the shortcode atts are escaped. As a plugin author, I’ll have had time to update my plugins to handle escaped angle brackets however I so choose.

              What happens to every existing HISA I have? Do they still work?

              What happens when I go into an old post with an HISA and edit it and save? Does it retain what I had before or change it do the new?

              How can we ensure that we’re not just moving the problem? I have a feeling we’d actually just be trading one set of problems for another by allowing them to render the HTML on the front end though I’d have to deep dive more into the API to be sure… And part of the issue there is in what people have their shortcodes doing when they process :/

              The issue is that we don’t want the old code to work. Or rather we cannot give anyone assurance it will continue to work due to ongoing security revelations. We pretty much know we’re going to break things. Badly.

      • Robert Chapin 2:10 am on October 9, 2015 Permalink | Log in to Reply

        To paraphrase what Helen said in the meeting, the issues with our enclosures of [shortcode:foo=bar] is that we were now actually encouraging people to do the thing we didn’t want them to do in the first place.

        I just need to point out that’s not correct. The value was never placed inside a delimiter like that, which would not make sense in the first place.

    • bobbingwide 5:56 pm on October 8, 2015 Permalink | Log in to Reply

      My comments:
      4.4 – restriction on shortcode names is absolutely fine by me
      4.5 – filter priority changing also goes a long way to satisfying my requirements
      4.6 – also acceptable

      but not having the HTML in shortcode attributes, because you don’t like the new syntax and/or haven’t worked through the requirements, shouldn’t exclude it from the roadmap.

      If WordPress doesn’t deliver the goods then I imagine that some developers will just deliver the replacement functionality that’s required. Therefore, IMHO, the requirements should be documented as an enhancement, and then developed as a feature plugin.

      • Ipstenu (Mika Epstein) 6:15 pm on October 8, 2015 Permalink | Log in to Reply

        The issue is WHY we needed to sort out a new syntax in the first place. We’re attempting to support something that was a bad idea to start with, is potentially dangerous, and messes with rendering posts. Building any alternative means we’re condoning something we believe aren’t best practices, and that’s a terrible idea for supportability.

      • Scott Fennell 11:12 pm on October 8, 2015 Permalink | Log in to Reply

        “What happens to every existing HISA I have? Do they still work?”
        I’m proposing that they still render on the front end until 4.7, yes.

        “What happens when I go into an old post with an HISA and edit it and save?”
        I’m proposing that on 4.5, your HISA angle brackets would be escaped upon save. What happens to that on the front end depends on how diligent your plugin author is. I know you have some qualms about that, but it’s better than passing the burden to the end user to edit their shortcodes.

        “Does it retain what I had before or change it do the new?”
        On 4.5, it escapes angle brackets in your HISA, that’s all.

        “How can we ensure that we’re not just moving the problem?”
        Well, at the risk of being tongue-in-cheek, I know I’m solving the problem if core doesn’t break my shortcodes! I’m in an agency context where there are not malicious or even adventurous post editors. I know not everyone is in that position, but I think it’s under-voiced here on the make blog (I digress).

        More to your point, how do we know we’re not just moving the problem? I don’t know — I thought the problem was angle brackets in shortcode atts. Is that not the problem?

  • Daniel Bachhuber 7:42 pm on October 5, 2015 Permalink
    Tags: , , , , shortcodes,   

    Shortcake (Shortcode UI) chat summary – October 5th, 2015 

    Present: @danielbachhuber, @goldenapples, @matth_eu

    Logs: https://wordpress.slack.com/archives/feature-shortcode/p1444071794000007

    • Matt’s making process on support for encoding HTML in attributes. Gallery functionality is also almost done, but there’s one small bug.
    • Than started work on trying to add some filters that can be used to handle floated/non-block previews. It still some work to go, as it’ll involve overriding some methods deep in mce.view.
    • Daniel will hit up the backlog when he has a moment, as there are a number of unanswered open issues.
    • We discussed inline editing and agreed upon an ideal abstraction .

    Next chat: same time and place

    Next release: v0.6.0 – Tuesday, November 3rd

     
  • Robert Chapin 3:42 pm on September 29, 2015 Permalink
    Tags: , , shortcodes   

    Shortcode Roadmap Draft Two 

    This is the second draft of the Shortcode API Roadmap. It describes in broad terms the changes that might take place across versions 4.4 through 4.7. This roadmap gives notice to plugin developers that significant changes in plugin design may be needed for compatibility with future versions of the Shortcode API. This roadmap also identifies steps taken to minimize the impact on plugin developers to allow most plugins to continue working with only small changes.

    This roadmap is based on decisions made at the meeting in #feature-shortcode, as well as feedback on previous posts, and consultation with the core developers.

    Our need for a roadmap arose from specific problems in the old code.  There are performance problems in parsing shortcodes, and we need to fix those problems with backward compatibility in mind.  Recent security patches illustrated the problem of not being proactive about security hardening.  Bloat in content filters is another big problem in itself.

    Please comment on this new draft.  We will have another meeting next Wednesday at 17Z, which is 2015-10-07 1700. Trac tickets that were nominated for closure at the last meeting will be closed today, with references to this draft.

    4.4 – Introduce Multiple Enclosures

    The two items on the 4.4 Milestone help us move toward our goals of security hardening without breaking websites.  A new delimiter in the shortcode syntax will enable plugin authors and users to always place their HTML between the shortage tags rather than inside of them.  At the same time, the names of registered shortcodes will be slightly restricted by disallowing angle braces in shortcode names.  It should be possible to implement both of these changes immediately with no impact on existing content or plugins.

    New Delimiter

    A new addition to the shortcode syntax along with documentation of how it works will be created in the 4.4 development cycle.  Its purpose is to allow more than one HTML enclosure in a single shortcode. Use of this new delimiter is optional or as needed.

    Enclosure Delimiter:  [parent:attr=]
    
    Usage:  [shortcode] Content HTML [shortcode:name=] Attribute HTML [/shortcode]
    
    Example:  [photo link_to="twentysixteen/"] Here is <b>my</b> caption. [photo:media=] <img src="00.twentysixteen-260x300.png" width="260" height="300" /> [/photo]
    
    Formally:
        delimiter   =  begin code-name middle attr-name end
        begin       =  "["
        code-name   =  1*( ALPHA / DIGIT / unreserved / %x80-FD )
        middle      =  ":"
        attr-name   =  *( ALPHA / DIGIT / "-" / "_" )
        end         =  "=]"
        unreserved  =  "!" / "#" / "$" / "%" / "(" / ")" / "*" /
                       "+" / "," / "-" / "." / ";" / "?" / "@" / 
                       "^" / "_" / "{" / "|" / "}" / "~"
    

    In the examples above, the “name” part of the new delimiter works the same way as an attribute name. The main difference when using this new style of attribute (the enclosure) is improved support for HTML inside the value text. One basic reason why this works better is because our HTML filters do not need to understand how to look in the middle of individual shortcode tags. All of the HTML is located between shortcode tags, making the shortcode and HTML codes easy to process separately.

    (More …)

     
    • J.D. Grimes 4:14 pm on September 29, 2015 Permalink | Log in to Reply

      First, thank you so much for your continued hard work on this.

      I do have one point of clarification: will this mean that : will not be allowed in a shortcode or attribute name? I only ask because I think I have seen it used before (although I can’t find an example at the moment).

      • Robert Chapin 4:19 pm on September 29, 2015 Permalink | Log in to Reply

        The plan is to allow : in shortcode names, but with the caution to avoid name collisions. If my shortcode is named caption:title= then I can expect some serious technical problems in the future.

    • Arunas Liuiza 4:27 pm on September 29, 2015 Permalink | Log in to Reply

      Now that’s something that might actually work! Good job, guys.

    • Greg Ross 4:58 pm on September 29, 2015 Permalink | Log in to Reply

      A fabulous second draft. It was great to see everyone participate in the meeting and the feedback from everyone considered and explored!

    • Mike Nelson 6:35 pm on September 29, 2015 Permalink | Log in to Reply

      +1

    • Stanislav Khromov 8:48 pm on September 29, 2015 Permalink | Log in to Reply

      Will this roadmap bring support for nested shortcodes that use the same base? For example:

      [shortcode][shortcode][/shortcode]

    • Sallie Goetsch 11:01 pm on September 29, 2015 Permalink | Log in to Reply

      This seems much more usable and comprehensible than the last proposal.

    • Scott Fennell 1:14 am on September 30, 2015 Permalink | Log in to Reply

      Nice progress here, thank you for your work on this!

      I still find the syntax in this proposal to be awkward, by my primary concern is preventing shortcodes on hundreds of live sites from breaking.

      Let’s say I have offending shortcodes in widget text fields, widget titles, and post excerpts. The shortcodes are offensive both for having shortcodes in HTML attributes, and for having HTML in shortcode attributes. Given this situation, I’m intrigued by the point you mentioned,

      “This will allow plugins to easily register a WordPress cron that will scan all posts in small batches and convert the shortcodes in old content.”

      Can you go into more detail on how robust the core support for this might be, and how one might extend it to capture fields other than just post content?

      • Ipstenu (Mika Epstein) 1:32 pm on September 30, 2015 Permalink | Log in to Reply

        There was no way to do this without the syntax being slightly awkward to some degree. We’re trying to back port in something shortcodes were never supposed to use, so the balance of security vs usability was really hard. This was the best version we were able to come up with, while still addressing all the concerns :/

        Shortcodes in html, like wrapping a-href around a shortcodes, won’t be changed.

        HTML in shortcodes is what’s going away, and the core support for the cron job isn’t written yet, so we can’t tell you how robust it will be just yet. Is there a particular feature you’re concerned with?

        • Scott Fennell 4:39 pm on September 30, 2015 Permalink | Log in to Reply

          “core support for the cron job isn’t written yet, so we can’t tell you how robust it will be just yet. Is there a particular feature you’re concerned with?”

          Forgive me for being a little out of sorts here: I’ve been using WordPress at a relatively large scale across many client sites for about 5 years now, and this thread is the first occasion where a core initiative has threatened to break many of my sites upon update. So, I’m not sure what a realistic expectation might be.

          In an ideal world:
          1) I write a small plugin where I feed a list of shortcode names and also a list of field names (example idea: post[‘content’], post[‘excerpt’], widget[‘title’], user[‘description’], some_type_of_object[‘some_meta_key_for_that_object’] ) to a cron function.
          2) I network-activate the plugin.
          3) In cron-land, wordpress core sweeps through all of those fields on all of my sites and does whatever it has to do such that my pages look exactly the same way they did before.

          This ideal-world solution would cure what ails my shortcodes both in the case of html tags in shortcode atts, and shortcodes in html atts.

          This seems like an impossibly tall order. I’m trying to “play the character” (if you will) of someone who’s exposed to a massive amount of breakage on this initiative, who’s hoping for as much support from core as possible.

          • Robert Chapin 5:10 pm on September 30, 2015 Permalink | Log in to Reply

            This ideal-world solution would cure what ails my shortcodes both in the case of html tags in shortcode atts, and shortcodes in html atts.

            Hi Scott, shortcodes in HTML attributes are not something that can be converted within the core API. At best, we can offer ways to find those codes. We need to hear about the expected impact if the codes stop working. We need to know if the 4.7 Milestone is a reasonable deadline for plugin authors to write their own updates.

            • Scott Fennell 5:22 pm on September 30, 2015 Permalink

              Thank you for helping explain that more specifically. I appreciate it!

              The breakage I’m talking about might be something like a link href that would then 404 upon update — a link or two or three on hundreds of sites.

              4.7 is a reasonable timeline to write plugin updates, sure — that’s not the question.

              The question is, what is a reasonable timeline for my staff to log into hundreds of sites and update shortcodes in all the possible text fields where they might occur? I would argue (from my ideal podium in my ideal world) that there is *no* realistic timeline for that.

              “At best, we can offer ways to find those codes.”
              That would go a long ways, and might even get me to shut up 😀

              However, this thread still makes me very uncomfortable. I consider myself to be a half-decent plugin developer. I don’t hack core, I use core methods, and I read the docs and the source code before using core methods. Yet I am now in the position of having a mess. It’s actually pretty embarrassing for me at work — I feel like I’ve exposed the rest of my agency to a massive loss of goodwill if we have client-facing errors on update.

              If this initiative goes through, you’re now in the position of saying, “WordPress is pretty good as long as you have enough staff to fix stuff by hand upon updating core”. That’s not how I currently think of WordPress. I think that would be a sad turn of events.

              I know I have huge expectations from core — thanks for entertaining them!

            • Ipstenu (Mika Epstein) 6:47 pm on September 30, 2015 Permalink

              But shortcodes within HTML isn’t going to break is it?

              I can still do img src=SHORTCODE if I wanted to. I mean, I’m weird, but I could do it.

            • Scott Fennell 7:01 pm on September 30, 2015 Permalink

              @ipstenu

              I’m weird but I also want to do that! This is a very common use for us:

              <a href='[client_url]'>some text</a>

              Perhaps I am mis-reading, but by my reading, that use case is in jeopardy:

              “To fully enforce the separation of shortcodes and HTML codes, the shortcode API of version 4.7 would also ignore any shortcode tag or attribute that exists inside of an HTML attribute.”

            • Robert Chapin 7:12 pm on September 30, 2015 Permalink

              But shortcodes within HTML isn’t going to break is it?

              They are highly restricted since 4.2.3. In hindsight, they should have been totally blocked since the beginning. This is a “doing it wrong” situation where we just don’t have many options.

            • Ipstenu (Mika Epstein) 7:13 pm on September 30, 2015 Permalink

              Ugh. Obviously I missed that somehow. Multiple times.

              <a>some text</a> would have to become [client_url text='some text']

              Obviously you’d have to update the shortcode there, but a regex search COULD do that:

              Search: <a>(.*?)</a>
              Replace: [client_url text='%1']

              http://wp-cli.org/commands/search-replace/ for example but also https://github.com/interconnectit/Search-Replace-DB can help with that.

            • Scott Fennell 8:26 pm on September 30, 2015 Permalink

              Thank you for those suggestions, Ipstenu.

              However, if that were to go through, you’d be in the position of saying, “WordPress is pretty good, but you can only use it at a large scale if you are willing to run large DB search & replaces upon some core updates”. I don’t care for the sound of that.

            • Ipstenu (Mika Epstein) 9:32 pm on September 30, 2015 Permalink

              @scofennellgmailcom Not exactly 🙂

              WordPress is pretty good, and you can use it at a large scale if you’re fully aware of what your plugins and themes are doing, and they’re doing things the right and intended way to start with. Some core updates will be made in the name of security and some edge cases will always be adversely impacted. If your company lives and dies by your WP site, it’s in your best interest to do exactly what you are doing now 🙂 Keep in touch with issues like this and fix your site WELL ahead of any planned deadline changes.

            • Scott Fennell 10:19 pm on September 30, 2015 Permalink

              @ipstenu,

              That’s a fair interpretation. I agree with you.

              Certainly, there are things I’d rather do with my staff hours than worry about shortcodes breaking, but also I’m worried about the WP community taking a hit. There have to be a lot of agencies like us, but very relatively few of them monitor the core blogs like this.

              WordPress can’t be everything to everyone, but here’s my voice: I want WordPress to not break stuff. I don’t care if the source code is bloated and carries lots of deprecated API’s. I just want it to not break stuff. If I wanted the leanest, most non-legacy, most modern block of code, I’d build sites on whatever HTML9 Responsive Boilerstrap JS happens to be popular this week.

          • Robert Chapin 5:34 pm on September 30, 2015 Permalink | Log in to Reply

            At the next meeting, I think we should explore the possibility of making the HTML attribute parser an opt-in feature that is disabled by default. This would at least give us some middle ground between performance and breaking things.

            • Scott Fennell 5:40 pm on September 30, 2015 Permalink

              Absolutely. In our situation, it’s a “trusted network”. Meaning, we would never have a situation where arbitrary members of the public can register as a user, not even as a subscriber.

          • Ipstenu (Mika Epstein) 10:21 pm on September 30, 2015 Permalink | Log in to Reply

            @scofennellgmailcom – Oh I 100% agree we shouldn’t break things. The problem is the name of the game isn’t ‘deprecated’ or ‘bloated’ but ‘unsafe.’ :/ (Believe me, I’ve been fighting not to break things this whole way, I was very vocal in the first round of this).

            • Scott Fennell 10:48 pm on September 30, 2015 Permalink

              I realize there are some naive assumptions here, but if you’re able to humor me: Given the assumption that we trust all of the users on our network, would you be able to explain in more detail what the security issue is?

            • Robert Chapin 10:50 pm on September 30, 2015 Permalink

              It’s almost a footnote at this point, but where it says

              Implement security context in API

              This is an idea that would give “trusted” capabilities to shortcodes. Currently we do not have that feature. At all. The downside is that would definitely create an inconsistent experience for different users. But it’s still on the table as a possibility.

            • Ipstenu (Mika Epstein) 1:57 am on October 1, 2015 Permalink

              Two things

              1) Even on a trusted network, you can’t trust people not to be stupid

              2) Inconsistent experiences are bad.

              The first one is a big issue, frankly. You may trust your users. I don’t trust my users to know right and wrong when it comes to technical things. That’s not their job, it’s not something I would ever expect them to learn. That’s MY job, to save them 🙂

              I do not, never have, never will trust my users to know “Don’t paste the random code you picked up on the internet!”

            • Scott Fennell 11:07 pm on October 1, 2015 Permalink

              I hear you, but can you go into more detail on what sort of vulnerability it is? A user could deface their own site, or something worse?

            • Robert Chapin 1:09 am on October 2, 2015 Permalink

    • chris@vendiadvertising.com 2:20 pm on September 30, 2015 Permalink | Log in to Reply

      The first draft was quite honestly very scary. This new draft is spot on! Thank you everyone for the hard work on balancing convenience and security!

    • jadpm 2:56 pm on September 30, 2015 Permalink | Log in to Reply

      Juan from the Toolset dev team here.

      I agree with the comments above: the first draft for this changes was a little scary, but this one is indeed quite nice. From our point of view, there are two main things to consider here:

      First, the solution to kind of allow HTML in attribute values is clever. Having multiple enclosures is fantastic, and enables all sort of new posibilities since they wrap actual HTML output. I am thinking on shortcodes for conditional output with else statements. Realy cool, and looking forward to be able to use it.

      Second, although we do depend on shortcodes being used in nHTML attributes, we do understand also that this opens the door to all kind of security issues. We will have plenty of time until 4.7 to update our shortcodes, and provide ways to avoid such structures (maybe even using the feature above). Hopefuly the cron jobs will be enough for us to update existing content.

      We will be keeping an eye on the progress of this, and we might even give a hand. 🙂

    • bonger 3:22 pm on September 30, 2015 Permalink | Log in to Reply

      The HTML-centric thrust of the Shortcodes Roadmap – which appears to be driven by security issues – seems wrong to me. The latest 4.3.1 security update caused me to do some research on the history of the security issues, and it seems to me – based on my no doubt partial understanding – that the approach adopted in 4.2.3 (and continued in 4.3.1) to fix the issues was wrong. Obviously that’s a bold, probably foolish statement, but even if it’s wrong the explanation of its wrongness will be useful.

      The proper response to the security issues was to 1) add capabilities to shortcodes, with KSES-like checks on their attributes, 2) use placeholders for shortcodes, 3) fix comprised content in the database.

      1) Capabilities need to be added to shortcodes (this idea comes from @Kleor in https://core.trac.wordpress.org/ticket/33116#comment:25). Everything’s become skewed by the fact that untrustworthy contributors/authors can insert shortcodes into their content. Thus shortcodes supposedly have to operate in an untrustworthy environment. This is wrong. By default all shortcodes (default capability ‘unfiltered_html’) should be ignored in such content, except for specially hardened shortcodes whose capability (‘edit_post’) allow it, and who undergo a KSES-like check. This should be done on post save, same as KSES, not at run-time. The vast bulk of other shortcodes can then assume that they operate in a trustworthy environment – which they do implicitly anyway.

      2) Parsing is a security issue. Taking shortcodes and their messy attributes out of the equation is needed to ensure real future-proof security. Shortcodes need to be parsed first, without regard to HTML, and replaced by placeholders. This can be made to work seamlessly with the current set-up. So the HTML parsing focus of the Shortcodes Roadmap – and current implementation – is wrong. (Note that this step is separate to the do_shortcode execution of shortcodes, whose priority is unaffected and remains moot.)

      3) The remaining problem is the stored comprised content, entered by untrustworthy contributors/authors, which presumably exists on sites and is patched to be harmless by the changes introduced in 4.2.3 and 4.3.1. This will need an elective database update to fix – elective so that sites with trustworthy contributors/authors can either elect not to update or whitelist their shortcodes first to avoid getting them messed about. (Patching databases is a yucky thing to have to do – so perhaps this was a major contributing reason as to why the current approach was chosen??)

      The restrictions on shortcode attributes introduced (and planned) for security reasons can then be lifted, and the new attribute syntax becomes a good-to-have feature, which can be used to make shortcodes more expressive, but is not a particularly immediate concern.

      • bonger 3:59 pm on September 30, 2015 Permalink | Log in to Reply

        *comprised === compromised

      • Robert Chapin 1:47 am on October 2, 2015 Permalink | Log in to Reply

        I can only assure you that all of these ideas were explored either publicly or privately before we started drafting the roadmap. Saying that the current approach is “wrong” will not accomplish anything.

    • Ipstenu (Mika Epstein) 11:07 pm on October 2, 2015 Permalink | Log in to Reply

      Bother… We need to rethink this a little with regards to HTML attributes.

      Not allowing shortcodes as html attributes means we will outright break some plugins. I remembered a plugin like this: https://wordpress.org/plugins/relative-site-url-on-content-save/

      So I dug deeper and hit https://wordpress.org/plugins/post-link-shortcode/screenshots/ (which has a nice example of _doing_it_wrong() right there)

      I think this may be bigger and deeper and really messier than we’d hoped. If we were just breaking some edge cases I’d be worried but that’s tolerable. Now we’re talking about a bit more.

      Also there’s a plugin called shortcoder – https://wordpress.org/plugins/shortcoder/ – that uses the colon.

      • Robert Chapin 3:28 am on October 3, 2015 Permalink | Log in to Reply

        I’d still like to get HTML attributes as an opt-in only feature. I mean, if we can’t do anything then we can’t, but I’m open to alternatives.

        The colons should be no problem. No plans to block or conflict with that in most cases.

  • Robert Chapin 4:18 am on September 4, 2015 Permalink
    Tags: , , shortcodes   

    Shortcode Roadmap Extended Discussion 

    We saw a great amount of feedback on the first draft of the Shortcode API Roadmap.  The resounding concensus was that the shortcode syntax we already know and love should not be replaced.

    Now we need to bring that enthusiasm and more feedback to the first official meeting to help create a second draft of the roadmap.

    On Wednesday at 17Z, which is 9 Sep 2015 17:00.

    It is scheduled in the #feature-shortcode channel.

    In case you can’t make it to the meeting, here are some of the items up for discussion.

    Parser Problems

    The biggest issue with keeping the BBCode style syntax is that we don’t have a scalable way to make these shortcodes work in PCRE.  The current pattern searches for all registered shortcodes by name, because searching for matching pairs of braces leads to backtrack limitations and segfaults.  If someone has an idea or knows a good library, now is the time to tell us!  With that said, we are also bringing some new proposals of our own.  One theoretical solution should make it possible to preview the first word of each shortcode tag so that the full search pattern then only includes the names of shortcodes already found in the input, rather than the names of all registered shortcodes.  This is the best idea so far and could be easy to implement.

    HTML vs. Shortcode Syntax

    We still want to expand the shortcode syntax to allow multiple enclosures.  So which new tags would work best internally and still look nice for users?  We are now thinking something more like this:

    [shortcode] HTML [=part2=] HTML [/shortcode]

    Let’s also have a brief discussion about preparing for the eventual removal of HTML-in-shortcodes and shortcodes-in-HTML combinations. As we saw in 4.2.3, it is better to plan for this rather than allowing it to become an urgent surprise update.

    Version Issues and Breaking Things

    We still need to plan out a change of filter priorities so that shortcodes will be processed first, before paragraphs and curly quotes. How much impact will this have on plugins now that we will be changing the way existing shortcodes work instead of adding a whole new system? Is it adequate to offer paragraphs and curly quotes as an opt-in only feature?

    Will we need weekly meetings after this first one?

    Are there other tickets or features, such as nested shortcodes, that need to be roadmapped?

     
    • leehodson 5:33 am on September 4, 2015 Permalink | Log in to Reply

      Two ideas:

      Backwards Compatibility

      a) WordPress records page creation dates and page update times. Leave the existing parser, though deprecated, to parse shortcodes in any content that was last updated before the new parser’s release.

      b) Deter use of the old style shortcode syntax by displaying a popup warning (or by highlighting text) in the content editor when old style tags are suspected of being added to new content or when old content is updated. This would double as a public information notice to alert editors to the new shortcode syntax.

      c) Provide a plugin to detect known old shortcodes and suspected unknown shortcodes. Editors and admins can then manually update them to the new syntax or risk automatic update. I guess a ‘shortcode updated’ flag could be added to page metadata to determine use of the new shortcode parser.

      New Syntax

      Yesterday I suggested back-to-back square brackets e.g [ and this ] used together like this [] be used as the new shortcode delimiter. This is easy to remember, easy to look at and shouldn’t cause too much damage to content readability wherever single brackets that are used to encase regular text are automatically converted erroneously into the new shortcode delimiter syntax.

      Self closing (two forward slashes at the end)

      []tag[//]

      Encasing (single forward slash)

      []tag[/] content [/]tag[]

      Nested

      []tag[/] content [] nested self closing [//] more content [/]tag[]

      []tag[/] [] nested self closing [//] [/]tag[]

      Question:

      What if a maximum tag name length were introduced? That should help reduce work performed by the new parser.

      • chriscct7 6:34 am on September 4, 2015 Permalink | Log in to Reply

        Under the back the drawing board discussions, a new syntax isn’t needed, except where HTML is in shortcode attributes which will use the multiple enclosers to continue being able to take in HTML via shortcodes as shown above. The consensus of Draft 1 was if at all possible not to replace the new syntax

      • kjbenk 4:43 pm on September 4, 2015 Permalink | Log in to Reply

        I love the idea of creating a new plugin that will notify site owners that there post contains deprecated shortcode tags and what also what posts they are. Would you consider developing this / would you want some help?

        • leehodson 8:59 pm on September 5, 2015 Permalink | Log in to Reply

          Please do run with it. I have little free time at the minute but I’m happy for you to build the idea into a plugin.

          I’m surprised WordPress doesn’t yet have a compatibility settings page for enabling / disabling deprecated features. Would make it easier to manage backwards compatibility and allow faster evolution of WP core.

    • Weston Ruter 6:02 am on September 4, 2015 Permalink | Log in to Reply

      The biggest issue with keeping the BBCode style syntax is that we don’t have a scalable way to make these shortcodes work in PCRE.

      Maybe we’re doing it wrong by using regular expressions in the first place. If regular expressions shouldn’t be used to parse [X]HTML, per the famous stackoverflow answer, should they be used to parse shortcodes either? HTML with shortcodes is not a regular language but a context-free one (ish).

      What if instead of using regular expressions we used PHP’s own HTML parser in DOMDocument This would, nevertheless, require an initial regular expression replacement to convert shortcodes into HTML tag syntax. The add_shortcode() function would also need to allow shortcodes to be registered indicating whether they are empty or not.

      So then take a look at this gist which has a proof of concept for the initial parse of the post_content containing raw shortcodes: https://gist.github.com/westonruter/27307ea745701b6abf88

      So then you’d have a DOMDocument with span[data-shortcode] elements which could then be traversed over the DOM tree and for each element, starting with the inner-most nested (depth-first), it could send the content off to the shortcode handler and then the response could then replace that DOMElement. The tree walker would then step up the tree and handle any shortcode that wrapped that shortcode, also replacing that placeholder span element with the output from the shortcode handler.

      I note some dovetailing here between shortcodes and Web Components.

      • Nick Haskins 11:49 am on September 4, 2015 Permalink | Log in to Reply

        This was my initial thought as well after reading todays post. It seems to me that finding a better way to parse would be far more easier, and a lot less time consuming vs introducing new syntax, educating, rewriting docs, and dealing with the fallout from such a large change.

      • Robert Chapin 12:29 pm on September 4, 2015 Permalink | Log in to Reply

        If we have reasonable benchmarks and some way to roadmap this after disallowing the HTML/shortcode combinations, then yes. So, my main concerns are whether it runs as fast as PCRE, and that we can’t convert shortcodes to HTML placeholders in the current system. Also, the “initial regular expression replacement to convert shortcodes into HTML” implies a strange level of redundancy. If we are parsing by PCRE then what’s the point of the conversion?

    • Pascal Birchler 8:37 am on September 4, 2015 Permalink | Log in to Reply

      As per this comment by nacin, I’d love to see this happening with the updated shortcode system:

      (Random thought, if anyone wants to have some fun, we should rewrite the API to use hooks under the hood, and ditch $shortcode_tags. And then break everyone reaching directly into $shortcode_tags.)

    • Greg Ross 1:28 pm on September 4, 2015 Permalink | Log in to Reply

      Parser Problems:

      I agree with @WestonRuter, PCRE is probably not be the solution to parsing shortcodes. However if there is still a desire to use them and a need to make them unique, perhaps something like:

      `[wp] html [/wp]` or `[sc] html [/sc]`

      makes more sense than some of the current complex proposals.

      Just to make it clear, I’m still all for keeping the current syntax.

      HTML vs. Shortcode Syntax

      Here’s a question, why do you want to expand the syntax to include multiple enclosures? What problem are you trying to solve?

      The most obvious one would be using that syntax as a type of if/else to conditionally display content. It would seem to make more sense to wrap the content in divs with classes and then conditionally set the class styles to display/hide them.

      Is there a better way than multiple enclosures to solve the problem?

      If that is still the way to go, perhaps a syntax like:

      `[shortcode] HTML [\part2] HTML [/shortcode]` or `[shortcode] HTML [part2 \] HTML [/shortcode]`

      (I like the flow of the first better but the second is more consistent)

      Version Issues and Breaking Things

      If it comes down to not being able to fix the real issues with shortcodes while maintaining the current syntax without breaking the old shortcodes, I would be in favour of using a new syntax rather than break the old shortcodes.

      Shortcodes are embedded in far too much content to break without giving users lots of time to address it.

      This would seem to be a great use of the feature plugin system to flesh out what a new shortcode system would look like and what impacts it would have on sites.

      Here’s what I’d suggest, WP 4.4 should include enough hooks in the shortcode parser to allow for a plugin to completely replace it (maybe it already does, haven’t dug that deep in to it).

      Then a new feature plugin is created to start working out the technical implementation of the new shortcode system. One of the core tenants of the plugin must be that it doesn’t break the old shortcodes. Either by using the same syntax and just extending it or by using a new syntax altogether.

      Once the plugin is mature, merge it in to core.

      If a new syntax for shortcodes is used, start alerting users as they edit content and use the old syntax of the new syntax.

      Then, at some point in the future, a discussion can be had about when or if the old syntax should be deprecated.

    • chatmandesign 2:57 pm on September 4, 2015 Permalink | Log in to Reply

      If there’s value to distinguishing between self-closing shortcodes and shortcodes that contain other content, I think it would make sense to simply mimic XML syntax:

      [gallery/]

      This is a pretty small change that could be introduced as the strongly preferred syntax for self-closing tags, without necessarily eliminating support for the old style tags at all. Perhaps a very lightweight function could sweep through and handle all of these first, if that would cut back on the amount of work a beefier routine needs to perform. Just a thought, since the original proposal suggested it would be valuable to distinguish self-closing shortcodes.

      • Robert Chapin 7:57 pm on September 4, 2015 Permalink | Log in to Reply

        This already exists actually. Nobody uses it. 🙂 Kind of a moot point but I think this feature was not implemented correctly and now we are stuck with it.

    • Mark Root-Wiley 4:12 pm on September 4, 2015 Permalink | Log in to Reply

      A lot of this parsing stuff is over my head, but two ideas that I’ll throw out (and which probably won’t work, but what the heck):

      • What about as a new syntax? stuff for the unclosing variety. It’s fairly unique, similar to the old, and even subtly discourages nesting within html. I even wonder if this would be easier to work with if you move toward a different parser that likes HTML. This would also presumably mean broken shortcodes don’t get displayed on the front end (although I suppose the editor would escape those angle brackets?). This probably doesn’t work…
      • Also wonder if it would be possible to add a prefix to all shortcode tags as part of a solution, so [my-shortcode] has to become [wp-my-shortcode] or [shortcode-my-shortcode] or [wpsc-my-shortcode]
    • PhilipBarrington 5:55 am on September 5, 2015 Permalink | Log in to Reply

      Perhaps a lot would be less traumatic if there was a button on the TinyMCE panel that allowed you to insert shortcode and handled the code, whatever you decide it is going to be. Given the proliferation of shortcode in wordpress and plugins I suspect there needs to be a longer lead time. Part of me suspects that it is a little like replacing the car because the ash tray is full. I admit I don;t understand the problem.

    • Ipstenu (Mika Epstein) 4:37 am on September 6, 2015 Permalink | Log in to Reply

      Normally I’d just make my own blog post, this is long, but I think it’s appropriate here. Thank you again, Robert, for starting this discussion!

      Parser Problems

      PCRE stands for Perl Compatible Regular Expressions – Regular expressions is how we pattern search in code. (Not everyone knows all the jargon, don’t feel bad if you didn’t).

      Parser issues are huge. Yes. They’ve always been a mess. If you want to have a look at how messy, here’s https://wordpress.org/plugins/shortcode-reference/ – this will list all your shortcodes and may help some of us visualize the issue differently.

      We’re parsing to ‘detect’ shortcodes, though, correct? I suppose it would be too much to ask to have the API add it to an option in wp_options so all registered shortcodes are listed? And then that list is parsed before content is output? My gut suggests that would slow things down, especially on long posts, but SQL like that is my third weakest wheelhouse with WP.

      HTML vs. Shortcode Syntax

      I think I know what we’re addressing, but we should perhaps break down what we have shortcodes for today:

      1. No-option shortcodes [button] or – These have no input, we just toss it in and it works.
      2. Some options – These have limited input, maybe a URL or a size variable.
      3. Shortcodes as design elements (too complex to show) – These are the ones with HTML etc in them.

      The issue is not 1 or 2. Those are necessary evils right now when it comes to things like Multisite where we cannot insert iframes etc, and where embeds are sadly not appropriate. Also a shortcode should continue to work, even if I flip back and forth between GUI and HTML editors. In as far as that goes, the absolute death of shortcodes by firing them into the sun will always be stopped because TinyMCE eats HTML. I admit it makes me wonder why/how/what people are doing with these HTML-using shortcodes…

      Actually that’s a good point. Does someone have a really good example of a shortcode doing it ‘wrong’? Crafting a new API roadmap without every being on the same page as to what’s ‘intended’ and what was ‘wow, what are you doing?’ is bound to make a conversation go in circles.

      At its heart, and I know many of my peers detest shortcodes, they are perfect for what they were meant to be. Small, simple, easy to use short codes to insert content that would either be massively complex otherwise (gallery), impossible (embeds in a pre auto-embed world, iframes), or annoyingly repetitive. They do work quite well for that.

      The shift of short-code-for-design is where, I think, much of this has come up.

      Version Issues and Breaking Things

      We absolutely cannot use curly’s. That would kill 40% of the user base (if Nacin’s tweeted stats of non-English users is accurate). They don’t have easy access to them, and right now I feel like I should send cookies to all the coders who don’t have English keyboards.

      Breaking things is also something I have admittedly strong opinions on. We’ve been doing this too much, and we need to stop. Shortcodes are so wildly used, this is something I fear could break WordPress and chase off users for simpler platforms that are more restrictive. I know, we painted ourselves into a corner here and in the fight of ‘break vs secure’ I am a firm ‘Be more secure!’ proponent, but we saw the tip of the iceberg.

      That’s kind of why I would love to see a few examples of plugins we’re pretty darn sure WILL break. Let’s get a real world idea of who we’re going to destroy and maybe we can do something about it? I feel like we’re guessing, and I’m a little uncomfortable.

      I don’t want to name names like ‘you were wrong!’ just ‘You’re the wild west, now let’s see what you did so we can try to fix it!’ So I really hope everyone understands that one… It’s a hard thing to ask 🙁

    • bonger 10:09 pm on September 8, 2015 Permalink | Log in to Reply

      I think the pre-processor filter idea that’s been floating around that replaces stuff with numbered placeholders (same idea as the pre tag replacer at the start of wpautop()) could be the way to go to alleviate a lot of the parsing and other problems.

      For shortcodes it could use the full get_shortcode_regex() (with the first-word improvement) to parse all shortcodes and replace them with placeholders, eg “[shortcode_name n]”, where n is a unique number (I think this makes more sense than a HTML placeholder). This then makes parsing by filters down the line much more straightforward. It also eliminates the need for the current and planned restrictions on HTML input in shortcodes, restoring backwards compatibility (though this appears to be a contentious issue so I could be missing something here).

      The shortcode pre-processor could also restore the left-to-right processing of shortcodes by parsing for tags containing shortcode placeholders and replacing them in turn with placeholders, enabling them to be processed in-line by do_shortcode_tag() for KSES security. I’ve a prototype of this which I’ll post to the ticket https://core.trac.wordpress.org/ticket/33134 as a demonstration.

      Using the same structure other pre-processor filters could also be added, eg to replace HTML comments with placeholders (““), and then to replace blob-like tags (script/style/object etc) with comment-like placeholders (““) etc, simplifying the parsing (and lessening the intrusion in the case of wpautop()) for filters down the line.

      This still leaves the initial get_shortcode_regex() parsing issue, but the first-word search looks like a great improvement. Also for large alternations trie optimizations might be worth doing, as implemented by eg http://search.cpan.org/~dankogai/Regexp-Optimizer-0.15/lib/Regexp/Optimizer.pm (in Perl). Lots of plugins use prefixes to namespace their shortcodes so it could be a big win. Or not.

      The HTML parameter syntax looks good. Would the parts be named and passed through the atts array, eg “[=param1=]”? Could they be optionally introduced by the shortcode name, eg “[=shortcode param1=]”? (Verbose, but might be useful for nesting. Or not.)

      • bonger 10:12 pm on September 8, 2015 Permalink | Log in to Reply

        Sigh.

        I think the pre-processor filter idea that’s been floating around that replaces stuff with numbered placeholders (same idea as the pre tag replacer at the start of wpautop()) could be the way to go to alleviate a lot of the parsing and other problems.

        For shortcodes it could use the full get_shortcode_regex() (with the first-word improvement) to parse all shortcodes and replace them with placeholders, eg “[shortcode_name n]”, where n is a unique number (I think this makes more sense than a HTML placeholder). This then makes parsing by filters down the line much more straightforward. It also eliminates the need for the current and planned restrictions on HTML input in shortcodes, restoring backwards compatibility (though this appears to be a contentious issue so I could be missing something here).

        The shortcode pre-processor could also restore the left-to-right processing of shortcodes by parsing for tags containing shortcode placeholders and replacing them in turn with placeholders, enabling them to be processed in-line by do_shortcode_tag() for KSES security. I’ve a prototype of this which I’ll post to the ticket https://core.trac.wordpress.org/ticket/33134 as a demonstration.

        Using the same structure other pre-processor filters could also be added, eg to replace HTML comments with placeholders, and then to replace blob-like tags (script/style/object etc) with comment-like placeholders etc, simplifying the parsing (and lessening the intrusion in the case of wpautop()) for filters down the line.

        This still leaves the initial get_shortcode_regex() parsing issue, but the first-word search looks like a great improvement. Also for large alternations trie optimizations might be worth doing, as implemented by eg http://search.cpan.org/~dankogai/Regexp-Optimizer-0.15/lib/Regexp/Optimizer.pm (in Perl). Lots of plugins use prefixes to namespace their shortcodes so it could be a big win. Or not.

        The HTML parameter syntax looks good. Would the parts be named and passed through the atts array, eg “[=param1=]”? Could they be optionally introduced by the shortcode name, eg “[=shortcode param1=]”? (Verbose, but might be useful for nesting

    • bobbingwide 10:26 am on September 9, 2015 Permalink | Log in to Reply

      I’ve written a summary of my requirements for shortcode syntax, formatting and editing here

      http://www.oik-plugins.com/2015/09/shortcodes-requirements-for-input-editing-and-formatting/

      Please take these into account later today.

      Here’s a brief summary of my responses to the Extended discussion items above.

      If PCRE doesn’t satisfy the requirements don’t use it.

      First you have to document all the sensible ways that square brackets might be included in content and then decide a strategy for determining whether or not the stuff that follows is expected to be treated as a shortcode.

      Removing stuff – don’t do it; be really, really open about the security issues.

      Consider the usefulness or otherwise of the space character.

      Paragraphs and curly quotes and all other manner of strange filters should be revisited
      along with preventing the Visual editor from messing with my beautifully hand crafted content.

    • Jacob N. Breetvelt 9:03 am on September 10, 2015 Permalink | Log in to Reply

      Changing shortcode priority so that formatting is done after shortcode processing will be disastrous for my plugin wp photo album plus. I woulod rather see a possibility to add the priority as an option to add_shortcode(). Many plugins and themes run content filters that destroy the html generated by my plugin. An example here: https://wordpress.org/support/topic/first-image-of-gallery-not-aligned-with-rest?replies=6 where a content filter ‘my_formatter’ adds empty p tags in the midlle of my code. Running a slideshow on this site will copletely crash. Many times i was confronted with inline script function calls with completely damaged arguments, all due to formatting ( i.e. destructing ) content filters run with an higher priority than the do_shortcode priority.

      Changing the default priority of do_shortcode to be before formatting filters, without the possiblity of the proposed addition to do_shortcode might result in me quitting the wordpress development.
      PLEASE DO NOT DO THIS!!!

    • Jacob N. Breetvelt 4:21 pm on September 11, 2015 Permalink | Log in to Reply

      This is a screenshot when the priorities are ok:
      http://wppa.opajaap.nl/wp-content/wppa-pl/Screenshots/Prority-ok.JPG

      Here are two screenshots with do_shortcode priority = 9. The exact same page:
      http://wppa.opajaap.nl/wp-content/wppa-pl/Screenshots/Priority-faulty-1.JPG
      http://wppa.opajaap.nl/wp-content/wppa-pl/Screenshots/Priority-faulty-2.JPG

      So you can see the damage of wpautop()

    • Jacob N. Breetvelt 10:59 am on September 12, 2015 Permalink | Log in to Reply

      It is worthwile noting these two tickets about wpautop that cause the majority of problems:
      https://core.trac.wordpress.org/ticket/33834
      https://core.trac.wordpress.org/ticket/33840

  • Robert Chapin 10:10 pm on September 2, 2015 Permalink
    Tags: , , shortcodes   

    Shortcode Roadmap Draft One (Proposal – Request for Comments) 

    This is the first draft of the Shortcode API Roadmap. It describes in broad terms a new feature set and migration that might take place across versions 4.4 through 4.7. This roadmap gives notice to plugin developers that significant changes in plugin design may be needed for compatibility with future versions of the Shortcode API. This roadmap also identifies steps taken to minimize the impact on plugin developers to allow most plugins to continue working with only small changes.

    The decision to create this roadmap arose from specific needs that are not met by the old code. Our old [ and ] delimiters were easily confused with the way those characters are commonly used in English quotations and sometimes even in URLs. The proposal to use [{{ and }}] instead should allow a better balance between being able to type in the shortcodes and avoiding confusion with any other input. With these more unique delimeters, we will be able to process registered shortcodes more efficiently because we will not have to search for them by name. Unregistered shortcodes will have more consistency because we can find them more accurately.

    Old style delimeters also gave no strong indication whether or not a closing tag was required. The proposal to use }$] as the delimeter of a shortcode with a following closing tag increases the efficiency of regular expressions, because the search for a closing tag will only happen as needed.

    Adding the new style of shortcode syntax provides an opportunity to make significant API changes. One of those major changes is to ensure that shortcodes are processed before paragraphs and before curly quotes. This will lead to greatly simplified code in related functions that currently must find and avoid shortcodes every time they run. We also have an opporunity to re-think the way shortcodes are filtered, and to give plugin authors more control over those filters when registering their shortcodes.

    4.4 – Introduce New Syntax

    A new syntax and documentation of how it works will be created in the 4.4 development cycle. Support for the new syntax will be introduced, allowing plugins to register for extra features. Core shortcodes will use the new syntax in all new posts. The old syntax will not change. Old posts will not be affected. Initial work on the syntax concept follows.

    Proposed New Syntax

    Self-Closing:  [{{shortcode}}]
    
    Attributes:  [{{shortcode  attr1="value1"  attr2='value2'  "value3"  'value4'  value5}}]
    
    Enclosing:  [{{shortcode}$] HTML [${shortcode}}]
    
    Multiple Enclosures:  [{{shortcode}$] HTML [${encl2}$] HTML [${encl3}$] HTML [${shortcode}}]
    
    Escaped Code:  [!{{shortcode}}]
    
    Stripped Unregistered Code:  [{{randomthing}}]
    
    Stripped Unregistered Enclosure:  [{{randomthing}$]  Content also stripped.  [${randomthing}}]
    
    Stripped Empty Code:  [{{ }}]
    
    Ignored Orphan:  [{{shortcode}$]
    
    Ignored Orphan:  [${shortcode}}]
    
    Ignored Orphan:  [${encl2}$]
    
    Ignored Context:  [{{shortcode <br> }}]
    
    Ignored Context:  <a href="[{{shortcode}}]">

    4.5 – Deprecate Old Syntax

    Starting in 4.5, plugins that register shortcodes without declaring support for new features will raise debugging errors to alert developers that support for the old shortcode syntax is ending.

    Old posts will continue to work normally while the old syntax is deprecated.

    4.6 – Upgrade Old Posts

    Old posts will be automatically converted to the new shortcode syntax. The Shortcode API will continue to provide deprecated support for old syntax so that there is no disruption during the conversion process.

    The API should be adequately abstracted so that old plugins are not affected by this conversion. However, as the new syntax will not support HTML inside of shortcode attributes, there is no guarantee that every shortcode will work the same way in 4.6 as in earlier versions.

    4.7 – End of Support for Old Syntax

    Old shortcodes will stop working in 4.7. Plugins that still produce the old shortcode syntax will be ignored by the Shortcode API.

    The upgrade to 4.7 will include a second pass of the conversion of old posts so that any old syntax that was added to posts during 4.6 will still get converted.

    In 4.6 or 4.7, if necessary, a filter could be added to automatically convert any old syntax still being produced by old plugins when new posts are created.

     
    • Daniel Bachhuber 10:15 pm on September 2, 2015 Permalink | Log in to Reply

      Old posts will be automatically converted to the new shortcode syntax. The Shortcode API will continue to provide deprecated support for old syntax so that there is no disruption during the conversion process.

      What about shortcodes stored in other fields?

      • Muhammad 10:18 pm on September 2, 2015 Permalink | Log in to Reply

        Hopefully, we’ll have helper functions to convert those into new format.

      • Robert Chapin 10:38 pm on September 2, 2015 Permalink | Log in to Reply

        Unless there are alternative ideas, I think plugins that do things in non-core ways will be expected to follow the roadmap with their own upgrades. If there are certain fields that are very commonly used in plugins, we should consider doing a core conversion of those, but with much caution.

      • Robert Chapin 11:10 pm on September 2, 2015 Permalink | Log in to Reply

        One alternative would be to leave the old syntax parsers available for plugin filters, while removing them from default filters.

    • AmirHelzer 10:16 pm on September 2, 2015 Permalink | Log in to Reply

      Thanks for the advanced notice on this huge upcoming change. Aren’t you worried that this process will basically break 99% of all existing WordPress sites? Almost all sites use shortcodes in one way or another (like image captions). doing automated updates to CONTENT is certainly possible, but also prone to corner cases and errors.

      It’s great to have a new API. Is it critically important to deprecate the old one?

      Also, for authors, it’s a lot nicer to have a simple [ ] syntax rather than a more complex sequence of characters.

      • Robert Chapin 10:44 pm on September 2, 2015 Permalink | Log in to Reply

        I think the conversion process will be seamless for most sites. This will mainly depend on corner cases such as plugins embedding shortcodes inside of HTML attributes.

    • Nick Haskins 10:18 pm on September 2, 2015 Permalink | Log in to Reply

      The advanced notice is great, but IMO this is absolutely ridiculous. Are there any benchmarks on how much more efficient it is having users type in more characters for a shortcode? Any benchmarks with how much more efficient core will be from having this change made? This is a HUGE change that will affect a VERY large subset of users.

      • chriscct7 10:40 pm on September 2, 2015 Permalink | Log in to Reply

        This is partially about efficiency. This has a lot to do with security, and basically making 4.2.3 impossible as possible to re-occur

        • Nick Haskins 10:45 pm on September 2, 2015 Permalink | Log in to Reply

          I’d love to see benchmarks on the regex processing of the current implementation against the proposed implementation. Re: security; ATM I’m failing to see how [{{this}}] is more secure than [this]?

        • kevinwhoffman 1:13 am on September 4, 2015 Permalink | Log in to Reply

          If the proposal has positive effects on security, it should mention them. A brief summary or a link to the issues with 4.2.3 would help inform that discussion. As of now the only focus seems to be on theoretical improvements to efficiency while complicating the process of using shortcodes.

      • FolioVision 11:31 pm on September 2, 2015 Permalink | Log in to Reply

        This is an absolutely hideous proposal with dreadful syntax. Is the goal of this WordPress project of ours not to bring software to the people? Syntax like this will be the beginning of a new and much better simple CMS which doesn’t do things in the ugliest, most complicated way possible.

        Code better database upgrades, test them properly before releasing and please stop trying to ruin the software for the end users (those who actually use the software and pay all the bills for us to build it).

        • Travis Northcutt 3:54 pm on September 3, 2015 Permalink | Log in to Reply

          “Syntax like this will be the beginning of a new and much better simple CMS which doesn’t do things in the ugliest, most complicated way possible.”

          The sky is falling!

    • Scott Taylor 10:18 pm on September 2, 2015 Permalink | Log in to Reply

      Let’s all note the “Draft One” portion of the title, and let’s all actively give feedback and participate in any decisions on the table here.

      • Nick Haskins 10:21 pm on September 2, 2015 Permalink | Log in to Reply

        My main concern is that this isn’t easier for authors, and that this will affect a very large portion of users and active sites. If I’m understanding correctly, is this being proposed because it’s more efficient for core?

        • Scott Taylor 10:23 pm on September 2, 2015 Permalink | Log in to Reply

          I have the same concerns, and hate the syntax. I also know that the current situation with shortcodes and content-parsing is potentially unstable and unsustainable.

        • Justin Sternberg 10:33 pm on September 2, 2015 Permalink | Log in to Reply

          I’m in agreement here with the concerns. This will break a lot of things because the short code API has been around so long. There’s just no way to account for all the hacks, workarounds, etc, and doing a migration effort on a WP update sounds like a mighty scary prospect.

          • leehodson 10:37 pm on September 2, 2015 Permalink | Log in to Reply

            Should be able to use the existing shortcode parser to recognize old format shortcodes then make the necessary changes to update them. It recognizes shortcodes efficiently enough at the moment.

            Can imagine breakage but as long as the updater shows the changes site admins ought to be able to determine where hands-on intervention is required.

          • FolioVision 11:33 pm on September 2, 2015 Permalink | Log in to Reply

            It seems a make work project. Are there not some important areas where we could make some real progress like integrated multilingual and better native galleries/media management (NOT reliant on external Automattic services)?

            The upgrade process here and cross-version incompatibilities are more than scary.

          • galbaras 11:44 pm on September 2, 2015 Permalink | Log in to Reply

            Backward compatibility should take into account site-specific customizations, i.e. declaring shortcodes in functions.php of a (child) theme. Most site owners use a “build and forget” approach, or at least neglect their sites for long periods. If a new version contains no support for old-style shortcode declarations, sites will break all over the web.

        • Solinx 11:09 pm on September 2, 2015 Permalink | Log in to Reply

          Yes. How am I going to explain anyone that this syntax is an improvement?

          What specifically are the reasons for not using {{shortcode}} and {{shortcode}} {{/shortcode}}? Or [[shortcode]] and [[shortcode]] [[/shortcode]]?

          Other than that the improvements to the API sound good.

          • Robert Chapin 11:18 pm on September 2, 2015 Permalink | Log in to Reply

            Lone curly braces are easily confused with PHP and other C-style code snippets. The latter square brace example is incompatible with the old syntax and would cause migration problems.

    • Andrew Ozz 10:20 pm on September 2, 2015 Permalink | Log in to Reply

      Was just about to say that this is a proposal. Scott beat me to it (as usual) 🙂

    • Pippin Williamson 10:22 pm on September 2, 2015 Permalink | Log in to Reply

      I would assume the answer is “yes”, but have there been benchmarks done that compare the differences in various syntaxes?

      Why [{{shortcode}}] and not {{shortcode}}?

      Most of the proposed syntax seems sane (though still curious on question above), but this one concerns me:

      [{{shortcode}$] HTML [${shortcode}}]

      For most users, that will be exceptionally unintuitive and we will almost certainly see a high error rate with users not being sure where and when a $ is used.

      • Nick Haskins 10:26 pm on September 2, 2015 Permalink | Log in to Reply

        Pippin I have to agree. If we absolutely have to move, how about something as simple as [{this}] HTML [${this}] . it’s easier to digest, and easier to remember that a $ ends a closing statement.

      • webaware 11:19 pm on September 2, 2015 Permalink | Log in to Reply

        The whole proposal sounds pretty horrible to me, but this suggestion by Pippin sounds moderately less ridiculous.

        Re: {{…}}, I would assume that’s because they are already used heavily in non-WP templating, especially JS templating.

        Essentially, this “fix” is coming rather too late in the piece and I think it’ll break more things than it solves. As a plugin writer heavily dependent on shortcodes, I can only see a greater number of support requests from broken shortcodes in my future if this goes ahead.

        Only Shortcake can save us now! 🙂

        • idealien 2:33 am on September 3, 2015 Permalink | Log in to Reply

          “Only Shortcake can save us now!”

          Seriously! If Shortcake were to be brought into core along with this (draft) proposed change – it would go a LONG way to mitigating the concerns many have raised around complexity.

          Without Shortcake:
          “Look – we took this feature you love and made it harder to use….But it’s safer / faster / better”

          With Shortcake:
          “Look – we made is hella-easy to use shortcodes without having to remember syntax type things. Oh, and that syntax is also getting harder b/c it needs to be safer / faster / better.”

          Car metaphor time… With Shortcake: You’re getting automatic transmission and manual is just a little bit harder to use. Without Shortcake: Manual transmission is now harder to use.

          Shortcake FTW!

      • Robert Chapin 11:33 pm on September 2, 2015 Permalink | Log in to Reply

        If we could get this to work…

        [{{shortcode}} HTML {{shortcode}}]

        Is that better? Or still confusing?

      • Mario Peshev 4:16 pm on September 3, 2015 Permalink | Log in to Reply

        I do agree with Pippin’s example with the dollar signs. Seems incredibly easy to mess up by users. I also noticed another syntax including an exclamation sign, which adds an additional layer of complexity for each and every user.

        +1 on the benchmark question. Given the several different supported syntax proposals, I’m curious to see the regex that is parsing those in a sensible and timely manner.

    • AmirHelzer 10:23 pm on September 2, 2015 Permalink | Log in to Reply

      Also, we deal a lot with shortcodes in Toolset plugins. I can’t see any evidence of any user confusing the [ ] characters for anything else than shortcodes.

      Please, besides the code changes, please also consider the huge amount of documentation update that such a change will require.

      Run a quick Google search for “WordPress shortcodes”. A few pages come from wordpress.org, but the majority are old pages, blog posts and tutorials. Nobody is ever going to update them.

      So, after making such a change, people will see two sets of documentation – with new syntax and with old syntax. You know that Google favours ‘old’ content, so the old tutorials that are never going to update will still explain to use [ ]. I don’t think that this will work out well.

      • chriscct7 10:34 pm on September 2, 2015 Permalink | Log in to Reply

        The team working on this is committed to the documentation updates that are necessary.

        • Pippin Williamson 10:35 pm on September 2, 2015 Permalink | Log in to Reply

          Core documentation isn’t an issue. It’s every plugin / theme developer in the world that also has to update their documentation.

          I don’t disagree with updating docs, just making it clear that’s the issue @AmirHelzer is raising.

          Also have to keep in mind the thousands (millions?) of blog posts and tutorials out there.

          • AmirHelzer 10:49 pm on September 2, 2015 Permalink | Log in to Reply

            Exactly. I’m sure that the core team would be quick to update documentation on wordpress.org. I’m talking about the many pages on other people’s sites that will likely never be updated.

    • Scott Fennell 10:25 pm on September 2, 2015 Permalink | Log in to Reply

      Thanks for your work on this!

      I’m trying to understand what I would need to do in order to prevent about 1,200 live blogs from breaking when this syntax is no longer supported:

      [my_shortcode before='<div class="my_class">' after='</div>']

      Andrew offered a helpful suggestion here:

      https://make.wordpress.org/core/2015/08/30/shortcodes-roadmap/#comment-27368

      If I understand his suggestion correctly, I’d be on the hook to do a search and replace in the database, which will be quite scary and expensive for my company. We have literally thousands of posts on thousands of blogs.

      Please, let’s find a way to move forward with this that leaves the old format unscathed.

      • Scott Fennell 10:28 pm on September 2, 2015 Permalink | Log in to Reply

        Sorry, meant to provide the following code.

        http://pastebin.com/1CMN731H

        Is there a reference somewhere for the comment syntax so I can stop posting gibberish?

      • Robert Chapin 11:49 pm on September 2, 2015 Permalink | Log in to Reply

        If the situation doesn’t support

        [{{my_shortcode class="my_class"}}]

        Then the alternative proposed in the roadmap is

        [{{my_shortcode}$] Optional Content [${before}$]<div class="my_class">[${after}$]</div>[${my_shortcode}}]

        We will also likely encode any HTML in attributes during a migration, so you would have the option of decoding that in your plugin.. security implications could be significant though.

        • Scott Fennell 4:10 pm on September 3, 2015 Permalink | Log in to Reply

          “We will also likely encode any HTML in attributes during a migration”

          Can you explain what you mean by migration, exactly? My imagination is running wild a bit with what you might mean by that. Do you mean to propose that, upon this update, the new version of WordPress will automatically fix my post_content across many gigabytes of posts, without me having to do anything to the posts?

          • Aaron D. Campbell 4:29 pm on September 3, 2015 Permalink | Log in to Reply

            That’s definitely the goal. Basically using the existing parser to pull shortcodes and replace them with the new syntax.

            • Scott Fennell 7:43 pm on September 3, 2015 Permalink

              Sorry for sounding clueless here but that would be … amazing. Is there a precedent for this level of work in a similar update? I’m having trouble imagining how this would be possible given a very large database like mine. Would it run as a cron job perhaps?

    • leehodson 10:30 pm on September 2, 2015 Permalink | Log in to Reply

      The new syntax is very confusing to the eye. Users would find it complicated. I feel we could make it less confusing by using back-to-back opening and closing brackets.

      For example

      Self Closing:

      []shortcode[]

      Enclosing:

      []shortcode[] Content [/]shortcode[]

      Note the forward slash between the brackets of the closing tag on the enclosing shortcode.

      Is there a reason this wouldn’t work?

    • Peter Wilson 10:51 pm on September 2, 2015 Permalink | Log in to Reply

      I’d be happier with {{selfclosing}} and {{enclosing}} {{/enclosing}} or similar. Mixing the formats seems fraught.

      Upgrading old content is a necessary evil, backward compatibility on the deprecated format defeats the purpose somewhat. Seeing some benchmarking results would be great.

      As theme and plugin developers, we have been caught out by an ill-defined API and security problems in the past, a long term roadmap makes the best of a bad situation. I’d rather a two year defined process than a repeat of 4.3.2

      • webaware 12:46 am on September 3, 2015 Permalink | Log in to Reply

        [ignoring the double-moustache issue here…] but what signifies that the self-closing shortcode is self-closing? The problem is not merely one of mixed use, it’s the calling syntax that the regex needs to accommodate.

        • Peter Wilson 1:38 am on September 3, 2015 Permalink | Log in to Reply

          Okay, understood – that makes sense.

          {{selfclosing /}} and {{enclosing}} {{/enclosing}}, perhaps. That makes sense to me as a developer, so may be bad for a user.

          Could be double moustache, square, anything. Looking for the balance between performance and teaching.

        • Curtiss Grymala 3:44 pm on September 3, 2015 Permalink | Log in to Reply

          In HTML, what signifies that a self-closing tag is self-closing? Hint: it’s not the / at the end; that’s totally optional; in fact, even the closing HTML tag is optional in a lot of cases.

          For instance:

          <img src="/my-image.png" alt="">

          and

          https://gist.github.com/cgrymala/d61c3cacd6fb07bdcf9a
          (a complete HTML page that is 100% valid HTML)

          are both perfectly valid HTML code samples, even if they do make those of us accustomed to XHTML syntax cringe.

          In a perfect world, there would be some way to indicate when registering the shortcode itself whether or not it’s supposed to be self-closing, but that’s not always possible (I believe there are plenty of use-cases where a single shortcode works one way when it’s self-closing, and works a slightly different way when it’s wrapped around content).

          That said, I would definitely support the idea of at least trying to indicate that in the shortcode registration (for instance, if my shortcode is only ever written to be a self-closing shortcode, let me indicate that when I register it; if it’s only ever intended to wrap around content, let me indicate that when using add_shortcode()).

          • webaware 11:09 am on September 4, 2015 Permalink | Log in to Reply

            Yes, and a parser for HTML is a pretty complex piece of code, much more significant than a handful of regular expressions. Ultimately, a true parser might be needed if we want to keep shortcodes the way they are. But perhaps that discussion is better held on the new post

    • Scott Fennell 10:52 pm on September 2, 2015 Permalink | Log in to Reply

      Othe use case for shortcodes I’m hoping you don’t break: Shortcodes in widget titles and widget text fields. This has been a gloriously simple and helpful solution for our agency, and many others I’d suspect.

      • williampatton 12:36 am on September 3, 2015 Permalink | Log in to Reply

        Shortcodes can be parsed anywhere by passing it through `do_shortcode()` which is what I expect your themes are doing to make use of them in widget titles and content.

        Your use case should be just fine so long as the shortcodes used obey the newer syntax 🙂

    • KalenJohnson 11:10 pm on September 2, 2015 Permalink | Log in to Reply

      I’m most likely horribly biased since I’ve started the discussion about [https://core.trac.wordpress.org/ticket/33472 templating engines], but aren’t those shortcodes looking an awful lot like templating engine syntax?

      I’m mostly confused as to why users apparently would be able to remember all those different syntax’s, yet theme and plugin developers wouldn’t, or rather don’t have built in functionality to.

      Also, I also feel rather strongly that there are MANY options now that are much preferable to shortcodes. Things like ACF, Pods, etc. They are not built in to WP core like shortcodes, but really, shouldn’t we focus on getting the Fields API set up correctly rather than continuing to pursue having complicated layouts in one single WYSIWYG editor (not so much WYSIWYG when it’s littered with shortcodes).

      • Jonathandejong 5:52 am on September 3, 2015 Permalink | Log in to Reply

        I’m with this guy ^ Complicated layouts in a single WYSIWYG field just feels wrong. Altho I recognize there’s a lot more to it than just the editor (do_shortcode(), widget fields etc.).

    • Aristeides Stathopoulos 11:33 pm on September 2, 2015 Permalink | Log in to Reply

      I love it!
      It was about time we use something more unique for shortcodes…

    • nick6352683 11:51 pm on September 2, 2015 Permalink | Log in to Reply

      Priorities core developers, priorities. With so many things to still fix and add to the core, you are going to spend time and energy on this, and for the lamest excuse for doing so? Who uses square brackets for quotations? Way to break millions of web sites, for nothing!

      • webaware 12:48 am on September 3, 2015 Permalink | Log in to Reply

        [I hate to pop your bubble but] many people use brackets for essentially parenthetical in-sentence comments. 🙂

    • Jon Brown 11:53 pm on September 2, 2015 Permalink | Log in to Reply

      Shortcode API, yay! This is an exciting proposal, but I’ve got to echo others that the proposed syntax looks like a disaster. I get it, but I can’t imagine users will. I get avoiding the sane syntax of handlebars, angular, C, etc… but do we really have to be _that_ different. I really hope there is a better way.

      On core updating old shortcodes. I’m sorry but no, just no. Don’t you dare go changing _my content_ on a WP update. Worse still is only planning on updating post_content and ignoring all the other places shortcodes show up. Custom Fields for one, but also in PHP files.

      I’d humbly suggest two significant change to this proposal (beyond finding better syntax).
      1. Don’t plan on actually dropping support for the old shortcodes. Do show deprecated notices, just don’t stop supporting the old syntax until WP 6.0. Of all the deprecated cruft I’d love to see dropped, this will never be one of them.
      2. Build a separate utility/plugin a la the WordPress XML importer that can update old shortcodes in a user controlled fashion. Bonus if it actually scans PHP files and identifies shortcodes there. This could then either bulk update, or step through shortcodes one by one for the super paranoid.

      Summary: Simpler Syntax, No real deprecation, Separate plugin for converting old shortcodes.

    • Alex Mills (Viper007Bond) 11:56 pm on September 2, 2015 Permalink | Log in to Reply

      Going to be… interesting migrating SyntaxHighlighter over to this. Maybe this is the motivation I need to finally finish my rewrite from scratch.

    • Peter Chester 12:44 am on September 3, 2015 Permalink | Log in to Reply

      I love clarity. I love easily parseable content. However, this seems like it’s destined to cause WordPress sites around the world to break. I’m sure we’ll see heaps of bare old shortcodes from this change.

      I’m not sure I understand why it’s a problem to check that a shortcode is actually a shortcode before trying to convert it. Why are we doing this?

      Also, I have a hard time teaching people the existing simple shortcode. I expect the error rate will be quite high for people with a more complicated new format.

      > Our old [ and ] delimiters were easily confused with the way those characters are commonly used in English quotations and sometimes even in URLs.

      What are some examples? I haven’t run into this at all in all my years of working with WordPress.

      • webaware 12:59 am on September 3, 2015 Permalink | Log in to Reply

        Very common to have brackets in quotes, to fill in implied references, like this:

        “I won’t be rushing out to get my daughters vaccinated [for cervical cancer], maybe that’s because I’m a cruel, callow, callous, heartless bastard but, look, I won’t be.” Tony Abbott, November 9th, 2006

        Also for excerpted quotes, like this:

        […] We shall go on to the end, we shall fight in France, we shall fight on the seas and oceans, we shall fight with growing confidence and growing strength in the air, we shall defend our Island, whatever the cost may be, we shall fight on the beaches, we shall fight on the landing grounds, we shall fight in the fields and in the streets, we shall fight in the hills; we shall never surrender [… continued]

        For URLs, consider that PHP only handles query parameters as arrays when the parameter name has [] or [key], and some URLs pass such parameters.

    • Greg Ross 12:47 am on September 3, 2015 Permalink | Log in to Reply

      I agree with many of the above comments about the syntax, it’s cumbersome and confusing.

      HTML seems to get by just fine with a single character for it’s delimiters and I see no reason a shortcode can’t as well. Perhaps square brackets are too common, fine let’s change to something else (curly brackets seem reasonable).

      And don’t reinvent the wheel, HTML already has a standard for self-closing and encapsulated tags ( and ) use the same format for shortcodes ( {shortcode /} and {shortcode} {/shortcode}).

      This would also allow both shortcode API’s to run side by side as plugin authors and content creators updated to the new format. Perhaps even make the new syntax a feature plugin for the next few releases until it is merged in to core and then move the old API out to a plugin for several more releases.

      The timeline seems far to aggressive, shortcodes were added way back in WP2.5 and how you want to yank out the old format in just three point releases? The amount of code and content that will require updating is significant.

      My final thought is about the assertion in the proposal that shortcodes *should* only do certain things (like not pass in html as attributes). Keep shortcodes flexible, don’t artificially limit them to certain things. While it may take more to support that flexibility, it would seem to have paid off greatly in the past looking at all the things people have done with them.

      • Ben Huson 4:11 pm on September 3, 2015 Permalink | Log in to Reply

        Completely agree on not re-inventing the wheel for self-closing and encapsulated tags. Stick with the same convention as HTML using the trailing/leading slash within the syntax.

    • Justin Tadlock 1:00 am on September 3, 2015 Permalink | Log in to Reply

      On syntax:

      Users know shortcodes. WP wasn’t the first system to have them. Anyone remember the old BBCodes?

      Open bracket. Type short tag. Close bracket. Simple. Intuitive. Hard to break if you’re not doing anything crazy. Let’s make sure this simplicity is at the core of any discussion on the topic.

      I’d pretty much put the [{{shortcode}}] syntax into the non-starter category. I could get behind {{shortcode}} if we must change syntax, but mixing and matching brackets is destined to fail.

      On updating:

      One thing to watch out for when auto-updating old shortcode syntax in the post content is those of us who output code examples, especially those of us who write in Markdown and don’t need to convert our [ and ].

    • Maxime Jobin 1:04 am on September 3, 2015 Permalink | Log in to Reply

      My main problem is understanding what real problem does it solve ?

      The proposed syntax is really confusing. Isn’t there a way to simply “improve” the actual syntax instead of changing it like this ?

      Also, wouldn’t it a good idea to abstract the concept in the editor. That would mean displaying a visual representation of the shortcode. I would see that as Facebook highlights someone you tag in a status/comment.

      But, that would mean “forcing” developers to specify expected/accepted attributes.

      Aren’t we ready for something more powerful than writing shortcodes ? For devs, it’s perfect. But for non-tech, it’s hard to understand and use. Changing that would mean a lot of confusion.

    • Justin Busa 1:07 am on September 3, 2015 Permalink | Log in to Reply

      If custom fields aren’t going to be migrated to the new syntax, then it would be nice if there was some sort of filter that would allow plugin developers to migrate fields themselves.

      Also, in terms of sites breaking, it would be nice if there was some sort of constant that would enable the old API. Reason being, lets say a site has shortcodes defined in the theme or is using do_shortcode in theme files. Enabling the old API would instantly fix the site while allowing the developer time to update the theme’s code.

    • Ipstenu (Mika Epstein) 1:18 am on September 3, 2015 Permalink | Log in to Reply

      My major concerns have been raised already but I’ll stress if from a different angle.

      If we learned anything from the shortcode change to secure them, we cannot keep breaking users sites. I know that we’re talking about a long term plan to possibly remove shortcodes as we know them by 4.7, but even with all the notification in the world, we will break sites.

      We have 3 releases of WP a year. 4.7 is 18 months away at the outside. That’s a lot of information to get out there really fast and really clearly, AND a lot of code to fix.

      We have to consider:

      • All the plugins and themes on the planet we will break (because we will, they won’t read or test). We have to degrade them as gracefully as humanly possibly. Continuing to say “Well the developers were notified and should have updated” now that we’re as big as we are is not sustainable.
      • All the very (legitimately) angry end users who are broken because they didn’t upgrade plugins and themes (or the themes/plugins didn’t get updated). People were rightly angry last time. It’s the end users, not the developers, who will be hardest hit by this change.
      • Communicating clearly to the users that it’s now {{gallery}}. That’s going to be very hard. Incredibly hard. Updating their old posts (keeping in mind Justin’s Markdown caveat and those who use them as an aside – I know I know) is easier than making sure everyone knows what to do. At best we can keep tabs on the ones built into WP and perhaps use the logic we have in the visual editor NOW to convert them, but we have to figure out how to make sure everyone knows. This is nothing like the move of Menus to customizer. That was confusing, but the users could see what happened. This is a legit change, your old way is no longer going to work. That is huge.
      • The number of users who have premium themes and plugins that do not get update alerts. These people are simply not going to know they need to update and this is not their fault. We should never be breaking them if there’s possibly any alternative.
      • Users will be upgraded by their hosts vis-a-vis one-click installs and managed hosting so they will have up to date WP and out of date plugins/themes. So yes, many users will be on 4.7 and then a theme from 2014. It sucks, it’s the reality, we know it’s the reality, we cannot stick our heads in the sand.
      • Plugins that are already using {{template}} tags in their code. Yeah, I’ve seen it. Most of them use it for search/replace within their own code, but we’ll want to make sure we check for everyone in the repo who might be doing it on their own.
      • Gabe Shackle 1:55 am on September 3, 2015 Permalink | Log in to Reply

        This sums up my feelings perfectly: “we cannot keep breaking users sites”

        Ending support for the current style of shortcode syntax within 18 months will absolutely break sites and frustrate end-users, site owners, and developers in a huge way.

        The best way to make developers stop using a platform is to force them to repeatedly have to retrain their clients and refactor their code.

    • Jean-Francois Arseneault 1:19 am on September 3, 2015 Permalink | Log in to Reply

      Clients, you know, the small business owers, who we train to use their site, and the 3-5 shortcodes they might need, already have a hard time understanding these “square brackets”.

      The draft I read seems to me like the start of “pleasing the machine, and not the human”

    • Luke Carbis 1:20 am on September 3, 2015 Permalink | Log in to Reply

      I’m concerned about client-brain-backward-compatibility. Any changes to the shortcode syntax should still work with my clients brain.

      For example, I provide my client with a site, and teach them to use [button] to insert a button. We finish our contract, and I am no longer in touch with them.

      First the client updates WordPress to 4.4. Nothing changes. Then 4.5. They’re in a production environment, so they don’t see the warnings. Then 4.6. They don’t happen review old posts / pages, but if they do, they don’t realise that their shortcodes have changed.

      Then the client updates to WordPress 4.7, but unlike me, they don’t read the Make WordPress Core blog. They don’t know about the changes. They don’t know that what they’ve learned has become deprecated and now unsupported.

      The client publishes a new page, and they put the button in the way they’ve been taught, which has always worked. But now, it doesn’t work. It’s broken. They call me.

      The client is now angry because a WordPress update cost them money, and confused because the new syntax is much harder to understand.

      I recognise that there are always changes to WordPress, and that the nature of incremental releases is that a user must learn to deal with change. But as far as I know, these changes have always been easy enough for a reasonable user to intuitively navigate. i.e. changes to the menu design, or updates to the nav menu UI.

      I have three questions:

      1. Has there ever been a change to WordPress which could not be intuitively learned by a reasonable user?

      2. Is it okay to introduce unintuitive changes to WordPress?

      3. How can we make this change intuitive to a reasonable user?

      • Luke Carbis 1:29 am on September 3, 2015 Permalink | Log in to Reply

        Here’s a suggestion to answer my third question.

        • When the user enters a shortcode (with either the [simple] or the [{{new}}] format), check if the shortcode exists.
        • If it does, replace the text shortcode with a GUI placeholder element
        • Also, behind the scenes (i.e. in the HTML editor), replace the [simple] shortcode with the [{{new}}] format

        The user experience doesn’t get worse – it gets better. My client enters a shortcode as they always have. They see a visual indicator to show that it was recognised and accepted. Behind the scenes, the new format is saved.

        • Weston Ruter 5:52 am on September 3, 2015 Permalink | Log in to Reply

          Yeah, if this new syntax goes in then GUI placeholders (TinyMCE views) are going to have to be the answer, which also seems like Shortcake.

    • webaware 1:30 am on September 3, 2015 Permalink | Log in to Reply

      Upon some further thought, I think that really there are two problems that are presented:

      1. brackets are already commonly used elsewhere, e.g. quotations, URLs with array parameters

      I see this as being a problem only when developers register shortcodes for real words. Using a sensible prefix for shortcodes prevents this. Unfortunately, there’s been a push for shortcode portability which directly undermines that, e.g. by recommending shortcodes like [product …] for products. [and of course, that prevents combining plugins with similar intent on the same site, but whatever…]

      Has it actually been a problem thus far? I imagine that anyone writing “blah-de-blah [product name] de-blah” will quickly discover that they should use parentheses instead if they don’t want to trigger the shop plugin they’ve installed. Content editors learn. They already have stuff they need to learn, and this is a very simple one that they learn pretty damned quickly anyhow.

      Is it necessary to strip “unregistered shortcodes” when we can just assume that they are part of normal sentence structure? So some websites will display the odd shortcode when its code [e.g. plugin] isn’t active; so what?

      2. self-closing shortcodes don’t look like they are self-closing until they don’t have a closing shortcode

      … which is easily resolved by demanding that they close with /] similarly to XML and HTML.

      To me, this is THE performance problem, and it is easily resolved by not allowing self-closing shortcodes without a terminating /].

      Now, bear in mind also that cleaning up old code might require changing content in:

      • wp_posts.post_content
      • wp_posts.post_excerpt (some plugins / themes use it for rich text)
      • wp_postmeta.meta_value
      • wp_usermeta.meta_value
      • custom tables
      • code (e.g. directly calling `do_shortcode(‘[product …]’);`

      And it has to take into account all sorts of edge cases.

      There’s only so much auto-upgrading that can be achieved. The changes proposed WILL BREAK WEBSITES. Even simply requiring self-closing shortcodes to have /] WILL BREAK WEBSITES. All I can see coming from this proposal is pain. “There will be a great gnashing of teeth” etc. 🙁

    • Jean-Francois Arseneault 1:47 am on September 3, 2015 Permalink | Log in to Reply

      Pardon the pink-colored glasses for a second, but why does it seem that our only choice moving forward, to improve the state of shortcodes, is to change *how* shortcodes work, and not *if* there isn’t a better / newer way to accomplish the goal of inserting content into a page ?

      In the spirit of seing the forest from the trees, why are we still trying to desperately manage mixed-source content using text-based shortcodes ?

      Why are we still using a paradigm where the main content area (if talking about posts) is still an area where we can add content, but also perform layout decisions *within* the content? Where’s the separation of concerns in this way of doing things?

      I think Page Builders are the start of a better solution for managing all these different contents within a page, but perhaps forcing plugin authors to expose their functionality into native widgets and ‘page builder elements’, vs letting them add shortcodes anywhere to render content might be an approach that is easier to grasp, from a UI perspective, for *end-users*.

      Sure, developers, integrators… we can handle everything you throw at us, but in the end, the sites are built for end users, for real-life applications and business purposes. End users are the ones that matter.

      Simplicity always wins.

      • Peter Wilson 2:19 am on September 3, 2015 Permalink | Log in to Reply

        The Shortcake feature plugin covers providing a UI for shortcodes.
        https://wordpress.org/plugins/shortcode-ui/

      • Chris Van Patten 2:30 am on September 3, 2015 Permalink | Log in to Reply

        Although I’m generally in favour of this proposal, I think this merits some consideration too. Is there potentially a way to eliminate shortcodes — as in a manually entered bit of pseudo-tag-like code — altogether? I think Shortcake is an interesting approach, but it still depends on the shortcode API in the backend. What if shortcodes were deprecated, and replaced with an entirely new concept for inserting dynamic “chunks” of content that doesn’t fall back on parsing through plaintext?

        Replacing shortcodes is probably a much-longer term play, but it does mean that instead of patching the old they can be replaced with something newer, more flexible, and more in tune with the direction core has been moving already.

      • John Teague 1:53 pm on September 4, 2015 Permalink | Log in to Reply

        I totally agree. We need to be thinking of an intuitive replacement for shortcodes altogether. That provides incentives for developers and users to make the change.

        I really appreciate the hard work by @Robert Chapman and others putting a lot of effort into solving the increasingly unsustainable shortcode system. Sadly, I think the proposal in its current form, will confuse already error prone users, and likely break a lot of sites.

    • Chris Van Patten 2:36 am on September 3, 2015 Permalink | Log in to Reply

      As a supplement to my other comment (which was a nested reply), I’m generally in favour of this proposal. I think there’s merit in discussing completely new alternatives to shortcodes, but that’s a semi-separate issue.

      Like many others, I’m not sold on the suggested syntax, but it’s obviously still early days.

      One other thought: why not make escaped code the default, and make unescaped the exception with its own special syntax? Are there backcompat reasons that wouldn’t work? Seems like (potentially) an easy win for security.

    • Andrew Nacin 3:03 am on September 3, 2015 Permalink | Log in to Reply

      Thanks for your feedback, everyone. Developers, truly, honestly, really: Thank you for paying attention. I often feel that requests for comments usually fall on deaf ears, and I know I am not the only committer who feels that way. This has been one of the most substantial and productive threads on make/core I’ve seen, and it’s encouraging to see.

      Here are some thoughts on the proposal:

      • This was clearly labeled a “draft”. I am glad many of you noticed that this was not a decree, but a call for feedback. (And thank you again for obliging.)
      • The vision of shortcodes should be something like “it should be easy and intuitive for non-technical authors to embed and enrich their posts” (I have cribbed this from @matt).
      • This vision does not actually mean that shortcodes are the solution. I actually have a strong dislike for shortcodes. I think they are a terrible user experience, and that even something like Shortcake is still only a marginal improvement.
      • Some changes to shortcode parsing is absolutely necessary in order to keep WordPress secure. Yes, this is about security, not just general sanity.
      • The proposed syntax significantly clashes with the proposed vision. And given all of your feedback, we’re clearly going to have to go back to the drawing board. Please note that we still need to do something, but maybe we can think further outside the box.
      • And for goodness sake, PLEASE STOP RELYING ON HTML IN SHORTCODE ATTRIBUTES. This also does not align with the outlined vision, either, and it’s what largely precipitates proposals like these.
      • Thank you @miqrogroove for your incredibly hard work on this. Honestly, folks, many of you simply cannot see how much work Robert has poured into shortcodes over the last year, as it takes place in private security discussions. He’s been a tremendous asset to the project.

      I’m looking forward to future make/core threads getting this much high quality feedback. Now if only it can also carry over to more Trac tickets. 😀

      • Solinx 9:21 am on September 3, 2015 Permalink | Log in to Reply

        Thanks!

        “I’m looking forward to future make/core threads getting this much high quality feedback. Now if only it can also carry over to more Trac tickets.”

        I think this could be helped by making Trac more accessible and by providing a better overview of what is currently going on.

        1. Trac
        When I first heard of Trac I had to get there through google. The “Get Involved” page didn’t help at all. Why not add a link to both the tickets dashboard and the ‘next release’ tickets list at https://make.wordpress.org?

        And why not link to related tickets, github repo’s, slack groups, etc. at the bottom of these blog posts? That makes it a whole lot easier to get to the right places to get involved.

        2. Progress overview
        It is difficult to keep track of all that is going on. The blogs provide updates, but not an overview. Perhaps you could introduce something such as: http://dev.modern.ie/platform/status/adownloadattribute/

        Also here you could link to the related tickets, github, etc.

      • Nick Haskins 12:11 pm on September 3, 2015 Permalink | Log in to Reply

        “Now if only it can also carry over to more Trac tickets.”

        The problem with Trac, for me at least, is that it’s not part of my daily workflow. I’m a bit intimidated by Trac, not to mention making patches. I take part in conversations on Github all day, because that’s how we manage code and issues for my job. I suspect that if these conversations were on Github, with PR’s being allowed there would be a lot more participation. After all, I don’t know any developers outside of WordPress who use SVN in their workflow.

        • NateWr 2:08 pm on September 3, 2015 Permalink | Log in to Reply

          Thanks for outlining this response @nacin. I’d second thinking outside the box. I suspect a lot of the problematic uses of shortcodes stem from people trying to plug gaps in other kinds of functionality (layout, class assignment, conditionals). Maybe solving some of those problems would take a bit of heat off of shortcodes, and make them a smaller attack vector. Or maybe not. Can’t say I really understand the security implications of shortcodes.

        • John Teague 2:33 pm on September 3, 2015 Permalink | Log in to Reply

          Totally agree with you on this Nick. Think how moving to Git would invite solid developers to submit great fixes and features? It’s not that a lot of us can’t use Trac/SVN. It’s just who the hell wants to.

          • Helen Hou-Sandi 4:06 pm on September 3, 2015 Permalink | Log in to Reply

            I imagine that a lot of the workflow folks think of centers around GitHub and pull requests as opposed to Git itself, but just a note that there is a proper Git mirror that many people patch against, including a number of committers. https://make.wordpress.org/core/2014/01/15/git-mirrors-for-wordpress/

            • KalenJohnson 4:49 pm on September 3, 2015 Permalink

              Yes that’s very true, it is Github itself which has become such a boon for developers to be able to quickly and easily find issues, discuss them, and create pull requests. Trac seems very antiquated by comparison, and submitting patch files is not something the rest of the open source web development world does that I’ve seen.

              I can understand how large the move to Github would be for WP, but don’t underestimate that it could really remove WP from the silo it’s in and open up to better quality issues, more issue discussion (as Nacin is looking for), and possibly more patches as people are much more familiar and willing to make pull requests.

            • J.D. Grimes 6:15 pm on September 3, 2015 Permalink

              Yes, the workflow on Trac is needlessly complex. I’d +1 moving to GitHub, or something where we can have PRs instead of patches.

            • John Teague 1:58 pm on September 4, 2015 Permalink

              My bad. I meant github.

      • dlouwe 1:07 am on September 4, 2015 Permalink | Log in to Reply

        “This vision does not actually mean that shortcodes are the solution. I actually have a strong dislike for shortcodes. I think they are a terrible user experience, and that even something like Shortcake is still only a marginal improvement.”

        Very much this. Users shouldn’t have to learn syntax to make use of shortcode-like functionality. They can already use the Visual editor to insert and manipulate HTML without ever knowing what HTML is, and that should be a clear goal for whatever shortcodes end up being in the future. It doesn’t much matter what the new syntax looks like if the only people who are going to be editing it directly are people who have business doing so. As a developer, I don’t mind jumping through some extra hoops to make sure my users have an easy and intuitive experience!

      • raamdev 3:23 am on September 4, 2015 Permalink | Log in to Reply

        > The vision of shortcodes should be something like “it should be easy and intuitive for non-technical authors to embed and enrich their posts”

        I agree. That makes this a good time to redesign shortcodes from scratch, to rethink how we would create something that provides the same flexibility and power of the existing Shortcode API while also more accurately fulfilling the goal of being something that is “easy and intuitive for non-technical authors to embed and enrich their posts”.

        The ideas that came to mind while reading your comment were: Do shortcodes need to be something that is entirely configurable from within the post editor? What if they were instead something that pointed to a configuration, a configuration stored in the database that allowed a plugin or theme to determine what should be returned? (I’m thinking something like `[{{shortcode id=”plugin_slug_123″}}]`, i.e., something like a shortcode with a single attribute.)

        Then a new API could provide plugin and theme developers with a way of building a UI that would allow users to configure ‘shortcodes’ (the configuration being stored in the database, e.g., as `plugin_slug_123`), similar to the way the Plugin API allows plugins to register meta boxes for post types.

        My feeling is that creating something that is both easy and intuitive for non-technical users will require getting rid of syntax as much as possible and replacing that with a UI.

        Building a whole new feature to replace the Shortcode API would also make it easier to transition away from the current Shortcode API: Eventually WordPress Core could disable the old Shortcode API by default (allowing site owners to re-enable it, perhaps using an MU Plugin, with the knowledge that it’s no longer recommended) while plugin and theme developers could update their code to support the new feature (their motivation to do so will be that future versions of WordPress won’t have the old Shortcode API enabled).

    • markkaplun 5:48 am on September 3, 2015 Permalink | Log in to Reply

      [might not be needed after so many comments but…] I want to also voice my opinion against the proposed. I can see how software developer *might* be happy with this kind of syntax but this is very far from freely written text. The need to balance the number of braces is a no go, it is just too easy for my cat to erase one of them and making me waste an hour trying to figure out what is wrong because it will not look obvious. I actually don’t feel too strongly about the syntax of the complex variants, as they require some kind of programming state of mind, but the self closing one should be extremely simple.
      In addition, The use of the $ sign for anything in this context is ridiculous. In free written text it indicates some reference to money or value and the reason it used in regexp is because it is used in free text but rarely in programming.

      Proposals
      1. Whatever is the final syntax, syntax highlighting for shortcodes should be part of both the tinymce and the “text” editors. This will make it easier to identify shortcode and find broken ones.

      2. Since this will be such a big chagne, might as well fix [the lack of] permission checks. On a multi author site everyone can use all shortcodes. There is no reason why a contributor should be able to insert a contact form via a short code. Sure the editor is supposed to review and remove it, but there is no point in giving the ability in the first place.

    • umchal 6:21 am on September 3, 2015 Permalink | Log in to Reply

      The biggest problem of shortcodes from my view is that they remain in the posts even after the plugin which uses the shortcode gets uninstalled.

      I suggest using the HTML comment syntax.

      • HTML

      This way, users do not have to concern about removing them form the post. I suggested this a while back (https://wordpress.org/ideas/topic/commentcode-alternative-to-shortcode).

    • Florian Girardey 7:04 am on September 3, 2015 Permalink | Log in to Reply

      How can you possibly want to break a lot of websites with 4.7 with end of support of the current shortcode syntax and absolutely rejecting the idea of using a recent version of PHP…

      I like the idea of improvement and this roadmap is a good idea but i don’t understand the logic behind the WordPress team right now.

      • BrashRebel 12:43 pm on September 3, 2015 Permalink | Log in to Reply

        If you want to understand, read the whole discussion. In particular @nacin‘s comment https://make.wordpress.org/core/2015/09/02/shortcode-roadmap-draft-one/#comment-27484

        • Florian Girardey 5:51 pm on September 3, 2015 Permalink | Log in to Reply

          I understand why this roadmap is necessary and i absolutely agree.

          But the arguments advanced for such a changement are closely the same that a PHP version update, i don’t understand how the WordPress team can accept this roadmap but also refuse a PHP version update. That’s all

          • Ipstenu (Mika Epstein) 7:18 pm on September 3, 2015 Permalink | Log in to Reply

            While I understand the comparison, it’s a logical fallacy.

            A minimum requirement is just that. A minimum. WP’s minimum is PHP 5.2.4 and MySQL 5.0.15. We still say mod_rewrite is optional. Supporting older versions doesn’t make the codebase insecure in and of itself, and there is no external dependency update that can fix the current security issues with shortcodes.

            The point of this post is to ensure, when we do fix shortcodes, we do it in as graceful and as minimally intrusive a way as humanly possible. To not repeat the shortcode sins of the past. Your concern, while well meaning and valid, has a marked tendency to detract from that.

    • Cristian Antohe 7:10 am on September 3, 2015 Permalink | Log in to Reply

      We’re using Mustache template tags in both our plugins and I can tell you {{tag}} is in no way simpler then [tag]. Once you start adding {{!tag}} or {{^tag}} HTML {{/tag}} things start to go wrong fast. Users simply have a hard time understanding how that works.

      Start to mix up template brakes like [{{tag}}] and that’s just stupid. It’s un-intuitive and will give a lot of headaches to ALL plugin developers who also actively support their plugins.

      • Cristian Antohe 7:19 am on September 3, 2015 Permalink | Log in to Reply

        With these more unique delimeters, we will be able to process registered shortcodes more efficiently because we will not have to search for them by name. Unregistered shortcodes will have more consistency because we can find them more accurately.

        The new syntax is making it harder for users to use shortcodes for the sake of developers who have a hard time working around the constraints that come with using [tag].

    • leemon 7:25 am on September 3, 2015 Permalink | Log in to Reply

      Sorry to say this, but that new syntax is hideous. If this goes forward as it stands, I predict major breakages, lol

    • Mike Schinkel 8:08 am on September 3, 2015 Permalink | Log in to Reply

      Reviewed all the comments and did not see anyone suggest what @cfc suggested on @miqrogroove‘s blog post which just use Custom HTML Elements:

      http://www.miqrogroove.com/blog/2015/shortcode-v44/#comment-515

      They seem perfectly suited for this:

      http://w3c.github.io/webcomponents/spec/custom/#custom-element-lifecycle

      What would they look like?

      Old Shortcode: [foo]bar[/foo]

      New Shortcode: <wp-foo>bar</foo>

      One I saw that idea by @cfc everything else seemed like painful overkill. Besides, shortcodes are very closely aligned with the goal of the custom HTML element proposal with shortcodes being server-side and custom HTML element being client side.

      • Benny 5:56 pm on September 3, 2015 Permalink | Log in to Reply

        I like the idea of custom HTML elements / web components replacing shortcodes. But that still means we can’t phase out the old shortcode syntax because it would break so many sites. How would you suggest to solve that problem? Maybe by adding you’re “You are doing it wrong” notices (for ever)?

      • Michael Davis 10:53 pm on September 3, 2015 Permalink | Log in to Reply

        I hope this gets more discussion. Particularly, what would be potential shortcomings of this method compared to what the new syntax would bring.

      • Knut Sparhell 1:18 am on September 4, 2015 Permalink | Log in to Reply

        I like this idea of custom HTML elements. Registered elements gets executed either server-side or client-side. An advantage is that it’s quite obvious that such elements can’t be used in attribute values, and attributes can’t contain such elements.

        What has to be solved is the slow deprecation of the old shortcode syntax, and if old shortcodes should be replaced upon upgrade, at some point.

        I also look to the more tag and page tags, that are HTML comment tags. When shortcodes was first introduced I was surprised the HTML comment tag was not considered.

        Custom HTML elements seem very future proof, outright elegant, even if the draft never becomes a recommendation or they will never be supported by browsers (for the client side types).

      • webaware 11:09 pm on September 4, 2015 Permalink | Log in to Reply

        Much love. Was thinking about how NextGEN Gallery does its galleries these days (as a decorated image tag it parses out), but this sounds even better.

      • Braad 12:57 am on September 5, 2015 Permalink | Log in to Reply

        +1 for web components. Great idea Mike!

    • chrishoward 8:24 am on September 3, 2015 Permalink | Log in to Reply

      Sorry, I see these new syntax changes as ridiculous.

      The current syntax is untidy and cumbersome enough, and now you want to throw all these braces in it? We should be making shortcodes easier to use, not harder.

      Have you forgotten the user totally?

      These are great for coders. We love “mustaches” and parameters!

      But I pity the user having to key in a multiple-enclosured shortcode.

      Personally, I believe shortcodes should be invisible to the user, just as HTML is. Show the code in text mode, sure, but not in WYSIWYG. In WYSIWYG mode shortcodes should only be entered via a button and form, and then represented in full if possible (like the gallery shortcode) or by a clickable icon.

      For day-to-day users, this new syntax is a big step backwards.

    • asarosenberg 8:45 am on September 3, 2015 Permalink | Log in to Reply

      I agree with previous comments that this suggestion is very bad for users. People can understand shortcodes if they are somewhat familiar with HTML or online forums but the suggested syntax is unreadable to anyone who is not a programmer.

      An easier way to solve the problem with wanting to use brackets for other things is to allow people to escape brackets for example by ignoring brackets prefaced by backslash, star or whatever char that makes sense. If you want WordPress to be a CMS and not just a blog tool, shortcodes are more important than brackets in text.

      As for deprecation and conversion consider that there are shortcodes in meta fields. Many themes use page templates with multiple wp_editors and shortcodes. To avoid massive sudden breakage the old format should work for at least a year or two after the changes have been implemented and announced.

    • Arunas Liuiza 9:07 am on September 3, 2015 Permalink | Log in to Reply

      While I do get the technical reasoning behind this proposal, but I am really worried changes like that will soon make WordPress only usable for people with PhD in Computer Science. Seriously, I think something like 简shortcode简 would be easier to use and understand than the proposed syntax for the average WordPress *user*.

    • andreasnrb 9:27 am on September 3, 2015 Permalink | Log in to Reply

      The first thing I think that should be done before even touching or discussing syntax is to clearly define the purpose of shortcodes and come up with appropriate use cases.
      And just to ignore my first statement. Why not go in other direction and introduce custom HTML tags instead? You register your tag, what attributes it accepts with sanitation callbacks.
      <wp-button href=””>Click me</wp-button>
      register_tag(‘button’,array(‘href’=>’esc_url’))
      Even makes it possible to integrate with HTML5 custom tags later on.

      • andreasnrb 9:31 am on September 3, 2015 Permalink | Log in to Reply

        Yes I know it encodes when written in the HTML window. So it should be combined with a insert button in the menu.

    • HasinHayder 11:56 am on September 3, 2015 Permalink | Log in to Reply

      Now you need a Ph.D in CS to use WordPress 🙁

    • FolioVision 12:11 pm on September 3, 2015 Permalink | Log in to Reply

      I see that doing this will open new possibilities, but I think the basic shortcodes should stay without change:

      Self-Closing: [shortcode]
      Attributes: [shortcode attr1=”value1″ attr2=’value2′ “value3” ‘value4’ value5]
      Enclosing: [shortcode] HTML [/shortcode]
      Escaped Code: [[shortcode]]

      If you want to add more features (like nested shortcodes here), I think yout can do it without breaking too much stuff:

      Multiple Enclosures (this one should work if they could look for closing tag for [shortcode] – a regex won’t suffice though): [shortcode] HTML [encl2] HTML [encl3] HTML [/shortcode]

      I’m not sure about the rest though, but these special cases could probably use the new syntax:

      Stripped Unregistered Code: [{{randomthing}}]
      Stripped Unregistered Enclosure: [{{randomthing}$] Content also stripped. [${randomthing}}]
      Stripped Empty Code: [{{ }}]
      Ignored Orphan: [{{shortcode}$]
      Ignored Orphan: [${shortcode}}]
      Ignored Orphan: [${encl2}$]
      Ignored Context: [{{shortcode
      }}]
      Ignored Context:

      Do you see any issue with keeping the basic shortcodes unchanged?

    • jakeparis 12:46 pm on September 3, 2015 Permalink | Log in to Reply

      I think that the new syntax is difficult and burdensome to the average user. I would suggest:

      a) Make at least the basic “Self-Closing” shortcode operate without the brackers (just as it does now).
      b) The “Enclosing” style should close with a slash, not a $. There is not precedent for using a $ as a closer in any programming language that I know of — therefore that makes this difficult to remember. The slash it a much better choice.

    • J.D. Grimes 1:26 pm on September 3, 2015 Permalink | Log in to Reply

      I do not know how serious the security implications might be, but perhaps we could avoid removing the old syntax at all, and just disable it on new sites, like we’ve done with other deprecated features.

      I think the real lesson here though, is that shortcodes don’t need to be fixed, they need to be removed. We would still have to figure out how to maintain some amount of backward compatibility for old sites, but it might be more productive to work on a broader roadmap that would actually make shortcodes obsolete. Others, including @nacin, have hinted at this. Is a syntax overhaul really necessary as an intermediate step? Is it even plausible as an intermediate step? What if instead we approached it like this:

      1. Bring Shortcake (or some other “shortcode UI”) into core. This would still use the shortcode API, but would provide a UI for it so the user never has to directly manipulate the shortcode string.
      2. Start using an alternate, secure method of indicating the insertion points of the different content bits when this UI is used.
      3. Disable the shortcode API on new sites.
      4. Let the old sites get slowly broken and hacked over time, or break them ourselves all at once with an update.

    • JakePT 2:28 pm on September 3, 2015 Permalink | Log in to Reply

      I’ve got a couple of things I want to say. Firstly, I really need clarification on what these security issues with Shortcodes are. I read the post on the 4.2.3 change and it didn’t really help. Maybe it’s because it’s happening in an area of WordPress development that I just haven’t stepped into, but I’m having trouble understanding what contexts Shortcodes are a problem in. Isn’t their output defined by the plugin? Why isn’t the security problem with the *plugin*? Aren’t authors the only ones who can use them anyway? What could they do with them that’s so bad? I’m having trouble accepting such a drastic proposal when it doesn’t come with a clear rationale.

      But my biggest problem by *far* is that this would be so so so much easier to swallow if there was *any vision at all* for the future of the content editor. A single TinyMCE field just doesn’t cut it anymore. What happened to the content blocks idea that came up around 3.8/3.9? WordPress needs to address the problem of editing and creating more complex layouts, not breaking functionality that is only being abused because of this lack of vision and progress.

      Shortcodes suck, but don’t make them suck more without something better to replace them.

    • Stefano Aglietti 3:02 pm on September 3, 2015 Permalink | Log in to Reply

      I don’t know if someone already rised the question, but apart that making shortcode delimiter like the ones propose make user in a complicated situation.

      Maybe english people don’ìt know that { and } are not present in alla keyboards directly as the [ and ] are, in Italian keyboard [ and ] are generated using ALT-GR+è or ALT-GR++ and keyboards show ALT-GR basic sign like €@## and [ ].

      To get { and } you net to type ALT-GR+SHIFT+è and ALT-GR+SHIFT++. Most people don’t know about this combination (i think only who write code does) and the parentesys are not written on any key.

      This change would be a a really complication for italian users and i supect a lot of non english keyboard suffer from the same problem. Please when you think to change something like this for end users, use ALWAYS the KISS Method, [{{ is not KISS at alli suggest adding after the square bracket somthig that is awailable on alla keyboards like ! or ? or % or & but for sure nothing is really hide in system and that 99% of people ignore that existes or how to get it leke curly braces

    • Curtiss Grymala 3:04 pm on September 3, 2015 Permalink | Log in to Reply

      First of all, let me say “Thank you” for beginning the process of thinking about this issue.

      As we begin to move forward, has anyone thought about trying to identify where & how shortcodes are currently used, then trying to come up with the best solution for each of those situations? Maybe a shortcode isn’t the best solution for all of the situations in which it’s being used; it’s just the only viable solution we’ve had up until now.

      The bottom-line is, we are still going to need placeholders of some sort within WordPress. There are perfectly reasonable uses of placeholders, regardless of the syntax they use.

      Some are within HTML tags, such as <div class=”[my-custom-class extras=’one-third first’]”> or <img src=”[some-img-url]” alt=”This is my awesome image”/> or <a href=”[my-home-page]” title=”Return to [my-site-name]”>. There are perfectly valid & viable reasons to need to use some sort of placeholder inside of HTML (even if it’s not necessarily inside of the Visual Editor). If we stop allowing these sorts of things, then we’re going to end up with more plugins implementing silly things like [custom-html tag="a" href="{{home-page}}" title="Return to {{site-name}}"]Go Home[/custom-html] which will introduce potentially even bigger security issues and more usability issues for the users.

      Some stand on their own, such as We currently have [number-of-users] users on the site.

      Some output HTML, such as .

      Some are, IMHO, poor attempts to get around security restrictions in the editor, such as [iframe] (there are many plugins that implement this, by basically allowing the editor to enter an HTML tag in normal syntax, but replacing the < and > with square brackets).

      Some need to be nested in order to work properly, such as something like:
      [accordion]
      [accordion-title]My accordion item title[/accordion-title]
      [accordion-content]Some content to appear within the accordion item[/accordion-content]
      [/accordion]

      Some are also completely related to formatting the content, attempting to “fix” the problem of only having one content area, and trying to keep editors from needing to enter HTML on their own, such as the various implementations of column shortcodes (this is a very similar reason that implementations like the accordion shortcodes above exist, as well).

      There are many more perfectly valid & useful implementations of the current shortcode API.

      Each of these may have a completely different and much better solution than the others.

      Finally, as WordPress has evolved very, very far from a simple blogging platform, we need to stop thinking that post titles and post content are the only features that people are using.

      There are plenty of places where shortcodes might appear within the Options table, possibly even inside of serialized data. For instance, shortcodes could, potentially be used within widgets or theme options.

      Shortcodes could absolutely be used, within plain-text or serialized data, within the postmeta table (there are a ton of places that custom fields on a post could include shortcodes) or possibly even the usermeta table.

      The problem I see with the original discussion we had after 4.2.3, and the problem I see with some of the comments on this post, are that there still seem to be people saying “You shouldn’t have used shortcodes to do that; that was stupid, so it doesn’t matter if we break them.”

      The thing I appreciate about this original post, regardless of the fact that the proposed solutions are certainly more complicated than they should be, is that the core team is back to at least trying to solve the issue rather than continuing to feel like it’s okay to break sites just because they disagree with the use-case.

      Again, though, I want to thank @miqrogroove for bringing this to the table and beginning the process of trying to move forward.

    • Stanislav Khromov 5:01 pm on September 3, 2015 Permalink | Log in to Reply

      I’d like to echo the dismay most people feel with this proposed roadmap, and also stress that this can cause a fragmentation in the WP ecosystem.

      Consider that it’s easier than ever to not keep WordPress up to date with the latest version – security patches are backported all the way back to 3.7 and minor security fixes get applied automatically.

      By introducing big, “scary” changes that people feel uneasy about (which the comments prove include even seasoned developers), many will just not update. Consider the kind of fragmentation Python faced after they introduced syntax changes in version 3. Most people just didn’t want or need these changes. I feel the same way about this proposed roadmap.

    • Grant Palin 5:07 pm on September 3, 2015 Permalink | Log in to Reply

      One other gripe I have with the current shortcode syntax is that it sometimes confuses Markdown. Particularly when using within posts authored using WP-Markdown plugin. There’s a workaround, but still.

    • Jacob Santos 8:59 pm on September 3, 2015 Permalink | Log in to Reply

      Why was shortcodes added and what problem did they attempt to solve? What was their original purpose?

      The reason it is a bad idea is that it neglects the reason shortcodes were created and why they have the syntax they do. The history for the most part, I believe, was based in part on BBCode. If Handlebars existed and was more popular, then who knows.

      You will always have a security issue. Tough, but true. Introducing a wholly obtuse syntax will not change that and will make it more difficult for users and developers. Most likely you will just see an influx of plugins that put back the same behavior or those who move away from shortcodes altogether. It will also be a slap in the face once another security flaw is discovered.

      Unless you wish to remove the security issue by pushing people away from using shortcodes completely or creating their own solutions of various quality for security.

      The timeline is far too short. If you go back as far as 3.7, then you must follow that pattern. Deprecate it and update it if you can, but you must support the current until 2018 or 2 years after 4.4 is released.

      • Jacob Santos 9:06 pm on September 3, 2015 Permalink | Log in to Reply

        Lest my argument be misconstrued; I am saying that any syntax change is a bad idea. Solve the security problems with the current or work around them. The bed is made, you have to deal with it.

        Biggest problem I have is that it looks like you are creating a new template engine. There are already those that exist. It would be better if you used them as several have already been audited by security professionals or could be audited by security professionals. They also have a community and those that support it outside of WordPress.

        If you are going to introduce a template engine to WordPress posts, which in and of itself is charge to be warily, you are better off going with something developed by professionals.

        Shortcodes were supposed to be an easy replacement for plugins without a lot of features. When did it become the end all, be all to implement a feature?

        If anything, the RFC should formally suggest where shortcodes should and should not be used, deprecate those usages and at a later date, remove support for them. If advanced templating is required, move to an actual established templating engine. Oh my god, I can’t believe I’m writing this.

        I must be in shock, because the horror of this is just beyond comprehension. I can’t imagine the discussions or the problems that lead to, “hey, why don’t we create a template engine for posts?”

        I have to conclude that this is an early or late April Fool’s joke.

    • chriscct7 2:44 am on September 4, 2015 Permalink | Log in to Reply

      As an update, the team working on the Shortcodes Roadmap, has been working on a new draft proposal which will be posted to make at some point in the near future, which is significantly different than Draft 1. Also, there will be a weekly meeting that will be announced during the same post.

      • John Teague 2:10 pm on September 4, 2015 Permalink | Log in to Reply

        Thank you Chris. And to everyone working to find a solution on this. Hope people realize how much time and effort has to go into this.

    • Ustimenko Alexander 4:45 am on September 8, 2015 Permalink | Log in to Reply

      Users must not pay for developers weakness. If you want something stron-n-cool, then incorporate twig syntax here, but this all is for DEVELOPERS an dnot for users. This syntax is super-extra-bad-thing for end user.

    • renzms 11:57 am on September 17, 2015 Permalink | Log in to Reply

      I’d also like to join in to say that this is more of a step sideways rather than forward/backward with this proposed roadmap.

      Whether a seasoned developer, or just a regular user with no coding experience, these shortcodes aren’t in any way user or developer friendly.

      I understand this is a draft, and a new proposal will be made, especially after all the sentiment seen here. So thanks for the warning, and not changing anything right away as drastic as this. I know this proposed change is due to a security issue, but it causes more problems down the road that developers/users just might not use WordPress anymore to begin with. I do hope a simpler and more unified solution is found.

    • Tom Belknap 8:08 pm on December 15, 2015 Permalink | Log in to Reply

      This isn’t quite such a draft when you consider that 4.4 is already here and already, it’s messing up my shortcodes. Specifically the nested ones.

      This might be the worst proposal since the rightfully-maligned “Settings API,” an innovation the devs needed to ditch in favour of the Customizer. How error-prone is this syntax?

      `[{{shortcode}$] HTML [${encl2}$] HTML [${encl3}$] HTML [${shortcode}}]`

      I thought this double curly brace business must have meant that you were calling out the things we were supposed to change. Then I saw you were serious about it.

  • Daniel Bachhuber 8:16 pm on August 31, 2015 Permalink
    Tags: , , , , shortcodes,   

    Shortcake (Shortcode UI) chat summary – August 31st, 2015 

    Present: @danielbachhuber, @matth_eu

    Logs: https://wordpress.slack.com/archives/feature-shortcode/p1441047764000146

    Next chat: same time and place

    Next release: v0.6.0 – Tuesday, November 3rd

     
  • Andrew Ozz 8:30 pm on August 30, 2015 Permalink
    Tags: , shortcodes   

    Shortcodes roadmap 

    The Shortcode API is well loved by developers. Thousands of plugins use it for many cool features.

    Unfortunately it wasn’t documented well when it was added. Even now the documentation is somewhat incomplete. The API was also very permissive, allowing many unintended user cases.

    The result of these early mistakes is that there are plugins which use shortcodes in very unintended ways: mixed with HTML tags, nested several levels deep, with huge attributes, etc.

    What are shortcodes:

    • With one word: placeholders.
    • Convenient way to add dynamic content inside post_content at run time.

    What shortcodes are not:

    • A way to conceal user input in post_content.
    • A way to store any type of data in post_content. There are better places and methods for that, like post meta.

    Shortcodes “live” in the same context as HTML tags. They should obey the same rules. Also — no interlinking between HTML tags and shortcodes. Think of the [ and ] being equal to < and >.

    Both <p title="<b>my title</b>"> and [paragraph title="<b>my title</b>"] should be illegal for the same reasons. Also <p title="[my-span]">. I know the current shortcodes parser mostly supports these, and some plugins use them, but that will probably need to change “for the greater good”.

    There is simply no good reason for trying to support mixing of shortcodes and tags with the current parser. These cases take longer time and more resources on every front-end page load. They require much more complex code to sanitize and ensure they are safe to run. If plugins cannot operate without mixing shortcodes and HTML tags, they will eventually have to implement their own placeholders and parsers, and ensure all data is sanitized properly. This will require a lot less time, effort and processing as the plugins would know what to expect.

    We’ve been talking about this with @miqrogroove for a while now. There are several very interesting suggestions in his post on the subject: http://www.miqrogroove.com/blog/2015/shortcode-v44/.

    We both agree that we need to create shortcodes roadmap, similar to the taxonomy roadmap. This will allow us to fix the shortcomings in the Shortcode API and clear the path for future improvements.

     
    • Scott Fennell 10:03 pm on August 30, 2015 Permalink | Log in to Reply

      Thank you for your work on this topic. There is one use-case that I have not seen come up yet in these discussions. Example:

      [my_shortcode before="<div class='my_class'>" after="</div>"]
      

      Where the output would be

      <div class='my_class'>whatever</div>
      

      That’s a situation I have across hundreds of sites. Does this use-case stand in jeopardy?

      I understand the need for a stricter parser for the greater good as you say, but count me in favor of doing this in a way that leaves a possibility for backwards compatibility.

      • Scott Fennell 10:05 pm on August 30, 2015 Permalink | Log in to Reply

        Sorry, my example got munched. Here’s what I meant to ask about:

        http://pastebin.com/1CMN731H

      • Ipstenu (Mika Epstein) 11:37 pm on August 30, 2015 Permalink | Log in to Reply

        Scott, as I understand it yes, that would be in jeopardy. Putting HTML output in a shortcode is something that should always be avoided.

        There is simply no good reason for trying to support mixing of shortcodes and tags with the current parser.

        That’s what he means. Mixing shortcodes and html tags is tricky, dangerous, and complicated.

        Better would be:

        [my_shortcode class=my_class]

        And then if you wanted to have an option of span or div, you’d make that a variable as well in the shortcode.

        [my_shortcode type=span class=my_span_class]

        Structural shortcodes are always weird, though, and kind of a janky way around the usual end goal which is “I need to put divs in my post and I can’t use pure HTML because if I swap to the GUI editor they’ll get yanked.”

        • Scott Fennell 4:10 pm on August 31, 2015 Permalink | Log in to Reply

          Thank you for the explanation, that’s very helpful. Two things:

          1) My company has about 1,200 live blogs/client websites that are using this use case. It seems like all of those use cases are going to break if we don’t intervene somehow. Would you agree? If so, do you have any advice for how I might do that? The best idea I can think of is to update the plugins that offer those shortcodes and, at the same time, do some kind of bulk search and replace across all of the posts in all of those blogs, perhaps using WP-CLI. This would be a very scary and expensive endeavor for my organization.

          2) If we can manage to steer the shortcode roadmap in a way that is backwards compatible, that will help instill trust in automatic updates. If not, it will erode trust in automatic updates.

          Any thoughts on this?

          • Andrew Ozz 5:38 pm on August 31, 2015 Permalink | Log in to Reply

            Of course there will be ways to prevent breakage. Most likely HTML tags in shortcode attributes will be entity encoded. Instead of using the before attribute directly, you will have to check if it has the expected value and then decode it.

            If you change your shortcode, you can do bulk replace in all posts, or you can “transition” to the new syntax when posts are edited.

            This will never be deployed in an automatic update. That’s why we are making a roadmap, to plan how to best handle the changes. Most likely this will happen over several releases, with adequate time for plugins to adapt.

            • Scott Fennell 6:47 pm on August 31, 2015 Permalink

              Right on. What is the best way for me to stay abreast of this particular issue? Will all of the great work happening in core, it can be hard to track issues of particular significance.

            • Andrew Ozz 7:39 pm on August 31, 2015 Permalink

              Best would be to follow this blog and watch the “shortcodes” tag. You can also subscribe to the trac tickets when they get created.

      • Robert Chapin 2:31 am on August 31, 2015 Permalink | Log in to Reply

        This was one of the scenarios discussed on Slack, summarized on my blog as well. It seems one of the features we need to add on the roadmap is multiple enclosures. Currently there is only one enclosure possible for each shortcode. For the extra flexibility, we would introduce a way to enclose multiple, separate HTML snippets.

      • Marcus 1:20 pm on August 31, 2015 Permalink | Log in to Reply

        Hi Scott, I have a similar scenario in one of my plugins, but I think (someone, correct me if I’m wrong!) it’s quite easy to get around whilst adhering to the new rules:

        [shortcode before="<ul>" content="<li>placeholders</li>" after="</ul>"]

        can simply become:

        <ul>[shortcode]<li>placeholders</li>[/shortcode]</ul>

        same effect, and frankly it’s easier to read!

    • pingram3541 3:47 am on August 31, 2015 Permalink | Log in to Reply

      This has been a concern of mine for a long time. You have plugins that have gained a very large following over the last few years such as WYSIWG back and front end “composers” that truly should have been built around post meta in the first place but instead use (exploit) the shortcode API and often have a significant impact on performance and security.

      However I can’t help but feel this has happened because of the lack of design control built into the core. It’s very attractive to someone who may know html and css very well but maybe not so much with php or even js. Especially when they’ve had shortcomings with false expectations of what the post content editor was truly intended for. I’ll admit early on I had these growing pains myself.

      The question is, has that changed? With the recent release of 4.3 and the new formatting shortcuts from the core team it seems business as usual – post content is a place for writing, not coding, however comparing it to the vast numbers that use it for designer drag and drop plugins and complex shortcode scripts with tons of attributes, I’d say we’re not focusing on what the public seems to be demanding, tools like webydo, webflow and the like. So do we just let that be and shrug it off as “yeah well they’re doing it wrong, its on them if they want to do it that way and pollute their content” or do we find a way to meet in the middle, tailor to both and make this the most flexible and designer friendly CMS ever?

      • Andrew Ozz 6:10 pm on August 31, 2015 Permalink | Log in to Reply

        Yes, as I mentioned above the current situation is result from lack of documentation and too permissive parsing.

        Shortcodes are generic placeholders. They are suitable for many different uses, but not for everything. Plugins that need to store complex structured data in post_content can use many ways to do so.

        For example, a simple span with data-myattr=(JSON encoded string) will probably work better and be much faster than a tree-like structure of nested shortcodes. Of course there are other ways to do the same thing and stay HTML compatible.

        Additionally the plugin implementing that will know what to expect and can sanitize it much better and “cheaper” than core trying to sanitize a number of generic placeholders with attributes.

    • smartpixels 5:07 am on August 31, 2015 Permalink | Log in to Reply

      Shortcodes can use a JS API may be react.js so we can render these tags inside WordPress editor a lot easier.I totally agree that the shortcodes API have been misused a lot.

    • jadpm 7:18 am on August 31, 2015 Permalink | Log in to Reply

      I think there are a series of premises here that just do not follow, but I also assume that some decisions have been talen already and we are just being given the oportunity to at least be informed of them. Not that bad after all.

      For example, with the simple definition given to shortcodes (“placeholders”) it just does not follow that they should not be used inside HTML attributes, or that they are same-class structures as HTML tags. Even less if you truly think that the right place to store metadata is, well, metadata. How do you show off that metadata on a theme-agnostic way, if you do not use placeholders?

      Following the same idea of just being placeholders (one must understand that they placehold… something else, right?), why can’t you placehold a classname, a width, a link URL, that is actually stored in postmeta? Sentences like “There is simply no good reason for trying to support mixing of shortcodes and tags with the current parser. ” just prove that some people have not put that into thought for a very long time, or have even looked at how this is being used in the wild at all.

      Yes, we all know about security concerns, and that we must address the problems, even by restricting the usage to what is being suggested here: a closer API that sets the tone of what is accepted or not (something we did not have, and resulted in calling “unaccepted” out of the blue). But let’s try to be open and clean here. Yes, there are legitimate usages of shortcodes mixed with HTML tags, and yes, there are security reasons to not allowing that. Let’s restrict that because of the security reasons, not because there are no legitimate uses.

      Just my 2c as part of the so called ones that “use shortcodes in very unintended ways”, meaning… as placeholders.

      • Andrew Ozz 6:57 pm on August 31, 2015 Permalink | Log in to Reply

        Shortcodes are “generic” placeholders meant to be treated as text in HTML context. They were actually meant to be user friendly way to add “dynamic” content to post_content and to be “user editable”, i.e. the users should have been able to type shortcodes in both the visual and text editors. A lot of the currently used shortcodes are like that.

        Of course you can use placeholders in HTML attributes. I’m sure there are some user cases where it may be useful to dynamically change attributes on front-end page load. However shortcodes are not suitable for that as implemented. Seems best would be for plugins that require that to implement their own placeholders. If we really need global generic placeholders for use in HTML attributes, best will be to introduce another syntax for them.

        The reasons for trying to change the current shortcodes usage are not just to improve security and increase speed. We should also analyze the “most complex” use cases and perhaps extend the API or implement another means of making them work well.

        • jadpm 7:12 am on September 1, 2015 Permalink | Log in to Reply

          So, as I said before, there are legitimate uses for shortcodes as generic placeholders beyond what “they were intended” for, even when there was no word on what that meant at all, and it is being defined now by restricting its usage.

          Again, as I said before, at least now we are being given a warning, which is not that bad after all. You might want to review your logic, because describing shortcodes as generic placeholders and then telling that someone needing a global generic placeholder might want to implement their own syntax seems, well, confusing. I can not find any difference between something generic and something globally generic, sorry.

          As a result, lots (I mean lots) of developers are going the WP_Embed route: moving the parsing of custom shortcodes introduced by plugins to an early stage, way before the do_shortcode callback in the the_content filter. Because, well, that is what happens when a legitmate use case is called non-existent. I wonder how that will affect security. Last time I asked, I was told security concerns could not be discussed.

          I really hope we will keep on being updated on changes coming to this API. Thanks for that, anyway.

          • Andrew Ozz 8:32 pm on September 1, 2015 Permalink | Log in to Reply

            By “global generic placeholder” I meant a placeholder that is defined in core for use both inside and outside HTML tags. That wasn’t clear, sorry.

            The idea is that plugins that really really need dynamic HTML attributes would implement their own, non-generic placeholders which will be used only in HTML attributes and would be much easier to parse and sanitize.

            Yeah, copying how WP_Embed::run_shortcode() works is not a good way forward. That may work for now but will probably change soon.

            If you have any security concerns that involve current code, please email security@wotrdpress.org. This is the only “proper” way to address them. If you have “security hardening” ideas, feel free to open ticket(s) on trac.

            We are creating the Shortcodes Roadmap to address all of these concerns, and to welcome ideas, opinions and eventually patches from everybody that is interested in using this API in the future.

    • bduclos 8:29 am on August 31, 2015 Permalink | Log in to Reply

      When you say “There is simply no good reason for trying to support mixing of shortcodes and tag”, do you mean in the attributes? Will we still be able to put HTML inside shortcode content? e.g .

      [shortcode]<h1>Title</h1>[/shortcode]
    • Dave Navarro, Jr. 4:20 pm on August 31, 2015 Permalink | Log in to Reply

      I have dozens of plugins that use shortcodes from inside of HTML attributes. So if you plan on killing that, you need to come up with an alternative. I very much disagree with the premise that shortcodes should not be used inside of HTML tags. The relationship between HTML and shortcodes is a marriage made in heaven. And removing the functionality is shortsighted in my opinion.

      • Andrew Ozz 7:12 pm on August 31, 2015 Permalink | Log in to Reply

        Yes, having placeholders in HTML attributes may be handy in some cases. For example: when front-end page loads require different attributes. However shortcodes (as currently implemented) are not suitable for that.

        Plugins that need to regularly change HTML attributes on the front-end will generally be better off defining their own placeholders or using data-myattr="needed values" and replacing that on runtime. There are several options that are better than using shortcodes.

    • Peter Wilson 3:56 am on September 1, 2015 Permalink | Log in to Reply

      A more defined API is great – thanks Andrew & others involved. A hearty +1.

      I back any effort to avoid mixing HTML within shortcodes or shortcodes within HTML attributes. Like all things editor, approach it as a tool for users, not coders.

  • Daniel Bachhuber 8:11 pm on August 24, 2015 Permalink
    Tags: , , , shortcodes,   

    Shortcake (Shortcode UI) chat summary – August 24th, 2015 

    Present: @danielbachhuber, @goldenapples, @miqrogroove, @azaozz

    Logs: https://wordpress.slack.com/archives/feature-shortcode/p1440442841000013

    • We triaged the remaining issues for v0.5.0. Daniel will be picking them up over the next day.
    • A big project for v0.6.0 will be to go through core’s feature plugin guidelines and identify what we need to change to be valid.
    • Spent time discussing @miqrogroove summary of shortcode problems, and proposed solutions

    Next chat: same time and place

    Next release: v0.5.0 – this week (a bit overdue)

     
c
compose new post
j
next post/next comment
k
previous post/previous comment
r
reply
e
edit
o
show/hide comments
t
go to top
l
go to login
h
show/hide help
shift + esc
cancel
Skip to toolbar