Idea: Uniform Resource Identifiers as an Alternative to Shortcodes

The idea is that any objects embedded in a post’s content have their own home and should be seen as separate resources. Data would be stored elsewhere.

Examples

  • https://example.com/resources/contact-forms/email-ella/
  • https://example.com/resources/charts/php-versions-wordpress/
  • https://example.com/author/iseulde/
  • https://example.com/galleries/yummy-chocolates/
  • https://example.com/surveys/wordpress-editor-2016/
  • https://external.com/some-map-or-form/

Object Types

This approach could be good for:

  • Forms such as contact forms, surveys and polls.
  • Visualised data such as diagrams, charts, graphs and tables.
  • Lists of items such as a galleries, playlists and lists of posts.
  • Albums or manually composed collections of items where the presentation is uniquely different from a normal list.
  • Any content that needs to be reused or organised separately. Anything that can be re-sourced. 😉
  • Any external resources.

This is not good for:

  • Layout.
  • Text conversions.

Inspiration

  • External embeds work similarly in WordPress through oEmbed.
  • Images, audio and video are embedded by their URL in HTML.
  • Media have their own “attachment” post type in WordPress.
  • Many plugins already have a separate post type to store their data.
  • I’ve seen some news media do this already for things for like graphs (post, resource).

Benefits

  • A URL (or “link”) is a concept that is already familiar to many users.
  • URIs are designed for these sort of things. Think about images — it would follow the same paradigm.
  • The content is treated as a separate resource, and can be reused, just like shortcodes, but it forces separation.
  • WordPress allows you to create “pretty” semantic URIs, so the resource can be described for a better experience.
  • A cool side effect is that others could embed these easily on their site through oEmbed. WordPress already supports this.
  • If there’s a problem, the URL will act as a fallback.

Implementation

A quick way to implement this for WordPress 4.4 and higher is registering a new post type. This will handle most things, you just need to provide a custom embed template with the template_include filter. An external resource can be filtered with the Embed API and TinyMCE View API, even if it doesn’t support oEmbed.

Challenge: Variants and Settings

Either each variant of an resource needs its own ID, or settings could be passed with a query string. I think the use of settings should be minimised — for example, columns for a gallery object may be better set per ID. Settings can certainly be useful for things like autoplay, for example YouTube allows you to set the start time.

Challenge: Alignment

This is great if the URL is added on a separate line, but aligning the object is not evident. This is a challenge for shortcodes too. At the moment the core galery shortcode does not allow you to allign it, and the caption shortcode has an attribute for it. Similarly the URl could have a query parameter for alignment, but that doesn’t sound ideal. Alternatively the paragraph could be floated, or it needs to be wrapped in a `div` element to float mid-text. Another approach is to always wrap the URLs in a (custom) tag that can have display information. Again, think about how images are embedded in HTML.

Possible solution:

<figure class="alignright">
 https://example.com/galleries/yummy-chocolates/
 <figcaption>Chocolates.</figcaption>
</figure>

Challenge: Namespace

Shortcodes would have a similar problem, though slug clashes are probably more likely. Ideally plugins should use their own prefix, but this may be seen as ugly. Another way to avoid clashes with other post types is a sub type for “attachments” or “resources”.

Challenge: Extra Queries

I don’t see this exactly as a challenge, as it’s the nature of the concept, but it may be used as an argument against it. Many shortcodes already make use of custom post types to store data and embedding media (or anything else) also requires extra queries. If and how this should be cached is another problem.


I would love to hear your thoughts on this alternative, especially from those who use shortcodes for this type of objects in the post content. If you already use a custom post type, why not take advantage of embedding instead of creating shortcodes? If you want to embed external resources, why wrap it in a shortcode?

If you have other alternatives, I’d love to hear about those too!

#editor, #media, #plugins, #shortcodes

Shortcake (Shortcode UI) chat summary – November 2nd, 2015

Present: @danielbachhuber, @goldenapples, @matth_eu

Logs: https://wordpress.slack.com/archives/feature-shortcode/p1446494424000273

  • We released Shortcake v0.6.0. Read through the full release notes.
  • Weekly meetings are on hold until January. Between now and then, we’ll be thinking about what we need to do to put forth a core proposal. @matth_eu might put together sketches.
  • We missed the boat on getting a Shortcake representative to the community summit, and are researching ways to helicopter @goldenapples to said community summit boat.

Next chat: sometime in January 2016

#chats, #feature-plugins, #meeting-notes, #shortcode-ui, #shortcodes, #updates

Shortcodes roadmap — clarifications

As mentioned in the initial shortcodes roadmap post, the main purpose of this roadmap is to find the best way for improving the shortcodes API and moving it forward. Currently it is slow, fragile, and attempts to handle a lot of edge cases. For this, the most important part is:

  • No shortcodes in HTML tags attributes.
  • No HTML in shortcodes attributes.

There are a few things that deserve to be clarified. Simple shortcodes are great. They are easy to understand and be typed directly by the users. Example: [gallery].

Unfortunately many plugins add complex shortcodes with many attributes and often with nested shortcodes. These are a nightmare for the users. They are not intended be typed directly and can be edited by some sort of UI. Using shortcodes to store this type of data in post_content is not a good idea. Since there is a UI for entering and editing, it would be better to use a simple shortcode to “hold the place”, and save all the data in post meta.

Many of these complex shortcodes also include HTML tags in their attributes. To keep that functionality, the second roadmap draft proposed an extended syntax that allows the shortcodes “content” (the text wrapped by [shortcode] and [/shortcode]) to be additionally separated by delimiters. That would allow for shortcode attributes to contain HTML tags that are stored in the shortcode content.

These delimiters are not intended to be typed directly by the users. They are intended for the plugins that have shortcode editing UI and cannot function without storing HTML in shortcode attributes.

At first look this makes the syntax needlessly complex. However after looking at how complex shortcodes are used now, it is relatively the same: these shortcodes cannot be typed directly and are useless without some sort of UI.

There have been questions about line breaks in shortcode content. It is possible to add support for this. However it will benefit only a very small amount of users. Since shortcodes “live” in HTML context, and line breaks are ignored there, typing in the Text editor and switching to the Visual editor will remove all line breaks. Typing in the Visual editor will add paragraph tags. So only users that never use the Visual editor and have to type long, complex shortcodes will see some benefit.

The Shortcode API Roadmap meeting is in #feature-shortcode today at 17z, which is 2015-10-14 1700.

#meeting, #roadmaps, #shortcodes

Shortcode Roadmap Draft Three

This is the third draft of the Shortcode API Roadmap. It describes in broad terms the changes that might take place across versions 4.4 through 4.6. This roadmap gives notice to plugin developers that significant changes in plugin design may be needed for compatibility with future versions of the Shortcode API. This roadmap also identifies steps taken to minimize the impact on plugin developers to allow most plugins to continue working with only small changes.

This roadmap is based on decisions made at meetings, feedback received on previous posts, and consultation with the core developers.

Our need for a roadmap arose from specific problems in the old code.  There are performance problems in parsing shortcodes, and we need to fix those problems with backward compatibility in mind.  Recent security patches illustrated the problem of not being proactive about security hardening.  Bloat in content filters is another big problem that complicates efforts to correct problems.

Please comment on this new draft.  We will have another meeting Wednesday at 17Z, which is 2015-10-14 1700.

4.4 – New Restriction on Shortcode Names

There is only one item on the 4.4 Milestone. It helps us move toward our goal of security hardening without breaking websites.  Names of registered shortcodes will be slightly restricted by disallowing angle braces.  It should be possible to implement this change immediately with no impact on existing content or plugins.

The < and > characters will be no longer allowed in the $tag parameter of add_shortcode(). Starting in 4.4, attempting to register an invalid shortcode name will result in a helpful error message.

These characters will be forbidden in shortcode names: [ ] < > / &

Also forbidden are the “non-printing characters” including spaces, tabs, new lines, and other control sequences.

Continue reading

#meeting, #roadmaps, #shortcodes

Shortcake (Shortcode UI) chat summary – October 5th, 2015

Present: @danielbachhuber, @goldenapples, @matth_eu

Logs: https://wordpress.slack.com/archives/feature-shortcode/p1444071794000007

  • Matt’s making process on support for encoding HTML in attributes. Gallery functionality is also almost done, but there’s one small bug.
  • Than started work on trying to add some filters that can be used to handle floated/non-block previews. It still some work to go, as it’ll involve overriding some methods deep in mce.view.
  • Daniel will hit up the backlog when he has a moment, as there are a number of unanswered open issues.
  • We discussed inline editing and agreed upon an ideal abstraction .

Next chat: same time and place

Next release: v0.6.0 – Tuesday, November 3rd

#chats, #feature-plugins, #meeting-notes, #shortcode-ui, #shortcodes, #updates

Shortcode Roadmap Draft Two

This is the second draft of the Shortcode API Roadmap. It describes in broad terms the changes that might take place across versions 4.4 through 4.7. This roadmap gives notice to plugin developers that significant changes in plugin design may be needed for compatibility with future versions of the Shortcode API. This roadmap also identifies steps taken to minimize the impact on plugin developers to allow most plugins to continue working with only small changes.

This roadmap is based on decisions made at the meeting in #feature-shortcode, as well as feedback on previous posts, and consultation with the core developers.

Our need for a roadmap arose from specific problems in the old code.  There are performance problems in parsing shortcodes, and we need to fix those problems with backward compatibility in mind.  Recent security patches illustrated the problem of not being proactive about security hardening.  Bloat in content filters is another big problem in itself.

Please comment on this new draft.  We will have another meeting next Wednesday at 17Z, which is 2015-10-07 1700. Trac tickets that were nominated for closure at the last meeting will be closed today, with references to this draft.

4.4 – Introduce Multiple Enclosures

The two items on the 4.4 Milestone help us move toward our goals of security hardening without breaking websites.  A new delimiter in the shortcode syntax will enable plugin authors and users to always place their HTML between the shortage tags rather than inside of them.  At the same time, the names of registered shortcodes will be slightly restricted by disallowing angle braces in shortcode names.  It should be possible to implement both of these changes immediately with no impact on existing content or plugins.

New Delimiter

A new addition to the shortcode syntax along with documentation of how it works will be created in the 4.4 development cycle.  Its purpose is to allow more than one HTML enclosure in a single shortcode. Use of this new delimiter is optional or as needed.

Enclosure Delimiter:  [parent:attr=]

Usage:  [shortcode] Content HTML [shortcode:name=] Attribute HTML [/shortcode]

Example:  [photo link_to="twentysixteen/"] Here is <b>my</b> caption. [photo:media=] <img src="00.twentysixteen-260x300.png" width="260" height="300" /> [/photo]

Formally:
    delimiter   =  begin code-name middle attr-name end
    begin       =  "["
    code-name   =  1*( ALPHA / DIGIT / unreserved / %x80-FD )
    middle      =  ":"
    attr-name   =  *( ALPHA / DIGIT / "-" / "_" )
    end         =  "=]"
    unreserved  =  "!" / "#" / "$" / "%" / "(" / ")" / "*" /
                   "+" / "," / "-" / "." / ";" / "?" / "@" / 
                   "^" / "_" / "{" / "|" / "}" / "~"

In the examples above, the “name” part of the new delimiter works the same way as an attribute name. The main difference when using this new style of attribute (the enclosure) is improved support for HTML inside the value text. One basic reason why this works better is because our HTML filters do not need to understand how to look in the middle of individual shortcode tags. All of the HTML is located between shortcode tags, making the shortcode and HTML codes easy to process separately.

Continue reading

#meeting, #roadmaps, #shortcodes

Shortcode Roadmap Extended Discussion

We saw a great amount of feedback on the first draft of the Shortcode API Roadmap.  The resounding concensus was that the shortcode syntax we already know and love should not be replaced.

Now we need to bring that enthusiasm and more feedback to the first official meeting to help create a second draft of the roadmap.

On Wednesday at 17Z, which is 9 Sep 2015 17:00.

It is scheduled in the #feature-shortcode channel.

In case you can’t make it to the meeting, here are some of the items up for discussion.

Parser Problems

The biggest issue with keeping the BBCode style syntax is that we don’t have a scalable way to make these shortcodes work in PCRE.  The current pattern searches for all registered shortcodes by name, because searching for matching pairs of braces leads to backtrack limitations and segfaults.  If someone has an idea or knows a good library, now is the time to tell us!  With that said, we are also bringing some new proposals of our own.  One theoretical solution should make it possible to preview the first word of each shortcode tag so that the full search pattern then only includes the names of shortcodes already found in the input, rather than the names of all registered shortcodes.  This is the best idea so far and could be easy to implement.

HTML vs. Shortcode Syntax

We still want to expand the shortcode syntax to allow multiple enclosures.  So which new tags would work best internally and still look nice for users?  We are now thinking something more like this:

[shortcode] HTML [=part2=] HTML [/shortcode]

Let’s also have a brief discussion about preparing for the eventual removal of HTML-in-shortcodes and shortcodes-in-HTML combinations. As we saw in 4.2.3, it is better to plan for this rather than allowing it to become an urgent surprise update.

Version Issues and Breaking Things

We still need to plan out a change of filter priorities so that shortcodes will be processed first, before paragraphs and curly quotes. How much impact will this have on plugins now that we will be changing the way existing shortcodes work instead of adding a whole new system? Is it adequate to offer paragraphs and curly quotes as an opt-in only feature?

Will we need weekly meetings after this first one?

Are there other tickets or features, such as nested shortcodes, that need to be roadmapped?

#meeting, #roadmaps, #shortcodes

Shortcode Roadmap Draft One (Proposal – Request for Comments)

This is the first draft of the Shortcode API Roadmap. It describes in broad terms a new feature set and migration that might take place across versions 4.4 through 4.7. This roadmap gives notice to plugin developers that significant changes in plugin design may be needed for compatibility with future versions of the Shortcode API. This roadmap also identifies steps taken to minimize the impact on plugin developers to allow most plugins to continue working with only small changes.

The decision to create this roadmap arose from specific needs that are not met by the old code. Our old [ and ] delimiters were easily confused with the way those characters are commonly used in English quotations and sometimes even in URLs. The proposal to use [{{ and }}] instead should allow a better balance between being able to type in the shortcodes and avoiding confusion with any other input. With these more unique delimeters, we will be able to process registered shortcodes more efficiently because we will not have to search for them by name. Unregistered shortcodes will have more consistency because we can find them more accurately.

Old style delimeters also gave no strong indication whether or not a closing tag was required. The proposal to use }$] as the delimeter of a shortcode with a following closing tag increases the efficiency of regular expressions, because the search for a closing tag will only happen as needed.

Adding the new style of shortcode syntax provides an opportunity to make significant API changes. One of those major changes is to ensure that shortcodes are processed before paragraphs and before curly quotes. This will lead to greatly simplified code in related functions that currently must find and avoid shortcodes every time they run. We also have an opporunity to re-think the way shortcodes are filtered, and to give plugin authors more control over those filters when registering their shortcodes.

4.4 – Introduce New Syntax

A new syntax and documentation of how it works will be created in the 4.4 development cycle. Support for the new syntax will be introduced, allowing plugins to register for extra features. Core shortcodes will use the new syntax in all new posts. The old syntax will not change. Old posts will not be affected. Initial work on the syntax concept follows.

Proposed New Syntax

Self-Closing:  [{{shortcode}}]

Attributes:  [{{shortcode  attr1="value1"  attr2='value2'  "value3"  'value4'  value5}}]

Enclosing:  [{{shortcode}$] HTML [${shortcode}}]

Multiple Enclosures:  [{{shortcode}$] HTML [${encl2}$] HTML [${encl3}$] HTML [${shortcode}}]

Escaped Code:  [!{{shortcode}}]

Stripped Unregistered Code:  [{{randomthing}}]

Stripped Unregistered Enclosure:  [{{randomthing}$]  Content also stripped.  [${randomthing}}]

Stripped Empty Code:  [{{ }}]

Ignored Orphan:  [{{shortcode}$]

Ignored Orphan:  [${shortcode}}]

Ignored Orphan:  [${encl2}$]

Ignored Context:  [{{shortcode <br> }}]

Ignored Context:  <a href="[{{shortcode}}]">

4.5 – Deprecate Old Syntax

Starting in 4.5, plugins that register shortcodes without declaring support for new features will raise debugging errors to alert developers that support for the old shortcode syntax is ending.

Old posts will continue to work normally while the old syntax is deprecated.

4.6 – Upgrade Old Posts

Old posts will be automatically converted to the new shortcode syntax. The Shortcode API will continue to provide deprecated support for old syntax so that there is no disruption during the conversion process.

The API should be adequately abstracted so that old plugins are not affected by this conversion. However, as the new syntax will not support HTML inside of shortcode attributes, there is no guarantee that every shortcode will work the same way in 4.6 as in earlier versions.

4.7 – End of Support for Old Syntax

Old shortcodes will stop working in 4.7. Plugins that still produce the old shortcode syntax will be ignored by the Shortcode API.

The upgrade to 4.7 will include a second pass of the conversion of old posts so that any old syntax that was added to posts during 4.6 will still get converted.

In 4.6 or 4.7, if necessary, a filter could be added to automatically convert any old syntax still being produced by old plugins when new posts are created.

#proposal, #roadmaps, #shortcodes

Shortcake (Shortcode UI) chat summary – August 31st, 2015

Present: @danielbachhuber, @matth_eu

Logs: https://wordpress.slack.com/archives/feature-shortcode/p1441047764000146

Next chat: same time and place

Next release: v0.6.0 – Tuesday, November 3rd

#chats, #feature-plugins, #meeting-notes, #shortcode-ui, #shortcodes, #updates

Shortcodes roadmap

The Shortcode API is well loved by developers. Thousands of plugins use it for many cool features.

Unfortunately it wasn’t documented well when it was added. Even now the documentation is somewhat incomplete. The API was also very permissive, allowing many unintended user cases.

The result of these early mistakes is that there are plugins which use shortcodes in very unintended ways: mixed with HTML tags, nested several levels deep, with huge attributes, etc.

What are shortcodes:

  • With one word: placeholders.
  • Convenient way to add dynamic content inside post_content at run time.

What shortcodes are not:

  • A way to conceal user input in post_content.
  • A way to store any type of data in post_content. There are better places and methods for that, like post meta.

Shortcodes “live” in the same context as HTML tags. They should obey the same rules. Also — no interlinking between HTML tags and shortcodes. Think of the [ and ] being equal to < and >.

Both <p title="<b>my title</b>"> and [paragraph title="<b>my title</b>"] should be illegal for the same reasons. Also <p title="[my-span]">. I know the current shortcodes parser mostly supports these, and some plugins use them, but that will probably need to change “for the greater good”.

There is simply no good reason for trying to support mixing of shortcodes and tags with the current parser. These cases take longer time and more resources on every front-end page load. They require much more complex code to sanitize and ensure they are safe to run. If plugins cannot operate without mixing shortcodes and HTML tags, they will eventually have to implement their own placeholders and parsers, and ensure all data is sanitized properly. This will require a lot less time, effort and processing as the plugins would know what to expect.

We’ve been talking about this with @miqrogroove for a while now. There are several very interesting suggestions in his post on the subject: http://www.miqrogroove.com/blog/2015/shortcode-v44/.

We both agree that we need to create shortcodes roadmap, similar to the taxonomy roadmap. This will allow us to fix the shortcomings in the Shortcode API and clear the path for future improvements.

#roadmaps, #shortcodes