Proposed roadmap: Tools for GDPR compliance

This is a proposed roadmap for adding privacy tools to core. The plan is to finalize it at the next #gdpr-compliance chat in Slack.

Main goal

Add tools to help site owners comply with the GDPR and other privacy laws and requirements.

Add notices for both registered users and commenters on what data is collected in core by default, and why.

  • Shorter texts in core with links to more information. Needs text.
  • Create these “more information” pages on Needs text.

Create some guidelines for plugins on how to get compliant.

A page (or several pages) on Needs text.

Add tools to core to facilitate compliance, and privacy in general.

There are few plugins that have started implementing these tools, so we have a nice head start.

The requests to see, download and delete/anonymize private data have to be with a confirmation (double opt-in) to avoid misuse. One possible solution would be to send a token by email when a user or a commenter has requested access to or deletion/anonymization of their private data. Then they will have to submit that token as a confirmation of their request.

TBD: shall we make this process automatic or should a site owner perform the action upon receiving the confirmed request?

  • For commenters. The stored private data is emails and IP addresses, the rest is public.
    1. Dialog for requesting to see and download their private data.
      TBD: should that data also contain the public portion?
    2. Dialog for requesting deletion/anonymization of the data.
      TBD: Deletion or anonymization? Or both and let the site owner decide?
    3. Ask for consensus for storing commenter cookies. This can be a (checked) checkbox under the comments form, something like “Save my name, email and site URL in my browser for next time I post a comment. More information”.
  • For registered users. All of the data stored by default is already visible in the user profile (except IP addresses if they have commented on the site), and most can be edited or deleted from there.
    1. Button for downloading their private data, including IP addresses if they have commented. Again, should that also contain the public data?
    2. Button for requesting deletion/anonymization of their account.

Add documentation/help for site owners on how to use these tools.

This should probably be another page under the Tools menu and contain short explanation of what privacy tools are available and how to use them. It could also contain the actual tools, for example an input field for anonymizing commenters by email address.

There are a few things that need clarification:

  • IP addresses may be considered personal data so they need to be deleted or anonymized. However do they need to be sent to the user when requesting to see or download their personal data? They are essentially third-party tokens used temporarily to access the Internet and the users have no control over them. Do other websites make them available?
  • Who are considered “controllers”? All admins on single install and all superadmins on multisite? Are admins on multisite controllers for their own site?

Please post your suggestions in comments so we can finalize the roadmap at the next #gdpr-compliance chat on Wednesday. Thanks @casiepa for helping with this!


Dev Chat Summary: August 16th (4.9 week 3)

This post summarizes the dev chat meeting from August 16th (agendaSlack archive).

Feedback on 4.9 goals

  • @johnbillion: as part of ongoing Security focus, planning on several user account security related changes, for example bringing some of the Multisite confirmation emails when you change your email address into single site
  • @flixos90: Multisite team is working on REST API endpoints for the multisite objects, like sites and networks, plus improvements to the users endpoint that make it compatible with multisite
  • @westonruter: just published 0.2.0 of the codemirror-wp plugin which adds an a11y option to the user profile screen to opt-out of syntax highlighting in the same way that a user can opt-out of the visual editor
  • @westonruter: feedback and help welcomed via the CodeMirror repo
  • @westonruter: Customizer drafting and scheduling (essentially porting the features from the Customize Snapshots plugin) is currently working through designs with @joshuawold, once that’s set this will move to implementation

Component roadmaps

  • @desrosj: Multisite team using weekly meetings to re-organize the Network & Sites component’s roadmap to align with Core’s current focuses and the component’s short and long term goals
  • @desrosj: We propose using as a location for components to house their roadmaps (multisite living at ``) as roadmap posts on the Make/Core blog slowly get pushed away as new posts are published
  • @desrosj: The roadmap outline (brief descriptions of each item, a timeline, some tickets to focus on immediately) would live on these roadmap pages, and link out to make blog posts with specific details about each item.
  • Agreement to proceed with this with Multisite and assess how its working before pushing other components to do the same

General announcements

  • @koenhuybrechts: looking for update on #17268
  • @mte90: looking for update on #12955
    • @drewapicture: concern with potential performance hit due to the sheer number of times it’s called on a given page load; will leave a comment on the ticket
  • @mte90: looking for an update on #13657
    • No one around related to the Database component, will continue to look for updates via the ticket
  • @mte90: looking for an update on #40542 as well as my other patches

#4-9, #components, #core-customize, #core-multisite, #core-restapi, #dev-chat, #roadmaps, #security, #summary

Shortcodes roadmap — clarifications

As mentioned in the initial shortcodes roadmap post, the main purpose of this roadmap is to find the best way for improving the shortcodes API and moving it forward. Currently it is slow, fragile, and attempts to handle a lot of edge cases. For this, the most important part is:

  • No shortcodes in HTML tags attributes.
  • No HTML in shortcodes attributes.

There are a few things that deserve to be clarified. Simple shortcodes are great. They are easy to understand and be typed directly by the users. Example: [gallery].

Unfortunately many plugins add complex shortcodes with many attributes and often with nested shortcodes. These are a nightmare for the users. They are not intended be typed directly and can be edited by some sort of UI. Using shortcodes to store this type of data in post_content is not a good idea. Since there is a UI for entering and editing, it would be better to use a simple shortcode to “hold the place”, and save all the data in post meta.

Many of these complex shortcodes also include HTML tags in their attributes. To keep that functionality, the second roadmap draft proposed an extended syntax that allows the shortcodes “content” (the text wrapped by [shortcode] and [/shortcode]) to be additionally separated by delimiters. That would allow for shortcode attributes to contain HTML tags that are stored in the shortcode content.

These delimiters are not intended to be typed directly by the users. They are intended for the plugins that have shortcode editing UI and cannot function without storing HTML in shortcode attributes.

At first look this makes the syntax needlessly complex. However after looking at how complex shortcodes are used now, it is relatively the same: these shortcodes cannot be typed directly and are useless without some sort of UI.

There have been questions about line breaks in shortcode content. It is possible to add support for this. However it will benefit only a very small amount of users. Since shortcodes “live” in HTML context, and line breaks are ignored there, typing in the Text editor and switching to the Visual editor will remove all line breaks. Typing in the Visual editor will add paragraph tags. So only users that never use the Visual editor and have to type long, complex shortcodes will see some benefit.

The Shortcode API Roadmap meeting is in #feature-shortcode today at 17z, which is 2015-10-14 1700.

#meeting, #roadmaps, #shortcodes

Shortcode Roadmap Draft Three

This is the third draft of the Shortcode API Roadmap. It describes in broad terms the changes that might take place across versions 4.4 through 4.6. This roadmap gives notice to plugin developers that significant changes in plugin design may be needed for compatibility with future versions of the Shortcode API. This roadmap also identifies steps taken to minimize the impact on plugin developers to allow most plugins to continue working with only small changes.

This roadmap is based on decisions made at meetings, feedback received on previous posts, and consultation with the core developers.

Our need for a roadmap arose from specific problems in the old code.  There are performance problems in parsing shortcodes, and we need to fix those problems with backward compatibility in mind.  Recent security patches illustrated the problem of not being proactive about security hardening.  Bloat in content filters is another big problem that complicates efforts to correct problems.

Please comment on this new draft.  We will have another meeting Wednesday at 17Z, which is 2015-10-14 1700.

4.4 – New Restriction on Shortcode Names

There is only one item on the 4.4 Milestone. It helps us move toward our goal of security hardening without breaking websites.  Names of registered shortcodes will be slightly restricted by disallowing angle braces.  It should be possible to implement this change immediately with no impact on existing content or plugins.

The < and > characters will be no longer allowed in the $tag parameter of add_shortcode(). Starting in 4.4, attempting to register an invalid shortcode name will result in a helpful error message.

These characters will be forbidden in shortcode names: [ ] < > / &

Also forbidden are the “non-printing characters” including spaces, tabs, new lines, and other control sequences.

Continue reading

#meeting, #roadmaps, #shortcodes

Shortcode Roadmap Draft Two

This is the second draft of the Shortcode API Roadmap. It describes in broad terms the changes that might take place across versions 4.4 through 4.7. This roadmap gives notice to plugin developers that significant changes in plugin design may be needed for compatibility with future versions of the Shortcode API. This roadmap also identifies steps taken to minimize the impact on plugin developers to allow most plugins to continue working with only small changes.

This roadmap is based on decisions made at the meeting in #feature-shortcode, as well as feedback on previous posts, and consultation with the core developers.

Our need for a roadmap arose from specific problems in the old code.  There are performance problems in parsing shortcodes, and we need to fix those problems with backward compatibility in mind.  Recent security patches illustrated the problem of not being proactive about security hardening.  Bloat in content filters is another big problem in itself.

Please comment on this new draft.  We will have another meeting next Wednesday at 17Z, which is 2015-10-07 1700. Trac tickets that were nominated for closure at the last meeting will be closed today, with references to this draft.

4.4 – Introduce Multiple Enclosures

The two items on the 4.4 Milestone help us move toward our goals of security hardening without breaking websites.  A new delimiter in the shortcode syntax will enable plugin authors and users to always place their HTML between the shortage tags rather than inside of them.  At the same time, the names of registered shortcodes will be slightly restricted by disallowing angle braces in shortcode names.  It should be possible to implement both of these changes immediately with no impact on existing content or plugins.

New Delimiter

A new addition to the shortcode syntax along with documentation of how it works will be created in the 4.4 development cycle.  Its purpose is to allow more than one HTML enclosure in a single shortcode. Use of this new delimiter is optional or as needed.

Enclosure Delimiter:  [parent:attr=]

Usage:  [shortcode] Content HTML [shortcode:name=] Attribute HTML [/shortcode]

Example:  [photo link_to="twentysixteen/"] Here is <b>my</b> caption. [photo:media=] <img src="00.twentysixteen-260x300.png" width="260" height="300" /> [/photo]

    delimiter   =  begin code-name middle attr-name end
    begin       =  "["
    code-name   =  1*( ALPHA / DIGIT / unreserved / %x80-FD )
    middle      =  ":"
    attr-name   =  *( ALPHA / DIGIT / "-" / "_" )
    end         =  "=]"
    unreserved  =  "!" / "#" / "$" / "%" / "(" / ")" / "*" /
                   "+" / "," / "-" / "." / ";" / "?" / "@" / 
                   "^" / "_" / "{" / "|" / "}" / "~"

In the examples above, the “name” part of the new delimiter works the same way as an attribute name. The main difference when using this new style of attribute (the enclosure) is improved support for HTML inside the value text. One basic reason why this works better is because our HTML filters do not need to understand how to look in the middle of individual shortcode tags. All of the HTML is located between shortcode tags, making the shortcode and HTML codes easy to process separately.

Continue reading

#meeting, #roadmaps, #shortcodes

Outlining a possible roadmap for the Customizer

Planning for the future is a necessary and important part of the WordPress development process. As we consider the future of WordPress – both as a whole and individual features – we publish proposed roadmaps to encourage greater discussion and give insight into the core team’s thought process.

The process of creating a roadmap is just as important as the vision behind it and the final roadmap itself. This process gives the entire community an opportunity to research and document history, define what specific items can be accomplished to bring us closer to the vision, and outlines how those tasks fit together within a possible timeframe.

What follows is a potential roadmap for the Customize component. If you’re interested in the future of live preview in WordPress, now is the perfect time to get involved and leave your feedback.

A couple of months ago, the WordPress lead developers met with the maintainers of the Customize component to discuss the future of live preview in WordPress. The goal of the chat was to come up with a potential roadmap for both the component and for how live preview can improve the user experience of WordPress for all users.

The ultimate goal of live preview in WordPress is to create user trust and remove the “save and surprise” inherent in some of the backend features.

After a lot of discussion, the group decided to target the following goals over the next two years:

  • Considerably improve performance.
  • Continue iterating on current live preview features to ensure they are solid and as easy-to-use as possible, including theme browsing and installation, menus, and widgets.
  • Experiment with new and different user interfaces. If we were creating live preview today, what would it look like? In what ways can we ease the feeling that you’re looking through a “porthole”?
  • Removal of the ambiguous mode. Currently, the Customizer is contained in a sidebar without the admin toolbar, but ideally there is the admin and the theme, and no in-between. One direction this may go is enabling “Customize” on the front end to immediately load the Customizer controls.
  • Experiment with a guided new user experience (NUX). Live preview lends itself to site setup. How can we improve the live preview experience and combine it with the NUX? Consider a “setup wizard” use case and ensure the flow has no dead ends, i.e. users can customize everything in one.

Those overall goals for live preview in WordPress can be rewritten into some specific features that are in development or planned for the future of the Customize component. These include:

  • Transactions. This re-architecture of some of the Customizer internals improves compatibility with themes by loading the preview using a natural URL, and allows Ajax requests or even REST API requests to be previewed. It also allows the preview to be viewed independently of the Customizer, so changes can be shared for others to review. See #30937.
  • Selective refresh. Only a piece of the page will need to be refreshed when this backend feature is implemented. (Formerly known as “Partial Refresh”.) Currently, this is available for menus in the Customizer. This eliminates duplication of display between PHP and JS, keeping it DRY. See #27355.
  • Concurrency. Allows for “locking” settings using the Heartbeat API, improving the overall user experience by preventing users from overwriting each other’s changes. See #31436.
  • Revisions. Enables plugin developers to add features like draft, roll back, and scheduled changes (e.g. “change my background on January 1”). This builds upon transactions, as the setting changes are staged in a transaction, and this facilitates settings to be revisioned and for settings to be scheduled. See #28721, #31089.
  • Theme installation. Iterates on and completes the theme browsing experience.
  • Responsive preview. Iterates on the concept of live preview by giving users a better idea of what their site will look like on other devices. See #31195.
  • Bootstrapped Customizer. Lazy-load the Customizer into the current frontend view without having to leave the page. With selective refresh implemented, inline controls and frontend bootstrapping would be possible since full-page refreshes would no longer be required.
  • Improvements for both touch and small devices.

Beyond those features, the group identified some specific changes that should be prioritized, in conjunction with the features planned:

  • The sliding animation between panels should feel more like “moving panels” (see: iOS).
  • Keyboard navigation should be consistent and clear.
  • Identify “dead ends” in the interface and remove them, when possible. For example, prior to menus in the Customizer, it was not possible to customize that aspect of your site’s design with the Customizer.

The concepts surrounding live preview and the Customizer have been in development for a long time. Many of the concepts from Elastic Theme and the Visual CSS Editor have been incorporated over time. Over the next few years, experimentation with these concepts will likely take place in feature plugins. For example, this team may experiment with inline content editing, where it makes sense in the context of customizing a site. Another path for exploration is simple theme customization – e.g. change the header font, change the sidebar color, or change the width of the sidebar.

As with all components and new features, we shouldn’t be afraid to experiment and fail and should continually push for new experiments and ideas, especially in the context of feature plugins. Further, some of the above experiments may not make it into core, but are meant as a general direction that live preview should take in WordPress.

Taking these features together, below is a sequence outlining a possible roadmap for live preview and the Customize component in general, along with estimated targets. Please note that this is a proposed roadmap and is entirely dependent on contributor involvement. Additionally, many of these things will take place in a feature plugin prior to core inclusion.

  • Partial refresh. Performance Improvements. (Target: 4.4)
  • Responsive Preview. Transactions. (Target: 4.5)
  • Concurrency. Revisions. Theme Install. Beginning of NUX wizard. (Target: 4.6)
  • Focus on touch screen / small device improvements. (Target: 4.7)
  • Developer API improvements based on feedback from plugin developers. (Target: 4.8)
  • Improved UI after experiments in 2016. NUX “wizard mode.” (Target: 4.9)

Live preview is one of the most critical features in WordPress as we continually combat “save and surprise.” The Customizer in its current form provides an improved user experience to WordPress users when customizing their site’s design. Each feature mentioned above is a continuation of the live preview concept, building and improving upon the Customizer.

Everything above is just a proposal and we need your feedback to ensure it is the right direction. If you’re interested in any of the above, comment here with your feedback, or join the team in #core-customize.

This post was a collaboration between @helen, @nacin, @mark, @celloexpressions, @samuelsidler, and yours truly.

#customize, #roadmap, #roadmaps

Preparing your plugins and your client sites for termmeta

The biggest hurdle in the introduction of metadata for taxonomy terms (#10142) is compatibility with existing plugins and customizations. In this post, I’ll outline the most significant concerns, along with recommendations for next steps.

This post has two audiences: authors of publicly available plugins and developers of client sites.

I’ve just completed a scan of all the plugins on that mention ‘termmeta’ or ‘term_meta’ (and boy, are my arms tired!). The numbers there are promising – it’s a fairly small number of plugins that will be affected, and many of them have just a few active installations. I’ll share the results of my scan at the end of this post.

Of greater concern are customizations that developers have built for clients. I know that many larger dev agencies, and many individual developers, have custom, in-house libraries that they use for client sites, and many of these libraries implement termmeta in one way or another. The best of these libraries will be unaffected by termmeta in core: they use prefixed functions, they store data in wp_options rather than a custom table, they use function_exists() where appropriate, and so on. However, I’m certain that there are many libraries – in use on many, many WordPress sites – that do not adhere to these best practices. Sites using these libraries will react in unexpected ways – some of which involve fatal errors – if preventative steps are not taken by the developers before termmeta is rolled into core. Developers should update their libraries and client deployments as necessary to avoid lost data or site downtime.

The primary reasons for concern, listed in order of severity, are as follows:

  1. Function name clashes. The proposed core implementation introduces a number of new functions likely to be used in third-party termmeta implementations: add_term_meta(), delete_term_meta(), get_term_meta(), update_term_meta(), and update_termmeta_cache(). (See the latest patches on the Trac ticket for details.) Plugin should either use a prefix (eg myplugin_add_term_meta()), wrap the definitions in function_exists() checks, or disable the termmeta portion of the plugin based on a WP version check.
  2. Termmeta table deletion. A number of plugins have been found that DROP $wpdb->termmeta on plugin deactivation. Aside from being bad general practice, this could be extremely bad in the case that $wpdb->termmeta becomes a core table. Plugins should not drop {$wpdb->prefix}termmeta under any circumstances.
  3. Non-matching function signatures. I haven’t found any specific examples of this, but plugins that define their own versions of unprefixed functions (using function_exists(), as described above, should be sure that the signatures match. For example, if your plugin defines get_term_meta( $term_id, $taxonomy, $key = '', $single = false ), it will break when moving to the core implementation, which has the signature get_term_meta( $term_id, $key = '', $single = false ). Plugin authors should double-check that their use of unprefixed _term_meta() functions matches the parameter order and default values described in the proposed core implementation.
  4. Non-matching table schemas. A number of plugins create tables called {$wpdb->prefix}termmeta. Most custom tables that I’ve seen are very close to the proposed core implementation, aside from a shortened index length on the meta_key column, introduced throughout core in 4.2. In fact, all implementations I’ve seen are close enough that it’s likely that core will be able to continue to use the tables, leaving data intact. However, if there are significant mismatches – for example, different field names – it will cause serious problems. Plugin author should verify that tables called {$wpdb->prefix}termmeta have a compatible schema with the core implementation.

Here’s a list of the plugins I identified from that will be affected by core termmeta (sorted most popular first). All of these plugins violate at least one of the maxims above. A few of them are OK except for a mismatched meta_key index the termmeta table; the current plan is for core to take care of this index automatically on upgrade. Most of the remaining violations have to do with unprefixed function names. If your plugin is listed below, you are strongly urged to put out an update as soon as possible that addresses the concerns described above. (Some of the plugins are very old, and may be inactive and/or unmaintained. In most cases, they should still be updated, in case of legacy sites.) Don’t hesitate to contact me, in the comments or privately, if you need more details about how a specific plugin in the list below will be affected.

It’s looking likely that termmeta will be introduced in WordPress 4.4 (or perhaps 4.5). The time to act is now. Be a good Citizen of the Internet, and fix your plugins sur-le-champ.

#4-4, #roadmaps, #taxonomy

Shortcode Roadmap Extended Discussion

We saw a great amount of feedback on the first draft of the Shortcode API Roadmap.  The resounding concensus was that the shortcode syntax we already know and love should not be replaced.

Now we need to bring that enthusiasm and more feedback to the first official meeting to help create a second draft of the roadmap.

On Wednesday at 17Z, which is 9 Sep 2015 17:00.

It is scheduled in the #feature-shortcode channel.

In case you can’t make it to the meeting, here are some of the items up for discussion.

Parser Problems

The biggest issue with keeping the BBCode style syntax is that we don’t have a scalable way to make these shortcodes work in PCRE.  The current pattern searches for all registered shortcodes by name, because searching for matching pairs of braces leads to backtrack limitations and segfaults.  If someone has an idea or knows a good library, now is the time to tell us!  With that said, we are also bringing some new proposals of our own.  One theoretical solution should make it possible to preview the first word of each shortcode tag so that the full search pattern then only includes the names of shortcodes already found in the input, rather than the names of all registered shortcodes.  This is the best idea so far and could be easy to implement.

HTML vs. Shortcode Syntax

We still want to expand the shortcode syntax to allow multiple enclosures.  So which new tags would work best internally and still look nice for users?  We are now thinking something more like this:

[shortcode] HTML [=part2=] HTML [/shortcode]

Let’s also have a brief discussion about preparing for the eventual removal of HTML-in-shortcodes and shortcodes-in-HTML combinations. As we saw in 4.2.3, it is better to plan for this rather than allowing it to become an urgent surprise update.

Version Issues and Breaking Things

We still need to plan out a change of filter priorities so that shortcodes will be processed first, before paragraphs and curly quotes. How much impact will this have on plugins now that we will be changing the way existing shortcodes work instead of adding a whole new system? Is it adequate to offer paragraphs and curly quotes as an opt-in only feature?

Will we need weekly meetings after this first one?

Are there other tickets or features, such as nested shortcodes, that need to be roadmapped?

#meeting, #roadmaps, #shortcodes

Shortcode Roadmap Draft One (Proposal – Request for Comments)

This is the first draft of the Shortcode API Roadmap. It describes in broad terms a new feature set and migration that might take place across versions 4.4 through 4.7. This roadmap gives notice to plugin developers that significant changes in plugin design may be needed for compatibility with future versions of the Shortcode API. This roadmap also identifies steps taken to minimize the impact on plugin developers to allow most plugins to continue working with only small changes.

The decision to create this roadmap arose from specific needs that are not met by the old code. Our old [ and ] delimiters were easily confused with the way those characters are commonly used in English quotations and sometimes even in URLs. The proposal to use [{{ and }}] instead should allow a better balance between being able to type in the shortcodes and avoiding confusion with any other input. With these more unique delimeters, we will be able to process registered shortcodes more efficiently because we will not have to search for them by name. Unregistered shortcodes will have more consistency because we can find them more accurately.

Old style delimeters also gave no strong indication whether or not a closing tag was required. The proposal to use }$] as the delimeter of a shortcode with a following closing tag increases the efficiency of regular expressions, because the search for a closing tag will only happen as needed.

Adding the new style of shortcode syntax provides an opportunity to make significant API changes. One of those major changes is to ensure that shortcodes are processed before paragraphs and before curly quotes. This will lead to greatly simplified code in related functions that currently must find and avoid shortcodes every time they run. We also have an opporunity to re-think the way shortcodes are filtered, and to give plugin authors more control over those filters when registering their shortcodes.

4.4 – Introduce New Syntax

A new syntax and documentation of how it works will be created in the 4.4 development cycle. Support for the new syntax will be introduced, allowing plugins to register for extra features. Core shortcodes will use the new syntax in all new posts. The old syntax will not change. Old posts will not be affected. Initial work on the syntax concept follows.

Proposed New Syntax

Self-Closing:  [{{shortcode}}]

Attributes:  [{{shortcode  attr1="value1"  attr2='value2'  "value3"  'value4'  value5}}]

Enclosing:  [{{shortcode}$] HTML [${shortcode}}]

Multiple Enclosures:  [{{shortcode}$] HTML [${encl2}$] HTML [${encl3}$] HTML [${shortcode}}]

Escaped Code:  [!{{shortcode}}]

Stripped Unregistered Code:  [{{randomthing}}]

Stripped Unregistered Enclosure:  [{{randomthing}$]  Content also stripped.  [${randomthing}}]

Stripped Empty Code:  [{{ }}]

Ignored Orphan:  [{{shortcode}$]

Ignored Orphan:  [${shortcode}}]

Ignored Orphan:  [${encl2}$]

Ignored Context:  [{{shortcode <br> }}]

Ignored Context:  <a href="[{{shortcode}}]">

4.5 – Deprecate Old Syntax

Starting in 4.5, plugins that register shortcodes without declaring support for new features will raise debugging errors to alert developers that support for the old shortcode syntax is ending.

Old posts will continue to work normally while the old syntax is deprecated.

4.6 – Upgrade Old Posts

Old posts will be automatically converted to the new shortcode syntax. The Shortcode API will continue to provide deprecated support for old syntax so that there is no disruption during the conversion process.

The API should be adequately abstracted so that old plugins are not affected by this conversion. However, as the new syntax will not support HTML inside of shortcode attributes, there is no guarantee that every shortcode will work the same way in 4.6 as in earlier versions.

4.7 – End of Support for Old Syntax

Old shortcodes will stop working in 4.7. Plugins that still produce the old shortcode syntax will be ignored by the Shortcode API.

The upgrade to 4.7 will include a second pass of the conversion of old posts so that any old syntax that was added to posts during 4.6 will still get converted.

In 4.6 or 4.7, if necessary, a filter could be added to automatically convert any old syntax still being produced by old plugins when new posts are created.

#proposal, #roadmaps, #shortcodes

Shortcodes roadmap

The Shortcode API is well loved by developers. Thousands of plugins use it for many cool features.

Unfortunately it wasn’t documented well when it was added. Even now the documentation is somewhat incomplete. The API was also very permissive, allowing many unintended user cases.

The result of these early mistakes is that there are plugins which use shortcodes in very unintended ways: mixed with HTML tags, nested several levels deep, with huge attributes, etc.

What are shortcodes:

  • With one word: placeholders.
  • Convenient way to add dynamic content inside post_content at run time.

What shortcodes are not:

  • A way to conceal user input in post_content.
  • A way to store any type of data in post_content. There are better places and methods for that, like post meta.

Shortcodes “live” in the same context as HTML tags. They should obey the same rules. Also — no interlinking between HTML tags and shortcodes. Think of the [ and ] being equal to < and >.

Both <p title="<b>my title</b>"> and [paragraph title="<b>my title</b>"] should be illegal for the same reasons. Also <p title="[my-span]">. I know the current shortcodes parser mostly supports these, and some plugins use them, but that will probably need to change “for the greater good”.

There is simply no good reason for trying to support mixing of shortcodes and tags with the current parser. These cases take longer time and more resources on every front-end page load. They require much more complex code to sanitize and ensure they are safe to run. If plugins cannot operate without mixing shortcodes and HTML tags, they will eventually have to implement their own placeholders and parsers, and ensure all data is sanitized properly. This will require a lot less time, effort and processing as the plugins would know what to expect.

We’ve been talking about this with @miqrogroove for a while now. There are several very interesting suggestions in his post on the subject:

We both agree that we need to create shortcodes roadmap, similar to the taxonomy roadmap. This will allow us to fix the shortcomings in the Shortcode API and clear the path for future improvements.

#roadmaps, #shortcodes