Editor Technical Overview

As we start looking at the editor from a technical perspective it’s important we identify the main obstacles and requirements we face before we start conjecturing solutions. As @matt wrote before, the editor focus aims to make writing rich posts effortless. This has taken the path of treating a post as being composed of distinct pieces of content called blocks. These pieces should be easy to insert and manipulate, providing rich and contextual interfaces to interact with as you craft a post.

So how do we go about turning this into a reality? Content in WordPress is, fundamentally, HTML-augmented text; that is to say, it has no inherent data structure. This has been a very important aspect of WordPress and a force for the open web—it speaks to the sense of ownership and freedom WordPress gives you, since it’s always easy to get the full content of your publications—yet the lack of structure gets in the way of the goal to treat content as composed from individual pieces. (This reality also became an issue for the development of post formats a few years ago, but I digress.)

It’s relatively easy to add structure, but it’s not trivial to do so in a way that doesn’t harm data integrity, portability, and the cohesiveness of the post_content. So let’s lay out a first requirement:

① Shape of the Data: Portable Text

To ensure we keep a key component of WordPress’ strength intact, the source of truth for the content should remain in post_content, where the bulk of the post data needs to be present in a way that is accessible and portable, while still providing additional structure on top of HTML semantics for our editing tools. Data needs to be praised and respected. This additional structure would hopefully be invisible to the document’s display context, as it ensures the rendered content is viewable in situations that may not be aware of blocks at all. (Think of email clients, RSS, older editor versions, mobile editors, database dumps, etc.)

Storing things separately means post_content becomes either a pointer or duplicated data, which fragments the source of truth since they can get out of sync easily. (A few content block plugins do this by storing structured data in postmeta and pure data in post_content.) On the other hand, storing things together means structure can become gibberish if it’s not formatted properly before display.

How can we then offer users a great experience when creating or manipulating content without sacrificing the spirit of integrity and data reliability that is expected from WordPress? Good representations of the data would also make it easier to develop robust collaboration tools in the future by allowing us to lock things in a more fine grained way. I believe this is important to figure out soon to allow us to prototype quickly, so I’ll follow up with an initial proposal by the end.

Honouring HTML leads to a second requirement:

② Simplicity and Semantics of Representation

Unless we are improving the semantics of the document we should minimize what markup we add to identify a block; for example, avoid adding extra DOM elements or attributes, both for simplicity and standards sake. WordPress has been a champion of web standards, and we should not venture away from this quality. How can we add structure in a way that remains invisible to the output (as meaningful content as possible) but gives the necessary hooks to infer a structured view for editing purposes?

While we discuss how to structure the content to include invisible demarcation of blocks, the aspect of their nature leads to a third requirement:

③ Static & Dynamic Blocks

Blocks can be either static or dynamic. That is to say, some blocks can be stored with all the necessary aspects needed for their display, while others need to be processed before generating their final output (shortcodes, embeds, widgets, etc). This distinction is important because the most common two blocks people will naturally use are text and images. We should not break their clarity as we treat them as blocks.


 # Static
Here’s some text.

# Dynamic
[text id=123] // Pulls "Here's some text" from somewhere.

This conceptual separation of blocks is useful for designing our project, yet they generate abstract complexity which users should not be exposed to, leading to a fourth requirement:

④ Consistent Experience

One of the biggest benefits of blocks is that composing a post becomes more intuitive and reliable. Everything is inserted under the same assumptions; discovering what can be inserted is a natural part of inserting content. To the user all blocks behave in a consistent and familiar way, even if they provide tailored UIs for their controls.

The user should also be able to edit a post in a different system (mobile, REST, older version of core, apps like Mars Edit, etc), even if they lose the advantages of block editing. That’s another reason why post_content as source of truth matters, compared to storing a JSON structure in postmeta. Having things in two places means they can get out of sync depending on what tool you used to edit.

These nuances of data, UI, and display lead to a final and more general requirement, which is understanding the system we’ll be crafting:

⑤ The System

The editor experience has three fundamental aspects to its system: the UI used to manipulate a block; the demarcation of the block; the rendered output of the block. These are all separate concerns, from the tools we craft to edit a post, to the document syntax that holds the data structure, to the way the final output is generated to be displayed as HTML. (With static blocks that last aspect may be of no significant concern since the document doesn’t care about the presence of static blocks, it just displays them.)

Picturing these concerns as connected but fundamentally separate would help us figure out the best design and technical solutions for each stage, while avoiding us taking aggressive moves by coupling expectations. For instance, JavaScript is a natural technology to look at when it comes to the ability to manipulate and interact with content blocks, yet it may not be the best at all when it comes to rendering the final post to a viewer. Avoiding painting the whole flow under the same light should allow us to focus efforts, because we don’t have to change everything.

❶ Coda

As a final coda, and following @joen‘s design exploration, let’s keep in mind that our first goal should be to set up a reliable foundation to allow us to iterate quickly and test assumptions. I propose we focus initially on a few static blocks (text, image with and without caption, quote) to limit scope of the project.

In which ways can we fulfil ① and ② from the above requirements?

Shortcodes fall short in that they are not invisible, they are opaque, not standing to the scrutiny of semantics, and are also painful to parse. Alternatives could be data-* attributes in the HTML elements or custom elements (paving the way for web components, perhaps), yet we need to be careful with adding cruft.

One other possibility is to look at HTML comments as a way to provide explicit demarcation to post_content without affecting the node structure of the document for something that is inherently spurious to the HTML semantics. WordPress already uses comments for the more tag (<!--more-->) and splitting content into pages in a way that has proven to be quite robust.

It could look something like this:

<!-- wp:image -->
<figure>
  <img src="/">
  <figcaption>A picture is worth a thousand words</figcaption>
</figure>
<!-- /wp -->

There are drawbacks, benefits, and implications with each that we should discuss separately. Are there any other possible solutions?

Please, join us in #core-editor if you are interested. We’re holding weekly meetings in Slack, Wednesdays at 19:00 CET.

#editor

Dev Chat Summary: December 21st (4.7.1 week 2)

This post summarizes the dev chat meeting from December 21st (Slack archive).

4.7 Retrospective

  • Reviewing comments on 4.7 Retrospective post on Make/Core
  • We will go through comments and discuss if there are changes to our process that we should recommend
  • Goal is not to second-guess decisions that were made, the goal is to figure out if the process can be improved in future releases
  • Things to start doing:
    • “We failed at getting the field guide and email to plugin dev out early enough. We have aimed to have that out around beta 2 and usually end up getting it out around RC the last few releases. This time it came out the day before (field guide) and day after the release (email).”
      • Coming up with some documentation and ensuring that it’s not just owned by one person is a good way to improve it
      • We should also ensure it is included in the release checklist
    • “The posts explaining new features and changes are helpful, but I think that we need more of them. I follow the trac feed, and sometimes I know that as a plugin developer a particular ticket is important for me to take note of, but it can be difficult to unravel exactly what the final decision was just based on the changesets. An example of something that is going on right now is the a11y team’s work on removing excessive content from headings on admin screens. Often API changes get documented and UI changes don’t, but I’m a perfectionist and I like to stay up to date on the latest design/a11y evolutions as well. I can usually figure out what changes I might want to make in my plugins based on the changes in core, but I’m sure that often most plugin developers don’t even know that there was a change, if they don’t read every ticket.”
      • Request here is to have more Dev Notes and explanations about what is changing
      • It would help to spread the load of writing Dev Notes a bit, sometimes they are time consuming, especially if you’re not much for writing this sort of contextual summary
      • Some components generate a component summary dev note which is a good practice
      • Should we also maybe reach out to the docs team to see if they want/can help?
      • Anyone have ideas for how we get people involved with drafting the note even if they aren’t leading developers on a feature/component?
      • We do list out every change in the “this week in core posts” (shout out to the team that works on those!) already, so there isn’t anything that goes unannounced
      • The solution suggested is more posts, but the problem appears to be that people aren’t seeing changes that they think might affect them
      • Getting to Trac and subscribing to whatever feed is a little hidden. Even Slack notifications is hidden. A more public place would be good.
      • The solution could be to push people to the Week in Core posts, they already list every change categorized by components
      • Someone willing to lead a discussion (likely on make/core) on how to improve this?
      • Action Item: wait and see how lack of timed release cycles plays out
    • “We need to collectively review the “feature plugin merge guidelines” listed here. While development in plugins has become less prominent, most of the bigger projects merging into core in 4.7 (I would exclude the REST API since that’s less user-facing) skipped many of the steps here. A lot of the points don’t apply to most projects – to the point that the checklist is often forgotten entirely. But there remains a need for better quality control and an updated checklist or recommended merge considerations for larger projects should be created accordingly. These written guidelines can better inform merge decisions and assess readiness.”
      • Can we reasonably make full test coverage (covering basic use cases at least) a requirement there?
      • This may be null as feature plugins may not play a significant, or any, role in the future
      • More “wait and see how new process/focus shakes out”
      • Action Item: No more feature plugins
    • “On a related note, clearer procedures about backing out merged features are needed. Particularly if a feature goes through an extensive process to demonstrate readiness and is approved for merge, input on removing the feature during beta/RC should be solicited publicly via make/core posts and scheduled meetings, similarly to the initial merge decision. Otherwise, the decision to remove a feature can seem to ignore the value of the opinions that went into making the decision initially and may not be fully informed about the broader implications with respect to related aspects of a component. If work on a feature seems to stagnate as bugs accumulate during beta and a lead is considering pulling it due to lack of attention, contributors working on the feature should be notified so that they can address the issues or recommend removing the feature based on availability. Perhaps putting more trust in the feature teams and component maintainers that are most intimately familiar with a given project could help ensure that decisions are more broadly considered.”
      • Still a question of who really owns final decision/veto power; @matt as product owner likely
      • Whomever is leading the release has final decision. That’s why they’re a lead. That much should be clear.
      • Action Item: continue to communicate changes clearly and early
      • Release leads and core leads need to be trusted to prioritize based on goals for the release
      • When somebody is unable to solicit feedback, we need to have honest conversations about why this is happening
    • “Add automated acceptance testing for the user flows. If we add these for new features added, we can ensure they work across browsers. And in future releases, these tests can ensure that we don’t break existing flows. Run tests on BrowserStack. See #34693.”
      • Any volunteers to help work on this?
      • Anyone think automated acceptance testing for user flows is a bad idea?
      • It could be difficult to maintain
      • This is done at Automattic: https://github.com/Automattic/wp-e2e-tests
      • Action Item: keep exploring in the ticket
    • “more focus on Trac and tickets, every committer should try to follow Trac on a daily basis. Not to know 100% of the details of each ticket but at least to get a sense of what’s going on. Also, the number of tickets on Trac is increasing more and more, there’s the need of some serious ‘Trac Gardening'”
      • A big ask for every committer following Trac on a daily basis
      • Especially since the vast majority of committers aren’t being donated anywhere near full time (and a large number are 100% volunteer)
      • This is why there’s component maintainers, so that we don’t overburden each person
      • Trac Gardening is something anyone is welcome to do, you don’t need to be a committer
      • Trac Gardening would benefit from some mentorship to be more effective
      • If there could be some mentoring for this – an initial joint meeting to get people started might we get some more interest?
      • We could benefit from improved workflows for managing the resources we do have and to reduce the uncertainty in gardening/contributing in non-code ways
      • Trac Gardening can be a thankless task to a novice who comments on tickets, asks for dev-feedback and then nothing further happens for months. Perhaps the dev-feedback tag needs watching more rather than all tickets.
      • Action Item: generalize 4.7 Bug Scrubs page “to run a bug scrub, announce it here, talk to these people, look at this report in Trac, then ping people on the tickets listed”
  • Things to continue doing:
    • “The combination of a Git startup phase and Slack is excellent. At least for the Twenty Seventeen theme.”
      • GitHub likely helps get new contributors involved, but not sure they stick around
      • GitHub is easier to follow along, post mockups, get feedback, review code
      • GitHub better with searching, labelling, organizing, looking at PRs, realtime updates, making branches and then submitting PRs from branches, plus its app
      • Our current code review process is sub-optimal because there are no workflows to support it (e.g., line-by-line comments on changes)
      • It would be good to separate what is the project management tool vs. version control method
      • GitHub is sub-optimal when iterating on PRs. In Trac, you can make minor changes to a patch and upload it to a ticket. In Github, depending on permissions patch iteration is not straightforward.
    • “Weekly design meetings.”
    • “On the upside, having clear deadlines for when enhancements need to be merged into core is very helpful for prioritizing time and focussing resources. I hope we will continue some form product calendar in the spirit of “deadlines aren’t arbitrary,” even if they take a different rhythm.”
    • “increase the effort to involve different teams in collaborating on features development, where different skills and knowledge are needed, of course.”
  • @jorbin to work on a summary of what was discussed here and post it on Make/Core

General Announcements

  • Uncertain if anyone is planning on running a core dev chat next week (or any weeks going forward), so watch for agendas on Make/Core or other notifications in #core

#4-7, #4-7-1, #dev-chat, #summary

Merge Proposal Discussion: REST API Content Endpoints

There are discussion meetings and office hours in #core-restapi at 2016-10-14 14:00UTC and 2016-10-14 19:00UTC on Friday the 14th. Our next team meeting is on 2016-10-17 14:00UTC. Please attend some of all of these, because

We are meeting at 2016-10-18 01:00 UTC to make a decision on this merge proposal!

To that end, the below discussion points will be updated regularly, please leave comments on this post or join the conversation in #core-restapi.

Yesterday at the dev chat the API Team proposed the Content API Endpoints for merge in WordPress 4.7. There was popular support for this feature but as @jorbin and @helen noted that the lack of dissent suggested additional review is needed, so the API Team is continuing to seek thorough review & constructive criticism of the content endpoints, including the open questions previously shared on the week 7 and week 8 API team updates.

The merge proposal also engendered follow-up discussion in the #core-restapi channel about the benefit content endpoints bring to core, whether having such endpoints built in is quantifiably more beneficial than having them as a plugin, whether moving development from a plugin to core would slow development, and whether the endpoints as-proposed have been sufficiently reviewed for security and performance. We attempt to capture those questions & concerns (and the perspectives on them) below.

Security

Have the content API endpoints been thoroughly reviewed for security?

  • The REST API plugin has been on HackerOne for over a year with paid bounties for bugs
  • @barry has begun a security review

Performance

How does the API measure up against alternatives? Are there concerns about how the API could impact the servers to which it is deployed?

  • DeliciousBrains did a performance comparison with Admin AJAX and found the REST API to have a performance improvement (These tests have not yet been independently verified)
  • @mikeschroder notes in the comments that using the REST API in place of Admin-Ajax will also bring speed benefits by permitting many previously-uncacheable requests to be cached.

User Feedback

Are the content endpoints sufficiently well-tested & vetted by the community?

  • @matt questions whether feedback is coming too late in development for concerns to be acted upon
    • @rmccue notes that the v2 endpoints were created based on user feedback; REST API endpoints are being deployed by plugins and running on VIP, and the content endpoints have been in wide use across a variety of sites, leading to 90+ code contributors and far more developers’ support & feedback on the endpoints
  • @rmccue has also reached out to Phil Sturgeon for feedback and will follow up

Do Content Endpoints Benefit Core Development?

Will having these endpoints in core improve future core development, or solve any immediate problems?

  • @bradyvercher suggested that the content API endpoints would remove the need to write a variety of one-off ajax callbacks when developing future WordPress Core AJAX functionality
  • @westonruter notes that the customizer could dynamically create settings for posts and other kinds of content without having to wire up new admin-ajax handlers

Will Merging Negatively Impact API Development?

Will having to work through trac instead of GitHub cause development to atrophy?

  • @jjj argues that merging will slow development, because GitHub-hosted plugins are not bound to WordPress release cycles and have less overhead for features to be developed and deployed for testing. @jjj requested a plan for how the REST API will be developed going forward after the merge of these endpoints that would account for the added friction.
  • @krogsgard countered that core increases the visibility of a project like the content endpoints
    • The number of new contributors in this Slack discussion suggests that this merge proposal is bringing in new voices; whether this supports Brian’s point or not, the team is grateful for the breadth of perspectives being shared -Ed.
  • @rachelbaker suggested that the API endpoints are sufficiently inter-dependent with core WordPress code that maintaining the plugin separately amounts to maintaining a fork, and that such separated development is untenable long-term.
  • @matt hopes that a merge of these endpoints would slow release speed, but not development speed; @rmccue feels that development speed will stay the same or increase, and that tying releases to WordPress Core increases the stability and predictability of the API endpoints.
  • The versioning of the API endpoints supports forward compatibility

Do Content Endpoints Belong on Every WordPress Site?

What are the pros and cons to having every WordPress site have content API endpoints?

  • @rmccue suggests the API has network effects that can only be realized with a large install base. @krogsgard draws a comparison to RSS, the widespread availability of which enables everything from podcasting from WP to the use of apps like Feedly.
  • @matt suggests that the Atom API is a better analogue than RSS, which is an independent and pre-existing standard, and that network effects could be tested through inclusion in Jetpack
  • @joostdevalk notes that many plugins, like Yoast, have data tied to existing content such as posts and pages; either they expose the content through their own endpoints, or core does. If Core exposes content types through the API then plugins may build on top of that shared foundation, not independently reinvent the wheel. “if this doesn’t end up in core, we’ll start rolling our own API for stuff. Others will too. Interoperability won’t be there, for even the most basic stuff. I think this isn’t like RSS, I think this is much more like all of us using the same table space in MySQL.”
    • @shelob9 and @masonjames agree that merging the endpoints would create a consistent and reliable open “standard” that WordPress developers can use instead of continually reinventing how to read and edit post data over HTTP.
    • In response to the question “what prevents you from building on the endpoints in their plugin form,” @joostdevalk went on to note that plugin dependencies would make that a viable option, but that burden currently lies on the user. Plugin installation is not frictionless.
  • Can these endpoints be bundled? short takeaway: no
    • Woo bundled the API infrastructure before it was merged; doing so for content endpoints would require bundling prohibitively large amounts of endpoint code.
    • @nerrad worries that if plugins bundle different versions of the endpoints plugin, then those plugins may conflict if all bundled copies are not kept in sync.
      • @nerrad clarifies in the comments below that these worries also encompass the additional risk of conflicts when plugin authors each build their own versions of these content endpoints, instead of leveraging a shared standard: if two plugins each expose their own REST collection for posts, a developer working on a site with multiple such endpoints will need to decide which to extend, and will then have their extension tied to that specific plugin rather than to a core API.
  • @schrapel and @jorbin discussed that these content endpoints make it much easier to crawl a site, which also brings some potential performance concerns: no new content is exposed, but the process of aggregating it is easier and more easily automated.
  • In the  comments below @foliovision believes that merging the endpoints will be the best way to assert the level of back-compatibility that mid-size agencies need in order to confidently utilize the endpoints.

Please leave thoughts, questions, concerns, comments & experience in the comments below. Thank you!

Edited 2016-10-16 to include the below comments into the body of the post

#4-7, #rest-api

REST API Team Update, 4.7 Week 8

Summary: Beta 15 has been released, there are open questions that would benefit from your feedback, and the Content API Endpoints and OAuth Server are being proposed for merge as distinct, separate enhancements to the existing WordPress REST API infrastructure.

REST API v2 Beta 15 released

The 15th beta release of the REST API content endpoints plugin was released on October 7. This release builds on top of the recent Beta 14 to…

  • Add support for Post Meta, Term Meta, User Meta and Comment Meta within their parent endpoints
  • Introduce a settings endpoint to allow key site setting values to be retrieved & modified using the API
  • Introduce query parameters to query for posts that are NOT IN one or many terms of specific taxonomies
  • Resolve bugs, including bad comparison logic when updating comments.

Please try it out and report any outstanding issues; the REST API project gained its 90th code contributor this week and the team is deeply grateful for the energy and support of the broader WordPress community in testing out this merge-candidate plugin!

New Questions & Discussion Items

Items which have arisen through final ticket triage & review on which the team seeks feedback:

  1. Should the `filter` shim should be removed prior to merge? It is the majority position of the API team that `filter` be deprecated to dramatically improve the simplicity and consistency of API query functionality
  2. How should comments be handled for password-protected posts? Should the password be passed as a query parameter with the PUT/POST request, or is there a better option?
  3. Should the API match core’s logic when users with the `unfiltered_html` capability are creating or updating Posts or Comments?

Meeting Notes

At the weekly team meeting on October 10 the group reviewed open issues in the 2.0 milestone, which represents the candidate for our merge proposal shared last week.

Meeting attendees agreed to review open issues and pull requests individually, and to reconvene on Tuesday at 1500UTC to ensure all priority tickets had an owner.

At that meeting on October 11, the team reviewed the incoming feedback around the OAuth plugin (linked above). While the API team feels that having a built-in authentication solution provides a much-needed service, particularly to developers building mobile and desktop applications, the design and usability feedback we have received does indicate that the plugin needs more work.

OAuth’s place in the Merge Proposal

The API Team believes that the identified issues are resolvable, that the OAuth plugin is on track and that it should still be considered for merge in 4.7. However, after discussion within the team, input from @matt, and advice from @aaroncampbell and other core committers, we have edited our merge proposal to submit the Content API Endpoints and OAuth server as separate merge candidates. The API Team proposes both components for merge, but we submit the content endpoints for consideration independently of the OAuth1 server.

Content Endpoints Without OAuth

The Content API endpoints are stable, well-tested, and in wide production use across a variety of applications. Theme and plugin developers will benefit from having canonical, well-tested API endpoints in core, which may be used to query WordPress both from PHP code and from JavaScript applications running on the front-end or admin of WordPress. Sharing the endpoints for core data types enables increased consistency of what data is exposed and how it is persisted across different plugins, improving consistency and shortening development time by using . These themes and plugins have full read and write access to the API using the existing cookie & nonce authentication.

Mobile and desktop applications can leverage these same endpoints in a read-only capacity to create a variety of powerful reader-oriented applications and tools that expand the capability of what WordPress can do today, such as a unified reader for Make WordPress blogs and other experiments hypothesized by @jorbin.

Should OAuth 1 not be accepted for 4.7, secure write access for these external applications would still be only a plugin install away; and while having an OAuth server in core will provide a canonical approach for authenticating from remote applications, depending on the needs of a specific site or specific client application other authentication schemes may actually be preferable. Plugins exist for JWT Authentication and of course OAuth 2, and should OAuth 1 not be accepted for 4.7 these plugins may still be installed to enable an external application to opt-in to secure write access to your WordPress site.

In Summary

The API team submits for 4.7 merge consideration two enhancements to the REST API infrastructure: the Content API Endpoints for core WordPress datatypes, and an OAuth server which will reduce the setup time needed to securely interact with those endpoints from outside of WordPress. We believe these enhancements are each individually sufficiently tested and mature to meet the quality and security standards of WordPress Core, and each individually provides wide-reaching benefit to WordPress developers, and through them to the authors, readers & publishers of the web.

#4-7, #rest-api

New lead for 4.7: Helen Hou-Sandí

Due to some unexpected constraints on my time this year, I’m going to be stepping down as the 4.7 lead, and I’m happy to announce that Helen Hou-Sandí is stepping up to lead the release in my stead. I’ll still be behind the scenes providing whatever support is necessary, and I’m really looking forward to the release . You might remember Helen for her famous work on WordPress 4.0.

#4-7

REST API meeting summary, Feb 4

The core REST API team, members of the WordPress.com API team, and other interested developers met to discuss the REST API.  Full logs of the meeting are available on Slack.

Existing endpoints

The meeting opened with a discussion of the existing endpoints: what is and isn’t ready. @rmccue summarized:

  • The main focus has been on the 4 “core” objects in WordPress: posts, terms, comments, and users.
  • Posts
    • Password-protected posts are going to be ignored from the API until we have a good solution.
    • Meta
      • Post (and other) meta has been pushed out into a separate feature plugin led by @jeremyfelt. Discussion is on the github repo.
      • The main issue is that there’s no differentiation between meta fields. Meta could be added via the Custom Fields meta box, or by a plugin.
      • Enhancing register_meta() in core would allow opt-in to the meta REST API – requiring some core work. See the patch on #35658 for details.
      • There is no current way to explicitly register meta as public.
      • We need to support object meta (post, user, comment, term) and object types (custom post types, etc)
    • Media
      • Mostly complete (it’s a custom post type).
      • Some special handling around uploading media.
    • Autosaves and post previews
      • Tricky, but we think we have a solution for that moving forward.
      • Working on them in a separate feature plugin instead.
      • This will be an enhancement to the API in the future, and doesn’t block compatibility.
  • Terms
    • Work great.
  • Users
    • Works great; undergoing final review.
  • Comments
    • Works; custom comment types are harder until core better supports them.

Merge proposal and discussion

An extensive discussion ensued regarding the possibility and appropriateness of merging the endpoints into core. The discussion continued for over an hour; mainly covering whether we should merge the 4 core endpoints, or wait for a complete API before merging…  and what the definition of a complete API is.

The REST API team’s proposal is that we merge the 4 core objects in the REST API with full read, create, update, delete support, then add more peripheral features when they’re ready. 

@matt argued that we wait until the API supports everything you can do in wp-admin before merging.

The discussion revolved around these proposals and what is best for the WordPress project. Would merging the existing well developed endpoints sooner help or hinder developer adoption, and further iteration/improvement? Would delaying the endpoint merge help or hinder progress?

Here are some key comments from the discussion.

  • @rmccue  The approach to the API needs to change from “the API team creates all of the API” to “the API team controls the general shape of the API, and each team works with it”
  • @danielbachhuber Options / a Site Resource is more easily viable, as are Plugins and Themes. Widgets and Menus essentially need to be reinvented before they’ll work in a REST API
  • @jnylen It’s a question of when, and how much testing can be done first. Shipping an API that is read-only and incomplete in several key areas feels like a big mistake.
  • @kadamwhite we’re shipping an API with write capabilities but only cookie-based auth will be supported OOTB.
  • @matt  I would be pretty skeptical of merging a partial API into core…  no partial endpoints in core. let’s design a complete API, not half-do it.
  • @rmccue The API is specifically structured around progressive enhancement
  • @jorbin I think only supporting the four core object types allows for some amazing themes and amazing content editors to be created, but doesn’t allow for full wp-admin replacements. I’m ok with that.
  • @jnylen From what I’ve seen so far, I’m most concerned about shipping individual endpoints with missing features.
  • @jorbin I think the best way to pick up the pace is to get key things locked up and get the four core object types in core. This would help core develop API-first for those features
  • @codebykat On the WPCOM side, we’re committed to taking the plunge, but we’re not there yet which means things are untested.
  • @jnylen I think this needs more testing at large scale before shipping, including some of the more difficult and obscure features of WP.
  • @drew Shipping with full wp-admin replacement capability is unrealistic, imo. We need something flexible and stable that developers can use as solid jumping off point. All the rest can get separately iterated.
  • @mike I think that, realistically, to ship the API… we need an MVP, and I don’t think that defining our goalpost as “all of wp-admin” fits that criteria.
  • @matt Full wp-admin coverage is a firm goalpost, it gives us a super clear criteria for when it should be merged into core. is everything possible in wp-admin, possible through the API?

#meeting-notes, #rest-api

Welcome the 4.5 class of committers!

As announced in the State of the Word this year at WordCamp US by @matt, there are seven new committers to introduce.

Many of you have seen Michael Arestad‘s (@michaelarestad) design and front-end development contributions over the last couple of years, notably with the redesign of Press This in WordPress 4.2. His numerous, high quality contributions are a welcome addition to core. I personally am looking forward to his work on markup and styling, having relied heavily on his judgment for quite some time now.

WordPress 4.4 adds a new embed feature to WordPress, making it an oEmbed provider for the first time. Work on this new feature was done in a large part by Pascal Birchler (@swissspidy), who has been doing great work for the past few releases. Pascal’s clear communication and thorough support of the flow mindset are things we can all be inspired by.

Rachel Baker (@rachelbaker) is the co-lead of the REST API, a Comments component maintainer, and a major contributor to WordPress 4.4. Her work has made it possible for sites around the world to utilize the REST API, making WordPress a great application platform. Look for more of these contributions as the REST API iterates within core.

Likewise, Joe Hoyle (@joehoyle) is a major contributor to the REST API. As we prepare to commit the REST API endpoints in an upcoming WordPress release, there will be more and more to come from both him and Rachel.

As a Media component maintainer and a long-time contributor across many components and features, Mike Schroder (@mikeschroder) helped shepherd the responsive images feature plugin into core for WordPress 4.4. He was also a backup release lead for WordPress 3.9.

Throughout the WordPress admin interface, everywhere you look you’ll see the work of Mel Choyce (@melchoyce). Her design and experience contributions are long-standing and have benefited the entire ecosystem. As one of the maintainers of the Dashicons project, the icons you interact with daily are a big part of her contributions, as well as themes available in the WordPress.org Theme Directory.

Eric Andrew Lewis (@ericlewis) has been contributing in various forms for many years, exploring lesser-known areas, documenting them, and challenging assumptions. Most recently, you may have seen his work as a Media component maintainer or with the shiny updates feature in WordPress 4.2.

Additionally, Ella Van Dorpe (@iseulde), Konstantin Obenland (@obenland), Weston Ruter (@westonruter), Tammie Lister (@karmatosed), Andrea Fercia, (@afercia) and Ryan McCue (@rmccue [that’s one M, two C’s]) have all had their guest commit renewed.

Please join me in welcoming this great set of new committers!

#commit

Announcing the release leads for 2016

As announced during the State of the Word this year, we have a brand new selection of release leads for 2016.

Mike Schroder
Previously a co-lead for WordPress 3.9 and long time contributor, Mike Schroder (@mikeschroder) will kick off the year as the release lead for WordPress 4.5.

Dominik Schilling
Following WordPress 4.5, Dominik Schilling (@ocean90) will be the release lead for WordPress 4.6. Dominik has been a core committer for a couple of years now and was a backup release lead for WordPress 4.4.

Matt Mullenweg
Finally, closing out the year, Matt Mullenweg (@matt) will put on his release lead hat and lead WordPress 4.7. Matt previously led the WordPress 3.8 release.

Each of these release leads need your help! Every release is made by hundreds of contributors over many months, not just by its release lead. Additionally, every release lead needs a backup lead or two to help ensure the release moves forward at a solid pace. These backup release leads get great training for the real deal, as they often become future release leads (see both Mike and Dominik above!).

Are you interested in being a backup release lead? Just comment here to let Mike, Dominik, and Matt know.

#release-lead

Outlining a possible roadmap for the Customizer

Planning for the future is a necessary and important part of the WordPress development process. As we consider the future of WordPress – both as a whole and individual features – we publish proposed roadmaps to encourage greater discussion and give insight into the core team’s thought process.

The process of creating a roadmap is just as important as the vision behind it and the final roadmap itself. This process gives the entire community an opportunity to research and document history, define what specific items can be accomplished to bring us closer to the vision, and outlines how those tasks fit together within a possible timeframe.

What follows is a potential roadmap for the Customize component. If you’re interested in the future of live preview in WordPress, now is the perfect time to get involved and leave your feedback.


A couple of months ago, the WordPress lead developers met with the maintainers of the Customize component to discuss the future of live preview in WordPress. The goal of the chat was to come up with a potential roadmap for both the component and for how live preview can improve the user experience of WordPress for all users.

The ultimate goal of live preview in WordPress is to create user trust and remove the “save and surprise” inherent in some of the backend features.

After a lot of discussion, the group decided to target the following goals over the next two years:

  • Considerably improve performance.
  • Continue iterating on current live preview features to ensure they are solid and as easy-to-use as possible, including theme browsing and installation, menus, and widgets.
  • Experiment with new and different user interfaces. If we were creating live preview today, what would it look like? In what ways can we ease the feeling that you’re looking through a “porthole”?
  • Removal of the ambiguous mode. Currently, the Customizer is contained in a sidebar without the admin toolbar, but ideally there is the admin and the theme, and no in-between. One direction this may go is enabling “Customize” on the front end to immediately load the Customizer controls.
  • Experiment with a guided new user experience (NUX). Live preview lends itself to site setup. How can we improve the live preview experience and combine it with the NUX? Consider a “setup wizard” use case and ensure the flow has no dead ends, i.e. users can customize everything in one.

Those overall goals for live preview in WordPress can be rewritten into some specific features that are in development or planned for the future of the Customize component. These include:

  • Transactions. This re-architecture of some of the Customizer internals improves compatibility with themes by loading the preview using a natural URL, and allows Ajax requests or even REST API requests to be previewed. It also allows the preview to be viewed independently of the Customizer, so changes can be shared for others to review. See #30937.
  • Selective refresh. Only a piece of the page will need to be refreshed when this backend feature is implemented. (Formerly known as “Partial Refresh”.) Currently, this is available for menus in the Customizer. This eliminates duplication of display between PHP and JS, keeping it DRY. See #27355.
  • Concurrency. Allows for “locking” settings using the Heartbeat API, improving the overall user experience by preventing users from overwriting each other’s changes. See #31436.
  • Revisions. Enables plugin developers to add features like draft, roll back, and scheduled changes (e.g. “change my background on January 1”). This builds upon transactions, as the setting changes are staged in a transaction, and this facilitates settings to be revisioned and for settings to be scheduled. See #28721, #31089.
  • Theme installation. Iterates on and completes the theme browsing experience.
  • Responsive preview. Iterates on the concept of live preview by giving users a better idea of what their site will look like on other devices. See #31195.
  • Bootstrapped Customizer. Lazy-load the Customizer into the current frontend view without having to leave the page. With selective refresh implemented, inline controls and frontend bootstrapping would be possible since full-page refreshes would no longer be required.
  • Improvements for both touch and small devices.

Beyond those features, the group identified some specific changes that should be prioritized, in conjunction with the features planned:

  • The sliding animation between panels should feel more like “moving panels” (see: iOS).
  • Keyboard navigation should be consistent and clear.
  • Identify “dead ends” in the interface and remove them, when possible. For example, prior to menus in the Customizer, it was not possible to customize that aspect of your site’s design with the Customizer.

The concepts surrounding live preview and the Customizer have been in development for a long time. Many of the concepts from Elastic Theme and the Visual CSS Editor have been incorporated over time. Over the next few years, experimentation with these concepts will likely take place in feature plugins. For example, this team may experiment with inline content editing, where it makes sense in the context of customizing a site. Another path for exploration is simple theme customization – e.g. change the header font, change the sidebar color, or change the width of the sidebar.

As with all components and new features, we shouldn’t be afraid to experiment and fail and should continually push for new experiments and ideas, especially in the context of feature plugins. Further, some of the above experiments may not make it into core, but are meant as a general direction that live preview should take in WordPress.

Taking these features together, below is a sequence outlining a possible roadmap for live preview and the Customize component in general, along with estimated targets. Please note that this is a proposed roadmap and is entirely dependent on contributor involvement. Additionally, many of these things will take place in a feature plugin prior to core inclusion.

  • Partial refresh. Performance Improvements. (Target: 4.4)
  • Responsive Preview. Transactions. (Target: 4.5)
  • Concurrency. Revisions. Theme Install. Beginning of NUX wizard. (Target: 4.6)
  • Focus on touch screen / small device improvements. (Target: 4.7)
  • Developer API improvements based on feedback from plugin developers. (Target: 4.8)
  • Improved UI after experiments in 2016. NUX “wizard mode.” (Target: 4.9)

Live preview is one of the most critical features in WordPress as we continually combat “save and surprise.” The Customizer in its current form provides an improved user experience to WordPress users when customizing their site’s design. Each feature mentioned above is a continuation of the live preview concept, building and improving upon the Customizer.

Everything above is just a proposal and we need your feedback to ensure it is the right direction. If you’re interested in any of the above, comment here with your feedback, or join the team in #core-customize.

This post was a collaboration between @helen, @nacin, @mark, @celloexpressions, @samuelsidler, and yours truly.

#customize, #roadmap, #roadmaps

Shortcode Roadmap Draft One (Proposal – Request for Comments)

This is the first draft of the Shortcode API Roadmap. It describes in broad terms a new feature set and migration that might take place across versions 4.4 through 4.7. This roadmap gives notice to plugin developers that significant changes in plugin design may be needed for compatibility with future versions of the Shortcode API. This roadmap also identifies steps taken to minimize the impact on plugin developers to allow most plugins to continue working with only small changes.

The decision to create this roadmap arose from specific needs that are not met by the old code. Our old [ and ] delimiters were easily confused with the way those characters are commonly used in English quotations and sometimes even in URLs. The proposal to use [{{ and }}] instead should allow a better balance between being able to type in the shortcodes and avoiding confusion with any other input. With these more unique delimeters, we will be able to process registered shortcodes more efficiently because we will not have to search for them by name. Unregistered shortcodes will have more consistency because we can find them more accurately.

Old style delimeters also gave no strong indication whether or not a closing tag was required. The proposal to use }$] as the delimeter of a shortcode with a following closing tag increases the efficiency of regular expressions, because the search for a closing tag will only happen as needed.

Adding the new style of shortcode syntax provides an opportunity to make significant API changes. One of those major changes is to ensure that shortcodes are processed before paragraphs and before curly quotes. This will lead to greatly simplified code in related functions that currently must find and avoid shortcodes every time they run. We also have an opporunity to re-think the way shortcodes are filtered, and to give plugin authors more control over those filters when registering their shortcodes.

4.4 – Introduce New Syntax

A new syntax and documentation of how it works will be created in the 4.4 development cycle. Support for the new syntax will be introduced, allowing plugins to register for extra features. Core shortcodes will use the new syntax in all new posts. The old syntax will not change. Old posts will not be affected. Initial work on the syntax concept follows.

Proposed New Syntax

Self-Closing:  [{{shortcode}}]

Attributes:  [{{shortcode  attr1="value1"  attr2='value2'  "value3"  'value4'  value5}}]

Enclosing:  [{{shortcode}$] HTML [${shortcode}}]

Multiple Enclosures:  [{{shortcode}$] HTML [${encl2}$] HTML [${encl3}$] HTML [${shortcode}}]

Escaped Code:  [!{{shortcode}}]

Stripped Unregistered Code:  [{{randomthing}}]

Stripped Unregistered Enclosure:  [{{randomthing}$]  Content also stripped.  [${randomthing}}]

Stripped Empty Code:  [{{ }}]

Ignored Orphan:  [{{shortcode}$]

Ignored Orphan:  [${shortcode}}]

Ignored Orphan:  [${encl2}$]

Ignored Context:  [{{shortcode <br> }}]

Ignored Context:  <a href="[{{shortcode}}]">

4.5 – Deprecate Old Syntax

Starting in 4.5, plugins that register shortcodes without declaring support for new features will raise debugging errors to alert developers that support for the old shortcode syntax is ending.

Old posts will continue to work normally while the old syntax is deprecated.

4.6 – Upgrade Old Posts

Old posts will be automatically converted to the new shortcode syntax. The Shortcode API will continue to provide deprecated support for old syntax so that there is no disruption during the conversion process.

The API should be adequately abstracted so that old plugins are not affected by this conversion. However, as the new syntax will not support HTML inside of shortcode attributes, there is no guarantee that every shortcode will work the same way in 4.6 as in earlier versions.

4.7 – End of Support for Old Syntax

Old shortcodes will stop working in 4.7. Plugins that still produce the old shortcode syntax will be ignored by the Shortcode API.

The upgrade to 4.7 will include a second pass of the conversion of old posts so that any old syntax that was added to posts during 4.6 will still get converted.

In 4.6 or 4.7, if necessary, a filter could be added to automatically convert any old syntax still being produced by old plugins when new posts are created.

#proposal, #roadmaps, #shortcodes