Preparing WordPress for a JavaScript Future Part #1: Build Step and Folder Reorganization

In the past couple of months, I have been working on a patch to make WordPress fit with modern modular JavaScript programming practices. We’ve figured out what the concerns are with regard to JS code organization in WordPress and how to be future compatible, so we can easily allow modern JS practices and inclusion of projects like Gutenberg in WordPress.

This patch is the first big step towards an improved JavaScript development workflow for WordPress core and introduces the following major changes:

  • Introduces a required build step for WordPress develop. This means it will no longer be possible to run WordPress from the src directory, but I’ve taken multiple steps to make this transition acceptable and as smooth as possible, described below.
  • Fully backwards compatible reorganization of the JavaScript code in src. All JavaScript is placed under src/js and is organized in an improved directory structure that gives much more context and is future compatible. In production, everything is built back into its original location to ensure backwards compatibility.
  • Read a summary of all changes in this comment.

Why?

Here’s an overview of the directory structure that comes with the patch and the idea behind it.

src/js				| All the JavaScript source.
├── _enqueues			| Any script that ends up being enqueued.
│   ├── admin 			| Procedural scripts ran in the admin.
│   ├── deprecated		| All deprecated scripts.
│   ├── lib			| All standalone lib scripts.
│   ├── vendor			| All 3rd party deps that can't be managed with NPM.
│   └── wp			| All scripts that assign something to the wp namespace.
│       ├── customize		| Anything under wp.customize.
│       ├── editor		| Anything under wp.editor.
│       ├── media		| Anything under wp.media.
│       ├── utils		| Anything under wp.utils.
│       └── widgets		| jQuery widgets on the wp namespace.
└── media			| The media library.

This approach was taken for several reasons, almost all of which have to do with improving development workflows. This is broken down into the following concerns.

  1. We need to maintain backwards compatibility while providing flexibility and future compatibility for modern JS development workflows.
  2. We need to easily distinguish scripts that are enqueued from source modules that are bundled. We need to provide freedom in working with the source and organizing it in the best way with minimal to no impact on backwards compatibility.
  3. We need to organize the enqueued scripts in a structure that provides more context about their scope, contents and use.
  4. We need to have a better process for managing dependencies and vendor scripts.
  5. We need to make it intuitive to make JavaScript modules and APIs globally available using Webpack.
  6. We need to facilitate a scalable workflow for building and bundling code. (Allowing for a build step.)
  7. We need to lay the groundwork to publish individual JS packages to NPM straight from the WordPress source, so the entire GPL community can benefit from them.

For a more elaborate description of these concerns, please refer to the ticket.

How?

To make this transition as smooth as possible for code contributors, we’ve taken a couple of initial steps, with more to follow.

Separate index.php in src informing developers with steps to take

We’ve added a special page that people running WordPress from source will start seeing:

Notice notifying users they cannot run WordPress from source with steps to fix.

As the screen says, npm install && grunt build will be enough to trigger a build. After that, you should have a smooth development experience using grunt watch. The wording in the notification can still be improved. I’ve created a separate ticket to improve the wording of this message.

Optimized, (much) faster grunt watch

We’ve optimized grunt watch rebuild times to be much faster, typically around 1 second for JavaScript and almost instant for PHP. Developing with grunt watch should be a comfortable experience. Future optimizations in this area should still be possible and could make the developer experience even smoother.

Documentation

I’ve written documentation drafts, which has been reviewed extensively by the Core JavaScript team, though more feedback is always greatly appreciated. There are 4 documents in total with the following topics:

  • JavaScript code organization and development practices
  • JavaScript modules and Webpack
  • Managing dependencies
  • Setting up your local development environment

These documents can be integrated into existing documentation or added to the handbook as separate entries once this patch takes effect.

What’s Next?

In working on this, we’ve come to the conclusion that we can do a much better job at creating a seamless development experience for WordPress core. Some complexity cannot be avoided. But if we can find ways to not force that complexity onto contributors, we should do so. This patch is a good first step in enabling more modern development workflows and practices, but it also invites future iteration. To that end I’ve opened two more tickets:

The plan is for this change to be committed to trunk, so iteration can continue. I do acknowledge that, while we work on further improvements and optimizations, trunk might become slightly less comfortable to work with. This is the point where we need everyone’s help to get it right. Any help to improve the developer experience for WordPress core is greatly welcomed. You can think of feedback in the form of comments or tickets, tooling proposals, better use of existing tooling, optimizations of the build, anything that helps make the development experience for WordPress core more seamless.