A New Frontier for Core Development

In a little over a decade, we’ve made twenty five thousand commits to WordPress. WordPress (along with the web itself) has come a long way, but our development workflow has remained largely the same.

As a part 3.7, I’ll be leading an effort to revamp and streamline our development workflow. We’re going to bring all of our core components—our code, our tests, and our tooling—under one roof. Developers will be able to use and improve the tools we’re already working with day-to-day, and we’ll be able to add new tools to make working with WordPress even easier.

We’re also making sure that any changes are compatible with our current workflow, so you won’t have to change the way you work. These changes won’t break any existing checkouts or scripts that use core.svn.wordpress.org. Now that we’ve cleared that up…

develop.svn.wordpress.org

This week, we’re creating a new repository: develop.svn.wordpress.org.

The develop repository will be the new home for all core development. We’re going to sync over all of the commits from core.svn.wordpress.org and will reorganize this new repository to use a new directory structure:

The src directory

This is the WordPress core source. All of the existing WordPress core files will be moved into this directory. You’ll develop and test against this directory as usual, but the real power will come from using the rest of the tools in the repository. The contents will be optimized for development, not production (this means uncompressed files—including our internal libraries).

The tests directory

We’re going to pull our internal unit tests into this repository as well. This will make testing a cinch, and allow us to commit patches that include changes to both the source and the unit tests. We’ll also add JavaScript unit tests during the 3.7 cycle.

The tools directory

Our tools directory will include all of the little things that make our lives as developers easier. To start, we’ll include our internationalization tools, which will allow us to generate pot files from within core. We’ll also include a handful of scripts that some of us use to interact with Trac, like scripts to upload and apply patches.

The root directory

This frees up the root directory to include configuration files, allowing WordPress to interact more seamlessly with other applications and services like Travis CI and Composer.

A build process

We’ve had an implicit build process in the WordPress codebase for a considerable time now. Most of that process is now contained within bumpbot, a little script that lives on the WordPress.org servers. Whenever we update CSS or JavaScript files, bumpbot comes around, minifies them, and commits the result. Despite bumpbot’s efforts, this process is less than ideal. Our source files (in this case, JavaScript and CSS) live alongside our production-optimized files, which causes confusion and makes contributing to core considerably more difficult. Our built output—the files optimized for machines—shouldn’t live alongside the source code that’s optimized for developers, and beyond that, shouldn’t be tracked in source control.

We’re adopting Grunt to write a real build process and help us manage and run our various tools. Grunt is a JavaScript task runner that has a robust ecosystem of plugins focused around building modern web applications. We can include tools such as integrated testing, minification, linting (for PHP, JS, and CSS), image optimization, and LiveReload—and that’s just the beginning.

Using Grunt will be an optional but encouraged part of WordPress development. These tools are all built atop Node.js, which Grunt needs in order to function.

So what’s happening to core.svn.wordpress.org?

In a word, nothing. The contents of the core repository have always been optimized for production, and that’s not going to change. core.svn.wordpress.org will now serve as the repository for our built output: every time we make a commit that changes the build, the core build will be updated with a corresponding commit. As mentioned earlier, scripts and checkouts will continue to function as expected. When you download a zip of WordPress, it will still be generated from this repository.

Let’s get started

The develop repository will go live soon. For now, I’ve made a preparatory repository available to illustrate the structure I’ve described above, and allow us to experiment with this new workflow. You can learn more about how the tools are installed and used on the repository page. Please provide feedback on this post—the goal is not to build all of the code there first, but to provide insight into the new process.

Lastly, I’d like to say thank you to all of the contributors who helped shape this idea over the past few months, especially at WordCamp San Francisco’s hack day. Together, we’ve created a structure that elegantly addresses many of our existing problems and opens the door to countless possibilities for better development practices.

#svn

Team Gandalf Update

Last week marked the end of our second cycle and the beginning of our sprint to get things done. We began by doing just that: last Friday, we merged the theme customizer into core. There’s still quite a bit to do and polish, so please keep that in mind. Development is being trac(k)ed at #19910.

In the past week, we’ve introduced better JS APIs, working preview navigation (you can click URLs and it works, hooray), namespaced attributes (which will allow us to enable form submission within previews), and more. We also uncovered and fixed a fairly insane bug that stemmed from a low-level bug in Opera’s JS engine. Good times.

Moving forward, the main goals are to integrate more setting types (headers, widgets), improve the refreshing process, polish the UI, and remove any vestigial plugin cruft. Dominik’s time will be limited over the next few weeks due to exams, so if any of this sounds exciting to you, please let me know.

#customize, #gandalf, #team-update

Team Gandalf Update

Yesterday marked the halfway point of our second cycle. We’re moving along at a steady clip. The main goal for the coming week is to tie up any loose ends and begin integrating the plugin into core. Until then, follow our progress at #19910 and in the plugin repo.

We’ve made considerable progress in the past week. We completed a first pass for menu locations and have almost all of the elements we need for custom backgrounds (all that remains is the file upload dialog and fixing the color picker). We also added the ability to alter/choose a static front page within the customization (as decided in dev chat yesterday). Under the hood, we’ve made numerous improvements including working APIs (including rendering controls, previewing, and saving) for theme_mods and options, proper handling of multidimensional IDs, and a whole bunch of bug fixes.

#customize, #gandalf, #team-update

Team Gandalf Update

Our first cycle ended yesterday, and all is well. Follow along with the fun at #19910 in the plugin repo.

Dominik (ocean90) and I have implemented a base UI, and have properly sandboxed both the theme preview and the customization controls. Only the target theme can alter the controls/preview, the preview can be independently refreshed, and the controls/preview can communicate with each other.

We’re set to begin our second cycle. A tentative scope:

  • Develop APIs for adding/previewing/saving individual settings.
  • Implement custom background (properly, with save).
  • Core integration (decided in dev chat).
  • Work with the Headerators to implement custom headers.
  • Begin work on adding menu locations.

#customize, #gandalf

Team Gandalf Update

Wednesday marked the halfway point of our first iteration. The description of our current cycle can be found in #19910. Development is currently taking place in a plugin (you can find the repo here). For an overview of appearance improvements, see #19909.

We’ve accomplished most of the initial goals for this cycle and are now refining and building out the existing concepts. Currently, the two remaining points are to add an (initially blank) save action and to enable communication between the UI and preview via postMessage.

On Tuesday, we had a UX review with Jane where we decided to proceed with a stacked accordion UI to navigate between different components in the preview frame. We also decided to add a button to collapse the sidebar (similar to the “collapse menu” button in the admin menu, but larger and “more buttony”). Once postMessage support has been added, we’ll also experiment with a zooming feature to allow the user to get a bird’s eye view of the page. In addition to adding the save action and postMessage, ocean90 and I are hoping to get a head start on the accordion UI so we can break more tasks out to subteams.

We have another meeting with Jane in #wordpress-dev scheduled on Monday at 21:00 UTC that will double as office hours. If you have any questions or comments, feel free to swing by the meeting, drop a comment here, or ping me (koopersmith) or ocean90 in IRC. All in all, we’re on track to meet our two week deadline.

#customize, #gandalf