Community Summit: Contributing to WP-CLI

“Contributing to WP-CLI” was the second of two discussions we held at the Community Summit. For notes from the first, see details embedded in the feature development post.

We began the conversation by giving an overview to the current contribution process. Notably:

From the introduction, the conversation turned more free-form. In no particular order, some highlights:

  • One big challenge is that WP-CLI is a rather complex project and assumes a lot of knowledge from a contributor. When onboarding new contributors, they have to learn two things: the process for contributing, and how everything works (without reading the code). Although the internals page is reasonably helpful, it doesn’t cover the command execution flow. Having a flow document would be useful.
  • One observation is that potential contributors can enter the project in different ways (e.g. GitHub repo for a custom command vs. a third-party tutorial on how to use WP-CLI). Documentation primarily provides a linear path.
  • We’re seeing some contributors submit documentation but not a ton. It’s unclear whether this indicates the documentation is good enough, or whether the path to contributing documentation is too confusing. It’d be helpful to see more users open questions about the documentation, as a way of validating/improving the content.
  • Another challenge is that generally of WP-CLI tools is that the user knows the abstract problem, but not which command addresses the problem. Related to this:
    • It would be useful if a help command also included the URL for more information.
    • It’d be nice if you could easily see a tree of all commands.

Thanks to everyone who participated!

Feature Development Discussion Recap

This week’s WP-CLI office hours had been a special “feature development and the package index” edition, as was announced in the previous post: How should we embark upon new feature development?

Here’s a link to the chat log. The attendees were: @danielbachhuber, @dave_navarro, @grapplerulrich, @miyauchi, @modernnerd, @nerrad, @schlessera.

Chat summary:

We started with a quick recap of what the different approaches were that we already presented in the previous post:

  1. No package index, but community-driven feature development
  2. Submission proposal that is coupled to precise quality and maintenance requirements
  3. Two-tiered system with both an “official” index and a “community” index

Most of the discussion that immediately followed revolved around approach C, and what the best technical implementation of such a system would be.

It is clear that this involves a lot of work on the infrastructure that supports these different tiers. Some existing package indexes/managers were mentioned as a comparison: npm/packagist in terms of being a similar CLI tool, but also rpm as a package system that allows arbitrary repositories to be added as a package source.

When I told the group that we were considering going with approach A for now, while targeting some form of C as a longer-term goal, people seemed to agree in principle. The discussion that followed then made it rather clear that we are discussing two distinct problems and trying to find a solution for both at the same time:

  1. A mechanism to install external packages that allows to differentiate between “official” and “third-party” packages.
  2. A mechanism for discovery of new packages, that allows searching and/or browsing some type of collected index.

This lead the group to reason about these two problems separately, and ultimately allowed us to formulate the following plan:

  • For installing packages, we won’t have an actual “index” in place, we accept any Composer source (git repository, zip file, path to folder, …) as a package, with the added detail that a package identifier of vendor/package will default to the corresponding GitHub repository.
  • This makes it obvious what an “official” package is: any package under the wp-cli GitHub organization.
  • This also allows all third-party packages that are hosted on GitHub to be easily installed via such a shortened package identifier, without the need to add them to any sort of index.
  • In case this is needed, we will provide a backward compatibility mapping to make sure the packages from the current package index still work as expected.
  • The current package index will be retired. It will not be deleted, though, to keep legacy versions of WP-CLI working as expected.
  • “Discovery” is then an entirely different problem and will be solved through a separate (potentially third-party) project.

We will now start work on investigating this specific approach. Expect several issues to pop up on GitHub related to this.

This also means that the actual feature development will now be handled as was described for approach A. Ideas are collected within the wp-cli/ideas repository. The ones that get the most traction get included in the roadmap to build as new official packages, which means they will be part of the wp-cli GitHub organization. We will more clearly define our policies surrounding this process and include them in the contributor’s documentation.

We are still very grateful about any feedback we can get about this important aspect of WP-CLI, so don’t hesitate to share your thoughts in the comments below!

How should we embark upon new feature development?

Update July 11th:  We’re having a follow-up discussion in Slack (#cli channel) next Tuesday, July 18th at 16:00 UTC (9 am PT, 12 pm ET). Our goal is to get to the point where we have a rough sense of the path forward.

The WP-CLI package index is a directory of user-maintained commands. For a long while now, submissions have been put on hold. I’d like to unblock new feature development, but we first need to address the conundrum before us.

Originally, the package index was created as space for developers to share custom WP-CLI commands. A developer would write a new command, submit it to the index, and the command would be displayed on the website for others to discover. The command would also become installable through wp package install.

However, the package index suffered from the same problems that plague the WordPress plugin directory:

  • After a while, submitted packages are no longer actively maintained. Eventually, they become abandoned. Maintaining packages is commonly a solo-author activity when it needs to be a multi-author activity to be sustainable.
  • Over time, different implementations are submitted of the same feature. I actually ended up pausing acceptance of new packages when four of five submissions were near duplicates of existing packages.

Just like WordPress, the end-user experience is the priority for the WP-CLI project. It’s a bad user experience to have to choose from multiple poorly-maintained implementations of the same feature. It’s a much better user experience to have one high-quality, well-documented solution to a specific problem.

Not only that, but we’d much rather focus contributor effort towards maintaining common packages, rather than have spread amongst a number of one-off individual implementations. The maintenance burden of the command ecosystem is much easier to manage when somewhat centralized, than spread amongst numerous small projects.

Given what we know at the moment, our priorities are the following:

  • Provide an outstanding user experience.
  • Have a streamlined pipeline for adding needed functionality.
  • Be able to maintain and adopt packages to keep them available for the community.
  • Keep maintenance effort in check.
  • Encourage contributions in many forms.

What we want to avoid is any of the following:

  • Outdated / abandoned packages being endorsed to users.
  • Duplicated functionality causing confusion.
  • Maintainers being a bottleneck.

We were fortunate enough to be able to discuss this problem during the WordPress Community Summit. Brainstorming with other community members, we were able to identify three possible approaches so far:

A. No package index, but community-driven feature development.

Ideas are collected within the wp-cli/ideas repository or a similar tool. The ones that get the most traction or votes get included in the roadmap to build as new official packages.

Observations from the discussion:

  • Solves problems noted above by having WP-CLI be the sole source of endorsed packages.
  • Not practically feasible in terms of team effort.
  • Too slow to make progress and act on requirements.

B. Submission proposal that is coupled to precise quality and maintenance requirements.

To get included in the package index, you need not only ensure high quality but also commit to regular maintenance.

Observations from the discussion:

  • Adds strict procedures and requirements to the current package index.
  • Could include the provision that abandoning a package will cause it to be adopted by the WP-CLI project itself.
  • Growing team effort that will become problematic over time.

C. Two-tiered system with both an “official” index and a “community” index.

The “official” index is controlled and endorsed by the WP-CLI team, while the free-for-all “community” index will include a notice that use of these packages is at everyone’s own risk.

Observations from the discussion:

  • Completely open directory is valuable for innovation.
  • Submissions could default to the “community” index, and packages then need to submit a proposal to get “promoted” to the “official” one if they meet the requirements.
  • Adds cognitive overhead by having two sources of packages, with potentially two different mechanisms of installation.

None of these three approaches is a perfect fit for our priorities above, they just provide differing sets of benefits and drawbacks. In addition, we are quite sure there are other models out there that can potentially be adapted to meet our needs.

This is where we are hoping for valuable input from the community. There are some specific decisions we need to make:

  • What process should we follow for new feature development?
  • Is it possible to adapt the existing package index to help us achieve our priorities?
  • If yes, what form should it take?
  • If no, is there another mechanism might we employ?
  • What should we do with the pending submitted packages?

Have some perspective to share or process to suggest? Know of other projects that we can model our approach on? We’d appreciate your comment on this post, especially if it also includes pros, cons, expected maintenance burden effort, etc.

Please keep in mind that we may make a decision you don’t agree with. Our decision will be biased to reflect the priorities of the project.

We greatly welcome your input to the decision-making process, though, particularly to the degree that it’s respectful, introduces perspective we may not have considered, and represents a great deal of thought and consideration.


Good first issues for new contributors

Want to submit your first pull request to WP-CLI? We’ve identified a few good first issues for you to get your feet wet:

Read through the contributing guide for details on how to get started, or join us in the #cli channel with any questions you might have.

WordCamp Europe 2017 Contributor Day Recap

On Thursday, June 15th, WordCamp Europe 2017 in Paris had the very first Contributor Day where WP-CLI was officially represented as a separate team that people could contribute to.

On the official WordCampe Europe 2017 website, you can find the organization information that was sent to the attendees.

To get a sense of the event, you can visit the Contributor Day album by Olivier Gobet on Flicker.

Team Attendance

Over the course of the day, there were a dozen contributors joining the team to work on WP-CLI. Considering that the command line is not a mainstream topic in the WordPress world and that the team has only recently been officially added, we can consider this a success in itself.

After a small introductory round, it was clear that all of the contributors had prior experience with WP-CLI, either making regular use of it or even having already done development against it. This meant that we could skip the basic introduction and immediately dive into a more technical analysis of how the tool and its packages are architected.

We then split the team into two groups, depending on what the individual contributors preferred to work on: one group focussing on commands, and the other group focussing on the framework itself. The main difference between these two is the skill set needed: for the commands, the development closely mimics regular WordPress development, whereas for the framework, WordPress has not been loaded yet and the development is closer to pure PHP console development.

Things That Worked Well

Once we’ve split up the teams, we went through the Good First Issues page, and everyone was quickly able to find something to work on. This was a definite success, as I’ve often attended Contributor Days where some people were lost during the entire day, not really knowing where to focus their efforts on.

The fact that the code is managed through git and GitHub was well received by the contributors, as it made issue discovery easier and allowed for faster feedback cycles.

What made everything run smoothly as well was the fact that the contributors to join this team were mostly of above average technical proficiency. This meant that technical issues that were encountered were quickly taken care of, with people helping each other out, and there has even been an impromptu introductory session to test-driven development (TDD) by one of the contributors as well.

Things That Need To Be Improved

We are still facing some issues with the contributor onboarding process, mostly around documentation and testing.

One problem we faced was the fact that it is not obvious for most contributors that they need to create a test database first before being able to run the functional tests. Documentation could be clearer on this step, and we might even provide a hint in the console when the tests are being run without access to a database.

Another source of confusion was that there are different conventions (workflows, file names, …) of how to proceed depending on whether you want to work on a command, or the framework itself. This should all be streamlined so that every single wp-cli package will have the exact same requirements and support the same workflows.

Au Revoir

Big thanks to the organizing team of WordCamp Europe in Paris for the additional efforts they went through in order to find a space for our brand-new team! Although we hadn’t been included in the original planning (when they didn’t yet know about the WP-CLI team), we had a flawless experience and felt welcome as an integral part of the WordPress project.

Version 1.2.1 released

As it turns out, refactoring can introduce bugs. Who knew?!

WP-CLI v1.2.1 fixes a few regressions introduced in v1.2.0:

  • Properly registers commands to existing namespaces when defined by plugins [#4123, #4130].
  • Restores compatibility with using WP-CLI as a dependency of a Composer-based WordPress install [#4126].
  • Only forces use of /usr/bin/mysql on *nix systems [#4132].
  • Improves wordwrapping by using total columns in terminal, instead of an arbitrary width [#4133].

Want to help us catch these bugs earlier? Run wp cli update --nightly in your local and staging environments to use the latest and greatest.

Contributors to this release (4 total): danielbachhubergitlostschlesseraszepeviktor

Version 1.2.0 released

Happy release day!

After 325 merged pull requests, we’re excited to bring you WP-CLI v1.2.0, chock full of enhancements, bug fixes… and a bootstrap refactor.

But first…

We have a new logo!

Coming soon to a laptop near you:

Thanks to Chris Wallace and the crew at Lift UX for their work, as well as everyone who responded to my pings about feedback.

Commands abstracted to distinct packages

We’ve split up the project!

The main wp-cli/wp-cli repository now only contains the framework itself. All of the bundled commands can be found in separate repositories. For instance, the wp cache * series of commands are now located at

This abstraction provides a few benefits:

  • While developing, the tests are only run for the specific component you’re working on, making the feedback loop much shorter.
  • Individual command packages can be controlled and set up independently, opening up the opportunity for better collaboration.
  • Hotfixes and intermediary releases can be published for individual commands, that can then be updated through the built-in package manager.
  • Tests run really fast now.
  • When you submit a pull request, you don’t have to wait two hours for the tests to run.

For those using WP-CLI via Composer, you can contribute improvements to packages as you’d normally contribute to Composer dependencies. Use composer install --prefer-source to install dependencies as Git clones.

For those using WP-CLI via Phar downloadable, you can contribute improvements by installing the package (which will override the bundled version). For instance, with the cache command, this is:

$ wp package install
$ cd $(wp package path wp-cli/cache-command)

Check out #3728 for the original history. For a better understanding of the underlying infrastructure, read Alain’s posts about the new bootstrapping mechanism, managing command dependencies and dependency resolution mechanisms.

Next up: Contribution workflow

The investments in package abstraction and the bootstrap process are a part of our larger effort to improve the contribution workflow.

Ultimately, we’d like to make contributing to the project:

  • Effortless. While understanding the WP-CLI codebase does require a certain technical knowledge, those who can contribute to the codebase should be able to do so with as little overhead as possible.
  • Enjoyable. Making improvements to WP-CLI should be fun and rewarding. For contributors, this means well-defined entry points, sufficient workflow documentation, clearly-articulated vision, roadmap, and decision-making process, and so on.

For new contributors, we now have a Good First Issues page:

For committers, we now have the beginnings of a customized dashboard:

Work on the contribution workflow is an on-going process. We’ve barely scratched the surface.

Expect to see lots efforts on some of the key areas that make up the contributor experience:

  • Articulation of the decision making process for transforming ideas into new commands. We need to identify how we can sustainably transform ideas into maintained packages.
  • Improving the onboarding of new committers through documentation, tools and processes. Read the Committers credo to better understand our expectations.

Join the conversation during our weekly office hours, Tuesday, April 25 at 16:00 UTC, or at WordCamp Europe.

New commands

Want to sanity check your wp-config.php? Use wp config get list constants and globals defined in wp-config.php [#9]

$ wp config get --fields=key,value
| key                | value           |
| table_prefix       | wp_             |

Don’t want to remember where wp-config.php is on your filesystem? Use wp config path to get the path to wp-config.php [#7]

# Edit wp-config.php in your editor.
$ vim $(wp config path)

Curious how much your database weighs? Run wp db size to get the size of the database and its tables [#16]

$ wp db size --tables
| Name                  | Size   |
| wp_users              | 64 KB  |
| wp_usermeta           | 48 KB  |
| wp_posts              | 4 MB   |
| wp_comments           | 2 MB   |
| wp_links              | 32 KB  |
| wp_options            | 1 MB   |
| wp_postmeta           | 8 MB   |
| wp_terms              | 416 KB |
| wp_term_taxonomy      | 336 KB |
| wp_term_relationships | 736 KB |
| wp_termmeta           | 48 KB  |
| wp_commentmeta        | 2 MB   |

Everything else in v1.2.0

Command improvements

  • wp core install:
    • Generates 18 character password for admin user [#4002].
  • wp cron event run:
    • Only runs specified events when $args are passed with --due-now [#11].
  • wp db *:
    • Uses /usr/bin/env mysql instead of mysql when calling MySQL executable [#14].
  • wp db (drop|reset):
    • Presents database name to confirmation prompts for wp db drop and wp db reset commands [#12].
  • wp db export:
    • Adds --exclude_tables=<tables> option when exporting a database [#20].
  • wp db import:
    • Speeds up import process by disabling auto-commit and (unique and foreign) key checks [#3829].
  • wp language core install:
    • Processes --activate flag even when language is already installed [#3851].
  • wp language core (install|uninstall)
    • Permits multiple languages to be installed or uninstalled at the same time [#4, #5].
  • wp media import:
    • Error handling is improved in a variety of ways [#3755].
  • wp media regenerate:
    • Deletes existing PDF preview images upon regeneration [#3824].
    • Fixes media regeneration when an image has a dimension smaller than a registered image size [#5].
    • Adds --image_size=<size> to regenerate a specific image size [#9].
  • wp menu location assign:
    • Increases verbosity and error reporting [#3852].
  • wp package *:
    • Fixes path returned from get_composer_json_path() under Windows [#11].
  • wp plugin install:
    • Disables renaming behavior when installing from GitHub raw ZIP URLs [#3823].
  • wp plugin update:
    • Introduces --minor and --patch flags for limiting updates based on semantic versioning [#13].
    • Displays correct error when plugin fails to update [#3803].
  • wp (plugin|theme) update:
    • Adds --exclude=<name> argument to exclude plugins or themes from updating [#16].
  • wp post term *:
    • Introduces --by=id argument to explicitly handle terms as ids [#3896].
    • Pluralizes messages based on term count [#3898].
  • wp rewrite *:
    • Warns user when managing rewrites with --skip-plugins or --skip-themes, because rewrites may be missing [#3917].
  • wp scaffold (child-theme|_s):
    • Includes a default .editorconfig based on WordPress coding standards [#3902].
  • wp scaffold plugin-tests
    • Variety of improvements to scaffolded .travis.yml and circle.yml [#3919].
    • Caches Composer in scaffolded .travis.yml [#3816].
    • Installs PHPUnit 4.8.* for PHP 5.*, and PHPUnit 5.7.* for everything else [#6].
    • Tests PHP 7 versions in scaffolded circle.yml [#16].
  • wp server:
    • Permits environment variables to define a specific PHP binary [#3868].
  • wp user create:
    • Generates 24 character password for new users [#7].
  • wp user import-csv:
    • Disables core’s email notifications when updating a user [#3904].

Framework enhancements

  • Symfony 3.x components can be used with WP-CLI. We didn’t even need to break backwards-compatibility [#4067].
  • New hooks:
    • before_add_command:<parent command> lets you check runtime requirements before adding a command
    • after_add_command:<parent command> lets you depend on a specific parent command before triggering a given functionality [#4033]
  • Automatic command dependency resolution. If a sub-command depends on a parent command not yet registered, the addition of this sub-command is deferred until the parent is available [#4094].
  • Spelling suggestions. If you mistype a command, WP-CLI will now be smart enough to make suggestions about what you probably wanted to type, and suggest these corrected spellings. This helps with discovery and error resolution. Suggestions are provided for commands, parameters and aliases [#4004, #4008, #4109]
  • Help documentation wordwraps to 80 characters by default. It wordwrapped previously, but inconsistently so it was often broken [#4105].
  • Variety of testing framework improvements:
    • Variables are replaced in subdirectory paths [#4085].
    • Forces RUN_DIR deletion, safely, to ensure Behat doesn’t hang on cleanup [#4112].
    • Terminates all launched background processes in a cross-platform compatible manner [#4074].
    • Only applies @require-wp tags when WP_VERSION isn’t ‘latest’ or ‘nightly’ to ensure full test suite is run in these contexts [#4055].

Contributors to this release (43 total): 1naveengiri, aaemnnosttv, afragen, balbuf, behzod, carl-alberto, CodeProKid, danielbachhuber, davgothic, diablodale, diggy, dnmvisser, fjarrett, flaskboy, geekoun, gitlost, hason, JayWood, jeremyfrady, ka7, kcarwilemiller, lichtscheu, mbovel, MiteshShah, miya0001, Nikschavan, ntwb, petenelson, rahul3883, raquelmsmith, Rarst, ryanshoover, schlessera, Sidsector9, SosyalAlkolik, ssnepenthe, Steveorevo, tfrommen, tillkruss, timdream, trepmal, wp-make-coffee, zacksheppard

Command dependencies revisited

As I already described in Managing command dependencies, we have added two hooks that allow developers to explicitly state what type of requirements need to be fulfilled before loading their custom command.

This works just fine when you know about these hooks and make proper use of them.

But there’s a large number of third-party commands out there already that are not yet making use of these new hooks, obviously. As we want to provide a safe update path, we had to look into providing a mechanism for the commands that might break due to the bootstrapping and package distribution changes.

Automatic dependency resolution to the rescue

As it turns out, having these hooks available makes it very easy to solve the dependency resolution in a fully automated way.

If a sub-command depends on a parent command that is already registered, or if a command does not have a dependency at all, we add the command immediately, just as before. This is the standard behavior.

If a sub-command depends on a parent command that is not yet registered, adding the sub-command immediately would just break. Imagine wanting to add a scaffold my-plugin sub-command, when the scaffold command has not yet been registered. In this case, we hook up the sub-command addition to the after_add_command:<parent command> hook, and add the command to the list of deferred command additions.

Once we add a deferred command when its hook has been triggered, we remove that command from the list of deferred command additions.

If we processed all commands, any remaining additions on the list of deferred command additions are executed as a final step. This is needed for sub-commands like network meta, where the parent does not actually exist (and thus, the after_add_command:<parent command> hook is never called).

Backward compatibility and ease of use

With the above mechanism, all current commands that relied on the fact that bundled commands were always loaded first will still work, even if the bundled commands might now be loaded later, due to the fact that all bundled commands have been split out into their own packages.

What’s more, any command depending on any other command will now just work, no matter what the loading order is, as long as both commands are loaded eventually.

X-posting Proposal: WordPress Community Conduct Project

Please read + comment on the original post.

Proposal: WordPress Community Conduct Project

New bootstrapping mechanism to load the WP-CLI framework

The upcoming 1.2.0 release of WP-CLI will include a refactored bootstrapping flow that provides a more flexible way of loading the framework itself and setting everything up. This refactoring is needed to allow for some of the expected functionality to work in conjunction with bundled commands being provided through external packages and the general use of autoloading (Issue #3850 | Pull Request #3872).

What does the term “bootstrapping” mean?

In general terms, bootstrapping code is code that is not part of the actual logic meant to solve the problem, but rather code that is needed to prepare the environment so that the actual logic is able to run.

This usually means locating files and folders, setting constants, reading configuration, etc…

What problem does this change solve?

The current stable release has all bundled commands be part of the wp-cli/wp-cli repository as a big monolithic package. This has several drawbacks, like having a huge testsuite be run for every single change to the framework (and waiting for 2 hours on feedback from the tests), and coupling the update frequency of these commands to the framework itself (making a quick hotfix to one single command difficult).

To get around these problems, we’ve extracted all of the commands into separate repositories that exist on their own, come with their own testsuites and have their own release cycles. To be able to profit from these decoupled release cycles, though, you need to be able to update a bundled command independently of the framework. This means that the act of loading the framework needs to be strictly separate to the act of loading the bundled commands.

The new bootstrapping mechanism separates the procedural list of load operations into a set of isolated steps for which the order can be freely defined.

General flow

After defining some constants to locate the needed files, WP-CLI will load the bootstrap component and call the WP_CLI\bootstrap() function (source).

This function defines a list of bootstrapping steps that will then be loaded and processed in the provided order (source).

The list of bootstrapping steps is a basic array of fully qualified class names (source). Each of the classes implements the WP_CLI\Bootstrap\BootstrapStep interface, meaning that it provides a process( BootstrapState $state ) method (source).

Bootstrap steps

When the process( BootstrapState $state ) method of a WP_CLI\Bootstrap\BootstrapStep object is being called, that object will execute its tasks to take care of its responsibilities and then return the (potentially modified) $state.

Each step works in isolation, so that they can be added/removed/rearranged as needed.

The current code does not allow for the list of steps to be modified from outside code yet, while we collect data and feedback about this new mechanism. However, if we feel confident about the reliability and usefulness, we might open up the array of steps for modification with a later release. If you feel you have a valid use case for such modifications to the bootstrapping process, please let us know!

Bootstrap state

The WP_CLI\Bootstrap\BootstrapState object that is being passed from one step to the next is a very basic key-value store. It currently only knows about one key, BootstrapState::IS_PROTECTED_COMMAND, which is used to skip certain steps when dealing with protected commands (source).

The only protected command right now is wp cli info (source). This protection means that loading of external command packages and --required files is skipped, so that the command can not be overridden. In this way, you can always be sure that the wp cli info command is indeed the one provided by the framework.

Split autoloader

The current implementation uses a custom Composer plugin to split the Composer autoloader provided by the wp-cli/wp-cli package into two separate autoloaders (source). This is needed to allow autoloaders from external command packages to take precedence over the bundled commands. Otherwise, you would not be able to update bundled commands through the built-in package manager.

This is only a workaround to get around the fact that the “WP-CLI Framework” would actually need to be a separate repository from the “bundled WP-CLI”. If the repositories themselves should be split at a later date, this custom Composer plugin will not be needed and removed again. So don’t rely on this plugin or the functionality it provides to be available.