WP Plugin & Theme Checksums Project – Update

The WordPress Plugin & Theme Checksums Project has been kicked off!

Here’s a quick run-down of the current state of affairs:

  • The server side of the development will be managed in https://meta.trac.wordpress.org/ticket/3192, whereas the consuming client code development will be managed in https://github.com/wp-cli/ideas/issues/6 for now.
  • Project collaboration and coordination will happen in a dedicated #cli-checksums channel (not created yet, WIP: https://meta.trac.wordpress.org/ticket/3195)
  • It now looks like the most promising approach is to directly build the checksum generation into the existing .org infrastructure, as the code that is generating the ZIP downloads can easily be extended to generate and store the checksums at the same time. @dd32 is currently working on proof-of-concepts to evaluate the best approach for this in https://meta.trac.wordpress.org/ticket/3192.
  • The current timeline for following through with the project is:
    • Planning phase: Oct 3 – Oct 24
    • Implementation phase: Oct 25 – Nov 14
    • Integration phase: Nov 15 – Nov 22
  • Several companies have offered to support this project with development time or other help, and are already invested in differing degrees: DreamHostPagely, Plesk, Savvii.

UPDATE (Oct. 17th): We opted to keep project collaboration and coordination in the general #cli channel for now, instead of in a dedicated channel.

#checksums

Daniel likes these issues

WP-CLI’s next release, v1.4.0, is just around the corner: Tuesday, October 17th. If you’ve contributed to WP-CLI before, here are a few moderately advanced issues I’d love to see fixed in the next release:

Read through the contributing guide for a refresher on the process. Feel free to ask questions on the specific issue, or join us in the #cli channel with any questions you might have.

Good first issues for new contributors

Want to submit your first pull request to WP-CLI? We’ve identified a few good first issues for you to get your feet wet:

  • Link to code review guidelines from PULL_REQUEST_TEMPLATE
    We’ve published code review guidelines that now need to be incorporated into our PULL_REQUEST_TEMPLATE scaffolding. This issue involves a one-line change to a template file.
  • List dropin plugins in wp plugin list
    For those using a db.phpobject-cache.php, or similar, it’d be helpful to display them when running wp plugin list. This issue involves writing a bit of code with Behat functional tests around the behavior.
  • Add pre-built dictionary of “did you mean…” suggestions
    When a user mistypes a command argument, WP-CLI tries to suggest the correct argument. In certain cases, the algorithm needs enhancement from a dictionary of pre-defined corrections. This issue involves reviewing WP-CLI command arguments, writing a bit of code to produce a dictionary around the most sensible corrections, and enhancing our existing PHPUnit tests for the feature.
  • Handle all cases for Utils\wp_version_compare()
    As it turns out, WordPress has some version strings that version_compare() can’t handle. Our existing utility is inadequate for all of the cases, so we should update. This issue involves tracking down all potential version strings, writing a bit of code, and enhancing our existing PHPUnit tests for the feature.
  • Explain why certain tables can be skipped when using wp search-replace
    There are a couple of cases, indicated in the issue, where tables can be skipped or “missed” entirely. This issue involves adding some explanation of these cases to the existing command documentation.

Read through the contributing guide for details on how to get started. Feel free to ask questions on the specific issue, or join us in the #cli channel with any questions you might have.

WordPress Plugin and Theme Checksums Project – Announcement

Overview

WP-CLI provides a way for system administrators to verify the integrity of the WordPress core files. Through wp checksum core, you can easily verify that a given installation has not been tampered with. It not only checks whether the correct files are in place, but also that their content has not been changed. This is possible because WordPress provides an official API to check the expected core file checksums at https://api.wordpress.org/core/checksums/.

Having this kind of functionality for plugins and themes as well would be a huge security benefit. It would allow you to check the file integrity of an entire site, possibly in an automated fashion. However, there is no centralized way of retrieving the file checksums for plugins or themes yet, and the alternative of downloading the plugins and themes from the official servers first just to check against them is wasteful in terms of resources and bandwidth.

The aim of this project is to extend the checksum verification and its underlying infrastructure so that it can reliably and efficiently check the integrity of plugins and themes as well.

Project Stages

The project will be structured into four stages. Each stage will be followed by a detailed report, containing a summary of the stage’s efforts as well as a clear enumeration of decisions and results.

A. Initiation (← we are here)

During this initial project stage, we raise awareness of the project and discuss it with key stakeholders, sponsors, and volunteers.

We’ll evaluate the alternative approaches with all involved parties to distill the most viable path to a maintainable solution.

Finally, we’ll define a clear scope for the project, and the metrics that define its success. We plan for a working beta version by end of November, so we will want to keep the scope tight for this first iteration.

B. Planning

After we’ve decided on a specific route to follow, we can start planning the details of the solution we want to implement.

This stage will result in a project roadmap with milestones and their respective deliverables. It will also produce a list of requirements, like the provisional budget for infrastructure, the decisions needed or the estimated workload for each milestone.

C. Implementation

After we’ve planned all the technical details and broke down the work involved, we’ll start with building the infrastructure and implementing the client and server software.

The specifics of how this stage will be handled should have been laid out during the planning stage already, so this stage is all about execution and monitoring progress.

D. Integration

During the final project stage, we will move all code and infrastructure to reside under the official wordpress.org domain and complete the integration with the WordPress Core and the WP-CLI tool.

Get Involved!

This project will have a huge impact on the perceived and effective security of WordPress installations. It can greatly reduce the amount of malware-infested sites plaguing the internet, and through the substantial market share of WordPress, improve the general browsing experience for all net citizens.

If you want to get involved, you’d ideally meet the following criteria:

  • You have a vested interest in security and/or system administration in a WordPress context.
  • You can spare a consistent average of ~5+ hours/week (hopefully on your employer’s time).
  • You have experience with one or more of the types of components this project requires.

If this is you, please get in touch with us, either by commenting on this post or by joining the discussion in the following GitHub issue: https://github.com/wp-cli/ideas/issues/6.

We will have a formal kickoff during the next WP-CLI office hours in the #cli channel on October 3rd, 2017 at 16:00 UTC. Feel free to join the discussion and help us get this ball rolling.

Free WP-CLI stickers for your event!

Want to share your love of WP-CLI with your community?

Starting today, you can fill out this form to request stickers (free of charge) to distribute at your WP-CLI-related event (WordCamp, meetup, or otherwise).

Some fine print to be aware of:

  • Requests must be made at least four weeks in advance of the event by an official organizer of the event.
  • Offer is for up to 50 stickers to any geographic region Stickermule can ship to.
  • Stickers must be made available in a public common area and announced at the beginning or end of the WP-CLI session.

Feel free to reach out to danielbachhuber on Slack with any questions. Happy scripting!

X-post: Community Conduct Project – Kick off meeting scheduled for 17:00 UTC on the 5th September 2017

X-comment from +make.wordpress.org/updates: Comment on Community Conduct Project – Kick off meeting scheduled for 17:00 UTC on the 5th September 2017

Version 1.3.0 released

Happy release day! After 210 total merged pull requests, we’re excited to bring you WP-CLI v1.3.0.

Install packages with shortened identifiers

Recently, we have been discussing the future of the WP-CLI package index. Our conclusion was to deprecate the existing package index for now and provide a new mechanism for more easily installing external commands that are hosted on GitHub.

As of WP-CLI v1.3.0, whenever you provide a package identifier in the form of <vendor>/<package>, WP-CLI will first check the deprecated package index (for backward compatibility reasons), and then check for a GitHub repository that matches this identifier. This also accepts all version qualifiers/requirements that Composer can parse.

Examples:

# Install vendor/command from GitHub (uses https://github.com/vendor/command):
$ wp package install vendor/command

# Install version 1.0.5 of vendor/command:
$ wp package install vendor/command:v1.0.5

# Install commit 95ce52b of vendor/command:
$ wp package install vendor/command:dev-master#95ce52b

New commands

Wondering whether a specific string exists in your database? Wonder no more! Use the new wp db search to search through all text columns in your database for your specified string (or regex pattern) [#29, #33]:

# Search through the database for the 'http://' regular expression, printing stats.
$ wp db search 'http:\/\/' --regex --stats
wp_comments:comment_author_url
1:https://wordpress.org/
    ...
Success: Found 99146 matches in 10.752s (10.559s searching). Searched 12 tables, 53 columns, 1358907 rows. 1 table skipped:
wp_term_relationships.

Need easy access to the database prefix for chaining into other commands? Use wp db prefix to print it out [#22]:

$ wp db prefix --url=example.com/foo
wp_3_

Everything else in v1.3.0

Command improvements

  • wp config *:
    • Errors early when no wp-config.php can be found [#22].
  • wp config create:
    • Generates keys/salts locally and use WordPress.org API as fallback [#25].
  • wp config get:
    • Adds --constant=<constant> or --global=<global> to get the value of a specific constant or global [#16].
    • Indicates files included by wp-config.php [#18].
  • wp core (multisite-install|multisite-convert):
    • Use --skip-config to avoid addition of multisite constants to wp-config.php file [#18].
  • wp import:
    • Prevents non-existent directories from ending up in the list of files to import [#8].
  • wp media *:
    • Changes media noun to ‘items’ in most cases, to reflect multi-type nature of media [#18].
  • wp media import:
    • Adds --skip-copy flag to allow import of media from local filesystem without moving on disk [#21].
  • wp package install:
    • Adds support for short package identifiers [#22].
  • wp post term delete:
    • Implements --all flag to remove all terms from a post [#23].
  • wp scaffold *:
    • Creates phpcs.xml.dist instead of custom-named phpcs.ruleset.xml [#19].
    • Better support for symbolic links [#26].
    • Changes the grunt config for addtextdomain to override all text domains by default [#28].
  • wp search-replace:
    • Includes --format=count to only show number of rows affected [#14].
  • wp term (get|update|delete):
    • Introduces --by=<type> argument to get/update/delete term by slug [#27].
  • wp user *:
    • Support fetching users with an email address in the login field [#21].
  • wp super-admin remove:
    • Allows revoking super-admin of non-existent user [#6].

Framework enhancements

  • Fixes autoload file names for $custom_vendor condition [#4147].
  • Saves runtime config so it can be passed as args to Runner::run_alias_group() invocation [#4148].
  • Manually loads comments if opcache.save_comments is disabled [#4161].
  • Allows numbers in subcommand names and arguments [#4164, #4269].
  • Fixes double slash in boot-phar.php path [#4169].
  • Allows root use of wp cli info, in addition to wp cli update [#4177].
  • Updates SSH URL parser regexp to allow for null port number [#4182].
  • Add WP_CLI\Utils\get_home_dir() helper function [#4184].
  • Reduces included files (Behat/PHPUnit in particular) in built Phar [#4185].
  • Behat: Allows test DB user + pass to be set by environment variables [#4196].
  • Fixes output in JSON format in case of error while encoding [#4199].
  • Passes WP_CLI_STRICT_ARGS_MODE on to --ssh=<ssh> if set [#4207].
  • Displays a more helpful error message when site cannot be found [#4212].
  • Fixes broken indentation on Windows systems because of line endings [#4221, #4222].
  • Adapts --ssh=<ssh> flag to work with Docker and Docker Compose [#4240].
  • Checks for availability of proc_open/close in various scenarios [#4245].

Contributors to this release (45 total): aaemnnosttv, BhargavBhandari90, chetansatasiya, chriszarate, cjhaas, colemanedwards, danielbachhuber, davetha, drrobotnik, electrokit, emgk, emirpprime, erikjoling, fjarrett, freegenie, gitlost, greatislander, iansvo, Ippey, jalavoy, jameselks, joehoyle, johnbillion, @JPry, junaidbhura, kouratoras, lucatume, @mapk, mikeschinkel, miya0001, @murtzsarialtun, nikolov-tmw, pierre-dargham, plastikdreams, rahul286, ronaksampat, schlessera, Sidsector9, soulseekah, szepeviktor, tfrommen, vbaranovskiy-plesk, westonruter, wp-make-coffee, wpbullet

Good first issues for new contributors

WP-CLI v1.3.0 is coming soon!

Want to submit your first pull request to WP-CLI? We’ve identified a few good first issues we’d love to get in the next release:

Feel free to weigh in on the corresponding issue with any questions. Read through the contributing guide for details on how to get started, or join us in the #cli channel with any questions you might have.

Community Summit: Contributing to WP-CLI

“Contributing to WP-CLI” was the second of two discussions we held at the Community Summit. For notes from the first, see details embedded in the feature development post.

We began the conversation by giving an overview to the current contribution process. Notably:

From the introduction, the conversation turned more free-form. In no particular order, some highlights:

  • One big challenge is that WP-CLI is a rather complex project and assumes a lot of knowledge from a contributor. When onboarding new contributors, they have to learn two things: the process for contributing, and how everything works (without reading the code). Although the internals page is reasonably helpful, it doesn’t cover the command execution flow. Having a flow document would be useful.
  • One observation is that potential contributors can enter the project in different ways (e.g. GitHub repo for a custom command vs. a third-party tutorial on how to use WP-CLI). Documentation primarily provides a linear path.
  • We’re seeing some contributors submit documentation but not a ton. It’s unclear whether this indicates the documentation is good enough, or whether the path to contributing documentation is too confusing. It’d be helpful to see more users open questions about the documentation, as a way of validating/improving the content.
  • Another challenge is that generally of WP-CLI tools is that the user knows the abstract problem, but not which command addresses the problem. Related to this:
    • It would be useful if a help command also included the URL for more information.
    • It’d be nice if you could easily see a tree of all commands.

Thanks to everyone who participated!

Feature Development Discussion Recap

This week’s WP-CLI office hours had been a special “feature development and the package index” edition, as was announced in the previous post: How should we embark upon new feature development?

Here’s a link to the chat log. The attendees were: @danielbachhuber, @dave_navarro, @grapplerulrich, @miyauchi, @modernnerd, @nerrad, @schlessera.

Chat summary:

We started with a quick recap of what the different approaches were that we already presented in the previous post:

  1. No package index, but community-driven feature development
  2. Submission proposal that is coupled to precise quality and maintenance requirements
  3. Two-tiered system with both an “official” index and a “community” index

Most of the discussion that immediately followed revolved around approach C, and what the best technical implementation of such a system would be.

It is clear that this involves a lot of work on the infrastructure that supports these different tiers. Some existing package indexes/managers were mentioned as a comparison: npm/packagist in terms of being a similar CLI tool, but also rpm as a package system that allows arbitrary repositories to be added as a package source.

When I told the group that we were considering going with approach A for now, while targeting some form of C as a longer-term goal, people seemed to agree in principle. The discussion that followed then made it rather clear that we are discussing two distinct problems and trying to find a solution for both at the same time:

  1. A mechanism to install external packages that allows to differentiate between “official” and “third-party” packages.
  2. A mechanism for discovery of new packages, that allows searching and/or browsing some type of collected index.

This lead the group to reason about these two problems separately, and ultimately allowed us to formulate the following plan:

  • For installing packages, we won’t have an actual “index” in place, we accept any Composer source (git repository, zip file, path to folder, …) as a package, with the added detail that a package identifier of vendor/package will default to the corresponding GitHub repository.
  • This makes it obvious what an “official” package is: any package under the wp-cli GitHub organization.
  • This also allows all third-party packages that are hosted on GitHub to be easily installed via such a shortened package identifier, without the need to add them to any sort of index.
  • In case this is needed, we will provide a backward compatibility mapping to make sure the packages from the current package index still work as expected.
  • The current package index will be retired. It will not be deleted, though, to keep legacy versions of WP-CLI working as expected.
  • “Discovery” is then an entirely different problem and will be solved through a separate (potentially third-party) project.

We will now start work on investigating this specific approach. Expect several issues to pop up on GitHub related to this.

This also means that the actual feature development will now be handled as was described for approach A. Ideas are collected within the wp-cli/ideas repository. The ones that get the most traction get included in the roadmap to build as new official packages, which means they will be part of the wp-cli GitHub organization. We will more clearly define our policies surrounding this process and include them in the contributor’s documentation.

We are still very grateful about any feedback we can get about this important aspect of WP-CLI, so don’t hesitate to share your thoughts in the comments below!