Changes to the WordPress Core PHP Test Suite

Why were changes needed?

Dev Wapuu
Image credits: @marktimemedia

The WordPress test suite uses the industry standard PHPUnit tool to run the PHPPHP The web scripting language in which WordPress is primarily architected. WordPress requires PHP 5.6.20 or higher unit tests.

Over the years, PHPUnit has seen a number of changes, new assertions being introduced, different annotations and more, but most notably, as of PHPUnit 8.0, a void return type was added to the typical fixture setUp() and tearDown() methods.

This void return type is problematic in the context of WordPress, as return types in general were only introduced in PHP 7.0 and the void return type wasn’t introduced until PHP 7.1.
While WordPress still has a minimum PHP version of PHP 5.6, the void return type can not be introduced in the test suite as it would inhibit the tests from being run on PHP 5.6 and 7.0.

At the same time, having to run the tests on older PHPUnit versions (PHPUnit < 8.0) made it increasingly difficult to get the test suite to run on new PHP versions, like PHP 8.0 and the upcoming PHP 8.1 (expected end of November) as these older PHPUnit versions are no longer supported and are not being made compatible with newer PHP versions anymore.

Over the past few years, a number of different solution directions were explored and rejected, largely due to the large maintenance burden these would add to the small team of WordPress contributors maintaining the test framework.

With the upcoming release of PHP 8.1 as a driver, we took another look at this problem and the available tooling in the wider PHP field and a solution has now been implemented which should future-proof the test suite for, at least, a number of years.

The solution

The implemented solution is based on the external PHPUnit Polyfills library, which “allows for creating PHPUnit cross-version compatible tests by offering a number of polyfills for functionality which was introduced, split up or renamed in PHPUnit”.

The PHPUnit Polyfills also solves the void conundrum via a tailored TestCase using snake_case methods.

In effect, this means that the WP CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. test suite can now run on all PHPUnit versions between PHPUnit 5.7.21 up to the latest release (at the time of writing: PHPUnit 9.5.10), which allows for running the test suite against all supported PHP versions using the most appropriate PHPUnit version for that PHP version.

It also means that, as of mid August, the tests are being run against PHP 8.1 and fixes for PHP 8.1 compatibility are currently being made.

With the PHPUnit Polyfills in place, tests can now be written using the feature set of the highest supported version of PHPUnit.
The Polyfills library will fill in the gaps and ensure the tests can still run on lower versions of PHPUnit without problems.

What has changed?

  1. The Composer lock file has been removed.
    The version constraints in the composer.json file have been made stricter to ensure the developer experience is not negatively impacted by this with regards to coding standards checks.
  2. The PHPUnit Polyfills library at version ^1.0.1 has been added as a Composer dev dependency.
  3. All WordPress Core tests now use PHPUnit 9.x assertions and expectations.
  4. All WordPress Core tests now use snake_case fixture methods, i.e. set_up() instead of setUp() and tear_down() instead of tearDown().
  5. The minimum supported PHPUnit version has been raised to PHPUnit 5.7.21 (was 5.4.0).
  6. The WordPress Core test bootstrap file will no longer throw an error when the tests are being run with PHPUnit 8.x or 9.x.
  7. The WordPress Core test bootstrap file will throw an error when the PHPUnit Polyfills are not available or do not comply with the minimum version requirements.
  8. All WP Core native assertions now have an extra, optional $message parameter, just like all PHPUnit native assertions.
    Please use this parameter in all tests which contain more than one assertion to make debugging tests easier.
  9. The WP_UnitTestCase_Base::setExpectedException() method is deprecated and should no longer be used.
  10. The WP_UnitTestCase_Base::checkRequirements() method is deprecated and no longer functional, and in reality hasn’t been for a long time for anyone using it in combination with PHPUnit 7.0+.
  11. The copies of the PHPUnit SpeedTrapListener classes have been removed as they were never actively used in Core.
    Anyone who still wants to use the SpeedTrapListener can install it separately.
  12. The copies of the PHPUnit 9.x MockObject classes which were introduced in the WP Core test suite in WP 5.6 have been removed, as they are no longer needed when the tests are run on the appropriate PHPUnit version for the PHP version used.

While the above changes have been made in WordPress 5.9, a minimal selection of these changes has been backported to WordPress 5.2 – 5.8:

  1. The PHPUnit Polyfills at version ^1.0.1 is now a requirement for the test suites in WP 5.2 – 5.8 and this requirement will be enforced via the test bootstrap.
  2. … which makes all the polyfills for PHPUnit 9.x assertions and expectations available when running tests against WP 5.2 – 5.8.
  3. Additionally, snake_case wrapper methods have been added for the camelCase fixture method names, meaning that for WP 5.2 – 5.8, the snake_case fixture method names will work without needing further work-arounds, both for fixture declarations as well as for calling the parent::set_up() and the likes.

    There is one caveat to this: the backported implementation presumes a fixed order for calling the parent (camelCase) methods versus the child (snake_case) methods: for set_up*() methods: parent first, child second; for tear_down*() methods: child first, parent second.
    This is the standard order, but if you have a fixture method which diverges from this or doesn’t call the parent, you may get unexpected results.

These backports allow for backporting future (security) fixes for WordPress itself without having to make the accompanying tests compatible with older PHPUnit versions.

These backports will also make it more straightforward for extenders to continue to test their pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party/theme against multiple WordPress versions.

For full details about all the changes, please have a read through Trac ticket 46149.

Changes under the hood which should not be noticeable:

A new PHPUnit_Adapter_TestCase class has been added. This class is nested in-between the WP_UnitTestCase_Base class and the PHPUnit TestCase class and provides the PHPUnit cross-version adapter layer.

All PHPUnit assertion polyfill methods which previously existed in WP Core have been removed as they are no longer necessary now this functionality is provided via the PHPUnit Polyfills library.
All polyfills for assertions/expectations which were previously in WP are still available, they are now just provided via the Polyfills package.

As for the Docker set up: the PHPUnit container is no longer needed and has been removed from the docker-compose config.

What hasn’t changed:

  • The PHPUnit class aliases (for support of PHPUnit 5), which WP provided are still available, though shouldn’t be needed anymore.
  • You can still extend the WP_UnitTestCase class for your tests and will receive access to everything which was available before + more (i.e. a complete set of polyfills).

Future changes

There is a ticket open to rename some of the WordPress native test helper methods to handle the “doing it wrong” and WP native deprecation notices, as the current method names (too) closely resemble a PHPUnit native method name, which can easily lead to confusion and use of the wrong methods in tests.

When that ticketticket Created for both bug reports and feature development on the bug tracker. is actioned, this dev-note will be updated with the relevant information.

What does this mean for contributors to WordPress Core?

In general:

If you use Composer locally, please run composer update --with-all-dependencies (or composer update -W for short) from the root of your WordPress clone now to make sure your install is updated and to get the most appropriate versions of the dependencies for the PHP version you are running on.

Go on, do that now. This dev-note will wait patiently for you to come back.

You will need to run this command semi-regularly in the future (whenever the composer.json file has been updated).

For WP 5.9 and higher, please don’t use composer install anymore.

If, for example for backports, you need to install the dependencies for WP 5.8 or lower, in that case, you still need to run composer install.

🎓 Why?

The first time you run composer install locally, it creates a composer.lock file and when you run Composer again, it will look at your composer.lock file to install the “locked” versions again.

Previously, with the committed composer.lock file, the lock file was managed and updated centrally. However, that also meant that you often would be running the dev tools at a version which wasn’t the most appropriate one for the PHP version you are working under. This was getting more and more problematic for running the tests, which is why the file was removed.

Now the composer.lock file is no longer committed, you have to update it yourself to make sure you receive the latest version of the dev dependencies appropriate for your PHP version and within the version constraints set in the composer.json file.

For running the Core tests:

If you usually run the Core tests via Docker using the npm run test:php command, you can continue to do so and all should still work as expected.

If you usually run the Core tests via a Composer installed version of PHPUnit, again, you can continue to do so and all should still work as expected as long as you followed the above instructions to run composer update -W first.

If you usually run the Core tests via a PHAR file, you either have to run composer update -W once in a while or you have to set up a clone of the PHPUnit Polyfills repo. For more information about this last option, please see the set up information in the handbook.
If you are running locally on PHP 7.2 or higher, you may want to download a more recent PHPUnit PHAR file (PHPUnit 8 or 9) to benefit from the advances which have been made in PHPUnit.

If you are running the tests locally on PHP 7.2 or higher, you may notice the test runs being faster and the output being enhanced as the tests will now run on a more recent PHPUnit version.

💡 Pro-tip:

Now might also be a good time to verify that your local wp-tests-config.php file is still in sync with the wp-tests-config-sample.php file.

Similarly, if you use a local phpunit.xml overload configuration file, it is strongly recommended to verify that any changes made in the phpunit.xml.dist (and multisite.xml) file are synced into your local configuration.

For writing tests for Core:

You can now use the full range of assertions as available in PHPUnit 9.5 in your tests. Please use the most appropriate assertion available.

Test fixture methods MUST use snake_case method names from now on as per the below table.

Old nameNew name
setUpBeforeClass()set_up_before_class()
setUp()set_up()
assertPreConditions()assert_pre_conditions()
assertPostConditions()assert_post_conditions()
tearDown()tear_down()
tearDownAfterClass()tear_down_after_class()

The Make Core handbook page about writing tests has been updated with this information.
The page has also been enhanced with more handy tips and tricks, so please have a read through!

What does this mean for plugins/themes running integration tests based on the WP Core test suite?

It is a known fact that there are a lot of plugins/themes which use the WordPress Core test framework as a basis for their integration tests.

If your plugin/theme is one of them, these changes will impact you as well.

Step-by-step: how to make your test setup compatible with these changes and with higher PHPUnit versions:

  1. Run your tests against PHP 7.4 with PHPUnit 7.x and WP 5.8.1 and make sure there are no pre-existing errors/failures.
  2. Add PHPUnit Polyfills as a Composer require-dev dependency (or inherit it from WP).
  3. If you add the Polyfills as a requirement and only support WP 5.9 and higher, remove the requirement for PHPUnit in favour of letting the Polyfills handle it. This will prevent potential future version constraint conflicts.
    • If you still need/want to run your tests against older WP versions, keep the PHPUnit requirement and make sure it is set to ^5.7.21 || ^6.5 || ^7.5 and let CI (continuous integration script) handle removing that requirement for WP 5.9.
    • Or do it in reverse and remove the requirement for dev and add it back in CI for older WP versions.
  4. Make sure the Polyfills autoloader is wired in to your test bootstrap.
    • If you’ve chosen to “inherit the Polyfills from WP”, in this context that means that you use a full clone of WordPress and will install the Composer dependencies for WordPress before running the tests. In that case, you should be all set.
    • If you use only a partial clone of WordPress, like when your tests have been set up using the WP-CLIWP-CLI WP-CLI is the Command Line Interface for WordPress, used to do administrative and development tasks in a programmatic way. The project page is http://wp-cli.org/ https://make.wordpress.org/cli/ scaffold command, or if you don’t run WordPress’ Composer setup, please make sure you load the Polyfills autoloader in your test bootstrap before running the WP native test bootstrap.
      • If you include your Composer vendor/autoload.php file as your test bootstrap before you run the WP native test bootstrap, you’re all set already, the Polyfills autoloader will be included automatically.
      • Alternatively, you can add a require_once 'path/to/vendor/yoast/phpunit-polyfills/phpunitpolyfills-autoload.php'; in your test bootstrap before including the WP native test bootstrap.
      • As a last alternative, you can declare a WP_TESTS_PHPUNIT_POLYFILLS_PATH constant containing the absolute path to the root directory of the PHPUnit Polyfills installation in your plugin/theme’s own test bootstrap file.
        Again, this constant must be declared prior to running the WP native test bootstrap file.
  5. Search your codebase for declarations of the fixture methods, as well as calls to (parent) fixture methods, and replace camelCase with snake_case in the method names.
    Example:
    // Old:
    public function setUp() {
         parent::setUp();
         // Do something.
    }
    
    // New:
    public function set_up() {
        parent::set_up();
        // Do something.
    }
  6. Verify your tests run without errors after the changes by running them against PHP 7.4 on PHPUnit 7.x with WP 5.8 .
  7. Verify your tests run without errors after the changes by running them against PHP 7.4 on PHPUnit 7.x with WP trunk (= WP 5.9).
  8. While using WP trunk/5.9, switch to PHPUnit 8.x – look out for deprecation notices PHPUnit throws and just literally do what they tell you to do.
  9. While still using WP trunk/5.9, switch to PHPUnit 9.x – look out for deprecation notices PHPUnit throws and just literally do what they tell you to do.

Once you’ve run through these steps, your tests should be cross-version compatible with PHPUnit 5.7.21 – 9.5, able to run against the WordPress 5.2 to 5.9 branches and able to run on PHP 5.6 – 8.1.

Next, you may want to run your tests against PHP 8.0 and 8.1 using PHPUnit 9.x with WP 5.9 to see if your plugin/theme is compatible with these more recent PHP versions.

🚨 Pro-tip:

If you want your CI build to fail when PHPUnit encounters PHP native deprecation notices, make sure to add convertDeprecationsToExceptions="true" to your PHPUnit configuration file as the default value for this setting has been changed to false in PHPUnit 9.5.10/8.5.21.

Enabling this setting is strongly recommended for testing your plugin/theme against PHP 8.1, as PHP 8.1 introduces a lot of new deprecations.

What to do when running tests in CI against multiple WP/PHP combinations?

If you are running your plugin/theme integration tests against multiple WordPress and PHP combinations, you will most likely need to make some adjustments to your Continuous Integration (CI) script(s).

Which exact changes you need to make depends entirely on your specific setup. There is no “one size fits all” solution.

As a general rule of thumb:

  • WP 5.2 – 5.5 is able to run tests against PHP 5.6 – 7.4 with PHPUnit 5.x (PHP 5.6 and 7.0) – 7.x (PHP 7.1 and higher).
  • WP 5.6 – 5.8 is able to run tests against PHP 5.6 – 8.0 with PHPUnit 5.x (PHP 5.6 and 7.0) – 7.x (PHP 7.1 and higher).
  • WP 5.9 and higher is able to run tests against PHP 5.6 – 8.1 with PHPUnit 5.x – 9.x (use the most appropriate PHPUnit version for each PHP version).

Also see the PHP Compatibility and WordPress Versions and PHPUnit Compatibility and WordPress Versions pages in the Make Core handbook.

Other typical things to take into account and to work around when needed:

  • Is there a config - platform - php setting in your composer.json which fixes the PHP version to a specific version – typically PHP 5.6 – for installing dependencies ?
    If so, you may need to either selectively remove this setting or run Composer with --ignore-platform-reqs for certain WP/PHP combinations in your test matrix.
  • Has the composer.lock file been committed ?
    In that case, you may need to either selectively remove that file in CI before running composer install; or run composer update -W for certain WP/PHP combinations in your test matrix.
  • Do you use a complete clone of WP ?
    For WP 5.2 – 5.8, you’ll need to install the WP dependencies by using composer install.
    For WP 5.9 and higher, you’ll need to install the WP dependencies by using composer update -W.

To make sure you run the test against the right PHPUnit version, you may need to run (a variation on):

composer remove --dev phpunit/phpunit
composer update --dev yoast/phpunit-polyfills --with-dependencies --ignore-platform-reqs

💡 Did you know ?

If you use GitHubGitHub GitHub is a website that offers online implementation of git repositories that can can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged be the repository owner. https://github.com/ Actions to run your tests for continuous integration and PHPUnit and the PHPUnit Polyfills are your only external test dependencies, as of Setup-PHP 2.15.0 (expected soon), you can use the tools key in the shivammathur/setup-php action to install these:

- name: Setup PHP with tools
  uses: shivammathur/setup-php@v2
  with:
    php-version: '8.0'
    tools: phpunit-polyfills

For more information about the tools key for setup-php, see the action documentation.

For more information on how to wire in the PHPUnit Polyfills when installed via setup-php, see the FAQ section of the Polyfills documentation.

Anticipating some frequently asked questions

I’m a plugin/theme maintainer, but don’t use Composer, can I still run my integration tests?

Yes, but you do need to make sure that either the Polyfills are available via a Composer global or local installLocal Install A local install of WordPress is a way to create a staging environment by installing a LAMP or LEMP stack on your local computer. or via some other manner, like a clone of the repo.

If you haven’t looked at Composer before, now might be a good time to take a look at it.

I’m running my tests via another tool stack (like BrainMonkey, WP Mock, PHP Mock, WP Browser, PestPHP), how do the changes made to the WordPress test suite affect me?

Short answer: They don’t.

Long answer: if you want to run your tests against multiple PHP and PHPUnit combinations, you may still find the PHPUnit Polyfills library helpful to you.

If you’ve not heard of the above mentioned tools before and want to read up on them, here are some links:

I used the WP-CLI scaffold command to set up my integration tests. How do the changes made to the WordPress test suite affect me?

There is no automated way right now to adapt existing tests for which the initial was created via the WP-CLI scaffold command, to make use of the new setup.

The current recommendation is to go through the steps in “What does this mean for plugins/themes running integration tests based on the WP Core test suite?“, which might be more or less involved depending on what version of the scaffolded test setup you are currently using.

A future version of the WP-CLI scaffold plugin-tests command will provide an upgrade mechanism to automatically upgrade an existing test setup to the new requirements. This will include adding a fully Composer-based testing setup as a replacement for the current bootstrap logic, making a composer update possible in the future to keep up with further test setup changes.

If you’re interested in learning more about these plans for the future, please subscribe to the issue on GitHub to stay informed.

I’m using WP Test Utils for my unit and integration tests. How do the changes made to the WordPress test suite affect me?

WP Test Utils is a library offering utilities for both unit testing and integration testing for WordPress plugins and themes. WP Test Utils already includes the PHPUnit Polyfills.

For the unit testing part, which is based on BrainMonkey, you are not affected by the changes.

If you use the integration testing utilities, you will need to make the change from camelCase to snake_case for the fixture methods in your test suite and you can now potentially widen the PHPUnit version requirements for your integration tests (also see the information about this in step 3 of the “Step by step” guide and the information about adjusting CI scripts).

Presuming you were already using the PHPUnit Polyfills provided by the Test Utils to use modern assertions, that’s it. You’re done.

WP Test Utils will continue to handle the integration test bootstrapping, which allows for running the tests against multiple WordPress and PHP versions.

The first version of WP Test Utils which has full support for the test framework changes made in WP 5.9, is WP Test Utils 1.0.0.

WP Test Utils 1.0.0 also includes improved support for integration tests which were created using the WP-CLI scaffold command and support for running tests against WP versions which don’t include the backports, like WP 5.2 – 5.8 point releases released before today, as well as WP < 5.2.


Props to @hellofromtonya, @johnbillion, @dingo_d, @netweb, @sergeybiryukov, @swissspidy, @schlessera for reviewing ahead of publication.

#5-9, #build-test-tools, #dev-notes, #phpunit, #unit-tests

Recent Build/Test Tool changes and GitHub Actions update

It’s been a busy year so far for the Build/Test Tool component! Here are some notable changes to be aware of, and an update on the transition to using GitHub Actions for all automated testing.

NodeJS 14.x LTS support

NodeJS 14.x has been the active LTS version since April of 2020. While dependencies were updated to ensure support and related build scripts have worked on 14.x for some time now, the package.json file in CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. now officially recommends using NodeJS 14.15.0 and NPM 6.14.8.

The versions specified in the engines field of the package.json file have also been updated to specify a range of versions (>=14.15.0) instead of explicit versions (14.15.0). This should make it more clear to contributors that they can use any version newer than the one specified.

For more information on these changes, check out #51749 and #52455 on TracTrac An open source project by Edgewall Software that serves as a bug tracker and project management tool for WordPress..

Consistent tooling across all branches

The WordPress project’s current support policy is that only the most recent major version should be considered supported. At the time of this post, this means that 5.7 is the only maintained branchbranch A directory in Subversion. WordPress uses branches to store the latest development code for each major release (3.9, 4.0, etc.). Branches are then updated with code for any minor releases of that branch. Sometimes, a major version of WordPress and its minor versions are collectively referred to as a "branch", such as "the 4.0 branch".. However, security fixes are backported as a courtesy in an effort to promote a more secure web (currently) all the way back to the 3.7 branch.

Because of changes in TravisCI’s services, the 3.7-5.5 branches did not have working automated testing configured since the first week of December. To fix this, the GitHubGitHub GitHub is a website that offers online implementation of git repositories that can can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged be the repository owner. https://github.com/ Actions workflow files added to trunk needed to be backported. This could be accomplished for the 4.6-5.5 branches fairly easily after [49527-49533].

However, because build/test tool functionality has not been maintained in old branches, there were a few blockers and consistency issues that needed to be resolved before the necessary workflows could be backported further.

Outdated NodeJS versions

Because old branches are not maintained and only receive security updates as a courtesy, the version of NodeJS used in each branch largely reflected the active version of NodeJS when the branch was created. For example, the 3.7 branch used v0.10.48, 4.2 used v0.12.18, 4.3 used v4.7.2, etc..

This was recently made easier through the use of nvm (Node Version Manager) and .nvmrc files (see #51682), but the dependencies needed to run the local Docker environment (more on that below) do not support these older versions of NodeJS.

After [50187-50224], all older branches currently receiving security updates have been updated to support the most recent version of NodeJS LTS (currently 14.x) and all devDependencies specified in the package.json files have been updated to their latest versions.

The 3.7-4.9 branches also contained an npm-shrinkwrap.json file. This is a type of lock file that predates the newer package-lock.json file. Since all newer branches utilize a package-lock.json file to specify the exact desired versions of dependencies, all npm-shrinkwrap.json files have been replaced with package-lock.json ones in old branches for consistency.

Note: Because the dependencies are responsible for processing JSJS JavaScript, a web scripting language typically executed in the browser. Often used for advanced user interfaces and behaviors., SCSS, and CSSCSS Cascading Style Sheets. files, this will result in most of these files being updated in the next release of each old branch.

For more information on these changes, see #52341 on Trac.

Inconsistent tooling

Another side effect of only backporting security fixes to unmaintained branches is inconsistent tooling. Because tools that make contributing easier are updated and added in each release, switching to an older branch to create a patchpatch A special text file that describes changes to code, by identifying the files and lines which are added, removed, and altered. It may also be referred to as a diff. A patch can be applied to a codebase for testing. becomes much more difficult. The tools contributors are accustomed to using are not there or work differently, and then they have to spend time figuring out how things were done in the past before they can contribute.

All branches now contain the same basic scripts needed to work on WordPress locally. This includes:

  • npm run build
  • npm run watch
  • npm run grunt

Additionally, the grunt-patch-wordpress package has been updated to the latest version in all branches. It has also been added to the 3.7 and 3.8 branches where it was missing.

More information on these changes can also be found in #52341 on Trac.

Local Docker environment

Since WordPress 5.3, a local Docker environment configuration has been included in wordpress-develop to provide an easy way for contributors to configure their own development environment and serve as a more consistent testing environment (mainly for Core’s PHPUnit tests). This environment has also been used for all automated testing in branches 5.3 and newer since being introduced.

However, because of the blockers detailed above, this environment could not be backported to the 3.7-4.5 branches.

After those blockers were resolved, the local Docker environment was then merged into older branches in [50243-50251] ensuring all branches receiving security updates can use the Docker environment.

Note: PHPPHP The web scripting language in which WordPress is primarily architected. WordPress requires PHP 5.6.20 or higher 5.2 is not currently included in the testing workflows for older branches that supported this version. Adding this for true parity with the old TravisCI testing configuration is blocked by the local Docker environment’s PHP 5.2 PHPUnit image not containing the correct version of PHPUnit.

For more information on this, see #48301 (and the previous #47767) on Trac.

Transitioning to GitHub Actions for automated testing

If you’re unfamiliar with the ongoing transition moving all automated testing from TravisCI/Appveyor to GitHub Actions, the initiative introduction and follow-up posts will help bring you up to speed. This transition has been continuing, and is nearing completion.

As of [50324], automated testing has been restored for all branches still receiving security updates as a courtesy. This includes (where supported) PHPUnit testing, NPM testing, JSHint/PHPCSPHP Code Sniffer PHP Code Sniffer, a popular tool for analyzing code quality. The WordPress Coding Standards rely on PHPCS. linting, QUnit testing, and PHPCompatibility scanning.

In addition to restoring automated testing to branches receiving security updates, there have been a number of performance improvements to the workflows. Most notably, this has resulted in a roughly 70% total decrease in overall runtime for the PHPUnit test workflow.

Here are some additional changes related to GitHub Actions.

Note: All of the changes and improvements listed below have been backported through the 3.7 branch unless otherwise noted.

Limiting when workflows run

Because there are over 750 forks of the wordpress-develop mirror, it’s important to limit when workflows run appropriately. For private repositories, owners are given an allotment of free workflow minutes and are then charged for every minute used over that number. If no limitations are added, each workflow would run on every push event for every fork, including any additional public or private mirrors that are maintained. This would not only waste resources (running for every push to a fork is not necessary as most people open a pull request anyway), but could also unintentionally exhaust a user’s or organization’s private workflow minutes.

To help with this, all workflows have been updated to run only under the following conditions:

  • Every push event to wordpress-develop for the primary branch, branches >= 3.7, and all tags matching the pattern x.y.z that is >= 3.7.0.
  • Every pull request to wordpress-develop with the primary branch or branches >= 3.7 as the base.
  • Every pull request to a fork or private mirror repository with the primary branch or branches >= 3.7 as the base.

For pull request workflows, additional path filtering has been added so that workflows only run when relevant files are changed. For example, the JavaScriptJavaScript JavaScript or JS is an object-oriented computer programming language commonly used to create interactive effects within web browsers. WordPress makes extensive use of JS for a better user experience. While PHP is executed on the server, JS executes within a user’s browser. https://www.javascript.com/. testing workflow will only run when a JavaScript file, QUnit related file, or relevant configuration files (.jshintrc, package.json, etc.) are changed.

These conditions will help limit the number of workflow runs that occur throughout the contributor base without limiting the ability to test and verify changes.

For more information on these changes, see #52643 and #52667 on Trac.

Regular testing of old branches

In TravisCI, there was a feature to configure regular testing of a repository’s branches at given intervals. This was used to run the test suite for:

  • The currently maintained branch daily when there has not been a successful build in the last 24 hours.
  • The previous branch weekly when there has not been a successful build in the last 24 hours.
  • All other branches receiving security updates monthly when there has not been a successful build in the last 24 hours.

There is a schedule event for triggering workflows in GitHub Actions, but it only runs in the primary branch of a repository, so it cannot be used at the workflow level to ensure regular testing of branches.

Instead, a new workflow has been introduced that will run on a schedule and initiate all of the necessary workflows for old branches using the REST APIREST API The REST API is an acronym for the RESTful Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. It is how the front end of an application (think “phone app” or “website”) can communicate with the data store (think “database” or “file system”) https://developer.wordpress.org/rest-api/. to trigger a workflow_dispatch event. This workflow will test the two most recent branches twice per month, and all other branches monthly.

For more information on this, see #52653 on Trac.

Code coverage reporting

Generating a code coverage report for the code base has been supported for some time (see [42665]). But, reports have never been generated and aggregated on a regular basis. To correct this, a new workflow has been introduced to run a test coverage report daily (see [49834]).

The reports generated are now submitted to Codecov and can be viewed here.

This will hopefully give contributors interested in test coverage the ability to find areas of the code base with little to no testing, and provide some insight into how code coverage increases or decreases over time.

Note: The code coverage will only be reported for the primary branch.

For more information, see #52034 on Trac.

Additional improvements to GitHub Actions workflows

  • The NPM testing workflow has been generalized to not only verify the build tools work on Windows, but Linux and MacOS as well. Steps have also been added to test additional scripts, such as npm run build:dev and npm run grunt clean (see #52658).
  • The fail-fast option has been disabled for the NPM (see [50579]) and PHPUnit (see #52612) testing workflows. fail-fast is great for being alerted of a failure faster, but does not give the full picture of what conditions cause the failure.
  • The method of installing dependencies by use of npx install-changed has been replaced with using npm ci after comparisons found the latter was more performant (see #52660).
  • All restore-keys options for the actions/cache action have been removed in order to prevent the cache from snowballing due to lax cache key matching. This resulted in a 40% decrease in the NPM dependency cache size (see #52660).
  • The restapi-jsclient test group is no longer run separately. This group was never excluded in the phpunit.xml.dist file, so it already runs as part of the main test suite (see #52608).
  • The single site and multisitemultisite Used to describe a WordPress installation with a network of multiple blogs, grouped by sites. This installation type has shared users tables, and creates separate database tables for each blog (wp_posts becomes wp_0_posts). See also network, blog, site tests have been split into separate jobs that run in parallel. This has reduced the overall runtime for the PHPUnit workflow by over approximately 30% from the previous run (see #52548).
  • Because the test suite takes significantly longer to run on PHP <= 5.6, the PHPUnit workflow has been updated to run test groups that are considered slow (restapi, media, and external-http) in separate, parallel jobs. This reduced the overall time for the workflow by 34% from the previous run (see #52645).

Known transition steps remaining (updated)

The following items remain to achieve parity with the previous testing configurations on TravisCI.

  1. Add and configure SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. notifications. In addition to sending the results of the whole build of a core commit into #core, we may also want to consider a firehose channel for PRs. This may require all workflows to be combined into a single workflow if needed middleware cannot be found.
  2. Move to GitHub badges for build status indicators – note that these are per-workflow, which is different from the single badge for the entire Travis build for a given commit. However, GitHub does report an overall status for a commit/PR, so we may be able to use that information as well. It seems that the expectation in the greater developer community is that projects report status with a singular badge. Like the Slack notifications, this may require the workflows to be combined in the absence of middleware.
  3. Backportbackport A port is when code from one branch (or trunk) is merged into another branch or trunk. Some changes in WordPress point releases are the result of backporting code from trunk to the release branch. the workflow files to unsupported older branches receiving security updates.
  4. Finish backporting the local Docker environment to branches 3.7-4.5. This is blocked by:
    • wpdev-docker-images#46, which aims to fix the PHP 5.2 PHPUnit image to include the requred version of PHPUnit (3.6).
    • WP branches <= 4.5 are running a version of NodeJS that is too old for the needed NPM packages required to run the local Docker environment.
  5. Report test results to the Host Test Results page. Completed, but the MySQLMySQL MySQL is a relational database management system. A database is a structured collection of data where content, configuration and other options are stored. https://www.mysql.com/. version being tested is not currently being reported (see phpunit-test-runner#135).

Running tests from src (again)

Since the build process was overhauled in WordPress 5.1 (see #43309), automated testing has been running from the build directory. Running from build introduces several problems that makes contributing difficult:

  • Running a build is slow. It copies all the files and builds, validates, and minifies all the CSS and JS. None of this should be necessary for PHP testing.
  • Developing with grunt watch can give issues on some development environments that run in VirtualBox (like VVV), where changes aren’t being picked up. Having to rebuild manually after each change is a hassle.
  • A developer iterating on a patch in the source file has no way of knowing that their file is not actually being tested when running the tests, unless they run the build each time or start and run the file watcher. This is an easy step to forget.
  • PHP errors display a stack trace from build instead src.
  • Breakpoint debugging isn’t fun as it also uses the stack trace from build instead of src.

The build process was adjusted to allow building and cleaning src again in #44492, but the default directory for automated testing remained build because of some unit testunit test Code written to test a small piece of code or functionality within a larger application. Everything from themes to WordPress core have a series of unit tests. Also see regression. failures.

The remaining items blocking the PHPUnit test suite from running against the src directory have been fixed in [50441] and the default branch for testing has been switched back to src. The PHPUnit testing workflow has also been updated to run the Core test suite from the src directory. This change resulted in an approximately 28% decrease in the workflow’s overall run time from the previous run.

Note: This change has only been merged back to the 5.2-5.7 branches, as this was where it seemed reasonable to draw the line for this change when weighing effort vs. benefits.

For more information on this change, see #51734 on Trac. Related tickets: #43055, #44492, #45863.

Additional changes of note

Here are some other, additional build/test tool changes to make note of:

  • The Docker-based local environment now installs the WordPress Importer pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party into the tests/phpunit/data/plugins directory as part of the npm run env:install script. This eliminates the extra step required when running the unit test suite locally (see #49720).
  • MariaDB support has been added to the local Docker environment. To substitute MySQL for MariaDB, change the values of the LOCAL_DB_TYPE and LOCAL_DB_VERSION to mariadb and a valid MariaDB version in the .env file, in your local CLICLI Command Line Interface. Terminal (Bash) in Mac, Command Prompt in Windows, or WP-CLI for WordPress. configuration file (such as bashrc), or by setting the variable’s value in your session (see #51744).
  • The deprecated node-sass package has been substituted with the recommended replacement, DartSass (sass), which uses the same Javascript APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. (see #51763).
  • The svn:global-ignores and svn:ignore properties have been synchronized with the .gitignore file. These SVNSVN Subversion, the popular version control system (VCS) by the Apache project, used by WordPress to manage changes to its codebase. properties had fallen out of date, and several exclusions defined for contributors using GitGit Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. Git is easy to learn and has a tiny footprint with lightning fast performance. Most modern plugin and theme development is being done with this version control system. https://git-scm.com/. were not defined for those using SVN (see #49784).

Unmaintained branches and build/test tools going forwards

Even though these changes were merged all they through the 3.7, WordPress’ official stance continues to be that only the most recent release (5.7 at the time of this post) should be considered supported. The 3.7-5.6 branches will continue to only receive security fixes going forward (though there have recently been discussions about reducing the number of versions receiving security updates).

The changes above were merged into older branches because they were necessary to restore automated testing, or to maintain consistency in testing configurations across branches.

As changes are made to build/test tool areas of the code base going forward, component maintainers will use their judgement in determining which changes should be backported to these older branches. Changes can be grouped and backported as necessary. To start, reviewing these quarterly or after each major releasemajor release A release, identified by the first two numbers (3.6), which is the focus of a full release cycle and feature development. WordPress uses decimaling count for major release versions, so 2.8, 2.9, 3.0, and 3.1 are sequential and comparable in scope. can be used as the frequency.

Immense props go out to @johnbillion. Almost all of the performance related changes detailed above were his ideas and contributions. Props to @jorbin, @davidbaumwald, and @sergeybiryukov for peer review.

#build-test-tools, #github-actions

Continuing the transition to GitHub Actions for automated testing

Two months ago, the configuration files needed to run automated tests using GitHubGitHub GitHub is a website that offers online implementation of git repositories that can can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged be the repository owner. https://github.com/ Actions were added to WordPress CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress..

GitHub Actions allows us to automate software workflows directly in GitHub, triggered by GitHub events. By switching, we are able to take advantage of a unified interface, inline annotations for linting issues in pull requests, the broader open sourceOpen Source Open Source denotes software for which the original source code is made freely available and may be redistributed and modified. Open Source **must be** delivered via a licensing model, see GPL. ecosystem building and using Actions including existing work in Gutenberg, and free availability for public repositories. Note that private repositories do use the monthly bucket of included minutes.

Introducing GitHub Actions for Automated Testing

During this transition process, the workflows have been running smoothly with only small issues coming up (which have been fixed). The workflows are also performing better or identically to those run in TravisCI.

With the branching of 5.6, the workflows have been running for both trunk and the 5.6 branches. This has allowed them to be tested successfully in both the primary and non-primary branches.

Since [49162], an additional workflow that generates a code coverage report has also been introduced. Currently, this workflow generates coverage reports for both single and multisite installs and uploads the reports as artifacts to the workflow run. For now, this workflow runs weekly on Sunday at 00:00 UTC. #52141 has been opened to explore submitting the reports to an external service

Additionally, TravisCI has made further changes to their pricing models. As a result, no builds for the WordPress organization have run since the first week of December. When this happened, efforts to transition all repositories under the WordPress GitHub organization from TravisCI to GitHub Actions were accelerated.

As of today, all repositories have been fully transitioned to GitHub Actions except the WordPress Coding Standards (which has an open PR being actively refined), and the ones noted below. These are lower priority repositories because they are updated much less frequently than others. Here they are ranked in order of priority highest to lowest:

Pull requests are very much welcome!

The Appveyor build configuration (which has been replaced with the Test NPM on Windows workflow) was removed from Core in [49779,49809-49814], and the TravisCI configuration was removed in [49876-49898].

Forked repositories

To avoid workflows from running unnecessarily, conditional checks have been added to prevent all workflows from running in forked repositories. There are two exceptions to this:

  • When a pull request is opened to the official wordpress-develop repository from a fork.
  • When a pull request is opened to a forked repository from within a fork.

Known next steps (updated)

  1. Add and configure SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. notifications. In addition to sending the results of the whole build of a core commit into #core, we may also want to consider a firehose channel for PRs. This may require all workflows to be combined into a single workflow if needed middleware cannot be found.
  2. Backportbackport A port is when code from one branch (or trunk) is merged into another branch or trunk. Some changes in WordPress point releases are the result of backporting code from trunk to the release branch. the workflow files to actively maintained older branches.
  3. Finish backporting the local Docker environment to branches 3.7-4.5. This is blocked by:
    • wpdev-docker-images#46, which aims to fix the PHPPHP The web scripting language in which WordPress is primarily architected. WordPress requires PHP 5.6.20 or higher 5.2 PHPUnit image to include the requred version of PHPUnit (3.6).
    • WP branches <= 4.5 are running a version of NodeJS that is too old for the needed NPM packages required to run the local Docker environment.
  4. Move to GitHub badges for build status indicators – note that these are per-workflow, which is different from the single badge for the entire Travis build for a given commit. However, GitHub does report an overall status for a commit/PR, so we may be able to use that information as well. It seems that the expectation in the greater developer community is that projects report status with a singular badge. Like the Slack notifications, this may require the workflows to be combined in the absence of middleware.
  5. Report test results to the Host Test Results page. The MySQLMySQL MySQL is a relational database management system. A database is a structured collection of data where content, configuration and other options are stored. https://www.mysql.com/. version being tested is not currently being reported (see phpunit-test-runner#135).
  6. Switch to ESLint from JSHint, as the latter does not appear to easily support inline annotations, and the former is in broader usage including in core for docs, GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ and many community projects. See #31823 for more – volunteers very much appreciated here.

Committers: Until the Slack notifications are configured, please monitor the workflow runs associated with your commits to ensure they pass.

The goal is for the remaining repositories and items 1-3 above to be completed in January 2021.

Props @helen for peer reviewing this post.

#build-test-tools, #github, #github-actions