Recent updates to the unit testing infrastructure

Several changes have made their way into WordPress’ unit test suite and its infrastructure recently. If you maintain a WordPress project (such as a plugin or website) that has its own test coverage, you may be interested in porting some of these changes to your test suite too. Here’s a round-up:

  • Tests which require Multisite to be enabled, or tests which should be excluded from running when Multisite is enabled, can now be placed into the new ms-required and ms-excluded groups to skip them automatically when the test requires Multisite to be in use, or when it requires Multisite to not be in use, respectively. In addition, the default configuration in phpunit.xml now excludes such test groups as appropriate without them being marked as skipped, which removes unwanted noise from PHPUnit’s reporting when run in verbose mode. See #40531.

Note: Please note that this usage example has been updated from a previous version of this post.

Example usage:

/**
 * @group ms-required
 */
function test_something_on_multisite() {
	// Run your Multisite-only assertions here
}

/**
 * @group ms-excluded
 */
function test_something_without_multisite() {
	// Run your single-site-only assertions here
}
  • XDebug is now disabled when running tests on Travis in order to speed up the builds. This resulted in a huge reduction in build times, up to 40% in some cases. See #39978.
  • The PHPUnit version used when running tests on Travis is now explicitly coupled to the PHP version in order to avoid incompatibilities such as the one caused when PHPUnit 6 was rolled out. See [40269] for the main change and #40100 and #40086 for more info.
  • A compatibility shim for PHPUnit 6+ has been added, which means the test suite will no longer cause fatal errors if you want to update to PHPUnit 6+ locally or, for example, you want to require it through a Composer dependency. See #39822.
  • Several tests which were previously skipped when the environment wasn’t suitable will now cause failures, for example tests which depend on constants such as DISALLOW_FILE_MODS not being set. This is to avoid hiding tests that never run and therefore never fail if the assertions contained within them would otherwise fail. If the core test suite begins failing on your local environment as a result, you should address the cause. See #40533.
  • The number of PHP versions which older branches are tested against on Travis has been significantly reduced in order to speed up build times for patch releases. This is most beneficial when security fixes are released and backported. As a general rule, branch builds except the latest branch are now only tested on the oldest supported version (5.2), the latest in the 5.x branch, and the latest in the 7.x branch. See #40407.

#dev-notes