It’s been a busy year so far for the Build/Test Tool component! Here are some notable changes to be aware of, and an update on the transition to using GitHub Actions for all automated testing.

NodeJS 14.x LTS support

NodeJS 14.x has been the active LTS version since April of 2020. While dependencies were updated to ensure support and related build scripts have worked on 14.x for some time now, the package.json file in CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. now officially recommends using NodeJS 14.15.0 and NPM 6.14.8.

The versions specified in the engines field of the package.json file have also been updated to specify a range of versions (>=14.15.0) instead of explicit versions (14.15.0). This should make it more clear to contributors that they can use any version newer than the one specified.

For more information on these changes, check out #51749 and #52455 on TracTrac An open source project by Edgewall Software that serves as a bug tracker and project management tool for WordPress..

Consistent tooling across all branches

The WordPress project’s current support policy is that only the most recent major version should be considered supported. At the time of this post, this means that 5.7 is the only maintained branchbranch A directory in Subversion. WordPress uses branches to store the latest development code for each major release (3.9, 4.0, etc.). Branches are then updated with code for any minor releases of that branch. Sometimes, a major version of WordPress and its minor versions are collectively referred to as a "branch", such as "the 4.0 branch".. However, security fixes are backported as a courtesy in an effort to promote a more secure web (currently) all the way back to the 3.7 branch.

Because of changes in TravisCI’s services, the 3.7-5.5 branches did not have working automated testing configured since the first week of December. To fix this, the GitHubGitHub GitHub is a website that offers online implementation of git repositories that can can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged be the repository owner. https://github.com/ Actions workflow files added to trunk needed to be backported. This could be accomplished for the 4.6-5.5 branches fairly easily after [49527-49533].

However, because build/test tool functionality has not been maintained in old branches, there were a few blockers and consistency issues that needed to be resolved before the necessary workflows could be backported further.

Outdated NodeJS versions

Because old branches are not maintained and only receive security updates as a courtesy, the version of NodeJS used in each branch largely reflected the active version of NodeJS when the branch was created. For example, the 3.7 branch used v0.10.48, 4.2 used v0.12.18, 4.3 used v4.7.2, etc..

This was recently made easier through the use of nvm (Node Version Manager) and .nvmrc files (see #51682), but the dependencies needed to run the local Docker environment (more on that below) do not support these older versions of NodeJS.

After [50187-50224], all older branches currently receiving security updates have been updated to support the most recent version of NodeJS LTS (currently 14.x) and all devDependencies specified in the package.json files have been updated to their latest versions.

The 3.7-4.9 branches also contained an npm-shrinkwrap.json file. This is a type of lock file that predates the newer package-lock.json file. Since all newer branches utilize a package-lock.json file to specify the exact desired versions of dependencies, all npm-shrinkwrap.json files have been replaced with package-lock.json ones in old branches for consistency.

Note: Because the dependencies are responsible for processing JSJS JavaScript, a web scripting language typically executed in the browser. Often used for advanced user interfaces and behaviors., SCSS, and CSSCSS Cascading Style Sheets. files, this will result in most of these files being updated in the next release of each old branch.

For more information on these changes, see #52341 on Trac.

Inconsistent tooling

Another side effect of only backporting security fixes to unmaintained branches is inconsistent tooling. Because tools that make contributing easier are updated and added in each release, switching to an older branch to create a patchpatch A special text file that describes changes to code, by identifying the files and lines which are added, removed, and altered. It may also be referred to as a diff. A patch can be applied to a codebase for testing. becomes much more difficult. The tools contributors are accustomed to using are not there or work differently, and then they have to spend time figuring out how things were done in the past before they can contribute.

All branches now contain the same basic scripts needed to work on WordPress locally. This includes:

• npm run build
• npm run watch
• npm run grunt

Additionally, the grunt-patch-wordpress package has been updated to the latest version in all branches. It has also been added to the 3.7 and 3.8 branches where it was missing.

More information on these changes can also be found in #52341 on Trac.

Local Docker environment

Since WordPress 5.3, a local Docker environment configuration has been included in wordpress-develop to provide an easy way for contributors to configure their own development environment and serve as a more consistent testing environment (mainly for Core’s PHPUnit tests). This environment has also been used for all automated testing in branches 5.3 and newer since being introduced.

However, because of the blockers detailed above, this environment could not be backported to the 3.7-4.5 branches.

After those blockers were resolved, the local Docker environment was then merged into older branches in [50243-50251] ensuring all branches receiving security updates can use the Docker environment.

Note: PHPPHP The web scripting language in which WordPress is primarily architected. WordPress requires PHP 5.6.20 or higher 5.2 is not currently included in the testing workflows for older branches that supported this version. Adding this for true parity with the old TravisCI testing configuration is blocked by the local Docker environment’s PHP 5.2 PHPUnit image not containing the correct version of PHPUnit.

For more information on this, see #48301 (and the previous #47767) on Trac.

Transitioning to GitHub Actions for automated testing

If you’re unfamiliar with the ongoing transition moving all automated testing from TravisCI/Appveyor to GitHub Actions, the initiative introduction and follow-up posts will help bring you up to speed. This transition has been continuing, and is nearing completion.

As of [50324], automated testing has been restored for all branches still receiving security updates as a courtesy. This includes (where supported) PHPUnit testing, NPM testing, JSHint/PHPCSPHP Code Sniffer PHP Code Sniffer, a popular tool for analyzing code quality. The WordPress Coding Standards rely on PHPCS. linting, QUnit testing, and PHPCompatibility scanning.

In addition to restoring automated testing to branches receiving security updates, there have been a number of performance improvements to the workflows. Most notably, this has resulted in a roughly 70% total decrease in overall runtime for the PHPUnit test workflow.

Here are some additional changes related to GitHub Actions.

Note: All of the changes and improvements listed below have been backported through the 3.7 branch unless otherwise noted.

Limiting when workflows run

Because there are over 750 forks of the wordpress-develop mirror, it’s important to limit when workflows run appropriately. For private repositories, owners are given an allotment of free workflow minutes and are then charged for every minute used over that number. If no limitations are added, each workflow would run on every push event for every fork, including any additional public or private mirrors that are maintained. This would not only waste resources (running for every push to a fork is not necessary as most people open a pull request anyway), but could also unintentionally exhaust a user’s or organization’s private workflow minutes.

To help with this, all workflows have been updated to run only under the following conditions:

• Every push event to wordpress-develop for the primary branch, branches >= 3.7, and all tags matching the pattern x.y.z that is >= 3.7.0.
• Every pull request to wordpress-develop with the primary branch or branches >= 3.7 as the base.
• Every pull request to a fork or private mirror repository with the primary branch or branches >= 3.7 as the base.

For pull request workflows, additional path filtering has been added so that workflows only run when relevant files are changed. For example, the JavaScriptJavaScript JavaScript or JS is an object-oriented computer programming language commonly used to create interactive effects within web browsers. WordPress makes extensive use of JS for a better user experience. While PHP is executed on the server, JS executes within a user’s browser. https://www.javascript.com/. testing workflow will only run when a JavaScript file, QUnit related file, or relevant configuration files (.jshintrc, package.json, etc.) are changed.

These conditions will help limit the number of workflow runs that occur throughout the contributor base without limiting the ability to test and verify changes.

For more information on these changes, see #52643 and #52667 on Trac.

Regular testing of old branches

In TravisCI, there was a feature to configure regular testing of a repository’s branches at given intervals. This was used to run the test suite for:

• The currently maintained branch daily when there has not been a successful build in the last 24 hours.
• The previous branch weekly when there has not been a successful build in the last 24 hours.
• All other branches receiving security updates monthly when there has not been a successful build in the last 24 hours.

There is a schedule event for triggering workflows in GitHub Actions, but it only runs in the primary branch of a repository, so it cannot be used at the workflow level to ensure regular testing of branches.

Instead, a new workflow has been introduced that will run on a schedule and initiate all of the necessary workflows for old branches using the REST APIREST API The REST API is an acronym for the RESTful Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. It is how the front end of an application (think “phone app” or “website”) can communicate with the data store (think “database” or “file system”) https://developer.wordpress.org/rest-api/. to trigger a workflow_dispatch event. This workflow will test the two most recent branches twice per month, and all other branches monthly.

For more information on this, see #52653 on Trac.

Code coverage reporting

Generating a code coverage report for the code base has been supported for some time (see [42665]). But, reports have never been generated and aggregated on a regular basis. To correct this, a new workflow has been introduced to run a test coverage report daily (see [49834]).

The reports generated are now submitted to Codecov and can be viewed here.

This will hopefully give contributors interested in test coverage the ability to find areas of the code base with little to no testing, and provide some insight into how code coverage increases or decreases over time.

Note: The code coverage will only be reported for the primary branch.

For more information, see #52034 on Trac.

Additional improvements to GitHub Actions workflows

• The NPM testing workflow has been generalized to not only verify the build tools work on Windows, but Linux and MacOS as well. Steps have also been added to test additional scripts, such as npm run build:dev and npm run grunt clean (see #52658).
• The fail-fast option has been disabled for the NPM (see [50579]) and PHPUnit (see #52612) testing workflows. fail-fast is great for being alerted of a failure faster, but does not give the full picture of what conditions cause the failure.
• The method of installing dependencies by use of npx install-changed has been replaced with using npm ci after comparisons found the latter was more performant (see #52660).
• All restore-keys options for the actions/cache action have been removed in order to prevent the cache from snowballing due to lax cache key matching. This resulted in a 40% decrease in the NPM dependency cache size (see #52660).
• The restapi-jsclient test group is no longer run separately. This group was never excluded in the phpunit.xml.dist file, so it already runs as part of the main test suite (see #52608).
• The single site and multisitemultisite Used to describe a WordPress installation with a network of multiple blogs, grouped by sites. This installation type has shared users tables, and creates separate database tables for each blog (wp_posts becomes wp_0_posts). See also network, blog, site tests have been split into separate jobs that run in parallel. This has reduced the overall runtime for the PHPUnit workflow by over approximately 30% from the previous run (see #52548).
• Because the test suite takes significantly longer to run on PHP <= 5.6, the PHPUnit workflow has been updated to run test groups that are considered slow (restapi, media, and external-http) in separate, parallel jobs. This reduced the overall time for the workflow by 34% from the previous run (see #52645).

Known transition steps remaining (updated)

The following items remain to achieve parity with the previous testing configurations on TravisCI.

1. Add and configure SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. notifications. In addition to sending the results of the whole build of a core commit into #core, we may also want to consider a firehose channel for PRs. This may require all workflows to be combined into a single workflow if needed middleware cannot be found.
2. Move to GitHub badges for build status indicators – note that these are per-workflow, which is different from the single badge for the entire Travis build for a given commit. However, GitHub does report an overall status for a commit/PR, so we may be able to use that information as well. It seems that the expectation in the greater developer community is that projects report status with a singular badge. Like the Slack notifications, this may require the workflows to be combined in the absence of middleware.
3. Backportbackport A port is when code from one branch (or trunk) is merged into another branch or trunk. Some changes in WordPress point releases are the result of backporting code from trunk to the release branch. the workflow files to unsupported older branches receiving security updates.
4. Finish backporting the local Docker environment to branches 3.7-4.5. This is blocked by:
   • wpdev-docker-images#46, which aims to fix the PHP 5.2 PHPUnit image to include the requred version of PHPUnit (3.6).
   • WP branches <= 4.5 are running a version of NodeJS that is too old for the needed NPM packages required to run the local Docker environment.
5. Report test results to the Host Test Results page. Completed, but the MySQLMySQL MySQL is a relational database management system. A database is a structured collection of data where content, configuration and other options are stored. https://www.mysql.com/. version being tested is not currently being reported (see phpunit-test-runner#135).

Running tests from src (again)

Since the build process was overhauled in WordPress 5.1 (see #43309), automated testing has been running from the build directory. Running from build introduces several problems that makes contributing difficult:

• Running a build is slow. It copies all the files and builds, validates, and minifies all the CSS and JS. None of this should be necessary for PHP testing.
• Developing with grunt watch can give issues on some development environments that run in VirtualBox (like VVV), where changes aren’t being picked up. Having to rebuild manually after each change is a hassle.
• A developer iterating on a patch in the source file has no way of knowing that their file is not actually being tested when running the tests, unless they run the build each time or start and run the file watcher. This is an easy step to forget.
• PHP errors display a stack trace from build instead src.
• Breakpoint debugging isn’t fun as it also uses the stack trace from build instead of src.

The build process was adjusted to allow building and cleaning src again in #44492, but the default directory for automated testing remained build because of some unit testunit test Code written to test a small piece of code or functionality within a larger application. Everything from themes to WordPress core have a series of unit tests. Also see regression. failures.

The remaining items blocking the PHPUnit test suite from running against the src directory have been fixed in [50441] and the default branch for testing has been switched back to src. The PHPUnit testing workflow has also been updated to run the Core test suite from the src directory. This change resulted in an approximately 28% decrease in the workflow’s overall run time from the previous run.

Note: This change has only been merged back to the 5.2-5.7 branches, as this was where it seemed reasonable to draw the line for this change when weighing effort vs. benefits.

For more information on this change, see #51734 on Trac. Related tickets: #43055, #44492, #45863.

Additional changes of note

Here are some other, additional build/test tool changes to make note of:

• The Docker-based local environment now installs the WordPress Importer pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party into the tests/phpunit/data/plugins directory as part of the npm run env:install script. This eliminates the extra step required when running the unit test suite locally (see #49720).
• MariaDB support has been added to the local Docker environment. To substitute MySQL for MariaDB, change the values of the LOCAL_DB_TYPE and LOCAL_DB_VERSION to mariadb and a valid MariaDB version in the .env file, in your local CLICLI Command Line Interface. Terminal (Bash) in Mac, Command Prompt in Windows, or WP-CLI for WordPress. configuration file (such as bashrc), or by setting the variable’s value in your session (see #51744).
• The deprecated node-sass package has been substituted with the recommended replacement, DartSass (sass), which uses the same Javascript APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. (see #51763).
• The svn:global-ignores and svn:ignore properties have been synchronized with the .gitignore file. These SVNSVN Subversion, the popular version control system (VCS) by the Apache project, used by WordPress to manage changes to its codebase. properties had fallen out of date, and several exclusions defined for contributors using GitGit Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. Git is easy to learn and has a tiny footprint with lightning fast performance. Most modern plugin and theme development is being done with this version control system. https://git-scm.com/. were not defined for those using SVN (see #49784).

Unmaintained branches and build/test tools going forwards

Even though these changes were merged all they through the 3.7, WordPress’ official stance continues to be that only the most recent release (5.7 at the time of this post) should be considered supported. The 3.7-5.6 branches will continue to only receive security fixes going forward (though there have recently been discussions about reducing the number of versions receiving security updates).

The changes above were merged into older branches because they were necessary to restore automated testing, or to maintain consistency in testing configurations across branches.

As changes are made to build/test tool areas of the code base going forward, component maintainers will use their judgement in determining which changes should be backported to these older branches. Changes can be grouped and backported as necessary. To start, reviewing these quarterly or after each major releasemajor release A release, identified by the first two numbers (3.6), which is the focus of a full release cycle and feature development. WordPress uses decimaling count for major release versions, so 2.8, 2.9, 3.0, and 3.1 are sequential and comparable in scope. can be used as the frequency.

Immense props go out to @johnbillion. Almost all of the performance related changes detailed above were his ideas and contributions. Props to @jorbin, @davidbaumwald, and @sergeybiryukov for peer review.

#build-test-tools, #github-actions