WordPress 6.9.2 retrospective

The WordPress 6.9.2 release on March 10th wasn’t the smoothest, so some members of the Security Team did an internal retrospective to identify how the project can continue to improve release processes, with the aim of ensuring that users continue to have trust in minor releases. Here’s an overview of what went well, what didn’t go so well, and all the action points as a result.

6.9.2

The Security Team decided on a plan to fully ship 6.9.2 prior to starting the backports that are provided as a courtesy to older branches. Normally the commits to trunktrunk A directory in Subversion containing the latest development code in preparation for the next major release cycle. If you are running "trunk", then you are on the latest revision., the currently supported branchbranch A directory in Subversion. WordPress uses branches to store the latest development code for each major release (3.9, 4.0, etc.). Branches are then updated with code for any minor releases of that branch. Sometimes, a major version of WordPress and its minor versions are collectively referred to as a "branch", such as "the 4.0 branch"., and relevant backportbackport A port is when code from one branch (or trunk) is merged into another branch or trunk. Some changes in WordPress point releases are the result of backporting code from trunk to the release branch. branches are all completed before the release process starts, but this increases the time to release the supported branch and greatly increases the time pressure on the team.

The team agreed that it was a good idea to handle trunk and 6.9 prior to committing the backports, that the approach worked well, and that it facilitated shipping 6.9.2 as fast as possible.

6.9.3

Shortly after the release, an issue in 6.9.2 was reported on the support forums that affected classic themes using an unusual approach to template loading. The team paused the backport work to investigate, and decided that it warranted a fast-follow 6.9.3 release out of an abundance of caution.

The fix was shipped in 6.9.3, around eight hours after 6.9.2.

7.0 betaBeta A pre-release of software that is given out to a large group of users to trial under real conditions. Beta versions have gone through alpha testing in-house and are generally fairly close in look, feel and function to the final product; however, design changes often occur as part of the process. 4

It’s uncommon for a security release of WordPress to ship during the beta period of the next major releasemajor release A release, identified by the first two numbers (3.6), which is the focus of a full release cycle and feature development. WordPress uses decimaling count for major release versions, so 2.8, 2.9, 3.0, and 3.1 are sequential and comparable in scope.. This is not intentional, the situation simply doesn’t arise often. In the twenty year history of WordPress it appears to have happened three times previously (3.9.2, 4.0.1, and 6.3.2).

Shortly after the release of 6.9.2, several community members asked if a new beta of WordPress 7.0 would be released containing the security fixes. This was not originally planned but of course makes sense, as the project should encourage as many people as possible to test beta releases and not leave them on a known insecure version until the next scheduled beta.

As a result, WordPress 7.0 beta 4 was released at the same time as 6.9.3 and included both the security fixes from 6.9.2 and the template loading fix from 6.9.3.

6.9.4

Around twenty hours after 6.9.2 was released, the Security Team received a report that some of the fixes listed in the release announcement for 6.9.2 were missing from the package. The team quickly determined that three of the ten commits that were made to trunk did not get successfully merged into the 6.9 branch and were missing from the package as a result.

The merge commits were missed due to human error, but the problem should have been easily caught by the release process. There is no step in the minor releaseMinor Release A set of releases or versions having the same minor version number may be collectively referred to as .x , for example version 5.2.x to refer to versions 5.2, 5.2.1, 5.2.3, and all other versions in the 5.2 (five dot two) branch of that software. Minor Releases often make improvements to existing features and functionality. process to independently double check that all commits were successfully merged into the active branch. Sounds obvious in hindsight, but it’s a checklist oversight that’s never been spotted.

Backports

Officially only the latest version of WordPress is supported. The Security Team historically has a practice of backporting security fixes, as necessary, as a courtesy to sites on older versions in the expectation the sites will be automatically updated.

Backporting the fixes from 6.9.2/6.9.3 to 22 older branches proved to be very time consuming. While several of the branches were ready ahead of time, several were still being worked on close to release.

• Backports back to 6.4 were completed by end of day on Tuesday March 10th.
• Backports back to 5.3 were completed by end of day on Wednesday March 11th, mainly due to time constraints from contributors.
• Remaining backports were blocked by a bugbug A bug is an error or unexpected result. Performance improvements, code optimization, and are considered enhancements, not defects. After feature freeze, only bugs are dealt with, with regressions (adverse changes from the previous version) being the highest priority. with the wordpress.orgWordPress.org The community site where WordPress code is created and shared by the users. This is where you can download the source code for WordPress core, plugins and themes as well as the central location for community conversations and organization. https://wordpress.org/ svn server pre-commit hook that prevented pushing to the 5.2 branch and earlier, and issues with branches not syncing to the build server. The wordpress.org systems team members applied several fixes, and backports back to 4.7 were released by end of day on Friday March 13th.
• The 6.0 branch (which will be version 6.0.12) remains unreleased due to an unresolved issue with the build. At the time of writing this branch remains unprotected.

Backporting security fixes to a large number of branches (22 at the time of writing) continues to be a source of contention between the security team and project leadership. While the effort required from the security team to backport fixes to a branch is generally proportional to the age of the branch, the bulk of the work is actually taken up by the human processes that are needed to manage such a large number of branch releases. The team are in the planning stages of some work to increase automation and streamline the backporting processes so time from human contributors can be better spent elsewhere.

Action points

As a result of the issues with 6.9.2, its fast-follow releases, 7.0 beta 4, and backporting, the security team and coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. team have some action points that will be addressed in the coming days and weeks:

• Add unit testunit test Code written to test a small piece of code or functionality within a larger application. Everything from themes to WordPress core have a series of unit tests. Also see regression. coverage for handling stringable objects in the template_include filterFilter Filters are one of the two types of Hooks https://codex.wordpress.org/Plugin_API/Hooks. They provide a way for functions to modify data of other functions. They are the counterpart to Actions. Unlike Actions, filters are meant to work in an isolated manner, and should never have side effects such as affecting global variables and output. that was addressed in 6.9.3. While this is not strictly supported, the otherwise full front end breakage that can occur means we’re essentially locked in to supporting it now.
• Update minor release checklist page to:
  • Add info about double verification of merge commits
  • Update the TTL adjustment to be a requirement instead of a nice to have
  • Remove outdated noise from the checklist, including codex changes
  • As a general rule, there should be no reason to skip a step unless it’s clearly documented why that might happen
  • Add planning info about releasing a minor during a beta or RCrelease candidate One of the final stages in the version release cycle, this version signals the potential to be a final release to the public. Also see alpha (beta). phase
  • Before bumping versions and performing a tagging commit, the built asset on the Test Build Processes workflow should be used to confirm things work as expected.
• Backport related Build/Test Tool changes to older branches.
  • Ensure PHPUnit reports notices and warnings as exceptions.
  • Ensure all local environment scripts have catch() mechanisms in place.
• Create ways to make it easier to test built assets on the Test Build Processes workflow.
  • Playground supports testing this asset from a pull request but not an individual commit.
  • A script could be created to download the built asset and extract the zip file into the build directory locally for testing.

Longer term, Matt Mullenweg has asked what AI-assisted tooling can be used to review changes going into releases in order to assess risk of breakage, generally assist with review, and improve quality control. This is a wider concern not specific to security releases. Hopefully we’ll hear more about that in due course.

Thanks to @peterwilsoncc, @dmsnell, @audrasjb, and @sergeybiryukov for reviewing this post prior to publishing.