Andrew Ozz 12:54 am on January 8, 2020
Tags: 5.3 ( 80 ), retrospective ( 20 )

5.3 Release Cycle Post-mortem

By the Numbers

This cycle was longer, taking just over six months. The development (alpha) stage was 138 days, from May 8 to September 23. During that time there were 779 commits to 5.3 (trunktrunk A directory in Subversion containing the latest development code in preparation for the next major release cycle. If you are running "trunk", then you are on the latest revision.). Most active committers were @SergeyBiryukov with 426 (no surprises there), @pento 66, @afercia and @desrosj with 51 commits each.

The “betaBeta A pre-release of software that is given out to a large group of users to trial under real conditions. Beta versions have gone through alpha testing in-house and are generally fairly close in look, feel and function to the final product; however, design changes often occur as part of the process.” stage started on September 23 and lasted for 23 days. There were 274 commits. Most active committers were @SergeyBiryukov with 78, @afercia 42, @desrosj 34, and @whyisjake with 24 commits.

The “release candidaterelease candidate One of the final stages in the version release cycle, this version signals the potential to be a final release to the public. Also see alpha (beta).” stage started on October 16 and lasted for 27 days. There were 100 commits including documentation and build tools changes. Most active were @SergeyBiryukov with 39, @desrosj 20, and @johnbillion with 14 commits. There were 30 commits that were merged to the 5.3 branchbranch A directory in Subversion. WordPress uses branches to store the latest development code for each major release (3.9, 4.0, etc.). Branches are then updated with code for any minor releases of that branch. Sometimes, a major version of WordPress and its minor versions are collectively referred to as a "branch", such as "the 4.0 branch". after it was created on October 29.

In total the release cycle lasted for 188 days and there were 1,153 commits. Looking at the numbers, the busiest time was during beta with an average of nearly 12 commits per day. For comparison there were only about 5.5 commits per day during the “alpha” stage *.

Needed Improvements

There were also some irregularities during the 5.3 cycle.

Many “new” bugfixes were committed during beta. Generally these are added in alpha, and beta is reserved for testing and fixing any bugs introduced in alpha.
Several apparent enhancements were committed from partially related bugfix tickets: #34904, #47477, #47153, etc. Preferably there would be a discrete 1:1 relationship between tickets and changesets.
Some code that would be considered “alpha” (mostly untested and non-reviewed) was committed during beta. Preferably any code committed in beta is well-reviewed by multiple people with varied knowledge/experience with the existing code/areas of expertise.

A substantial departure from past releases was the inclusion of “global” CSSCSS Cascading Style Sheets. changes affecting all of the WordPress adminadmin (and super admin) during beta. They were committed from unrelated or partially related tickets: [46241], [46360], [46244], and most of the other changesets marked as “Improve and modernize user interface controls”.

It would have been preferable for the changes to be separated in new tickets, tested, reviewed, and then committed. Separating these into individual tickets would also have let enhancements be isolated and prepared for the next major releasemajor release A release, identified by the first two numbers (3.6), which is the focus of a full release cycle and feature development. WordPress uses decimaling count for major release versions, so 2.8, 2.9, 3.0, and 3.1 are sequential and comparable in scope..

Individually, none of the irregularities listed above are a big problem, but when taken all together it introduced avoidable complexity. In this particular case, many small styling bugs were introduced during beta, which is far too late in the release cycle. Again, individually they were fixable, but their late introduction and lack of discrete ticketing left folks without enough time to find and fix them all.

Moving Forward

There are few steps that can be taken to avoid such irregularities in the future:

Add some “Release Cycle 101” to the developer documentation. This will ensure that people without prior experience are on the same page. Generally during alpha anything goes. Trunk can be broken and fixed every day as the contributors iterate on enhancements and bugfixes. Beta is for testing by the users and to fix the bugs that were introduced in alpha. RCrelease candidate One of the final stages in the version release cycle, this version signals the potential to be a final release to the public. Also see alpha (beta). is “we think it’s ready, find the edge cases!”
Add a requirement for a code review for all changes committed during beta. Generally this has been the case for most commits during both alpha and beta. Requiring a review before merging works very well in GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/. It’s about time to make it “official” on TracTrac An open source project by Edgewall Software that serves as a bug tracker and project management tool for WordPress. too.
Remove the “(blessed)” part from the “task” ticketticket Created for both bug reports and feature development on the bug tracker. type on Trac. It seems it gives the wrong impression that a ticket can be “blessed” by anybody to allow it to bypass the release cycle rules.
Restrict use of the “task” ticket type on Trac only to the project leadership. Bypassing (our own) release cycle rules is not something that can be done lightly and carries a lot of responsibilities. Also, perhaps add a requirement for at least two reviews for each commit/code change in this case.
Perhaps extend the section on creating and using Trac tickets to specifically disallow unrelated or partially related changes to be made. This may be somewhat hard to determine, but when in doubt, better to create a new ticket instead of exponentially extending an existing ticket. In 5.3 there are several examples of such “run-away” tickets: #47477, #47153, #34904, etc.

* The slower commit pace during alpha is partially due to new features being developed, and not committed until deemed “ready”. Another reason is perhaps that complex enhancements and bugbug A bug is an error or unexpected result. Performance improvements, code optimization, and are considered enhancements, not defects. After feature freeze, only bugs are dealt with, with regressions (adverse changes from the previous version) being the highest priority. fixes need more research, reviews and testing before committing. Yet another reason is that perhaps most contributors focus more on the new features, enhancements and more complex bug fixes leaving the “easy” bugs for later.

#5-3, #retrospective

Timothy Jacobs 1:30 am on January 8, 2020

I think it’s worth mentioning that the proposed scope was not published until August 7th and the kickoff wasn’t until the 21st. So despite the large amount of time between 5.2 and 5.3, to me the release didn’t feel particularly slow. In particular, media retries dev felt very fast and barely made the RCrelease candidate One of the final stages in the version release cycle, this version signals the potential to be a final release to the public. Also see alpha (beta). deadline. For me at least it was fairly stressful.

I think identifying areas where scope crosses multiple components as early as possible would be a tremendous help. From a REST APIREST API The REST API is an acronym for the RESTful Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. It is how the front end of an application (think “phone app” or “website”) can communicate with the data store (think “database” or “file system”) https://developer.wordpress.org/rest-api/. perspective, we already had tickets in progress by the time the scope was announced. I also think we didn’t reactReact React is a JavaScript library that makes it easy to reason about, construct, and maintain stateless and stateful user interfaces. https://reactjs.org/. fast enough to move our limited resources to the designated scope.

Both of those things would be more easily doable if the release focus was known earlier. Moving forward, could the focus be announced at the beginning of the cycle or perhaps even while the previous cycle is closing RC?
- Andrew Ozz 1:48 am on January 8, 2020
  
  …media retries dev felt very fast and barely made the RCrelease candidate One of the final stages in the version release cycle, this version signals the potential to be a final release to the public. Also see alpha (beta). deadline. For me at least it was fairly stressful.
  
  Yeah, it turned out to be quite more complex than expected, was very stressful for me too. The good part is we managed to do it in time 🙂
  
  Moving forward, could the focus be announced at the beginning of the cycle or perhaps even while the previous cycle is closing RC?
  
  Yes, this would be great, perhaps not always possible. Sometimes the ideas/need for new features and enhancements come in later, even during the kick-off.
  
  Bugfixes are (mostly) unaffected by the release focus(es). Each team/component maintainers can prioritize which bugs should be dealt with next, which would require cross-component consideration/reviews, and which are more likely to introduce problems/regressions and should get in “early” to allow for better testing.
Daniele Scasciafratte 8:36 am on January 8, 2020

Just a comment, maybe the alpha period need to be more long as period to give to the various teams time also with few resources to follow everything?

I think that there was also the problem of the new theme in the 5.3 with few tickets about bugs, and a new theme is unusual for a wordpress release.
Jb Audras 10:52 am on January 8, 2020

Thank you @azaozz for sharing your thoughts about this release workflow,

It would have been more fair not to limit the problematic points to the tickets you mentioned in almost each section of this post, but also to mention other issues: Twenty Twenty development, Big images changes, other media changes, or About page last minute fixes, etc.

In any case, I understand the fundamental issues you raised in this post, and thank you for sharing that, very interesting and helpful.
- Jb Audras 12:01 pm on January 8, 2020
  
  I think we should also consider some other points in this post-mortem post (or maybe we should write a second one?). Here is some personal thoughts about the release process:
  
  – have a detailed release focus concerning enhancements and new features
  
  5.3 scope was “polishing current interactions and make the UIs more user friendly”. Although this is what we actually did in 5.3, I don’t think it’s covering all the important features that landed in this release. It’s hard for users to expect big changes in media handling by reading this scope.
  
  – get the dev notesdev note Each important change in WordPress Core is documented in a developers note, (usually called dev note). Good dev notes generally include a description of the change, the decision that led to this change, and a description of how developers are supposed to work with that change. Dev notes are published on Make/Core blog during the beta phase of WordPress release cycle. Publishing dev notes is particularly important when plugin/theme authors and WordPress developers need to be aware of those changes.In general, all dev notes are compiled into a Field Guide at the beginning of the release candidate phase. as soon as possible and widely communicate them
  
  CSSCSS Cascading Style Sheets. changes dev notedev note Each important change in WordPress Core is documented in a developers note, (usually called dev note). Good dev notes generally include a description of the change, the decision that led to this change, and a description of how developers are supposed to work with that change. Dev notes are published on Make/Core blog during the beta phase of WordPress release cycle. Publishing dev notes is particularly important when plugin/theme authors and WordPress developers need to be aware of those changes.In general, all dev notes are compiled into a Field Guide at the beginning of the release candidate phase. raised a lot of comments, and Big image changes dev note raised a way more comments. If the first change was (sort of, and far from ideally) announced with the scope of the release, the second one wasn’t announced at all. I think this kind of changes should be prepared ahead of the betaBeta A pre-release of software that is given out to a large group of users to trial under real conditions. Beta versions have gone through alpha testing in-house and are generally fairly close in look, feel and function to the final product; however, design changes often occur as part of the process. cycle while we are working on features development. Maybe the marketing team could help us to make a better technical and functional communication during the beta phase?
  
  Thanks to @francina we are going to set up a working group to improve the release process. Lot of things could be improved, and I don’t think it’s a question of tickets, but “merely” a question of clearly define what a major WordPress release is supposed to be 🙂
- Andrew Ozz 12:21 am on January 9, 2020
  
  It would have been more fair not to limit the problematic points to the tickets you mentioned in almost each section of this post, but also to mention other issues: Twenty Twenty development, Big images changes, other media changes, or About page last minute fixes, etc.
  
  The problems I’m trying to address are with “what is appropriate and not appropriate during betaBeta A pre-release of software that is given out to a large group of users to trial under real conditions. Beta versions have gone through alpha testing in-house and are generally fairly close in look, feel and function to the final product; however, design changes often occur as part of the process.”, i.e. alpha is for development and beta is to release a version that can be tested by the users and fix the bugs that were introduced during the development.
  
  Another problem is the need of “having a 1:1 relationship between a ticketticket Created for both bug reports and feature development on the bug tracker. and committed changes”, i.e. not commit things that are enhancements from (unrelated or partially related) bugfix tickets, or changes that are way out of scope for the particular ticket, etc. These should be handled from new tickets which would make it a lot easier to follow, test, and review. Arguably if all the CSSCSS Cascading Style Sheets. changes that were added in 5.3 beta were handled properly, we would have been able to fix most (all?) of the styling bugs they introduced.
  
  It’s true, a lot of Twenty Twenty fixes were added during beta and RCrelease candidate One of the final stages in the version release cycle, this version signals the potential to be a final release to the public. Also see alpha (beta)., but that followed the release cycle properly. All were discovered during the (user) testing and fixed.
  
  The “Big Image” changes followed the cycle properly too. Bugs were discovered during beta and RC and fixed.
  
  Historically there have always been About page last-minute fixes. It is the only open ticket that is “allowed” at RC as some changes are still pending. I agree, ideally it should be 99.9% ready at that time, and it’s worth looking at improving the way it’s handled so it can be finalized before RC 🙂
  
  Thanks to @francina we are going to set up a working group to improve the release process. Lot of things could be improved, and I don’t think it’s a question of tickets, but “merely” a question of clearly define what a major WordPress release is supposed to be.
  
  Right, but don’t think the basic release cycle stages would ever change. They’ve been set since the 70’s (or maybe even earlier). Alpha is longest and is for development (enhancements, new features, bugfixes). Beta starts with releasing a version that can be tested by a wider user audience and is for fixing the bugs introduced in alpha. RC is for final testing by even larger user audience, looking for any edge cases that have been missed during the beta testing. It’s not appropriate to continue “development” during beta. The reason is quite simple: new bugs introduced during beta are harder to discover and fix. This is generally what happened in the 5.3 release cycle.
  - Jb Audras 2:00 pm on January 11, 2020
    To clarify: I’m not saying the alpha/betaBeta A pre-release of software that is given out to a large group of users to trial under real conditions. Beta versions have gone through alpha testing in-house and are generally fairly close in look, feel and function to the final product; however, design changes often occur as part of the process./RCrelease candidate One of the final stages in the version release cycle, this version signals the potential to be a final release to the public. Also see alpha (beta). cycle should change 🙂
    I understand your feedback about those tickets scope as well and I share your thoughts.
    
    But I personally think this post-mortem is missing some items/things that really need improvements:
    
    Communicate the scope of the release focus by focus and component by component, instead of providing a single sentence to describe the whole release focus.
    
    Communication about the actual changes that are going to land, ahead of these changes.
    
    Earlier work on the dev notesdev note Each important change in WordPress Core is documented in a developers note, (usually called dev note). Good dev notes generally include a description of the change, the decision that led to this change, and a description of how developers are supposed to work with that change. Dev notes are published on Make/Core blog during the beta phase of WordPress release cycle. Publishing dev notes is particularly important when plugin/theme authors and WordPress developers need to be aware of those changes.In general, all dev notes are compiled into a Field Guide at the beginning of the release candidate phase./proper documentation, in order to release them earlier in the release cycle.
    
    Better testing of the big features, and publish posts on Make/CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. about the results of those tests (maybe something like we did with Adminadmin (and super admin) CSSCSS Cascading Style Sheets. changes, although it was a big post: we could probably simplify the analysis).
    
    In my opinion, it would be nice to also take those improvements into account, and to me, I believe it caused more issues than the 3 tickets you specifically raised in the post mortem 😉
    
    By the way, I thank you again for sharing your feedback, it’s very appreciated and worthwile for everybody 🙂
    - Andrew Ozz 8:06 pm on January 12, 2020
      
      But I personally think this post-mortem is missing some items/things…
      
      This is a mostly “technical” post-mortem looking at how things were done during the release and what needs fixing. It’s not an attempt to “redesign” the release process 🙂
      
      Communicate the scope of the release focus by focus and component by component…
      
      Communication about the actual changes that are going to land, ahead of these changes.
      
      Frankly don’t think that’s needed or even possible. Example: what would be the “scope” for the “Script Loader” component for 5.4? How do you “communicate” a bugbug A bug is an error or unexpected result. Performance improvements, code optimization, and are considered enhancements, not defects. After feature freeze, only bugs are dealt with, with regressions (adverse changes from the previous version) being the highest priority. fix or enhancementenhancement Enhancements are simple improvements to WordPress, such as the addition of a hook, a new feature, or an improvement to an existing feature. change before it is coded, tested, reviewed and committed? Also, bugs and enhancements are reported/requested during the development stage, often in the process of fixing/working on related code.
      
      Generally bug fixes and small enhancements (and changes that can be considered partial bug fix and partial enhancement) are always in scope. For example: adding a new filterFilter Filters are one of the two types of Hooks https://codex.wordpress.org/Plugin_API/Hooks. They provide a way for functions to modify data of other functions. They are the counterpart to Actions. Unlike Actions, filters are meant to work in an isolated manner, and should never have side effects such as affecting global variables and output. so plugins can do their job better, adding more default image sizes, rotating images according to EXIF orientation, optimizing images for web use, etc. etc. All of these would denote one of the focuses as being on the “Media” and “Upload” components, but it’s not feasible, and often not even possible to determine beforehand what can be completed in time for the release. Of course all of the above changes should be made during the development stage and if there are specific code/technical changes there should be dev. notes for them published when the code is finalized.
      
      Earlier work on the dev notesdev note Each important change in WordPress Core is documented in a developers note, (usually called dev note). Good dev notes generally include a description of the change, the decision that led to this change, and a description of how developers are supposed to work with that change. Dev notes are published on Make/Core blog during the beta phase of WordPress release cycle. Publishing dev notes is particularly important when plugin/theme authors and WordPress developers need to be aware of those changes.In general, all dev notes are compiled into a Field Guide at the beginning of the release candidate phase./proper documentation, in order to release them earlier in the release cycle.
      
      Not sure that would make sense in some/most cases. The dev. notes should be published after the code is finalized. That’s usually at the beginning of the RCrelease candidate One of the final stages in the version release cycle, this version signals the potential to be a final release to the public. Also see alpha (beta). stage.
      
      Better testing of the big features, and publish posts on Make/CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. about the results of those tests.
      
      “Better testing”, definitely! Publish test results here, on make/core… Why? Testing results belong in TracTrac An open source project by Edgewall Software that serves as a bug tracker and project management tool for WordPress., on the relevant tickets. Same as unit tests, e2e tests, etc.
      
      Frankly it seems that lately make/code is being “repurposed” from being developer oriented to being a “catch all” user oriented info site. Don’t think that’s appropriate, really. Thinking that perhaps we’ll need to look at setting some guidelines and “expectations” when posting here 🙂
      
      (maybe something like we did with Adminadmin (and super admin) CSSCSS Cascading Style Sheets. changes, although it was a big post: we could probably simplify the analysis)
      
      Right. Don’t think these posts were entirely appropriate here. The testing results should have been added to the appropriate Trac tickets. I agree, the fact that there weren’t appropriate tickets for each change made that pretty hard to do, but thinking next time new tickets should be created if they don’t exist.
Robert Anderson 12:51 am on February 19, 2020

Add some “Release Cycle 101” to the developer documentation. This will ensure that people without prior experience are on the same page. Generally during alpha anything goes. Trunktrunk A directory in Subversion containing the latest development code in preparation for the next major release cycle. If you are running "trunk", then you are on the latest revision. can be broken and fixed every day as the contributors iterate on enhancements and bugfixes. BetaBeta A pre-release of software that is given out to a large group of users to trial under real conditions. Beta versions have gone through alpha testing in-house and are generally fairly close in look, feel and function to the final product; however, design changes often occur as part of the process. is for testing by the users and to fix the bugs that were introduced in alpha. RCrelease candidate One of the final stages in the version release cycle, this version signals the potential to be a final release to the public. Also see alpha (beta). is “we think it’s ready, find the edge cases!”

We already have a How the Release Cycle Works page in the Handbook which could be updated. I suspect the problem, though, is less that this page doesn’t exist, and more that it isn’t discoverable and constantly re-communicated e.g. in weekly chats.

Add a requirement for a code review for all changes committed during beta. Generally this has been the case for most commits during both alpha and beta. Requiring a review before merging works very well in GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/. It’s about time to make it “official” on TracTrac An open source project by Edgewall Software that serves as a bug tracker and project management tool for WordPress. too.

I’ve primarily worked in Gutenberg and so am biased here, but yes I think mandatory code reviews help with both code quality and knowledge sharing and have proven very useful in developing the blockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. editor. (I didn’t even realise they weren’t mandatory in CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress.!)

Remove the “(blessed)” part from the “task” ticketticket Created for both bug reports and feature development on the bug tracker. type on Trac. It seems it gives the wrong impression that a ticket can be “blessed” by anybody to allow it to bypass the release cycle rules.
Restrict use of the “task” ticket type on Trac only to the project leadership. Bypassing (our own) release cycle rules is not something that can be done lightly and carries a lot of responsibilities. Also, perhaps add a requirement for at least two reviews for each commit/code change in this case.
Perhaps extend the section on creating and using Trac tickets to specifically disallow unrelated or partially related changes to be made. This may be somewhat hard to determine, but when in doubt, better to create a new ticket instead of exponentially extending an existing ticket. In 5.3 there are several examples of such “run-away” tickets: #47477, #47153, #34904, etc.

Merging Task (blessed) and Task together sounds good to me since the distinction is not clear and ultimately what makes a task “blessed” is if a component maintainer or release leadRelease Lead The community member ultimately responsible for the Release. advocates for it and/or a committercommitter A developer with commit access. WordPress has five lead developers and four permanent core developers with commit access. Additionally, the project usually has a few guest or component committers - a developer receiving commit access, generally for a single release cycle (sometimes renewed) and/or for a specific component. commits it.

I’ve not had enough experience working in Trac to comment on your other suggestions. Could we discuss this at the next DevChat to gain some feedback from committers?

Comments are closed.

Welcome!

Communication

By the Numbers

Needed Improvements

Moving Forward

Share this: