Andrew Ozz 1:37 am on October 11, 2024
Tags: proposal ( 38 ), trac ( 59 )

Proposal: Make unit test tickets easier to distinguish

This is a follow-up of a conversation in #core on Slack.

Currently there is no easy way to identify TracTrac An open source project by Edgewall Software that serves as a bug tracker and project management tool for WordPress. tickets specifically for adding unit tests. Some are added to the Build/Test Tools component, however the tests are not really “tools”. The reason for this is that unit tests, just like build and test tools, are “non-production code”. That means they can be committed at any time during the WordPress development cycle and do not follow the general milestones-based workflow.

Additionally, many tickets for adding unit tests are marked as “enhancements”. Following the release cycle workflow, enhancements cannot be committed during betaBeta A pre-release of software that is given out to a large group of users to trial under real conditions. Beta versions have gone through alpha testing in-house and are generally fairly close in look, feel and function to the final product; however, design changes often occur as part of the process. and RCrelease candidate One of the final stages in the version release cycle, this version signals the potential to be a final release to the public. Also see alpha (beta).. However, in practice unit tests can be committed at any time. It can be difficult to isolate these today, as the Trac search form lacks a filterFilter Filters are one of the two types of Hooks https://codex.wordpress.org/Plugin_API/Hooks. They provide a way for functions to modify data of other functions. They are the counterpart to Actions. Unlike Actions, filters are meant to work in an isolated manner, and should never have side effects such as affecting global variables and output. for such tickets, which makes triaging a bit harder.

Some suggestions

During the discussion in #core, several ideas were proposed about how this can be made to work better. And while there isn’t currently much traction on adding E2E tests to CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress., any option explored should take this test type into account. In no particular order:

Component: A new Tests component, which would be exempt from the general workflow, similar to Build/Test Tools. Example component filter.
Focus: A tests focus, similar to the existing docs focus. Example focus filter.
Keyword: A tests-only keyword to identify that it’s for unit/E2E tests only, and serve to filter tickets for triagetriage The act of evaluating and sorting bug reports, in order to decide priority, severity, and other factors. reports. Example keyword filter.
Milestone: An Any Time milestone could indicate when this type of ticketticket Created for both bug reports and feature development on the bug tracker. could be committed, being primarily reserved for tests- and docs-only tickets. Example milestone filter.
Type: A test ticket type, to clearly differentiate from the current defect (bug), enhancement, and task (blessed) types. Example type filter.
Type (alternative): A new non-production type could be introduced to broadly cover any ticket that does not affect build/production code. Example type filter.

I think that any of the proposed solutions would make searching and triaging test-oriented tickets a bit easier. Some are more flexible in that it’s easier to combine terms (focuses and keywords).

What are your thoughts on these ideas to make test-only tickets easier to distinguish? Please help continue the discussion in the comments below!

Thanks @costdev and @ironprogrammer for the suggestions and reviews.

#proposal, #trac

Jonathan Desrosiers 4:17 pm on October 14, 2024
Thanks for this proposal @azaozz! Finding an easier way to designate and find tickets that focus only on adding tests is a great idea. I’ve been thinking this through a lot, and I have decided that it is my strong preference to introduce a tests focus over all the suggestions. Here’s a bit of my rationale.

I think a good comparison for this is the REST APIREST API The REST API is an acronym for the RESTful Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. It is how the front end of an application (think “phone app” or “website”) can communicate with the data store (think “database” or “file system”) https://developer.wordpress.org/rest-api/.. There is a REST API component and a rest-api focus and they are designed to work together. Because of the nature of the REST API, it overlaps many other areas of CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. (themes, plugins, taxonomies, posts, etc.). The REST API component is for tickets related to the general foundation of the APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways.. But when tickets heavily overlap these other areas of the code base, they are placed in the most relevant component and the rest-api focus is applied.

Over time, I think this has worked great. Putting these tickets in more specific components helps ensure they get the proper attention form contributors and maintainers who are really familiar with the relevant areas of Core. I think the same thinking applied here would work really well.

In the end, tests are relevant to one specific area of the code base, so they should be placed in the most relevant component with the `tests` focus. Tickets that are making changes _and_ adding tests should use the has-unit-tests/needs-unit-tests keywords. Any tickets that are related to the underlying testing infrastructure should be placed in the Build/Test Tools component as that is more relevant.

One other benefit of this approach is that it gives more flexibility for surfacing tickets in reports. For example, we could create a new report that shows all tickets with tests focus organized by component.

Some other notes:
- It has long been a practice and understanding that build tooling or test related changes are allowed at any time in the release cycle (this is one of the rules that has never really been disputed when discussing what can and cannot be merged and when). If this is confusing, we should look to better codify this in the handbook to make it more clear when test related changes can be merged. I can see why having test only tickets as enhancements may seem confusing. But I think we can easily clarify that in some way.
- Keywords generally fall into two buckets according to the handbook: status-based and action based. tests-only doesn’t feel natural in either of those buckets.
- With the exception of Future Release and Awaiting Review, milestone should really only relate back to a specific cycle. If the ticketticket Created for both bug reports and feature development on the bug tracker. is closed as fixed, then it would sit in this “any time” bucket forever.
- Making a test ticket type feels like we’re abusing the point of ticket type. At the heart, adding a missing test is an enhancementenhancement Enhancements are simple improvements to WordPress, such as the addition of a hook, a new feature, or an improvement to an existing feature..
- Andrew Ozz 1:08 am on October 16, 2024
  
  it is my strong preference to introduce a tests focus over all the suggestions
  
  tests are relevant to one specific area of the code base, so they should be placed in the most relevant component with the tests focus. Tickets that are making changes _and_ adding tests should use the has-unit-tests/needs-unit-tests keywords.
  
  Thanks for looking into this and for the thoughtful analysis! Yes, I agree that having a tests focus would work well and match other similar cases. The only thing I’m a little concerned about it that it is possible to add one or more focuses to any ticketticket Created for both bug reports and feature development on the bug tracker., so there may be cases where the tests focus may be added by mistake to tickets that contain other code/patches and have to follow the milestones workflow. But seems that can be fixed during triagetriage The act of evaluating and sorting bug reports, in order to decide priority, severity, and other factors., so not a problem.
  
  Also agree that having a tests ticket type seems like an overkill. However maybe having a non-production type may make sense. This will make it very obvious the ticket is exempt from the general workflow. Or perhaps the name may be better as tools-docs-tests? But maybe can think about this later.
- Brian Alexander 6:28 pm on October 21, 2024
  
  +1 to preferring a new tests focus, for all the benefits and flexibility mentioned above.
  
  Would the appropriate location for clarifying the enhancement + tests (or docs, etc) focus be in the list on the Releasing Beta and RC Versions page? Or are there other exceptions that may warrant a dedicated page to clarify situations where confusion has occurred (e.g. syncs from GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ come to mind)? I don’t want to go overboard, but it’s helpful to be able to provide folks a direct link to “official” documentation when these questions come up.
Jonathan Desrosiers 2:18 pm on February 11, 2025

During the 6.8 cycle, this proposal came up as potentially being useful for tracking tickets solely focused on adding tests. As a result, I’ve gone and opened a Meta Trac ticket requesting the addition of a `tests` focus to give this a try.

If this approach does not work, we can re-evaluate other options. Thanks everyone!

Comments are closed.

Welcome!

Communication

Proposal: Make unit test tickets easier to distinguish

Some suggestions

Welcome!

Communication

Some suggestions

Share this:

Site resources