How to facilitate a Call for Testing

At a high level there are four parts to facilitating a call for testing within the WordPress project:

  • Determine what needs testing.
  • Write and share the call for testing.
  • Respond to feedback/close the feedback loopLoop The Loop is PHP code used by WordPress to display posts. Using The Loop, WordPress processes each post to be displayed on the current page, and formats it according to how it matches specified criteria within The Loop tags. Any HTML or PHP code in the Loop will be processed on each post. https://codex.wordpress.org/The_Loop..
  • Summarize the results.

The final two steps are sometimes optional, depending on how the call for testing is approached and whether someone is available to wrangle the feedback itself. Generally speaking, there are two kinds of calls for testing:

  • Self-serve: these calls for testing are centered on alerting folks to the need for testing and putting the responsibility on those testing to also close the loop on feedback they have by opening issues themselves. Here’s an example on testing features coming to 5.9 where folks are encouraged to test and report bugs on their own.
  • Facilitated: these calls for testing are more interactive and involve a point person(s) who engages with the feedback by opening issues for problems found, engaging directly with testers to ensure clarity, and summarizing feedback usually in a separate post. Here are examples from the FSE Outreach Program’s calls for testing and summary posts.

Remember that it’s better to alert the need for testing than not to test at all so, if there’s something that needs testing but you don’t have capacity to facilitate, it’s still valuable to offer a self serve option.

Determine what needs testing

If you want to help with calls for testing and you don’t have a specific feature you’re working on in mind, there are a few sources to pull from for insight around what would be most impactful to test. When in doubt, it’s always a good idea to ask in the open what folks would like more testing help on in the Core Dev and Core Editor meetings. Otherwise, here are sources to examine:

  • Features on roadmap posts for releases often shared 3-4 months before a release (5.9, 6.0, 6.1).
  • Features discussed in Core Editor and Core Dev meetings.
  • Feature projects that reach a level of stability or exploration and need feedback to continue.

Top ↑

Write and share the call for testing

Top ↑

Template

Use this handy template and checklist when creating your post. Or, skip all of this. Publishing anything and tagging it #needs-testing is sufficient and better than not posting.

What is it?

Describe the feature. Include screenshots. Describe any goals and gating criteria, such as parity requirements.

Where is it?

Describe how to get to the feature. Direct links are good.

What to test

Describe the primary flows to test. Consider using bulleted list or a visual record.

What to expect

List known issues and shortcomings (bug tracker links are helpful). Set expectations.

How to report

Provide tracTrac Trac is the place where contributors create issues for bugs or feature requests much like GitHub.https://core.trac.wordpress.org/. or githubGitHub GitHub is a website that offers online implementation of git repositories that can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged be the repository owner. https://github.com/ links, include the component.

Timeline for testing

Express urgency and priority. If there are target dates, provide them.

Top ↑

Checklist

  • Introduce yourself.
  • Explain the feature or update, including what’s needed to test (example: a specific version of GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ or WordPress).
  • Link to the feature.
  • List known issues.
  • Provide a short list of testing steps.
  • Mention how to provide feedback.
  • Include a date for the completion of the testing.
  • Include screenshots from multiple devices.
  • Keep a betaBeta A pre-release of software that is given out to a large group of users to trial under real conditions. Beta versions have gone through alpha testing in-house and are generally fairly close in look, feel and function to the final product; however, design changes often occur as part of the process. testing audience in mind.
  • TagTag Tag is one of the pre-defined taxonomies in WordPress. Users can add tags to their WordPress posts along with categories. However, while a category may cover a broad range of topics, tags are smaller in scope and focused to specific topics. Think of them as keywords used for topics discussed in a particular post. the post #needs-testing.

Top ↑

Sharing

Typically, calls for testing are shared on Make Core or on Make Test, depending on the feature and desired audience. Generally speaking, Make Core gets far more visibility so it’s best to share there when you need more attention on the effort from the start, especially if the efforts aren’t tied to a specific team.

If you’re struggling to get feedback, please view these high level suggestions.

Top ↑

Close the feedback loop

No matter what approach you’re using, the key is to ensure that information gets back to the right place. Usually, this means issues in Trac or issues in the Gutenberg GitHub repo.

Self serve calls for testing approach

This is a mostly hands off approach and solely requires clarity upfront in the call for testing you write. Any extra time you can spend ensuring that the instructions for where and how to report issues/feedback will be very high impact. This might include a short video around how to report or linking off to resources, like how to use Trac on Learn WordPress.

The biggest downside to this approach is that it requires a greater time investment of those reporting issues to both simply do the work to open an issue/comment on a current one and to know how to. This usually results in less engagement.

Facilitated calls for testing approach

This approach requires substantial involvement on behalf of those facilitating the call for testing. This varies wildly based on how many people respond to the call for testing as well. With this approach, those participating in the call for testing comment on the call for testing and those facilitating the call for testing respond with feedback, determining what needs open issues, what needs more clarity, etc. Here are some quick best practices:

  • Aim to respond within 1-5 days of someone leaving feedback, particularly if what you’re testing is more urgent or tied to a release. This both ensures someone’s effort is appreciated and that the feedback loop remains tight.
  • Replicate bugs that are found before reporting.
  • Make sure to search for currently open issues before opening new ones.
  • Comment on related issues with feedback as it’s relevant. Even if an issue is already known, it’s helpful for those working on them to have greater insight.

Mixed approach

Of note, you can always offer a mixed approach where you encourage folks to open issues directly as they are comfortable but have a fall back where you will facilitate closing the loop otherwise for those who aren’t in a position to do so for whatever reason.

Top ↑

Summarize the results

This section only applies if you are doing facilitated calls for testing. Summaries should be shared as soon as possible after a call for testing ends, ideally no longer than two weeks after. If what you are testing is less urgent, it’s okay to wait longer. These summaries should include the following as much as possible:

  • A high level summary with important takeaways.
  • List of any bugs.
  • List of feature requests.
  • Quotes/videos/images/etc to contextualize the more quantitive feedback and examine the broader experience being tested.

In terms of format, this will vary based on what’s being tested. For example, you might want to group related feedback for a feature under a heading (“Quote blockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. feedback”) or you might want to keep things more general (“General Usability feedback”). Here’s an example of a more general summary and an example of one with more specific headings based on common groupings of feedback.

Last updated: