Test Reports

What Is a Test Report?

A test report provides a documented record of a reported issue. It answers questions such as:

  • Does the issue occur for other users?
  • Under what conditions does the issue occur?
  • How can the issue be detected?
  • What are the impacts of the issue?
  • How does the issue appear to users?

Not only do test reports help substantiate the need to address an issue, but they provide important information to code maintainers so that patches address the correct feature of the software.

Top ↑

Key Test Report Features

The exact format of a test report may vary, but good reports always include the following pieces of information:

  • Environment: The operating system, software version, browser used, active plugins, and special configuration details (as applicable).
  • Steps to Test: A sequential list of actions to perform, and the expected behavior (i.e. what should happen) during the test.
  • Test Results: Clear indication of whether the test resulted in expected or unexpected behavior (such as whether a patch fixed an issue).
  • Supplemental Artifacts: Screenshots and animated GIFs, screencasts and videos, and log files (such as excerpts from debug.log) help provide additional context for reviewers to quickly identify notable behavior.

Use supplemental artifacts to help further explain the results of the test. There are no hard rules in what and how to provide this additional information, as the level of detail required can vary greatly between tickets.

Top ↑

Identify Expected and Unexpected Behavior

Visual cues that identify whether tests went as planned are useful for testers, code reviewers, and committers alike. TracTrac Trac is the place where contributors create issues for bugs or feature requests much like GitHub.https://core.trac.wordpress.org/. tickets, GitHubGitHub GitHub is a website that offers online implementation of git repositories that can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged be the repository owner. https://github.com/ issues, and PRs can become very lengthy, so calling out the most important details of a test using emoji or icons can save time.

“Expected” refers to behaviors, results, or observations of what is under test (see the following list) and are highlighted with a green checkmark emoji ✅:

  • When reproducing a reported issue, expected refers to the reported issue.*
  • When testing a bugfix patch, expected refers to the patch resolving the reported issue.
  • When testing a feature or enhancement, expected refers to what the feature or enhancement should do.

“Unexpected” refers to behaviors, results, or observations that should not happen (see the following list) and are highlighted with a red “X” emoji ❌:

  • When reproducing a reported issue, unexpected refers to any results that are not indicated by the issue reporter.
  • When testing a bugfix patch, unexpected refers to the patch not adequately addressing the issue or producing another bug or side-effect.
  • When testing a feature or enhancement, unexpected refers to anything that is not listed in the expected results (such as a bug, error, wrong UIUI UI is an acronym for User Interface - the layout of the page the user interacts with. Think ‘how are they doing that’ and less about what they are doing. displays, etc.) and/or the expected result doesn’t happen.

While at times it may seem counterintuitive, successful reproduction of the error as indicated in the ticket is considered “expected”, and should be marked accordingly (✅).

Top ↑

Where to Post Test Reports

The best location to post reports is to the corresponding ticket, issue, or PR (pull request) as a new comment.

Most tickets related to WordPress CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. can be found in Core Trac. When reporting test results in Trac, use WikiFormatting.

Issues tracked for the GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party are found in the Gutenberg GitHub repository, where it is helpful to provide reproduction test results in the reported issue. Patch test reports should be added to an individual PR’s comments. To format test report results in GitHub, use Markdown.

Top ↑

Types of Test Reports

There are three primary test report categories, and each comes with its own focus:

In some situations a Combined Report may be required.

Last updated: