Field Notes from WCEU 2022 Contributor Day

Olá! We’d like to express our gratitude to everyone who stopped by the Test Team (or CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress.:Test) table at this year’s WCEU Contributor Day 🙇. Your ideas, perspectives, and open discussions help foster initiatives critical to testing WordPress. Thank you to everyone who participated!

Participants at the event covered the following topics (some of which were also referred to #core-test in Slack):

Test Report Templates

  • In the proposed Test Reports guidelines, clarify how the green checkmark (✅) and red “X” (❌) emoji should be used in reports: expected vs unexpected.
  • Break out the report templates into subpages under a main “Test Report” description in the Test Handbook to improve readability.
  • Proposal to provide ticket creation templates for TracTrac Trac is the place where contributors create issues for bugs or feature requests much like GitHub.https://core.trac.wordpress.org/. issue reporting, similar to Gutenberg Issues (e.g for Bugs vs Enhancements).
    • @todo Investigate whether Trac supports pre-populated template options, for initial post and/or comments.

Easier Test Contributions

  • Improve/update the Test Handbook guidance for creating a local WordPress environment.
  • The desire for ephemeral test environments (no local installLocal Install A local install of WordPress is a way to create a staging environment by installing a LAMP or LEMP stack on your local computer. needed) to test PRs and patches. Some ideas:
    • Create a Core-targeted version of gutenberg.run.
      • Would need to support both PRs and individual Trac patch attachments.
    • Utilize a service similar to that used for calypso.live.
  • Add Test Handbook guidance for applying patches from GitHubGitHub GitHub is a website that offers online implementation of git repositories that can can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged be the repository owner. https://github.com/ or Trac, covering the various “best practice” methods.
  • Reiterate the importance of different environment flavors across the test contributor group (Docker/wp-env, VVV, Laravel Valet, Local, etc). There shouldn’t be a preference for “the best” or “one way” to run/test WordPress, since it should reflect the real-world variation across the WordPress community.
  • Assign a new week-in-test categoryCategory The 'category' taxonomy lets you group posts / content together that share a common bond. Categories are pre-defined and broad ranging. to Week in Test posts, for easier filtering of “where to start” in testing (currently grouped under the more generic summary category).

End-to-End (E2E) Testing

  • Questions as to where to begin E2E testing in WordPress:
  • It was noted that it’s common for E2E tests to fail intermittently, which can confuse and hamper development. This is often attributed to unexpected delays in DOM updates.
  • Consider that E2E testing can passively validate accessibilityAccessibility Accessibility (commonly shortened to a11y) refers to the design of products, devices, services, or environments for people with disabilities. The concept of accessible design ensures both “direct access” (i.e. unassisted) and “indirect access” meaning compatibility with a person’s assistive technology (for example, computer screen readers). (https://en.wikipedia.org/wiki/Accessibility) (“a11yAccessibility Accessibility (commonly shortened to a11y) refers to the design of products, devices, services, or environments for people with disabilities. The concept of accessible design ensures both “direct access” (i.e. unassisted) and “indirect access” meaning compatibility with a person’s assistive technology (for example, computer screen readers). (https://en.wikipedia.org/wiki/Accessibility)”) as a beneficial byproduct.
  • Add E2E section to Week in Test to increase awareness of this aspect of WordPress testing.

Tuesday Meetings Time Change

  • It was suggested that Tuesday meetings for <test-chat> and <test-triage> be shifted from 17:00 to 16:00 UTC to allow broader participation from European contributors. Please share your vote or thoughts here.

Props to @boniu91 for peer review of this post.

#meeting-notes

Test Team Chat Summary: 15 March 2022

The meeting started on SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. here.

@hellofromtonya made a summary of last week in Test:

  • Team agreed to first prioritize getting the @covers reviewed and committed as quickly as possible
    • @costdev and Tonya planned to do a deep review of @pbearne‘s PR
    • Then 6.0 took priority last week as Colin joined the 6.0 Release Squad
  • @Puja‘s e2e test was committed
  • Test Handbook – Team agreed to plan out how transform the handbook into a tool that empowers contributors to quickly get started testing.
  • PHPUnit Test Suite Restructuring – Team agreed to plan out how to restructure

@Ugyen asked if there’ll be a 5.9.3 release before 6.0, @audrasjb confirmed 5.9.3 release in the upcoming weeks.

@pbearne refreshed the @covers PR

@hellofromtonya agreed to move forward with the batch approach to make a review and commit process easier for this big PR

@tykoted joined his first meeting

@hellofromtonya reminded about the need of having one or two people that would lead the Test Handbook project. Together with @boniu91 agreed that summary post will be a good place to promote this opportunity.

@justinahinon described that moving e2e tests to Playwright in GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ is going quite well. He’s also working together with @juhise on moving existing tests to Playwright

@hellofromtonya reminded why we are in the process of switching to the Playwright

@pbearne offered Tugboat as a tool that could help the Test Team, he’ll provide more details soon.

Test Team is looking for a volunteer or two as lead(s) to shepherd Test Handbook forward. If you’re interested in this role, please leave a meesage.

#meeting-notes

Test Team Chat Summary: 1 March 2022

The meeting started on SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. here. With no agenda, the chat was open floor discussions.

6.0 Focus

@costdev asked: What projects or specific goals are targeted for 6.0? @hellofromtonya replied that 6.0 roadmap of the key features and refinements to be built for this major releaseMajor Release A set of releases or versions having the same major version number may be collectively referred to as “X.Y” -- for example version 5.2.x to refer to versions 5.2, 5.2.1, and all other versions in the 5.2. (five dot two dot) branch of that software. Major Releases often are the introduction of new major features and functionality.. The Test Team will support that effort.

4 Proposed Initiatives

Team discussed other projects and goals not necessarily specific to 6.0:

  • Initiative 1: Audit and reorganize the testing suite – which includes @covers and #53010.
  • Initiative 2: Ensure all patches ready for commit have test.
  • Initiative 3: Grow the e2e test suites.
  • Initiative 4: Testing Handbook – needs a lot of work to empower anyone to quickly get testing.

@hellofromtonya shared a bigger picture framework / thinking for testing:

Tests express the intended and expected behavior of how the code should and shouldn’t behave under different conditions. When tests do this, they are a wonderful source for learning.

Testing itself fuels the feedback loopLoop The Loop is PHP code used by WordPress to display posts. Using The Loop, WordPress processes each post to be displayed on the current page, and formats it according to how it matches specified criteria within The Loop tags. Any HTML or PHP code in the Loop will be processed on each post. https://codex.wordpress.org/The_Loop. back into the development process to know what does and doesn’t work as expected.

How?

  • Get more people involved in testing -> meaning the handbook, tooling, and code examples need to empower anyone to get testing.
  • Properly test both the expected (happy) and unexpected (unhappy) behavior in code at different levels > meaning tests need to be audited to ensure they are doing this and that the key areas have the tests needed to provide the feedback early enough.
  • Get the tests organized, readable, and usable.

The goals aren’t to build more tests. Rather, the goals are to build the tests that are needed to validate the code works as expected under different conditions to further help improve the user experience and get feedback fed back into the development cycles.

Where to start? What to prioritize?

Team discussed whether to start with planning the test suite reorganization or getting the @covers tags added / corrected.

The team decided to prioritize the @covers tags in order to ensure the code coverage reports are accurate to point to where more work is needed.

Team then discussed how to divide the auditing work to finish the @covers PR and get it merged.

Test Handbook

Restructuring the handbook raises the potential for more team members to work on the other initiatives. Since it feeds the numbers, confidence and accuracy of the team, it helps with everything else. Coverage reports won’t help increase the number of test team contributors, so I’m tempted to say that our first big drive should be to work on the handbook. If there’s things that need to be done in advance of handbook entries, such as establishing a new structure for the tests, that should be done in order to help 1) achieve the goal of restructuring and 2) provide information for the handbook entries.

by @costdev

Props to @boniu91 for proofreading.

#meeting-notes

Test Team Chat Summary: 20 July 2021

The meeting took place here in Slack. @hellofromtonya facilitated.

Discussion: Team Reps

Discussion starts here in Slack.

  • @lucatume asked “Who is this call open to?” Answer: all Test Team members
  • @lucatume asked “What are the responsibilities?” Answer: mostly administrative reporting duties including:
    • Gathering feedback and then writing the agenda for the biweekly team chat (example)
    • Facilitating test triage (example) and team chat (example)
    • Writing regular team recaps and posting them in Updates
    • Keeping an eye on the moving parts of the team to be able to report for quarterly updates (example)
  • Team discussed qualifications and refined them to be:

A rep is an active team member who is reliable and trusted, advocates for and is knowledgeable of one or more areas of testing, and wants to represent, nurture, and grow the team to serve the WordPress project.

Action: @hellofromtonya to publish Call for Nominations once peer review is complete. Done.

Update: Here’s the post: Test Team Reps: Call for Nominations.

Discussion: Team and Contributor Badges

Discussion starts here in Slack.

  • Reference: FSE Outreach Program badges proposal
  • Team discussed and agreed to use CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. Team’s model for awarding Test Contributor badge: 1 contribution = a badge
  • What is a test contribution?
    • @lucatume suggested to first define what is not a test contribution?
    • Do Release Party and Contributor Days participation count?
    • @hellofromtonya noted:

Broadly testing entire release is hard to capture the feedback. Right? But testing something that has a ticket or Call for Test (CfT) is trackable for not only testers but the Core folks who are working on that thing under test. Sharing how you tested (the test environment, theme, plugins, etc.), steps you did, results, and maybe even a screenshare …. this type of info is valuable.

Commenting: “It works” is not as valuable as a feedback.

Action: @hellofromtonya will start a proposal for discussion:

laying out some guidelines for how testing is captured, what testing is not, and then that will guide everyone to know what a testing contribution is and isn’t.

Update: Autogenerate e2e Tests Initiative

Discussion starts here in Slack.

#build-test-tools, #meeting-notes, #summaries

Test Team Chat Summary: 13 July 2021

The meeting took place here in Slack. @hellofromtonya facilitated.

This was the team’s first team chat in a year or more.

Discussion: What’s the team’s purpose?

@lucatume asked “what’s the purpose of the group?” Great question!

A healthy discussion happened around the history of the team and what role we serve in the project. The discussion starts here in the slack thread.

Some highlights are captured here:

  • Tonya shared the team’s history:
    • “The team is deeply rooted in validating and surfacing issues with user experience and usability. That’s where it started and lived in the space of testing workflows, screens, results, etc. through manual testing with users. Over the years, more and more testing has come to WordPress in the form of automated tests.”
    • “rebooting it [the team] to be all things QA and testing including triage, education, outreach, documentation, and expanding to multi-dimensional testing (automated testing, manual testing, usability – user – testing, and environment testing).”
  • Charter statement:
    • Tonya then shared “we as a team can formulate the actual charter statement. But here’s what I’m thinking (open to discussion):
    • “to create a high quality feedback loopLoop The Loop is PHP code used by WordPress to display posts. Using The Loop, WordPress processes each post to be displayed on the current page, and formats it according to how it matches specified criteria within The Loop tags. Any HTML or PHP code in the Loop will be processed on each post. https://codex.wordpress.org/The_Loop. into the project to continuously improve and elevate the quality and usability of WordPress”
  • Is the charter statement too generic?
    • @lucatume raised that it felt too generic as the terms “quality” and usability can different meanings.
    • @boniu91 shared “I think they need to be some kind of generic. We’re looking at WordPress from many different angles and point of views while testing.”

Action Item: Tonya to publish a vision of the team’s big picture of how the team can serve the project.

Discussion: Why did the team go dormant?

@lucatume asked why the team went dormant. The slack thread starts here.

Some of the reasons given include:

  • Testing and quality know-how
  • Interest in driving the team
  • Clear path for how QA and testing can serve the project

Discussion: autogenerating e2e tests

@lucatume raised a discussion about how to autogenerate e2e tests from written step-by-step processes or recording the actual testing process. The slack thread starts here.

The team agreed this is a great idea to help anyone contribute to testing.

Action item: @lucatume will work with @justinahinon and others to identify an auto-generator tool.

#build-test-tools, #meeting-notes, #summaries

Test Team triage meeting summary – June 1, 2021

The following is a summary of the bi-weekly Test Team triage meeting that occurred on 2021-06-01 13:00. A full transcript can be found here in the #core-test channel in the Make WordPress Slack. Meeting was moderated by @hellofromtonya.

Triage meetings are held every other Tuesday at 13:00 UTC. Reference to triage meetings announcement.

Announcements

Following links were shared:

Review of Action Items

@hellofromtonya shared tickets #34012 and #16773 are scheduled for testing.

@francina shared that she didn’t work on #39827 but it’s still in her queue. @boniu91 to send a copy of the premium pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party license for testing.

Ticket Triage

Ticket #13822

  • Status: ready for testing but may be impacted by new nav blockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. feature
  • Action: park this ticket and will reassess once new nav block feature is ready

Ticket #39589

  • Status: needs testing to reproduce issue with current WP version
  • Action: @boniu91 will test. Done
  • Followup: @boniu91 tested and found still an issue

Ticket #39603

  • Status: likely needs refresh, unit tests need improvement, needs performance testing
  • Action: @hellofromtonya will take this one

Ticket #31354

  • Status: not ready for testing as discussion ongoing (though stalled out)
  • Action: none

Ticket #12104

  • Status: needs a refresh and may need unit tests
  • Action: @hellofromtonya to comment on ticket. Done

Ticket #30499

  • Status: not ready for testing as discussion ongoing
  • Action: none

Ticket #25349

  • Status: needs testing but may not relevant today
  • Action: check if <!--more--> is still relevant

Ticket #40006

  • Status: skip as it’s awaiting review
  • Action: none

Ticket #37674

  • Status: skip as it’s a close candidate
  • Action: @desrosj will close the ticket. Done

Ticket #12104

  • Status: needs a refresh and may need unit tests
  • Action: @hellofromtonya to comment on ticket. Done

Ticket #31798

  • Status: needs consideration against the block editor and may not be applicable today ??
  • Action: none

Ticket #11477

  • Status: close candidate but followup items need tickets
  • Action: @francina taking this ticket to check if corresponding tickets were created

Ticket #40525

  • Status: needs a code review before ready for testing
  • Action: none

Open Floor

CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. 2e2 P2P2 P2 or O2 is the term people use to refer to the Make WordPress blog. It can be found at https://make.wordpress.org/.

@francina and @justinahinon raised creating a P2 to announce adoption of Jest + Puppeteer as the e2e setup for Core and need for handbook updates. This is a follow-up to a previous P2 by @francina.

Action: Create a draft for team review

Restart Team Meetings

@francina raised suggestion to restart the team’s meetings.

Team agreed to do these meetings same day and time but on alternating Tuesdays starting next week.

Action: @hellofromtonya to add to meetings calendar. Done

Action: @hellofromtonya to gather topics from team and create agenda for 1st team meeting

#meeting-notes, #summaries

Test Team triage meeting summary – May 18, 2021

The following is a summary of the bi-weekly Test Team triage meeting that occurred on 2021-05-28 13:00. A full transcript can be found here in the #core-test channel in the Make WordPress Slack. Meeting was moderated by @hellofromtonya.

Triage meetings are held every other Tuesday at 13:00 UTC. Reference to triage meetings announcement.

Summary

Following links were shared:

Started triage session using this ticket query report. During the meeting, this report was refined to this report.

Ticket #50683

  • Status: not ready for testing as there are no testing instructions
  • Action: @boniu91 will ask for testing info and add needs-testing-info keyword. Done
  • @francina observed that the reporter has been responsive when pinged

Ticket #37110

  • Skipped as no actionable changes for 5.8 to test
  • Action: none

Ticket #50924

Trac link https://core.trac.wordpress.org/ticket/50924

  • @Daria noted that there are no “how to reproduce” instructions, steps could be determined by the team, and would want to confirm with the reporter
  • Action: @Daria will write up the instructions and then ask the reporter to confirm. Done

Ticket #50781

  • Status: not ready for testing as discussions are still ongoing
  • @boniu91 noted that the steps to reproduce are clear but that discussions are still ongoing
  • @boniu91 asked: should the needs-testing keyword be removed?
    • @hellofromtonya explained that removing it would mean losing the tracking to know the ticket will need to be tested when ready
    • Agreed to keep it
  • Action: none

Ticket #42140

  • Status: not ready for testing or testing instructions as discussions ongoing
  • How to tell? dev-feedback keyword
  • Refined query report to exclude tickets with this keyword
  • Action: none

Ticket #34012

Ticket #16773

  • Status: needs a refresh and unit tests
  • Action: @hellofromtonya took this one since its PHPPHP PHP (recursive acronym for PHP: Hypertext Preprocessor) is a widely-used open source general-purpose scripting language that is especially suited for web development and can be embedded into HTML. http://php.net/manual/en/intro-whatis.php. version based, needs a refresh, and needs tests.
  • Update: Ticket is a duplicate and was closed.

Ticket #52050

Ticket #39827

  • Status: ready to test but …
  • @boniu91 noted it needs a premium pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party to test it
  • @francina asked if this is plugin territory and not coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress.
  • Action: @francina to check if they have a license to the premium plugin

#meeting-notes, #summaries

Test Team triage meeting summary – May 4, 2021

The following is a summary of the bi-weekly Test Team triage meeting that occurred on 2021-05-04 13:00. A full transcript can be found here in the #core-test channel in the Make WordPress Slack. Meeting was moderated by @hellofromtonya.

Triage meetings are held every other Tuesday at 13:00 UTC. Reference to triage meetings announcement.

Summary

Team planned to triage each 5.8 ticket marked as needs testing to determine if:

  • it is in a testable state
  • if yes, ask for volunteers to:
    • do the manual test
    • if needed, write automated tests.

Once the session started, shifted to discuss keywords and workflows for collective understanding.

Testing Keyword Combinations

Started reviewing ticket 37255 which has these testing keywords:

This ticket is not in a testable state. Team decided to discuss the meaning and usage of the keywords as well as how this team will know when it’s ready to be tested.

Discussed needs-testing keyword:

  • @hellofromtonya shared:
    • currently means => “it’ll need manual human testing once it’s ready” but doesn’t necessarily mean it’s ready right now for this testing.
    • can be added at any point in a ticket’s lifecycle.
    • Core Handbook defines it as: “One or more people are needed to test the solution”, but does not indicate when the keyword should be added.
  • Discussed removing keyword until it’s ready for testing:
    • @mai21 asked if the needs-testing keyword should be removed until all the testing details are added.
    • @francina added it “should be added only when it satisfied the has-testing-info requirement.
    • @hellofromtonya wondered how we’d know if a developer or maintainer wants the ticket to be manually tested if the keyword removed until ready.

Discussed workflow to know when a ticket is ready for Test Team action:

  • Discussed if a new keyword is needed such as ready-to-test
  • @boniu91 noted that “it’s pretty easy to sort the tickets in tracTrac Trac is the place where contributors create issues for bugs or feature requests much like GitHub.https://core.trac.wordpress.org/. using tags” and a ticket ready to test is noted as has-patch needs-testing has-testing-info.
  • @justinahinon suggested to make a P2P2 P2 or O2 is the term people use to refer to the Make WordPress blog. It can be found at https://make.wordpress.org/. post once the team agrees on how to add the tags/keywords

Team decisions:

Asking for testing info

@hellofromtonya shared the needs-testing-info script she used during 5.6 @hellofromtonya shared the needs-testing-info script she used during 5.6 and 5.7 cycles:

Please provide need more information to help testers manually test the patch (including at Test Scrubs):
– What are the steps to test?
– Are there any testing dependencies, such as a pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party or script?
– What is the expected behavior after applying the patch?

Team briefly discussed. Ran out of time.

Wrap up

Meeting went past the planned hour. @hellofromtonya wrapped up by encouraging further async discussion and Q&A.

Thanks to everyone who attended!

#meeting-notes, #summaries