Test Team Chat Summary: 3 August 2021

The meeting started on SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/., here.

Announcement: Test Team Reps nominations

Announcement starts here.

@hellofromtonya mentioned that two testers were nominated for this position to work together, both of them accepted the nomination, @hellofromtonya @boniu91

Team had no objections to those nominations

@hellofromtonya published and shared previously prepared introductions of newly nominated Test Team Reps

Discussion: What is needed for Testers to make their contribution easier and better

The discussion starts here.

@francina shared her list, people present on the meeting agreed:

  • Clear testing instructions in tickets
  • Clear, unified, synched instructions to setup different testing environments
  • Streaming to go through tests together and learn by doing
  • A roadmap and action plan to automated testing

@lucatume added, that’s not clear for him:

  • Where and how he could contribute in a best way. He’s willing to help with what he’s the best in: coding.
  • It’s also not clear where we should collect discussions about the tickets and other important things.

@hellofromtonya replied to the second point, saying that:

  • If the discussion is related to ticket, we should collect them in ticket. If the discussion happens on Slack, it’s a good practice to link it inside the ticket.
  • Slack is great for adding collaboration, seeking guidance, Team chats, etc.

@lucatume expanded the previous (first) point and for the most effective contribution, the following things should be clear:

  • A clear indication of what tickets will need test code, not manual testing.
  • A clear path to how test code can be contributed. If tools are required, what tools are required.
  • A definition of “good” and “bad” friction.

@hellofromtonya answered, that tickets with needs-unit-tests keyword are the ones for the PHPUnit side of things. We don’t have anything for the e2e tests though.

@lucatume suggested adding needs-e2e-tests keyword and the Team agreed that’d be useful.

@francina mentioned, that the current test setup is Jest + Puppeteer. We need developers to set up a tool for the Team that will make creating e2e tests easy.

Team agreed that it’s not possible to have 100% of automated tests, with no human review.

@francina shared what the Team needs right now to kick off the e2e testing:

  • Skilled QA engineers to setup up the infrastructure
  • Documentation to teach people to write testing specs
  • People to write the tests
  • Automated test engineers to review and maintain

@hellofromtonya said, that most likely, the bottleneck will be the group of automation test engineers to do the review, tuning, and maintenance work.

@lucatume explained how the “autogenerating” of e2e tests looks like (here)

@hellofromtonya confirmed the workflow:

  • Human does the manual test
  • Tooling records those steps
  • Tooling converts the steps into code

Mai mentioned, that generated code can be worse in terms of quality than the one written by a human. Team agreed that it’ll need to be refined and maintained. @hellofromtonya shared improved workflow:

  • Human tester does the manual testing steps
  • Tooling records those steps
  • Tooling converts the steps into code
  • Test code is attached to the ticket
  • Code is reviewed by a human is skilled in e2e tests and the thing under test
  • Once approved, a coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. committer commits the test code into the project
  • Skilled e2e test humans maintain the tests (including tuning and refinement)

Team agreed to start a pilot initiative:

  • Build a prototype
  • Start small with a handful of impactful e2e tests
  • Get those tests stable
  • Learn
  • Iterate

@lucatume will start creating the prototype
@francina offered reviewing the test handbooks

#build-test-tools, #test-team