The meeting started on Slack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/., here.
Announcement: Test Team Reps nominations
Announcement starts here.
@hellofromtonya mentioned that two testers were nominated for this position to work together, both of them accepted the nomination, @hellofromtonya @boniu91
Team had no objections to those nominations
@hellofromtonya published and shared previously prepared introductions of newly nominated Test Team Reps
Discussion: What is needed for Testers to make their contribution easier and better
The discussion starts here.
@francina shared her list, people present on the meeting agreed:
- Clear testing instructions in tickets
- Clear, unified, synched instructions to setup different testing environments
- Streaming to go through tests together and learn by doing
- A roadmap and action plan to automated testing
@lucatume added, that’s not clear for him:
- Where and how he could contribute in a best way. He’s willing to help with what he’s the best in: coding.
- It’s also not clear where we should collect discussions about the tickets and other important things.
@hellofromtonya replied to the second point, saying that:
- If the discussion is related to ticket, we should collect them in ticket. If the discussion happens on Slack, it’s a good practice to link it inside the ticket.
- Slack is great for adding collaboration, seeking guidance, Team chats, etc.
@lucatume expanded the previous (first) point and for the most effective contribution, the following things should be clear:
- A clear indication of what tickets will need test code, not manual testing.
- A clear path to how test code can be contributed. If tools are required, what tools are required.
- A definition of “good” and “bad” friction.
@hellofromtonya answered, that tickets with needs-unit-tests
keyword are the ones for the PHPUnit side of things. We don’t have anything for the e2e tests though.
@lucatume suggested adding needs-e2e-tests
keyword and the Team agreed that’d be useful.
@francina mentioned, that the current test setup is Jest
+ Puppeteer
. We need developers to set up a tool for the Team that will make creating e2e tests easy.
Team agreed that it’s not possible to have 100% of automated tests, with no human review.
@francina shared what the Team needs right now to kick off the e2e testing:
- Skilled QA engineers to setup up the infrastructure
- Documentation to teach people to write testing specs
- People to write the tests
- Automated test engineers to review and maintain
@hellofromtonya said, that most likely, the bottleneck will be the group of automation test engineers to do the review, tuning, and maintenance work.
@lucatume explained how the “autogenerating” of e2e tests looks like (here)
@hellofromtonya confirmed the workflow:
- Human does the manual test
- Tooling records those steps
- Tooling converts the steps into code
Mai mentioned, that generated code can be worse in terms of quality than the one written by a human. Team agreed that it’ll need to be refined and maintained. @hellofromtonya shared improved workflow:
- Human tester does the manual testing steps
- Tooling records those steps
- Tooling converts the steps into code
- Test code is attached to the ticket
- Code is reviewed by a human is skilled in e2e tests and the thing under test
- Once approved, a core Core is the set of software required to run WordPress. The Core Development Team builds WordPress. committer commits the test code into the project
- Skilled e2e test humans maintain the tests (including tuning and refinement)
Team agreed to start a pilot initiative:
- Build a prototype
- Start small with a handful of impactful e2e tests
- Get those tests stable
- Learn
- Iterate
@lucatume will start creating the prototype
@francina offered reviewing the test handbooks
#build-test-tools, #test-team