Test Team Chat Summary: 12 October 2021

The meeting started on SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. here.

@hellofromtonya summarised last week in tests:

  • Team cleared large number of tickets that needed help.
  •  5.9 e2e strategy session was held and the strategy was set.

5.9 e2e strategy session was a big step forward for WP CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress., as:

  • We are getting started with bringing e2e tests to Core.
  • We now have clear boundaries of where e2e tests will live in Core and GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/.
  • We have a process/workflow from idea => PR ready => code review => commit.

@hellofromtonya mentioned, that the next step is to do the same for PHPUnit integration tests. Both of the above will become a 5.9 Test Roadmap

Open Floor

@justinahinon shared his concerns about shipping Playwright to Gutenberg now as it’ll need lots of work. He thinks that it’s better to make this experiment in the Core before.

@hellofromtonya shared her thought that Puppeteer and Playwright might exist in the suite together

@costdev agreed

@hellofromtonya wrote the Bigger Scope which is:

  • Teaching contributors
  • Creating tool that will convert from Puppeteer to Playwright
  • Changing to linters

@costdev reminded, that we need to agree on standard for writing e2e with Playwright

@hellofromtonya mentioned that she’s working on the Webfonts APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways.. The API will handle the theme.jsonJSON JSON, or JavaScript Object Notation, is a minimal, readable format for structuring data. It is used primarily to transmit data between a server and web application, as an alternative to XML. webfonts. There are some tickets that need unit/integration tests:

  • https://core.trac.wordpress.org/ticket/52252
  • https://core.trac.wordpress.org/ticket/47642
  • https://core.trac.wordpress.org/ticket/49985

If anyone’s interested in writing tests please leave comment either on Slack or in ticket itself.

Hallway Hangout: Pattern Party Testing Walkthrough (6 October)

This is a summary of a Hallway Hangout that was wrangled in the #fse-outreach-experiment channel as part of the FSE Outreach Program. It was intended to be a casual walkthrough of the current call for testing with @sparklingrobots taking the lead sharing their screen. Big thank you for being brave enough to meander around the call for testing publicly.

As a reminder, there is still 1 week left to participate so please join in if you can.

Attendance:

 Thank you to everyone who attended 🙂 @karmatosed @mkaz @shaunandrews @paaljoachim @annezazu @sparklingrobots @richtabor

Video Recording:

Feedback Overview

Throughout the session, some specific issues were noted as problem areas to follow up on, including some previous reported and some new:

  • It’s not clear when you are adding a theme blockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. outside of the Query LoopLoop The Loop is PHP code used by WordPress to display posts. Using The Loop, WordPress processes each post to be displayed on the current page, and formats it according to how it matches specified criteria within The Loop tags. Any HTML or PHP code in the Loop will be processed on each post. https://codex.wordpress.org/The_Loop. vs within. In this case, it was confusing when the Post Comment Count block wasn’t properly displaying the number of comments and it wasn’t clear the issue was that it needed to be within the loop. There’s an open issue to discuss this general phenomenon.
  • She had a desire to change the post excerptExcerpt An excerpt is the description of the blog post or page that will by default show on the blog archive page, in search results (SERPs), and on social media. With an SEO plugin, the excerpt may also be in that plugin’s metabox. length, which has an open enhancement issue.
  • There was an issue with the post title overflowing outside the bounds of a Query Loop block pattern. Need to replicate and open an issue!
  • There was a strange Inserter issue where you couldn’t find any theme blocks based on the current block you were selecting. Need to replicate and open an issue!
  • The call for testing itself needed to have updated instructions as you need to install GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ first before activating a block theme. This was updated live on the call.

Outside of specific points of feedback about the experience, there were also high level themes that became apparent as @sparklingrobots made her way:

  • They were confused about where various settings were expected to be, sometimes seeking out the block toolbar and other times the block sidebarSidebar A sidebar in WordPress is referred to a widget-ready area used by WordPress themes to display information that is not a part of the main content. It is not always a vertical column on the side. It can be a horizontal rectangle below or above the content area, footer, header, or any where in the theme.. For example, it was unclear at a glance how to change width of overall query and how to change the number of posts displayed.
  • The placeholders for various theme blocks were underwhelming and often not informative enough to know quite how to customize.
  • She was confused by how by selecting one Query Loop pattern, they instead ended up with two Query Loops! This is part of one of the default patterns included in the Query Loop block and could cause confusion long term when thinking about having more complex patterns.
  • There was a desire for more dimension controls for various blocks including the Columns Block and Post Comments block. This is under active iteration!

We ended the call talking about how the future of the Query Loop block powering more user friendly variations, an integrated block pattern library, an overhaul of the IA of the various design tools, & more will help ease this current experience. For now though, we’re in an in between state where loads can still be learned to improve the default tools themselves.

Next Steps

@annezazu will follow up next week after she returns (she’s out Thurs/Fri) to replicate and open issues.

#fse-hallway-hangout, #fse-outreach-program

Test Team Chat Summary: 28 September 2021

The meeting started on SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. here.

@hellofromtonya introduced agenda of the meeting and shared a post that describes where the testers are needed during the week. At the end of introduction, Team Test Update was shared.

The team was able to decrease the numbers of tickets.

Modernizing to the Latest PHPUnit

@hellofromtonya described how this changes impact CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. testers. Anyone that’s running the PHPPHP PHP (recursive acronym for PHP: Hypertext Preprocessor) is a widely-used open source general-purpose scripting language that is especially suited for web development and can be embedded into HTML. http://php.net/manual/en/intro-whatis.php. test suite or buil automated integration tests, should read the dev note.

For running the PHP test suite with Docker, nothing changes in your workflow:

npm install
npm run build:dev
npm run env:start
npm run env:install

and for running the tests:

npm run test:php

for running a specific group:

npm run test:php -- --group groupname

When not using Docker to test, then there are extra steps. The handbook and Dev Note provide the details.

5.9 Test Roadmap

In 5.9 release, Test Team will introduce the roadmap.

Everyone agreed that it’s necessary and help with the process of analyse and future improvement. @hellofromtonya also mention that we’ll implement a longer term vision, called North Star.

@justinahinon shared his document that is a great roadmap for automate tests focal group.

Focal Groups

@hellofromtonya shared an idea to have 5 separate focal groups in Test Team, each will need folks to shepherd the group:

If anyone is interested in being a co-lead of any group, it’s definitely possible and the Team is looking forward for new members.

@hellofromtonya shared expectations:

  • Shepherd your focal group – meaning be active in it, help contributors, know the status, help move work forward, stay aligned to the roadmap, and help to develop the roadmap
  • Each week – share an update for your group (can go in the team’s update)
  • When blocked or need more help, pingPing The act of sending a very small amount of data to an end point. Ping is used in computer science to illicit a response from a target server to test it’s connection. Ping is also a term used by Slack users to @ someone or send them a direct message (DM). Users might say something along the lines of “Ping me when the meeting starts.” Team Reps or another group for help

Last words

Team agreed that we need a bit of advertisment of our group.

Test Team Chat Agenda for 28 Sep 2021

Here is the agenda for the upcoming Test Team Chat scheduled for 2021-09-28 13:00.

This meeting is held in the #core-test channel in the Making WordPress slackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/..

Highlights

Agenda

  • Changes to coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress.’s PHPPHP PHP (recursive acronym for PHP: Hypertext Preprocessor) is a widely-used open source general-purpose scripting language that is especially suited for web development and can be embedded into HTML. http://php.net/manual/en/intro-whatis.php. test suite -> What does this mean for test?
  • 5.9 Test Roadmap
  • Focal group updates
  • Any blockers?
  • Open floor

Open Floor

Do you have something to propose for the agenda? Please leave a comment below.

Can’t make the meeting? Share anything relevant for the discussion in a comment below.

Props: @boniu91 for peer review.

#agenda, #build-test-tools

5.9 End-to-End (e2e) Working Strategy Session Agenda – RESCHEDULED for 6 Oct 2021

With efforts in progress to introduce end-to-end (e2e) testing in CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress., this working session brings folks together to collectively define the big picture goals and what to accomplish during the 5.9 cycle (i.e. roadmap). These goals can then serve as the North Star 🌟 to guide contributors for maximizing impact.

When: RESCHEDULED for 2021-10-06 10:00
Where: Zoom call (link will be shared in the Making WordPress slackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. #core-test channel prior to the start of the session)
Topic: Strategize e2e goals for the 5.9 release cycle

Agenda

  • Define big picture goals of what to accomplish during the 5.9 cycle
  • Identify and prioritize areas of Core that could most benefit from e2e tests
  • Discuss if any GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ tests can be ported over to Core
  • Strategize stepping stones
  • Identify key areas of contributor needs and responsibilities

Resources

Props to @justinahinon for proofreading.

#agenda, #build-test-tools

Test Team Chat Summary: 14 September 2021

The meeting started on SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. here.

Short explanation for first time attendees

@hellofromtonya reminded, that this bi-weekly meeting is where people who test WordPress CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. gather to discuss important things regarding test contribution. Testing includes manual testing, attempting reproduce reported issues, and automated testing.

You can find the agenda meeting here.

Week in Test introduction

@hellofromtonya described the purpose of this weekly publication, in short, it includes:

  • Parts of the core where testing help is needed this week
  • Learning and reading opportunities

Team agreed that it’s very helpful.

Last week’s Team update introduction

@hellofromtonya reminded that it’s a Team update, an overview of what has happeed last week. It also includes stats that are related to our team.

The Team agreed that the queue of the items that need testers’ attention is long.

Focal group updates

@hellofromtonya explained that it’s a time for testers to share updates for testing project initiatives. This section has started with:

  • PHPPHP PHP (recursive acronym for PHP: Hypertext Preprocessor) is a widely-used open source general-purpose scripting language that is especially suited for web development and can be embedded into HTML. http://php.net/manual/en/intro-whatis.php. 8.1 fixes are underway
  • Modernization to Latest PHPUnit project is nearly finished. Backporting is next, then dev note
  • PHPUnit Test modernization is ongoing

@hareesh added:

  • The latest version of jQuery UIUI UI is an acronym for User Interface - the layout of the page the user interacts with. Think ‘how are they doing that’ and less about what they are doing. was recently merged to Core. Need a lot of testing.

Documentation strategy

@francina opened this discussion, asking what instructions we want to give. The main reason was related to setting up environment.

@hellofromtonya proposed to create a docs empowering everyone to contribution, we need clear entry points for:

  • Local machine setups
  • How to do different types of testing
  • How to create test reports

Team agreed that this documentation should be present in Make Test website, Make Core should only refer to it.

@hellofromtonya mentioned, that PHPUnit docs page lacks a reference to where contributors can set up their local machines. That should be a priority.

@francina @juhise @hellofromtonya agreed that the Team should start documenting the steps for Docker testing environment and later repeat the same pattern for other solutions. On of the concerns is that screenshot are getting outdated very quickly.

@mkaz shown blockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience.-editor’s development site setup instructions based on wp-env. Team agreed that all projects should be pointing to one place where everything related to development environment is stored and maintained.

@hellofromtonya proposed a step-by-step instrctions of creating documentation for different types of test environments:

  • Start with Docker
  • Setups on Make Test with reference in Make Core
  • Figure out the doc info structure / strategy
  • Doc for different OS: Windows, Mac, and Linux
  • Link to tooling’s official docs
  • Get feedback from contributors
  • Refine
  • Repeat

@francina proposed working session with setting up environments. It’s going to happen on 2021-09-23 14:00

People with different OS will be essential for the successful meeting.

@boniu91 shared a reminder that on 2021-09-17 13:15 the Test Scrub is going to happen taking care of the following tickets:

#build-test-tools, #core-test, #test

Test Team Chat Agenda for 14 Sep 2021

Here is the agenda for the upcoming Test Team Chat scheduled for 2021-09-14 13:00.

This meeting is held in the #core-test channel in the Making WordPress slackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/..

Agenda

  • Week in Test: where you can contribute and learn this week
  • Review last week’s Team update
  • Focal group updates
  • Open floor

Open Floor

Do you have something to propose for the agenda? Please leave a comment below.

Can’t make the meeting? Share anything relevant for the discussion in a comment below.

Props: @boniu91 for peer review.

#agenda, #build-test-tools

Test Team Chat Summary: 17 August 2021

The meeting started on SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/., here.

Explanation for first time attendees, what is the Test Team Chat?

@hellofromtonya described that this is our time together to talk about things for our team: blockers, needs, roadmap, learning, docs, etc.

Reminder about the poll related to schedule fo our meetings

@boniu91 reminded, that there’s open poll to decide when we’ll be meeting in the future, twice a week:

  • Tuesday meeting
  • Friday test scrub session

@hellofromtonya asked to post the answers as comments inside the mentioned post.

Update: Modernize to Latest PHPUnit work

@hellofromtonya described what is the goal:

  • Run on the latest PHPUnit version
  • Use the PHPUnit Polyfills to shim backwards for backwards-compatibility (which enables PHPPHP PHP (recursive acronym for PHP: Hypertext Preprocessor) is a widely-used open source general-purpose scripting language that is especially suited for web development and can be embedded into HTML. http://php.net/manual/en/intro-whatis.php. cross-version testing using the matching PHPUnit version)
  • etc

And also what’s the current status:

  • All supported PHP versions are now running with their matching PHPUnit version :white_check_mark:
  • Old workarounds are removed :white_check_mark:
  • Tests are running on PHP 8.1 :white_check_mark:
  • The test suite’s assertions and expectations have been updated to PHPUnit 9.x :white_check_mark:
  • Backports are in progress
  • Dev Note is in progress and should be out very soon

CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. tests were blocked for over 2 years from running PHPUnit 8 and 9. They are not blocked anymore!!

@pbearne asked if we have any list of backports that needs to be finished
@hellofromtonya write a list of PRs:

There are 2 PRs for backports:

Plus, there’s still work to do in #53904

Also, the backports are waiting for PR 1587 to be fully reviewed and committed, we need to be sure that all paths are covered.

@lucatume offered, that he can take a look

Open Floor

@boniu91 asked if there’s any calendar with future WP Core releases
@francina answered that not yet

@pbearne asked what’s the plan for the @covers
@hellofromtonya answered that the plan for @covers is to address those during the reorganization and namespacing effort.

Why?
That’s when we’ll be thoroughly reviewing each test. At that point, we’ll know what each test is actually covering.

Why wait?
To avoid duplicating effort. We’re focusing first on PHP 8.1 and modernizing for latest PHPUnit. Once that’s done, then we plan to switch to the reorganization, namespacing, covers effort.

Ways of contribution

@hellofromtonya mentioned that there are many ways of contribution:

  • On the PHP side of things, there’s about ~10% code coverage in the automated tests.
  • Adding more unit and integration tests will also help the e2e effort too.
  • Once we get the tests modernized, then comes: improving the docs

For those who don’t know how to build automated tests, you can contribute too:

  • Help identify if there’s a test for a function or public method
  • If not, create a new ticket for it

@boniu91 commented that it would be good to have some kind of document that would describe the possible ways of contribution for new Test Team members.

Team agreed, @francina agreed to lead the effort, @hellofromtonya and @boniu91 offered help.



#build-test-tools

Test Team meetings schedule – poll

Test Team is meeting twice a week:

  • Tuesdays 13:00 UTC bi-weekly for Triage Sessions
  • Tuesdays 13:00 UTC bi-weekly for Team meetings
  • Fridays 13:15 UTC for Test Scrubs

This schedule was discussed while ago, when the team wasn’t crystalized as it’s right now. This is why it’s a good time to ask members of this Team whether the time of meetings is okay, or shall we reschedule it.

Please post in the comments your opinion about that, we’ll be collecting votes until 30th of August and then make a decision.

Thank you for reading!

#build-test-tools, #core-meeting

Test Team Chat Summary: 3 August 2021

The meeting started on SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/., here.

Announcement: Test Team Reps nominations

Announcement starts here.

@hellofromtonya mentioned that two testers were nominated for this position to work together, both of them accepted the nomination, @hellofromtonya @boniu91

Team had no objections to those nominations

@hellofromtonya published and shared previously prepared introductions of newly nominated Test Team Reps

Discussion: What is needed for Testers to make their contribution easier and better

The discussion starts here.

@francina shared her list, people present on the meeting agreed:

  • Clear testing instructions in tickets
  • Clear, unified, synched instructions to setup different testing environments
  • Streaming to go through tests together and learn by doing
  • A roadmap and action plan to automated testing

@lucatume added, that’s not clear for him:

  • Where and how he could contribute in a best way. He’s willing to help with what he’s the best in: coding.
  • It’s also not clear where we should collect discussions about the tickets and other important things.

@hellofromtonya replied to the second point, saying that:

  • If the discussion is related to ticket, we should collect them in ticket. If the discussion happens on Slack, it’s a good practice to link it inside the ticket.
  • Slack is great for adding collaboration, seeking guidance, Team chats, etc.

@lucatume expanded the previous (first) point and for the most effective contribution, the following things should be clear:

  • A clear indication of what tickets will need test code, not manual testing.
  • A clear path to how test code can be contributed. If tools are required, what tools are required.
  • A definition of “good” and “bad” friction.

@hellofromtonya answered, that tickets with needs-unit-tests keyword are the ones for the PHPUnit side of things. We don’t have anything for the e2e tests though.

@lucatume suggested adding needs-e2e-tests keyword and the Team agreed that’d be useful.

@francina mentioned, that the current test setup is Jest + Puppeteer. We need developers to set up a tool for the Team that will make creating e2e tests easy.

Team agreed that it’s not possible to have 100% of automated tests, with no human review.

@francina shared what the Team needs right now to kick off the e2e testing:

  • Skilled QA engineers to setup up the infrastructure
  • Documentation to teach people to write testing specs
  • People to write the tests
  • Automated test engineers to review and maintain

@hellofromtonya said, that most likely, the bottleneck will be the group of automation test engineers to do the review, tuning, and maintenance work.

@lucatume explained how the “autogenerating” of e2e tests looks like (here)

@hellofromtonya confirmed the workflow:

  • Human does the manual test
  • Tooling records those steps
  • Tooling converts the steps into code

Mai mentioned, that generated code can be worse in terms of quality than the one written by a human. Team agreed that it’ll need to be refined and maintained. @hellofromtonya shared improved workflow:

  • Human tester does the manual testing steps
  • Tooling records those steps
  • Tooling converts the steps into code
  • Test code is attached to the ticket
  • Code is reviewed by a human is skilled in e2e tests and the thing under test
  • Once approved, a coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. committer commits the test code into the project
  • Skilled e2e test humans maintain the tests (including tuning and refinement)

Team agreed to start a pilot initiative:

  • Build a prototype
  • Start small with a handful of impactful e2e tests
  • Get those tests stable
  • Learn
  • Iterate

@lucatume will start creating the prototype
@francina offered reviewing the test handbooks

#build-test-tools, #test-team