The Test Team helps manage testing and triage across the WordPress ecosystem. They focus on user testing of the editing experience and WordPress dashboard, replicating and documenting bug reports, and supporting a culture of review and triage across the project.
If you’d like to help test Full Site Editing, please join the FSE Outreach Program. You can find current calls for testing for this program here and you can join the fun in #fse-outreach-experiment.
Welcome to this week’s edition of Week in Test! This post is a curated list of where you can get involved (i.e. where testers of all skill levels and expertise are needed), learning opportunities, and some reading to keep you informed.
This week focuses on helping with 5.9 new features and enhancements. Remember, feature freeze is 9 Nov.
Looking for ways to contribute? The following tickets and patches need contributors.
Who? All contributors (not just developers) who can set up a local testing environment, apply patches, and test per the testing instructions.
The following tickets need testers to manual test and provide feedback (test report):
Who? Any QA or PHPPHP PHP (recursive acronym for PHP: Hypertext Preprocessor) is a widely-used open source general-purpose scripting language that is especially suited for web development and can be embedded into HTML. http://php.net/manual/en/intro-whatis.php. developer contributors who can (or is interested in learning how to) build automated PHPUnit tests.
The following tickets need PHPUnit tests build:
wp_list_users() template tagwp_http_validate_url to control which ports are allowed for remote requestsWho? Any contributor.
The following new tickets need testers to attempt reproducing the reported issue and then providing a test report with the results:
open_basedir warnings on classic post edit screens with TinyMCE plugins presentProps to @boniu91 for peer review.
Welcome to this week’s edition of Week in Test! This post is a curated list of where you can get involved (i.e. where testers of all skill levels and expertise are needed), learning opportunities, and some reading to keep you informed.
Looking for ways to contribute? The following tickets and patches need contributors.
Who? All contributors (not just developers) who can set up a local testing environment, apply patches, and test per the testing instructions.
The following tickets need testers to manual test and provide feedback (test report):
Who? Any QA or PHPPHP PHP (recursive acronym for PHP: Hypertext Preprocessor) is a widely-used open source general-purpose scripting language that is especially suited for web development and can be embedded into HTML. http://php.net/manual/en/intro-whatis.php. developer contributors who can (or is interested in learning how to) build automated PHPUnit tests.
The following tickets need PHPUnit tests build:
_embed and _fields query parameters in the same querymonthnum query var is set without the year QVWho? Any contributor.
The following new tickets need testers to attempt reproducing the reported issue and then providing a test report with the results:
Props to @boniu91 for peer review.
A working session was held today to discuss big picture goals and what to accomplish during the 5.9 cycle.
Consensus reached as to where e2e tests will live within each repository:
All recent major changes to Core with user facing functionalities are top priorities to receive e2e tests as part of WordPress 5.9. These areas are:
tl;dr:
Along with the previous areas to test, the following priority would be to port some Core tests that are currently implemented in Gutenberg (e2e-test-utils package). Concerned tests are those related to the login feature.
The next step after this would be to improve the login tests to make them more performant. This includes for instance cookie based authentication across all tests in Core and Gutenberg.
These items are not necessarily part of 5.9 Goals. However, these are part of the bigger picture Test Roadmap.
tl;dr:
e2e-test-utils package (maintained by Gutenberg)All agreed Playwright for e2e tests would indeed have a lot of advantages for both Core and Gutenberg. However, consideration must be given for breaking changes and impacts for extenders using the test utils. Kai noted the utils could be made compatible with both Puppeteer and Playwright, meaning no breaking changes.
Also, a migration plan will need to be discussed on developed once the test utils are ready for Playwright. Part of this plan is to figure out how to change Gutenberg and Core as well as how to upgrade existing, not yet committed patches/PRs. The how and when is yet to be determined.
As there is still work to do to prepare for Playwright, the team agreed to continue with the roadmap to build e2e tests in Core.
Props @hellofromTonya for peer review and proofreading.
Welcome to this week’s edition of Week in Test! This post is a curated list of where you can get involved (i.e. where testers of all skill levels and expertise are needed), learning opportunities, and some reading to keep you informed.
Looking for ways to contribute? The following tickets and patches need contributors.
Who? All contributors (not just developers) who can set up a local testing environment, apply patches, and test per the testing instructions.
The following tickets need testers to manual test and provide feedback (test report):
clean_dirsize_cache()Who? Any QA or PHPPHP PHP (recursive acronym for PHP: Hypertext Preprocessor) is a widely-used open source general-purpose scripting language that is especially suited for web development and can be embedded into HTML. http://php.net/manual/en/intro-whatis.php. developer contributors who can (or is interested in learning how to) build automated PHPUnit tests.
The following tickets need PHPUnit tests build:
clean_dirsize_cache()$post in update_post_cache()_embed and _fields query parameters in the same querymonthnum query var is set without the year QVWho? Any contributor.
The following new tickets need testers to attempt reproducing the reported issue and then providing a test report with the results:
get_header_image_tag doesn’t add srcset/sizesReproducing the reported issue is the first step in a new defect ticket’s lifecycle. Why? In order to fix a bug, first step is confirm the bug is reproducible and is due to WordPress CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. itself (and not a third party like a plugin or theme).
Props to @boniu91 for peer review.
Howdy, good people! 👋
In the spirit of improving the E2E developer experience, I’d like to make a case for migrating CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress.’s browser automation library to Playwright. I was asked to write this post after opening an experimental pull request, where I migrated a selected portion of specs to Playwright, making them available for running both locally and on CI. That happened some time ago, so there’s already been quite a bit of discussion going on there! Having said that, I’m going to try making the case again here, taking the feedback I’ve received so far into consideration. I also encourage you to check out the PR to see the implementation details and Playwright advantages in action.
It’s easier to write stable tests.
There are a few reasons why. Please read on to find out which ones I think are the most relevant for the project.
Playwright’s API is almost identical to Puppeteer’s, which means that the developer transition should be close to effortless. I think that’s a big factor here, as it significantly lowers the cost of this transition also from the migrationMigration Moving the code, database and media files for a website site from one server to another. Most typically done when changing hosting companies. side. A good thing to start with! 🤞
From the documentation:
Playwright performs a range of actionability checks on the elements before making actions to ensure these actions behave as expected. It auto-waits for all the relevant checks to pass and only then performs the requested action. (…)
I think it’s the number one reason for the stability improvement over Puppeteer. In practice, it means the following:
Most of the DOM changes happen asynchronously, so in order to avoid flaky behavior, a test usually explicitly wait for an element before performing an action on it. A good example would be the most frequently used click action, which usually looks like this when performed with Puppeteer:
const button = await page.waitForSelector( 'button' );
await button.click();With Playwright, thanks to the auto-waiting mechanism it becomes just:
await page.click( 'button' )This is thanks to the stability check, which makes sure the element has a bounding box and is not moving before the action is performed. I think this is a major advantage because it allows to fully test the application, including the CSS animations, which are an integral part of the user interface.
In my PR, as a proof of concept, I removed the code that disables CSS animations and the forced reduced motion, which slowed down the refactored tests (21 suites) by around 37 seconds. This number will grow with every test, but judging by the current data it shouldn’t be more than a few minutes in total. I’d say the trade-off would still be worth it, but this can be discussed and decided upon later.
What do you think about testing without animations? Should they be enabled if it’s possible, even for the cost of extra wait?
In general, all of the above comes down to writing less and simpler code to get the same or better results than Puppeteer. If you go to my PR, you’ll notice that there are more lines removed than added in the refactored tests!
With Playwright, the tests and test utilities are easier to write and follow, and the environment requires less customization (e.g. disabled animations) which actually makes it closer to what users are experiencing.
Are there any downsides to the auto-waiting mechanism?
There were some concerns about how this mechanism could affect the performance tests. Because it could potentially become a blocking factor for this migration, I decided to migrate Gutenberg’s performance specs to Playwright as a proof of concept and see what happens. So far, thankfully it looks like there isn’t much difference between Puppeteer and Playwright — the performance metrics are very close, which would be the desired outcome.
Do you think there could be any downsides to the auto-waiting mechanism? Please let me know in the comments!
This part has changed a bit due to some feedback received in the PR. Originally, I mentioned text selectors and layout-based selectors as the number one reason for making the tests and utilities easier to write and follow, as well as making them more resilient. While prioritizing user-facing attributes is still a good practice for most applications, GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ is a bit different in this regard. Apparently, CSS classes are considered to be the API there, so they change less often than the user interface. Nevertheless, as some folks noticed advanced selectors are still a big win as they’d allow to, e.g. drop the use of cumbersome XPath selectors and more with powerful selector chaining. Currently, with Puppeteer, CSS selectors can only be used.
Learn more about Playwright’s advanced selectors from https://playwright.dev/docs/selectors.
The built-in Inspector is also a big advantage of Playwright. It’s quite intuitive and has some neat features like stepping through the script while running headfully or dynamically recording actions to a script — a really convenient way of quickly drafting the test. See the video below for a short demo of the script recorder.
Playwright offers a complete tracing solution. Trace can be recorded and stored in a zip file, which then can be viewed via the Trace Viewer GUI:
On the image above, you can see that the trace is displayed in a form of a film strip. Each frame can contain Before, Action, and After snapshots visualizing a complete action execution. On the left-hand side is a list of all the actions Playwright performed. Each of them can be inspected in more detail in the section on the right-hand side, where you can switch between the action log, location in the source code, and the network log.
I think it’s great to have that kind of functionality out of the box. It also shows how Playwright is intended to be a complete testing solution. With Puppeteer, there aren’t really any first-party tools for debugging, as far as I’m aware – The suggested way is to either slow down the tests in headful mode with DevTools open or use the Node.js Debugger when running headlessly.
Learn more about Playwright’s debugging tools from https://playwright.dev/docs/debug.
If there’s a goal to expand testing to browsers other than Chrome, it wouldn’t be an issue for Playwright as it supports all other major players: Firefox, WebKit, and Microsoft Edge. At the time of writing this, Puppeteer supports only Chrome and Chromium, and the official support of Firefox is currently experimental.
Learn more about Playwright browser support from https://playwright.dev/docs/browsers.
Playwright has a first-party test-runner, which is very similar to Jest test-runner (currently used for Puppeteer) but written from scratch. It contains a lot of end-to-end testing utils, tooling, concurrency, reporting, assertions, artifacts, etc., and extensive configuration support. Another quite nice thing to have without having to install and rely on third-party libraries!
Learn more about Playwright Test from https://playwright.dev/docs/intro.
I think it deserves a mention here, as it’s easy to navigate and really well written, in my opinion. Please take a few minutes and check it out for yourself at https://playwright.dev/docs/api/class-playwright – maybe you’ll find even more reasons to switch to Playwright? 😉
Writing good, stable E2E tests is often a struggle. If there’s a chance of improving that, especially with such a low cost, then it should be done. I would be happy to work on this task if there’s a consensus to move it forward.
Thanks for reading. I’m looking forward to all the feedback!
Props to @hellofromtonya and @boniu91 for proofreading!
Here is the agenda for the upcoming Test Team Chat scheduled for 2021-09-28 13:00.
This meeting is held in the #core-test channel in the Making WordPress slackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/..
Do you have something to propose for the agenda? Please leave a comment below.
Can’t make the meeting? Share anything relevant for the discussion in a comment below.
Props: @boniu91 for peer review.
Welcome to this week’s edition of “Week in Test”, a curated list of where you as a contributor can get involved (testers needed), learning opportunities, and some reading to keep you informed.
Looking for ways to contribute? The following tickets and patches need contributors.
Who? All contributors (not just developers) who can set up a local testing environment, apply patches, and test per the testing instructions.
The following tickets need testers to manual test and provide feedback (test report):
clean_dirsize_cache()Who? Any QA or PHPPHP PHP (recursive acronym for PHP: Hypertext Preprocessor) is a widely-used open source general-purpose scripting language that is especially suited for web development and can be embedded into HTML. http://php.net/manual/en/intro-whatis.php. developer contributors who can (or is interested in learning how to) build automated PHPUnit tests.
The following tickets need PHPUnit tests build:
clean_dirsize_cache()$post in update_post_cache()_embed and _fields query parameters in the same querymonthnum query var is set without the year QVWho? Any contributor.
The following new tickets need testers to attempt reproducing the reported issue and then providing a test report with the results:
Reproducing the reported issue is the first step in a new defect ticket’s lifecycle. Why? In order to fix a bug, first step is confirm the bug is reproducible and is due to WordPress CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. itself (and not a third party like a pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party or theme).
Props to @boniu91 for peer review.
With efforts in progress to introduce end-to-end (e2e) testing in CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress., this working session brings folks together to collectively define the big picture goals and what to accomplish during the 5.9 cycle (i.e. roadmap). These goals can then serve as the North Star 🌟 to guide contributors for maximizing impact.
When: RESCHEDULED for 2021-10-06 10:00
Where: Zoom call (link will be shared in the Making WordPress slackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. #core-test channel prior to the start of the session)
Topic: Strategize e2e goals for the 5.9 release cycle
Props to @justinahinon for proofreading.
Welcome to this week’s edition of “Week in Test”. This post highlights where you as a contributor can get involved (testers needed), learning opportunities, and some reading to keep you informed.
Looking for ways to contribute? The following tickets and patches need contributors.
Who? All contributors (not just developers) who can set up a local testing environment, apply patches, and test per the testing instructions.
The following are tickets and patches that need testers to manual test and provide feedback (test report):
clean_dirsize_cache()Who? Any QA or PHPPHP PHP (recursive acronym for PHP: Hypertext Preprocessor) is a widely-used open source general-purpose scripting language that is especially suited for web development and can be embedded into HTML. http://php.net/manual/en/intro-whatis.php. developer contributors who can (or is interested in learning how to) build automated PHPUnit tests.
The following tickets need PHPUnit tests build:
clean_dirsize_cache()$post in update_post_cache()_embed and _fields query parameters in the same querymonthnum query var is set without the year QVWho? Any contributor.
Since last week, there are 10 new tickets which need testers to attempt reproducing the reported issue and then providing a test report with the results.
Reproducing the reported issue is the first step in a new defect ticket’s lifecycle. Why? In order to fix a bug, first step is confirm the bug is reproducible and is due to WordPress CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. itself (and not a third party like a pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party or theme).
Props to @boniu91 for peer review.
The meeting started on SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. here.
@hellofromtonya reminded, that this bi-weekly meeting is where people who test WordPress CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. gather to discuss important things regarding test contribution. Testing includes manual testing, attempting reproduce reported issues, and automated testing.
You can find the agenda meeting here.
@hellofromtonya described the purpose of this weekly publication, in short, it includes:
Team agreed that it’s very helpful.
@hellofromtonya reminded that it’s a Team update, an overview of what has happeed last week. It also includes stats that are related to our team.
The Team agreed that the queue of the items that need testers’ attention is long.
@hellofromtonya explained that it’s a time for testers to share updates for testing project initiatives. This section has started with:
@hareesh added:
@francina opened this discussion, asking what instructions we want to give. The main reason was related to setting up environment.
@hellofromtonya proposed to create a docs empowering everyone to contribution, we need clear entry points for:
Team agreed that this documentation should be present in Make Test website, Make Core should only refer to it.
@hellofromtonya mentioned, that PHPUnit docs page lacks a reference to where contributors can set up their local machines. That should be a priority.
@francina @juhise @hellofromtonya agreed that the Team should start documenting the steps for Docker testing environment and later repeat the same pattern for other solutions. On of the concerns is that screenshot are getting outdated very quickly.
@mkaz shown blockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience.-editor’s development site setup instructions based on wp-env. Team agreed that all projects should be pointing to one place where everything related to development environment is stored and maintained.
@hellofromtonya proposed a step-by-step instrctions of creating documentation for different types of test environments:
@francina proposed working session with setting up environments. It’s going to happen on 2021-09-23 14:00
People with different OS will be essential for the successful meeting.
@boniu91 shared a reminder that on 2021-09-17 13:15 the Test Scrub is going to happen taking care of the following tickets:
You must be logged in to post a comment.