WordPress core is now using Playwright for all browser-based tests

Early last year, a plan was outlined to phase out all Puppeteer usage across WordPress coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. and GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ step by step. Since then, a lot of progress has been made:

  • The @wordpress/e2e-test-utils-playwright package has become more mature and already provides many useful utilities that cover the needs for core and Gutenberg.
  • The @wordpress/scripts package, which is widely used in the ecosystem, now not only supports Puppeteer but also Playwright, making it straightforward for projects to migrate their existing setups.
  • The majority of e2e tests in Gutenberg have already been migrated from Puppeteer to Playwright, as can be seen in this tracking issue. This includes all performance tests.

Now, another major milestone has been reached: all browser-based tests in WordPress core now use Playwright! This includes end-to-end (e2e) tests, performance tests, and visual regressionregression A software bug that breaks or degrades something that previously worked. Regressions are often treated as critical bugs or blockers. Recent regressions may be given higher priorities. A "3.6 regression" would be a bug in 3.6 that worked as intended in 3.5. tests. See #59517 for full details.

How does this affect contributors?

As mentioned in the original proposal, Playwright makes it easier to write stable tests. Thanks to a built-in test runner with superior debugging tools, auto-waiting, and advanced selectors support, Playwright offers the tools necessary to write tests that are easy to understand and cause less headaches because of random timeouts or similar. It also supports many browsers out of the box, which will help strengthen the end-to-end test coverage in core.

Since Playwright is originally a forked version of Puppeteer, it offers a very similar APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways., so contributors writing and debugging tests should be familiar with the way things work.

If you want to get familiar with Playwright, check out Gutenberg’s migration guide as well as the official Playwright documentation. The migrationMigration Moving the code, database and media files for a website site from one server to another. Most typically done when changing hosting companies. guide can be used to migrate Puppeteer-based tests from unmerged patches/PRs to the new setup.

End-to-end tests are still run using npm run test:e2e, with npm run test:e2e -- --ui running them in an interactive mode for easier debugging.

Playwright support was just added recently, so you might need to run npm install first to ensure the latest dependencies are installed.

Note: If you previously ran visual regression tests using npm run test:visual, the storage location for visual snapshots was changed from tests/visual-regression/specs/image_snapshots to tests/visual-regression/specs/snapshots. You can safely remove the files from the old folder if they show up in version controlversion control A version control system keeps track of the source code and revisions to the source code. WordPress uses Subversion (SVN) for version control, with Git mirrors for most repositories..


Props to @mamaduka for reviewing this post.

#e2e-tests, #gutenberg, #playwright, #puppeteer

Migrating WordPress E2E tests to Playwright

Context

The idea of using Playwright as an alternative to Puppeteer was initially mentioned in early 2020 during a JavaScriptJavaScript JavaScript or JS is an object-oriented computer programming language commonly used to create interactive effects within web browsers. WordPress makes extensive use of JS for a better user experience. While PHP is executed on the server, JS executes within a user’s browser. https://www.javascript.com/. chat (meeting recap). At that time, the project was still in its early stages, and it was decided to postpone the decision until Playwright was more mature.

A few months ago, @bartkalisz proposed a proof of concept to migrate the GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ E2E test suite from Puppeteer to Playwright. This proposal was discussed several times in the above PR, during an editor chat, and during test team meetings.

WordPress test team members, Gutenberg team members, and Playwright engineers met to discuss in detail how this migrationMigration Moving the code, database and media files for a website site from one server to another. Most typically done when changing hosting companies. could be done and what the possible blockers might be. You can find the minutes of that meeting here.

The infrastructure for writing Playwright tests in Gutenberg has been implemented by @kevin940726 here.

The purpose of this post is to present a migration plan to Playwright; as well as the implications this might have.

Areas in WordPress that will be affected by the migration

  • @wordpress/e2e-test-utils: It is a utility package with basic tests that are reused by other E2E test packages and suites in Gutenberg, in WordPress CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. and by third parties. This package is now deprecated and a new package @wordpress/e2e-test-utils-playwright was introduced internally for the migration process.
  • @wordpress/e2e-tests: This is the E2E test suite from Gutenberg. It contains tests for different parts of WordPress and Gutenberg. The package has also been deprecated, and new E2E tests exist now in the test/e2e folder.
  • WordPress Core E2E tests: This is the E2E test suite for WordPress Core. To date, it contains only a very small number of tests. It uses the @wordpress/e2e-tests-utils package, and will need to be migrated to use Playwright.

Migration plan

Here is the roadmap we will follow to perform the migration

  1. Add Playwright test infrastructure to Gutenberg (done)
  2. Gradually migrate Gutenberg E2E tests to Playwright:
    • Migrate tests from packages/e2e-tests to test/e2e one file at a time. See overview issue here.
    • Migrate test utils from packages/e2e-test-utils to packages/e2e-test-utils-playwright as required.
  3. Add Playwright test infrastructure to Core and migrate Core E2E tests to Playwright: once the e2e-test-utils used in Core are migrated, we can migrate the current suite of tests in Core to Playwright. 
  4. Remove old packages: remove @wordpress/e2e-tests, rename @wordpress/e2e-test-utils-playwright to @wordpress/e2e-test-utils.

Writing new tests during the migration

For now, developers may continue to write new E2E tests using Puppeteer. Once a significant amount of E2E tests have been migrated to Playwright, we can discuss enforcing that new E2E tests are written using Playwright. Waiting will ensure that enough test utils exist and that there are no problems with the new Playwright infrastructure.  See this issue for migration progress.

How to help

Helping migrate tests to Playwright is one way you can help right now. Playwright’s syntax is similar to Puppeteer’s, so it won’t be totally unfamiliar.

Here is some useful link to get you started:


Props to @kevin940726 @noisysocks for helping put this post together, and @aristath and @sergeybiryukov for reviews.

#e2e-tests, #gutenberg, #playwright, #puppeteer