5.9 End-to-End (e2e) Strategy Session Summary

A working session was held today to discuss big picture goals and what to accomplish during the 5.9 cycle.

• Meeting agenda
• Attendees: @hellofromtonya, @isabel_brison, @kevin940726, @netweb, @boniu91, @justinahinon, @juhise.

Where e2e tests live: GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ vs. CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress.

Consensus reached as to where e2e tests will live within each repository:

• Core: non-Gutenberg features, UIs, and UXUX UX is an acronym for User Experience - the way the user uses the UI. Think ‘what they are doing’ and less about how they do it. including:
  • new stuff being built during a release cycle
  • existing interfaces such as CustomizerCustomizer Tool built into WordPress core that hooks into most modern themes. You can use it to preview and modify many of your site’s appearance settings., login, pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party and theme installation/activation, interaction with list tables, settings page, etc.
• Gutenberg: its functionality being built within its repository

5.9 Core e2e focus areas

All recent major changes to Core with user facing functionalities are top priorities to receive e2e tests as part of WordPress 5.9. These areas are:

• The application passwords feature
• The uploading of new versions of plugins/themes feature
• The plugin dependency project
• The updater project
• Twenty Twenty-Two theme

Porting tests from Gutenberg to Core

tl;dr:

• Tests that test Gutenberg’s functionality will remain in its repository and not be ported to Core
• Core specific tests (such as login) can be moved to Core

Along with the previous areas to test, the following priority would be to port some Core tests that are currently implemented in Gutenberg (e2e-test-utils package). Concerned tests are those related to the login feature.

The next step after this would be to improve the login tests to make them more performant. This includes for instance cookie based authentication across all tests in Core and Gutenberg.

Other Roadmap Items

These items are not necessarily part of 5.9 Goals. However, these are part of the bigger picture Test Roadmap.

• Documentation: The goal is to help contributors quickly contribute to testing.
  • Test Handbook: Clearly document multiple workflows for folks to pick their onramp into testing
  • README: #53550 get its PR reviewed and merged
• Visual Regression: #49606 is an experiment to allow local vision regression testing.
  • From these learnings, plans can be crafted for how to build it into an automated CI process.
  • The challenges for the CI are storage of the artifacts and unreliability of testing these across different environments. A third party service may be possibility to explore in the future.
• Is Core a good experimental sandbox for Playwright?

Proposal to migrate to Playwright for e2e tests

tl;dr:

• Not a blocker to build Core’s e2e tests
• Requires changes in e2e-test-utils package (maintained by Gutenberg)
• Needs a migrationMigration Moving the code, database and media files for a website site from one server to another. Most typically done when changing hosting companies. plan

All agreed Playwright for e2e tests would indeed have a lot of advantages for both Core and Gutenberg. However, consideration must be given for breaking changes and impacts for extenders using the test utils. Kai noted the utils could be made compatible with both Puppeteer and Playwright, meaning no breaking changes.

Also, a migration plan will need to be discussed on developed once the test utils are ready for Playwright. Part of this plan is to figure out how to change Gutenberg and Core as well as how to upgrade existing, not yet committed patches/PRs. The how and when is yet to be determined.

As there is still work to do to prepare for Playwright, the team agreed to continue with the roadmap to build e2e tests in Core.

Props @hellofromTonya for peer review and proofreading.

#build-test-tools, #core-test, #summaries