E2E (End-To-End) Testing in Core Proposal

What

E2E (End To End) tests are a form of automated software testing which runs in the context of a user using an application. The intention is to test the complete “end to end” workflow that a user would take in various scenarios.

In web software development this means running and controlling a web browser pointing at a web page, either fully automatically by a machine, or by a user following the test steps.

Frameworks for running E2E tests allow a web browser to be controlled fully automatically, for instance by navigating to URLs, filling out forms, and clicking on links and buttons. The tests operate by checking assertions about the expected content or behaviour of the page and triggering a test failure when they are not met.

Why

Testing individual aspects of an application’s code — for example with unit tests or integration tests — reduces the chance of low-level bugs in the code that powers an application. Unit testing helps maintain the quality of the code that powers the application.

In contrast, E2E testing tests the higher-level functionality of an application that the user sees, typically including the interface and the inputs and controls that the user interacts with.

Unit testing and E2E testing can, should, and do co-exist.

How

E2E tests are usually human-readable and provide logical steps for a user to perform. For example:

  • Given I am logged in as an Author
  • When I view the list of posts in the dashboard
  • And I see posts published by other users
  • Then I should not see an “Edit” link next to posts which are published by other users

The scenario above can potentially form a completely automated E2E test, depending on the framework in use and the built-in steps that have been defined for it. The tests are written in such a way that a human can follow the same steps and come to the same result.

This means anyone, not just a developer, can write an outline of an E2E test.

Implementing E2E tests in WordPress CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress.

WordPress Core right now has only one E2E test in place, to check that the wp-adminadmin (and super admin) page loads. GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/, on the other hand, has quite a few.

There are some tickets and pull requests open about this:
https://core.trac.wordpress.org/ticket/43395
https://core.trac.wordpress.org/ticket/49507

There is also a test environment but it is not currently being used as part of the automated test suite: https://make.wordpress.org/core/2019/08/05/wordpress-local-environment/

It would be beneficial to create a project to work on this in a structured way in order to maintain the stability and reliability of WordPress core as new features are added and changes are made.

Moving Forward

Months ago, this topic was mentioned multiple times and @pandjarov from SiteGround has offered to set up the test environment plus some engineering time to work on the tests.

We could either set up a new one or check that the existing one is still the right way forward.

To really kick things off, the Core Team has to come up with requirements for the test environment, its tests, and the preferred workflow, so the tests are really useful.

What is needed

  • A list of requirements – I think Core Committers, especially with a long history of contributing (the so-called historians) are best suited to start the list with the most pressing concerns
  • Evaluating the existing test environment (existing vs creating new)
    Ideas about what functionality in WordPress should be a priority in the automated tests, for example, the upgrade process, or managing users, or installing plugins
  • Developers who are familiar with Jest, which is the test framework in place in Gutenberg and WordPress core, so the test environment can be improved, and to write new tests
  • A fork of the wordpress-develop repo on GitHubGitHub GitHub is a website that offers online implementation of git repositories that can can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged be the repository owner. https://github.com/ where the project can be managed and PRs opened
  • Volunteers to work on this. It’s a big task because it starts from 0 – this is very important because what the previous post about this has shown, in my opinion, is that an environment is not enough 😉
  • Coordination with the Test team
  • For making it easier to communicate I propose to add a dedicated weekly chat in the #core-test channel – From September 6th I am available to do that.

Deadline to comment

To make sure that the project lifts off I think it needs enough interest and volunteers to actively work on it.

Please leave a comment before September 6th (so you have a month to think about it!) with the requirements and/or (possibly AND) your availability to be part of this.

Dev Chat Summary: July 11, 2018 (4.9.8 Week 2)

This post summarizes the weekly devchat meeting held Wednesday, July 11, 2018 (agenda | Slack archive)

4.9.7 Security and Maintenance Release Feedback

  • WordPress 4.9.7 was released on Thursday,  July 5 2018. This was a Maintenance & Security Release. Details here.
  • All 4.9.7 milestone tickets were pushed to 4.9.8. The last devchat summary and some other posts on Make/CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. have been updated to replace “4.9.7” mentions with “4.9.8”

4.9.8 Release Leads

As confirmed last week:

4.9.8 Schedule

  • BetaBeta A pre-release of software that is given out to a large group of users to trial under real conditions. Beta versions have gone through alpha testing in-house and are generally fairly close in look, feel and function to the final product; however, design changes often occur as part of the process.: July 17, 2018
  • RC: July 24, 2018
  • Final Release: July 31, 2018
  • Bug Scrubs: Monday at 14:00 UTC and Thursday at 21:30 UTC during the cycle.

4.9.8 Development Update

Updates from @pbiron:

Focus Lead and Component Maintainer Updates

GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/

JavaScriptJavaScript JavaScript or JS is an object-oriented computer programming language commonly used to create interactive effects within web browsers. WordPress makes extensive use of JS for a better user experience. While PHP is executed on the server, JS executes within a user’s browser. https://www.javascript.com/.

The JSJS JavaScript, a web scripting language typically executed in the browser. Often used for advanced user interfaces and behaviors. team posted notes on their meeting. Topics included:

  • Deprecation Strategy
  • WordPress package to be merged into Gutenberg
  • Merging WordPress JavaScript coding standards in Gutenberg
  • Service Workers scope

General Announcements

Devchat Coordination Reminder

Jeff Paul is on vacation, so @whitneyyadrich, @audrasjb, @joemcgill, and @antpb are handling PM-y devchat duties this month: collecting agenda items, publishing an agenda, running the actual devchat meeting, and publishing a devchat summary.

Next Meeting

The next meeting will take place on Wednesday, July 18, 2018, 20:00:00 UTC in the #core Slack channel. Please feel free to drop in with any updates or questions. If you have items to discuss, but cannot make the meeting, please leave a comment on this post so that we can take them into account.