This is an overview of running and writing tests for WordPress. Automated testing is running test cases where manual intervention is not required to run each one. This is usually in the form of writing test suites which have multiple test cases and a library and command line tool that runs the test suite or suites. The test suite execution is usually manual, from the developer choosing which suites on the command line to run, but this isn’t required. The process could be automated and looked over from time to time to ensure that when the code changed, no problems were introduced.

Installation #

1. Install PHPUnit. WordPress uses PHPUnit, the standard for unit testing PHP projects. Installing PHPUnit is usually accomplished using the PEAR Installer. PHPUnit has instructions here, and on its Github repository. After, you may run phpunit --version from the command line to check that PHPUnit is installed correctly, and can be found in your $PATH environment variable. On Windows and having trouble? Check out this guide and add support for ANSI Command Line Colors under Windows.

2. Check out the test repository. The WordPress tests live in the core development repository, at http://develop.svn.wordpress.org/trunk/:

$ svn co http://develop.svn.wordpress.org/trunk/ wordpress-develop
$ cd wordpress-develop

3. Create an empty MySQL database. The test suite will delete all data from all tables for whichever MySQL database it is configured. Use a separate database.

4. Set up a config file. Copy wp-tests-config-sample.php to wp-tests-config.php, and enter your database credentials. Use a separate database.

Top ↑

Running the Test Suite #

In the root directory — next to wp-tests-config.php, the tests/ folder, and the phpunit.xml.dist file, run:

$ phpunit

You should see output that looks roughly like the following:

....................................................SS.....SS 61 / 1303 ( 4%) 
...SSSSSS.............S..SSS..........SS................S.... 122 / 1303 ( 9%) 
..................................S.......................... 183 / 1303 ( 14%) 
..................S............S.................S.......S... 244 / 1303 ( 18%) 
.............S.......S.S.......S............................. 305 / 1303 ( 23%) 
....SSSS.S..SSS.........S.SS................................. 366 / 1303 ( 28%) 
............................................................. 427 / 1303 ( 32%) 
............................................................. 488 / 1303 ( 37%) 
............................................................. 549 / 1303 ( 42%) 
......................................................S...... 610 / 1303 ( 46%) 
...................SS............SS.......................... 671 / 1303 ( 51%) 
..............SS.....................S....................... 732 / 1303 ( 56%) 
.............SSS..SSSSSSS.................................... 793 / 1303 ( 60%) 
SS.........SSSS.SS........................................... 854 / 1303 ( 65%) 
....................................S........................ 915 / 1303 ( 70%) 
..........................S..S................SS.S........... 976 / 1303 ( 74%) 
..........................................S.............S.... 1037 / 1303 ( 79%) 
............................................................. 1098 / 1303 ( 84%) 
............................................................. 1159 / 1303 ( 88%) 
.....S.................S..................................... 1220 / 1303 ( 93%) 
.....................................S

Time: 02:24, Memory: 119.75Mb
OK, but incomplete or skipped tests!
Tests: 1258, Assertions: 5219, Skipped: 74.

What each symbol means:

  • . – Each dot signifies one “test” that passed.
  • S means a test was skipped. For WordPress, this usually means that the test is linked to a bug report, and that bug report remains unfixed.
  • F means a test failed. More output will be shown for what exactly failed and where.
  • E means a test failed due to a PHP error, warning, or notice.
  • I means a test was marked as incomplete.

Running Specific Tests #

By default, the Ajax tests (tests written for core’s use of wp-admin/admin-ajax.php) are not run. To run these:

$ phpunit --group ajax

To run the tests under multisite, you must switch to the multisite.xml configuration file:

$ phpunit -c multisite.xml

Or you may instead define the constant WP_TESTS_MULTISITE as true in wp-tests-config.php.

Additionally you can grunt phpunit and run PHPUnit tests, including the ajax and multisite tests.

To run an individual file, you can use the name of the class, or the full path to the file:

$ phpunit Tests_Dependencies_Styles
$ phpunit tests/dependencies/styles.php

To run a group of tests – defined by @group in code comments:

$ phpunit --group dependencies
$ phpunit --group themes

Many tests are marked with a @ticket annotation, which indicates they were the result of that WordPress Trac ticket. The test is skipped if the ticket is still open — in these cases, we treat the test as a “known bug” that will likely fail. To see information about skipped and incomplete tests, use --verbose:

$ phpunit --group shortcode
Tests: 37, Assertions: 71, Skipped: 5.

$ phpunit --group shortcode --verbose

There were 5 skipped tests:

1) Tests_Shortcode::test_tag_hyphen_not_tag
WordPress Ticket 17657 is not fixed

tests/includes/testcase.php:150
tests/includes/testcase.php:130

To forcibly run the skipped tests for an open ticket, you may specify that ticket as a group:

$ phpunit --group 17657
Tests: 2, Assertions: 3, Failures: 2.

You may also combine groups: (Depending on your platform you may have to wrap the groups in double quotes)

$ phpunit --group shortcode,17657,6562,14050
Tests: 37, Assertions: 77, Failures: 5.

To view all groups:

$ phpunit --list-groups

Top ↑

Writing Tests #

TODO. Go over basic test organization, files/classes/methods, WP_UnitTestCase, setUp, tearDown, rollback, factory methods. Link to list of assertions in the PHPUnit manual.

Top ↑

Contributing Tests to WordPress #

There are three primary ways to contribute:

Write tests for a reported bug. A great way to contribute is to write tests that demonstrate an existing bug report. The core developers are reluctant to consider patches for many sensitive areas in core without test coverage. Well-written tests can help confirm that a patch fixes a problem without side effects, and can prevent any regressions from occurring in the future. When tests are particularly needed or desired for a ticket to proceed, they receive the needs-unit-tests workflow keyword. You can submit tests for existing tickets directly on the WordPress core Trac. Bonus points for submitting a bug report with a test case included.

Write new tests to improve our code coverage. Many areas of WordPress do not have adequate test coverage. Pick a function, class, or component and write tests for it. You can submit these tests on the WordPress Trac.

Fix or improve our existing test cases. There are many opportunities for improvement in the existing tests. Some of them are ancient and others are slow or fragile. Some do not tests well in multisite or under certain conditions. Some individual tests try to test too much, and could be improved by using data providers, dependencies, and more narrow assertions.

Additionally, More than 75 tests (linked to a few dozen different tickets) are currently considered to be known WordPress bugs and therefore skipped. (If you run phpunit --verbose, you can see the complete output.) Given enough test coverage, these tests can drive these bug fixes. You may find something you want to work on among the skipped tests.

Top ↑

Further Reading #