WordPress needs automated accessibility testing

The Make WordPress Accessibility Team needs help wrapping its arms around WordPress Core as new features get created while helping refine the existing codebase to make it more accessible to people with disabilities. Our small team can’t be everywhere and test everything, but we’re working to gather an accurate picture of what needs our help the most.

Once we have a better idea of what needs our attention the most, how do we maintain a good grasp of it? Automated accessibility testing can help in a big way here, and I’d like to start working to incorporate it into Core. I brought this up in our last Accessibility Team chat.

What’s automated accessibility testing? Similar to unit testing, this would allow developers to run a set of tests during their local development workflow of patches for WordPress via a command line tool. These tests would output results that would alert developers to potential errors and help them fix things like missing alt attributes or unassociated form fields and labels. Automated accessibility testing isn’t perfect and it won’t help us overcome all of our challenges, but it can help us find simple, easy-to-fix errors, alert us to large trouble spots within the codebase that need more manual testing and allow us cover a bit more ground. It’s a nice compliment to the manual testing and education we’ve done so far.

What do we need to consider when selecting a tool?

WordPress Accessibility Team member Karl Groves has a nice blog post about this. He says you should consider three things:

  1. Flexibility and Freedom. These tools exist for one reason only: to work for you. Every single second you spend learning the UI, figuring out how to use it, or sifting through results is a second of development time that could go into actually improving the system. The tool must be flexible enough to offer the freedom to use it in any environment by any team of any size in any location.
  2. Accurate. Developers of accessibility tools want to find every single issue they possibly can. This unfortunately often includes things that are, in context, irrelevant or of low impact. They also may require an additional amount of human effort to verify. This is a disservice. Tool vendors need to understand that their clients may not have the time, budget, or knowledge to sift through false positives. The user should be able to immediately eliminate or ignore anything that isn’t an undeniable issue.
  3. Understandable The user of the tool needs to know exactly what the problem is, how much impact the problem has, where it is, and how to fix it. Overly general, overly concise, or esoteric result descriptions are unhelpful. If a tool can find a problem, then it can make intelligently informed suggestions for remediation

I think these are good goals to try to hit. Ideally, we want something that will have little to no impact on a developer’s workflow, allow us to select different test criteria and deliver accurate results we can act on.

As research, I’ve spoken to Jesse Beach. She’s a Senior Front End Engineer at Acquia Inc., a Drupal contributor, a contributor to QuailJS (a tool Drupal uses for automated accessibility testing) and a member of the Drupal Accessibility group. She’ll help us look at QuailJS as a possible automated accessibility testing approach. And hopefully, we can share accessibility knowledge between these two awesome open source projects! We talked about what we (WordPress Accessibility Team) want to accomplish with automated testing, how a tool might fit into a WordPress developer’s workflow, the future of QuailJS and a bit about accessibility in our different open source projects.

One other possibility on the short list of tools besides QuailJS includes Pa11y. We will likely look at others as well.

Next steps

  • Firm up a list of requirements for a tool
  • Experiment with some tools
  • Meet with Core team members to discuss
  • Develop test criteria

Your ideas and feedback are welcome!