Automated accessibility testing, and why we still need people

Automated accessibility testing is a key component of modern quality assurance. In my bachelor’s thesis, I investigated the extent to which accessibility can be automated and where the limits lie. In this article, I’ll show you why it’s only the interplay of tools, processes and human evaluation that really works.

Accessibility as a quality issue, not a one-off measure

In many projects, accessibility is still viewed as a post-development check. A test at the end, a report, a few fixes. In practice, this rarely works.

Especially in larger web applications, barriers do not arise in isolation, but systematically:

• through reused templates
• through component-based front-ends
• through continuous changes to content

This means that once problems arise, they quickly multiply across many pages.

Accessibility is therefore not a one-off issue, but a structural quality feature of digital systems.

Automated tests as a scalable foundation

This is precisely where automated accessibility tests come in.

As part of my bachelor’s thesis, I used various tools:

• Lighthouse
• axe-core
• Pa11y

The whole process was carried out using a script for each tool, enabling the systematic analysis of multiple URLs. The key advantage: scalability. Whilst manual checks can take several hours per page, automated tests deliver results in seconds and are reproducible.

They are particularly reliable at detecting:

• missing alternative text
• insufficient colour contrast
• missing accessible names for interactive elements
• structural HTML issues

This makes them ideal as a baseline in quality assurance.

Used correctly, they can be integrated directly into development processes:

• in build pipelines
• in deployment processes
• as regular regression tests

Accessibility is thus not checked retrospectively, but continuously ensured.

Where automation falls short

As powerful as these tools are, they have a clear limitation. Automated tests check rules. However, accessibility is more than just compliance with rules. In practice, only around 30 to 35% of the requirements of the Web Content Accessibility Guidelines can be reliably tested automatically. These form the technical basis, but cover only part of actual accessibility.

Many crucial aspects are context-dependent:

• Is alternative text truly understandable?
• Does the focus order make sense for keyboard use?
• Is the navigation structured logically?
• Does it actually work efficiently with screen readers?

These questions cannot be answered purely from a technical perspective. This became clearly evident in my analysis: Some of the problems were identified exclusively through manual checks.

This shows: Automation provides an important technical baseline, but it does not replace an assessment of the actual user experience.

The key point: combination rather than either/or

The central finding of my work is therefore not a comparison of tools, but a methodological approach.

A sensible approach to accessibility combines both perspectives:

Automated tests for:

• rapid and broad analyses
• continuous quality assurance
• verification of minimum technical requirements

Manual tests for:

• assessment of actual usability
• context-dependent requirements
• complex interactions

Only this combination enables an efficient and robust assessment.

What this means for projects

From a project perspective, this results in a clear framework for action:

Accessibility should

• be considered early in development
• be technically validated through automation
• be specifically checked manually

And above all: It should be integrated into processes, not just into tools. This is precisely where the difference lies between one-off measures and sustainable quality.

Conclusion

Automated accessibility tests are a central component of modern quality assurance. They enable scalability, speed and continuous verification of technical requirements.

Furthermore, the results clearly show: Accessibility cannot be fully automated.

In my investigation, the homepage was chosen as the reference for a full manual test, as it represents the majority of functionalities and components. The page was analysed entirely manually. In the process, a total of twelve violated WCAG success criteria were identified.

In direct comparison, the automated tools used were able to detect six of these twelve criteria on the same page. This comparison shows that a significant proportion of the technical requirements can be tested automatically, whilst an equally large proportion still requires manual assessment.

Taking into account the remaining manual testing effort and the necessary interpretation of the automated results, the study demonstrated that combining both methods can reduce the testing effort by around 50%.

The key factor is therefore not complete automation, but the targeted combination of automated and manual testing methods.