What automated accessibility testing can and cannot prove

Automated accessibility testing can show that parts of a page fail particular requirements: an image has no text alternative, a form field has no label, the page declares no language, or text has less colour contrast than WCAG requires. It cannot prove that a page, or a whole shop, conforms to WCAG 2.2 AA.

On what tools cannot do, the W3C’s guidance on selecting web accessibility evaluation tools says: “Tools cannot check all accessibility aspects automatically. Human judgement is required. Sometimes evaluation tools can produce false or misleading results. Web accessibility evaluation tools can not determine accessibility, they can only assist in doing so.”

The W3C expects both kinds of testing. Understanding Conformance, part of the W3C’s Understanding WCAG 2.2, says that all WCAG 2 success criteria are written as testable criteria, and adds: “Testing the success criteria would involve a combination of automated testing and human evaluation.”

The line between the two runs through the success criteria themselves. Many of them ask for something a program can find and for something a person has to judge. Success criterion 1.1.1 asks for a text alternative “that serves the equivalent purpose”. Success criterion 2.4.6 reads, in full: “Headings and labels describe topic or purpose.” A program can find the text alternative, the heading and the label. Whether they serve their purpose is a judgement about meaning.

A scanner built on a rule engine such as axe-core loads a page in a browser and runs a set of rules against the page as the browser renders it. Each rule looks for one kind of failure, such as a button without an accessible name, and reports the element where it found it.

axe-core, the open-source rule engine by Deque Systems that Validiscan’s scanner runs, has rules for WCAG 2.0, 2.1 and 2.2 at levels A, AA and AAA, and best-practice rules besides (axe-core README). Counted for this guide from the rule list of axe-core 4.13.0, the list puts 63 rules under WCAG levels A and AA, leaving out the rules it marks as experimental or deprecated. Together these 63 rules are mapped to 20 of the 55 success criteria that WCAG 2.2 has at those levels.

A scan sees what is on the page when the rules run. axe-core’s API documentation lists the steps of a test, and the last one is: “Repeat for any inactive or non-rendered content after making it visible”. A menu, a dialog or an error message that appears only after a click or a submitted form is tested only when something opens it first. Some checks therefore interact with the page: they move through it with the Tab key, or render it at a narrow width, and measure what happens. Validiscan’s checks of this kind are described below.

No. A failure that a rule reports is evidence that a requirement is not met. A pass shows only that the rule found no failure of the kind it looks for.

The W3C draws this line for its own test rules. ACT Rules describe how to test conformance with WCAG and WAI-ARIA, and an ACT Rule can test a single aspect of a requirement: “an ACT Rule might test that a table cell has a header, rather than testing success criterion 1.3.1 ‘Info and Relationships’ in its entirety.” When content fails an ACT Rule, it does not satisfy the corresponding requirement. When it passes, “no corresponding failures were detected”, which “does not necessarily mean that the content satisfies all aspects of the corresponding accessibility requirement.” ACT Rules are informative: they are not required for determining conformance to WCAG.

axe-core’s README states, among its principles: “It returns zero false positives (bugs notwithstanding).” It adds that axe-core returns elements as “incomplete” where it could not be certain, and that manual review is then needed. That is the project’s own statement about its design, not an independent measurement, and the W3C’s guidance quoted above warns, of evaluation tools in general, that they can produce false or misleading results. The design does make one thing clear: the number of reported failures is not a total. It counts what the rules could establish, not everything that is wrong.

Three rules of axe-core 4.13.0 show where the line falls:

• Images. The rule for images passes an image that has an alt attribute or, among a few other options, an ARIA label, and the pass message of its check reads “Element has an alt attribute” (check has-alt). The check tests that the attribute exists, not what it says. An empty alt, which marks an image as decorative, passes as well. Whether the text describes the image, and whether the image really is decorative, is for a person to judge.
• Video captions. The rule for captions passes a video that has a captions track. Without one, it cannot fail the video, because, as a comment in its code says, “media may use another caption method”. It returns needs review with the message “Check that captions are available for the element” (check caption). Whether the captions are accurate is beyond it.
• Colour contrast. The contrast check computes the ratio between text and its background, and it lists 15 reasons for returning needs review instead, among them a background image, a background gradient and an element overlapped by another one (check color-contrast). Text set on a background image, such as a banner, goes to needs review.

In Validiscan’s own assessment, an automated check can decide most cases for a few WCAG 2.2 success criteria. For most of them, a check can establish some failures, but a person decides whether the criterion is met. For about a third, only a person can decide.

This grouping is not a W3C classification. It is based on the text of each success criterion in WCAG 2.2, the W3C ACT Rules, the rule list of axe-core 4.13.0, and a W3C working draft, Challenges with Accessibility Guidelines Conformance and Testing, and Approaches for Mitigating Them, which is cited here as work in progress, as the draft itself asks. The draft sums up the general problem: “We know how to automatically test for and count the occurrences of relevant markup. However, we do not yet know how to automatically verify the quality of what that markup conveys to the user.”

Grouped this way, the 55 success criteria of WCAG 2.2 at levels A and AA fall into three groups:

The grouping describes what a check can establish in principle. What a particular tool tests is narrower, and differs from tool to tool.

In a full evaluation, tools assist and people evaluate. The W3C’s evaluation methodology gives tools two roles: helping to identify a representative sample, and assisting the people who then evaluate it.

WCAG-EM 2.0, the WCAG Evaluation Methodology that the W3C’s Accessibility Guidelines Working Group published as a Group Note on 23 July 2026, has five steps:

1. Define the evaluation scope.
2. Explore the target digital product.
3. Select a representative sample set.
4. Evaluate the selected sample set.
5. Report the evaluation findings.

On tools, WCAG-EM 2.0 says: “While most accessibility checks are not fully automatable, evaluation tools can significantly assist evaluators during the evaluation process and contribute to more effective evaluation.” Its example: “some accessibility evaluation tools can scan an entire digital product to help identify relevant samples for manual evaluation.”

WCAG-EM 2.0 also separates large surveys from the evaluation of one product. Surveys of many products are “typically carried out by primarily using automated evaluation tools”, and such evaluations “do not usually address the necessary qualitative depth of conformance review per product for which this methodology is designed.”

In one published test against a known set of barriers, none of the automated tools tested found more than half of them. The UK Government Digital Service (GDS) placed 142 accessibility barriers on a page of content on purpose and ran 13 automated tools on it. In its accessibility tool audit, last updated on 13 April 2018, the best tool found 40% of the barriers and the worst found 13%. Counting barriers that a tool flagged for a person to check, such as whether alt text descriptions were accurate, the best tool reached 50%. The project’s README sums it up: automated checkers are “good for finding simple and obvious problems, but aren’t able to detect many accessibility issues.” The figures come from tool versions of 2017 and 2018 and from a test page built for the purpose, not from a real shop, so they show a range, not a rate for any shop today.

Tools can report different numbers for the same page. They do not all test the same things, as the range in the GDS audit shows, and they do not all read WCAG the same way. axe-core’s guide to proposing rules says that “not all testers hold the same interpretation of the accessibility guidelines”, and that all of its rules must be consistent with the interpretation of Deque Systems, the developer behind axe-core.

Large automated surveys show the same limit from the other side. WebAIM’s analysis of the home pages of the top 1,000,000 websites, with data from February 2026, detected WCAG 2 failures on 95.9% of them. WebAIM adds its own caveat: “Because only automatically detectable WCAG failures were considered, this suggests that the rate of full WCAG 2 A/AA conformance was certainly lower than 4.1%.” The six failure types it found most often are all of a kind an automated rule can detect: low contrast text, missing alternative text for images, missing form input labels, empty links, empty buttons and missing document language. Such a list shows what tools can see. It is not a ranking of all barriers.

Validiscan’s scanner runs automated rules, and a person works through a manual checklist. Which success criteria the rules test, and which have an item on the checklist, is published criterion by criterion.

The scanner runs 102 rules: 79 rules of axe-core and 23 checks Validiscan wrote itself, many of them for what axe-core does not test, such as keyboard traps, focus order and reflow. Another 11 axe-core rules are switched off because one of Validiscan’s own checks tests the same thing, so one element never gives two findings, and 15 are not run because axe-core marks them experimental or deprecated, or because they test only level AAA. The rule catalogue lists every rule with its status.

Some of Validiscan’s own checks interact with the page. One “Follows the Tab sequence and detects loops and dead ends”; another “Renders pages at a width of 320 CSS pixels (the equivalent of 400% zoom)”. They measure behaviour. They do not judge meaning.

Of the 55 WCAG 2.2 success criteria at levels A and AA, 33 are tested by at least one rule the scanner runs. The other 22 are tested by no rule the scanner runs, and each of them has an item on the manual checklist of 35 items. The checklist also has items for some criteria that a rule tests only in part, such as keyboard operation, focus order and captions. For the other criteria that a rule tests, the checklist has no item. The criteria matrix shows, for each success criterion, the rules that test it and whether the manual checklist has an item for it.

For each checklist item, Validiscan records pass, fail or not applicable, with notes and, where there is one, an evidence file. The results are part of the Remediation Evidence Package, in its section “Manual review and excluded findings”. The package’s section “Scope and methodology” lists the limits of the evaluation, among them:

• “Automated checks find only part of all accessibility barriers. A criterion counts as supported only when automated checks fully cover it or a manual check passed.”
• “Content inside cross-origin frames and closed shadow DOM, such as payment fields and some consent tools and widgets, is not evaluated automatically.”

In Validiscan’s application, automated checks count as fully covering a criterion when a rule the scanner runs tests it and the manual checklist has no item for it. For such a criterion, a result of “Supports” in the package rests on automated checks alone. As of 6 October 2026, these include criteria that the grouping above leaves to a person to decide, such as 1.1.1 Non-text Content and 2.4.4 Link Purpose (In Context).

Results that a rule marks for review are not kept by the scanner today, so they do not appear as findings. The method page lists this among the limits.

The package states, criterion by criterion, conformance with WCAG 2.1 AA and WCAG 2.2 AA success criteria as evaluated on the listed pages and dates. This package is evidence of an evaluation and of remediation work. It is not a certificate and does not state legal compliance. The free check is automated only, with no manual checks, so the failures it reports are not a total.