Plausible Accessibility: Where AI Still Falls Short and Who to Call Instead

Every code reviewer knows the shorthand. LGTM: looks good to me. It's one of the most common approvals in software and for most of what we review it's a reasonable bar.

For accessibility it's an odd one. It names the one sense that many users and some reviewers don't use.

I started thinking about the phrase in September, when I re-ran the accessibility checks behind every component in this site's design system. Each Storybook story in the test matrix writes a small evidence record: which checks ran, which passed, which runner produced them.

The first run produced 949 records for the light theme. All green, every one with exit code zero and every one missing its axe result. I'd forgotten one environment variable, CI=1. Without it the test runner treated accessibility violations as warnings. The records came out without the check that mattered most. Nothing failed and nothing warned. It looked good to me.

I threw that run away. The name for what it produced stuck.

Plausible accessibility.

Plausible accessibility is output that has the shape of accessible work and passes for anyone who can't check it: a reviewer skimming a diff, a scanner that tests what it can measure, a model grading its own answer. Inaccessible work usually looks broken somewhere. Plausible work looks finished, so nobody goes back to check it until someone can't use it.

{
  "storyId": "navigation-languageswitcher-default",
  "mode": "light",
  "sourceSHA": "ab85ef6ed9a545a5e1b91a5ef4f7d6af1097c3f5",
  "capturedAt": "2026-10-15T08:36:30.037Z",
  "runner": "local-2026-10-15",
  "checks": {
    "accessibility-tree": {
      "passed": true
    },
    "axe-no-violations": {
      "passed": true,
      "axeViolations": 0
    }
  }
}

The median of an inaccessible web

AI is about to produce a great deal more of it. A language model writes the most likely continuation of what it has seen. For interface code, what it has seen is the public web.

The web isn't a good teacher. WebAIM's annual scan of the top million home pages found detectable WCAG failures on 95.9% of them in the WebAIM Million 2026 report, up from 94.8% the year before. It was the first rise in six years. The average page had 56.1 detected errors. The report names AI-assisted "vibe coding" as one likely reason pages are getting more complex. Those are only the errors software can detect.

So when a code model writes a button, it writes the web's average button and the web's average button fails. Michael Fairchild at Microsoft built a benchmark to measure exactly this: 1,280 samples across eight frontier models, with no accessibility instructions. In his May 2026 update the average pass rate was 12%. The best model passed a quarter of the time. The hardest prompt, a dark-themed shopping home page, never passed.

The obvious fix is to ask for accessibility. It helps, sometimes. Fairchild found that a one-line instruction took the pass rate from 12% to 37% and a short instruction file took it to 60%. A study of six AI design tools presented at W4A 2026, a web accessibility research conference, found the opposite: when designers added accessibility requirements to their prompts, compliance went down. The designers judged that 39% of the AI tools' violations would need a major redesign, against 22% for their own work.

Different tools, measured differently. What they share is a ceiling. Fairchild's passes are automated passes and Microsoft's own report says a perfect score doesn't mean a page conforms to WCAG.

Then there's the confidence. Eeva-Jonna Panula, known as Eevis, an Android accessibility specialist in Helsinki, asked an assistant to build an app and explicitly asked for accessibility. The reply listed fifteen accessibility features it had considered. In the code she found a helper file nothing used, a semantic role that doesn't exist in the framework and cards with a redundant "Tap for details" on elements TalkBack, Android's screen reader, already announces as clickable. The list was plausible. The app wasn't.

People who use assistive technology meet this from the other side. When sixteen blind and low-vision programmers used Copilot for two weeks, they worked faster and they also struggled to tell the tool what they meant, to check what it produced and to keep track of what it changed.

At scale, plausibility is sold by subscription. Overlays promise to make a site accessible with one script tag and disabled people have organised against them for years because overlays get in the way of the screen readers and settings people already rely on. In April 2025 the US Federal Trade Commission ordered accessiBe to pay $1 million and barred it from claiming its product makes sites WCAG-compliant without evidence. A small florist that installed another overlay still had a site a blind customer couldn't use, was sued and is now suing the overlay vendor. The florist bought the thing that promised to make the problem go away. The customer was the one locked out.

What the green runs hid

Once CI=1 was back, the September runs went green for real. Green turned out to be a low bar. In the weeks that followed, this site turned up three more problems that no gate in the repo had caught.

The W3C saw this coming. Its guide to evaluation tools says tools can not determine accessibility, they can only assist. WCAG-EM 2.0, its newest evaluation methodology, notes that most accessibility checks are not fully automatable. Even a passing automated rule only means no failure was detected; a pass typically requires further verification by human testers.

The failures cluster in five places, each a judgment that needs context the model never sees: what something is for, how it sounds, the order it happens in, what it's called and how much effort it costs.

What the picture is for

WCAG's first success criterion, 1.1.1 Non-text Content, asks for a text alternative that serves the equivalent purpose. Purpose is the hard part. The W3C images tutorial says the right alternative is determined by the author, depending on the usage, context and content. A vision model describes what's in the picture. It can't know why the picture is on the page.

A house icon that links to the homepage should be announced as "Home" and a model will describe it, accurately, as a house. A bar chart's meaning lives in its trend. The W3C's draft research on machine learning and accessibility notes that automatic descriptions often miss the values that matter.

Generative descriptions add detail, but the same draft observes that they change each time the same image is processed. Regenerate the alt text for a chart three times and you can get three different accounts of what it says. Pick your favourite. For a chart the better fix is usually a short written summary and a data table beside it, as the W3C complex images tutorial has recommended for years.

Captions that score well

Captions look like the most solved problem on this list. The W3C draft puts automatic speech recognition at roughly 85% accuracy and notes that the Deaf community still considers it lacking. 1.2.2 Captions (Prerecorded) (a Level A criterion) and 1.2.4 for live content also ask for things accuracy scores ignore: who is speaking and the sounds that carry meaning.

Drishti Bansal, a deaf student who uses captions every day, describes where they break: accents, quiet voices, an air conditioner in the room, technical terms, two people talking at once. Tone disappears entirely.

A September 2026 preprint surveying 216 viewers, most of them deaf or hard of hearing, found the standard accuracy metrics tracked viewers' ratings far worse for AI captions than for broadcast ones. The number on a vendor's slide and the audience are scoring different things. Bansal's own verdict is the better summary: "AI makes communication possible, not equal."

Sign language is further away still. Signed languages have their own grammar and the World Federation of the Deaf and the interpreters' association WASLI have cautioned against signing avatars as a replacement for human interpreters.

Order, focus and the keyboard

Generated components usually look fine. They fail when someone operates them with a keyboard. On the Frontend Masters blog, Durgesh Rajubhai Pawar walked through a typical AI-generated sidebar that rendered perfectly and showed up in the accessibility tree as an anonymous generic container: no landmark, no heading, nothing focusable. Ten failures in 29 lines, none of them visible on screen.

The modal is the classic case. One developer's AI-built dashboard let focus escape the dialog to the page behind it. A QA team auditing a booking site found a modal that never returned focus to the button that opened it. Their scan found 14 issues; a manual pass found 47 more.

2.4.3 Focus Order asks that the order preserve meaning and operability. The W3C's guidance says several different orders can pass. A rule engine can't pick the right one. Someone has to press Tab and pay attention.

AI products have the same problem. Mike Gower at IBM found a floating chat button that looked about twelve tab stops away and took 85 presses to reach. Streamed answers keep changing until they settle, which breaks live regions, the mechanism screen readers use to announce content that changes (4.1.3 Status Messages). Gower calls the usual workaround feast or famine: a periodic "still composing", then the whole reply at once.

Names that sound right

The first of this site's three came from its own header. The language button shows "EN". With English already active, its accessible name began EN, EN, then a separator, then "Switch to English. Show language options." It passed every automated check in the repo. It had a name and the name contained the visible label, so it satisfied 2.5.3 Label in Name.

A screen reader user would have heard a repeated code and an instruction to switch to the language they were already using. A human auditor could fail it under 2.4.6 Headings and Labels. No machine would. The cause was ordinary: the header handed the component an action phrase where its contract asked for the language's name and the component dutifully read both.

Eevis found the same family of problem in AI-generated Android apps from four different tools: an "Add yarn" button described as "Add new yarn", so some screen readers read both. A machine can confirm a name exists. Whether it makes sense, said aloud in that order, takes someone listening.

The fix took two rounds. The first stopped the repetition but kept "Show language options" in the name, so the name changed whenever the menu opened, duplicating what aria-expanded already says. A review caught that. The name is now just the label and the language. The test that keeps it that way is the part worth copying, because it asserts the exact accessible name in each state:

expect(trigger).toHaveAccessibleName("EN, English");
await user.click(trigger);
expect(trigger).toHaveAccessibleName("EN, English");
expect(trigger).toHaveAttribute("aria-expanded", "true");
expect(screen.getByRole("button", { name: "FI, Finnish" })).toHaveAttribute(
  "lang",
  "fi",
);

A presence check is what let the bug through. An exact-name check fails the moment a name stops making sense. It still isn't the whole story: a screen reader adds the role and state. Each reader phrases them its own way, so someone still has to listen to the result.

   - group "Language":
-    - button "EN, EN — Switch to English. Show language options": EN
+    - button "EN, English": EN

Meaning, memory and effort

The quietest failures are cognitive. When researchers audited six AI-generated websites for ASSETS 2025, a leading accessibility research conference, 52.93% of the 308 errors they found were cognitive accessibility issues and most cognitive needs sit outside WCAG itself. The W3C's Making Content Usable for People with Cognitive and Learning Disabilities covers what WCAG doesn't.

Fluent text makes this worse, because it reads as if someone already simplified it. The AAA criterion 3.1.5 Reading Level can be approximated with a readability formula. The formula will happily approve a short sentence that lost the one condition a reader needed.

Login is the clearest case. The W3C's note on the inaccessibility of CAPTCHA cites research in which software beat audio reCAPTCHA about 90% of the time, while blind, deafblind and cognitively disabled people remain locked out. The test meant to tell humans from machines mostly tells machines from disabled humans.

WCAG only partly covers it. 3.3.8 Accessible Authentication (Minimum), at AA, removes memory and puzzle tests from login but still allows object-recognition CAPTCHAs; 3.3.9 at AAA closes that gap and 1.1.1 requires a CAPTCHA to offer alternatives in more than one sense. Checking any of it means watching people with memory or attention disabilities go through the whole flow.

What the machines are good at

None of this means keeping AI out of accessibility work. The real question is which work to give it.

Automation is excellent at the measurable. The same pipeline that once wrote empty records caught a real failure in September: in dark theme, the user's chat bubble on this site rendered body text at 1.82:1 against its blue background and axe flagged it on every chat message story.

The same bubble hid the second of the three. Its list bullets were the exact colour of its background: type a list into the chat and they vanished, at a measured contrast of 1:1. 1:1 is the technical term for invisible. No axe rule checks marker colour, so axe caught 1.82:1 and couldn't see 1:1. The fix was one line of CSS. The check that now guards it is a Storybook play test that measures the computed marker colour against the bubble and fails below 4.5:1, since numbered markers are text.

Models are useful when the job is structured: drafting alt text for a person to edit, turning success criteria into a test plan or reviewing their own output. Fairchild's best result came from a two-step "generate, then review" loop, which reached 86% at about five and a half times the input tokens. He also found that pasting the whole of WCAG into the instructions often made the code worse. Short and specific works better, something like this in the instruction file your coding agent reads:

## Accessibility
 
- Use native elements first: <button>, <a href>, <label>, <dialog>. No click handlers on <div>.
- Every control has a visible label and its accessible name starts with that label.
- Dialogs move focus in, keep it inside and return it to the trigger on close.
- Errors are text linked to their field with aria-describedby, never colour alone.
- Announce status changes in a polite live region instead of moving focus.
- Respect prefers-reduced-motion and keep layouts working at 200% text zoom.
- Before replying, review your output against this list and fix what fails.

The last line is the review step. It costs tokens. It's cheaper than shipping the first draft.

The division of labour I trust is plain. Machines propose and measure. People decide. The people who decide whether something works are the ones who use it. Once a model's output is accepted because it sounds right, the work has slipped back into plausibility.

Who to call and when

The W3C's evaluation guidance asks for combined expertise (people who understand disability barriers, assistive technology and adaptive strategies) and for involving users, because testing with people finds what a conformance review misses. Those are two roles and they often meet in one person: many of the best accessibility specialists are disabled themselves. A specialist tells you whether something meets the standard and why. The people who use assistive technology every day tell you whether it works. You usually need both: the specialist while you build; users from the first prototype onwards, not as a final sign-off.

The cases from this article are sorted below by when to make the call; each names the team that owns it. Every entry comes from a failure described above or from a gap an expert reviewer added.

Three rules make these calls work. Give testers tasks, not pages: "change your delivery address" finds problems that "review the settings page" never will. Don't generalise from one or two people; different disabilities and assistive technologies surface different problems.

And pay people, properly. Disabled testers are experts doing skilled work, so budget for them as you would any other specialist, including their preparation time. Recruit through disabled people's organisations or specialist panels, with sign-up and scheduling that work with assistive technology. Let people use their own devices and settings. Ask for informed consent and keep only the data you need, since disability data is special-category data under the GDPR. Credit people's contribution when they want it.

If you need to make the case to leadership in Europe, it isn't optional anymore. The European Accessibility Act has applied since June 2025 and EN 301 549 is the standard used to show conformance. A plausible accessibility statement is now a compliance risk as well as a user one.

Where my own checks still fall short

I'd like to say the pipeline that missed 949 axe results is fixed. In the narrow sense it is: I now know to set the variable. But no package script sets it for me and when I went back a month later to reproduce the failure for this article, I couldn't. The same command now records the axe result either way. I don't know exactly what changed, which is its own kind of lesson.

The third failure came from recapturing evidence for the language button. Writing the evidence files made Storybook hot-reload the story under test, so one check hung for its full 60-second timeout instead of failing. It passed in 241 milliseconds without the update step. Automation that can't detect its own gaps tends to fail politely.

A small guard closes the silent-skip class in any pipeline that writes records like these. It fails when a check you rely on is missing, as well as when it fails:

for (const record of records) {
  if (record.optedOut) continue; // stories that disable axe on purpose
  const axe = record.checks["axe-no-violations"];
  if (!axe) throw new Error(`${record.storyId}: axe result missing`);
  if (!axe.passed) throw new Error(`${record.storyId}: axe violations`);
}

The accessibility-tree snapshots this design system commits for every story in its test matrix are good at one thing: catching structural change. They record roles, names and states. They don't record whether a name makes sense out loud, whether an announcement arrives at the right moment or whether a sequence of steps is tiring. Those are judgments. A snapshot of a bad judgment passes forever.

During the Helsinki Design System work, I worked with two accessibility experts from Siteimprove. Both have visual disabilities. That engagement ended with the project. The design system behind this site hasn't had a review like it, which is the gap this article is about.

Works for whom

Looks good to me is still a fine approval for code. For accessibility, it answers the wrong question.

Models will keep getting closer to likely. Getting from likely to right, for one particular person, is still work for people.

The useful questions are smaller and harder: works for whom, on what, doing which task and who told you?

If your team is working out where it needs a specialist or testing with disabled people, get in touch.