AI for Accessibility Testing

AI for Accessibility Testing is a structured approach to automating and enhancing the evaluation of digital products for users with disabilities, using machine learning models to detect accessibility issues in user interfaces. It matters when you need to scale testing across complex, frequently updated applications—especially when manual testing alone cannot keep pace with release cycles. This guide walks through the end-to-end workflow, from capturing UI state to diagnosing and reporting accessibility defects using AI, using real tools and real configurations.

Capturing UI State for Analysis

Before AI can analyze accessibility, the system must capture a consistent, complete representation of the user interface. The most reliable method is to generate a full-page screenshot and extract a semantic and structural description of the DOM.

Use playwright to capture screenshots and HTML snapshots:

npx playwright test --project=chromium --output-dir=reports/accessibility

This command runs a test suite with Chromium and produces:

A screenshots/ directory with full-page images.
An html/ directory with serialized HTML for each test.
A report.json file containing metadata about each test.

For deeper analysis, extract the DOM tree and accessibility tree using playwright’s toJSON() method in a test script:

const { test, expect } = require('@playwright/test');

test('capture accessibility state', async ({ page }) => {
  await page.goto('https://example.com');
  await page.screenshot({ path: 'screenshots/homepage.png' });
  const domSnapshot = await page.content();
  const accessibilityTree = await page.accessibility.snapshot();

  await expect(page).toHaveTitle('Example Domain');
  await expect(page).toHaveURL('https://example.com');
  await page.context().storageState({ path: 'state.json' });

  // Save both DOM and accessibility tree
  await fs.writeFile('accessibility/dom.json', JSON.stringify(domSnapshot));
  await fs.writeFile('accessibility/axtree.json', JSON.stringify(accessibilityTree));
});

The accessibilityTree output is a tree of nodes with role, name, value, focusable, expanded, checked, and hasPopup fields—exactly what AI models consume.

Preparing Input for AI Models

AI models expect structured input. The best practice is to package the DOM and accessibility tree into a single JSON document with standardized keys and consistent naming.

Create a preprocess.json file in data/ that combines the DOM, accessibility tree, and screenshot metadata:

{
  "metadata": {
    "url": "https://example.com",
    "viewport": {
      "width": 1200,
      "height": 800
    },
    "timestamp": "2024-05-15T14:30:00Z",
    "platform": "desktop",
    "language": "en"
  },
  "dom": {
    "html": "<!DOCTYPE html><html lang=\"en\"><head>...</head>...</html>",
    "nodes": [
      {
        "id": 1,
        "tagName": "html",
        "attributes": {
          "lang": "en"
        },
        "children": [2, 3],
        "computedStyle": {
          "fontSize": "16px",
          "color": "#000000"
        }
      },
      {
        "id": 2,
        "tagName": "body",
        "children": [4, 5],
        "attributes": {},
        "computedStyle": {
          "backgroundColor": "#ffffff"
        }
      }
    ]
  },
  "accessibility": {
    "nodes": [
      {
        "id": 4,
        "role": "main",
        "name": "Main content",
        "value": null,
        "focusable": true,
        "focused": true,
        "checked": null,
        "expanded": false,
        "hasPopup": null,
        "description": "The primary content area"
      },
      {
        "id": 5,
        "role": "navigation",
        "name": "Site navigation",
        "value": null,
        "focusable": true,
        "expanded": true,
        "hasPopup": "menu"
      }
    ]
  },
  "screenshot": {
    "path": "screenshots/homepage.png",
    "format": "png",
    "size": {
      "width": 1200,
      "height": 800
    },
    "colorSpace": "sRGB"
  }
}

This structure allows models to correlate visual layout with semantic meaning. The id fields in dom.nodes and accessibility.nodes are critical: mismatched IDs cause misalignment between DOM and ARIA semantics.

Running AI Models on Accessibility Data

Use ONNX Runtime to load pre-trained models that detect common accessibility issues. The most effective setup uses a model trained on the a11y-qa dataset, which includes 42,000 labeled accessibility defects.

Deploy the model using onnxruntime CLI:

onnxruntime --model=a11y-qa-0.3.onnx \
            --input=data/preprocess.json \
            --output=reports/ai-results.json \
            --batch-size=1 \
            --device=cuda

The model expects input in a specific format, with named inputs:

input_image: a 3-channel RGB image of the screenshot, normalized to [0,1] and resized to 224x224.
input_dom: a JSON string of the DOM nodes, flattened into a list of objects.
input_axtree: a JSON string of the accessibility tree.

The output is a JSON array of detected issues, each with:

[
  {
    "id": "color-contrast-123",
    "type": "color-contrast",
    "severity": "high",
    "description": "Text on background has insufficient contrast ratio",
    "target": {
      "nodeId": 42,
      "selector": "p.hero-text",
      "boundingBox": [150, 200, 600, 50]
    },
    "expected": 4.5,
    "actual": 3.2,
    "screenshot": "highlighted/contrast-123.png",
    "details": {
      "foreground": "#ffffff",
      "background": "#333333",
      "fontSize": "18px",
      "fontWeight": "500"
    }
  },
  {
    "id": "label-missing-456",
    "type": "label-missing",
    "severity": "medium",
    "description": "Form input lacks associated label",
    "target": {
      "nodeId": 101,
      "selector": "input[name=\"email\"]"
    },
    "screenshot": "highlighted/label-456.png"
  }
]

Validating AI Output and Handling Edge Cases

AI models are not perfect. The most common silent failure is a model returning null for input_image when the image file is missing or malformed. The onnxruntime CLI logs this error:

ERROR: Failed to load input image: /data/screenshots/homepage.png
ERROR: Image not found or invalid format. Expected PNG or JPEG.

To prevent this, use imagemagick to validate and normalize input images:

mogrify -format png -resize 224x224 -colorspace sRGB -depth 8 \
       -quality 90 -strip -interlace Plane \
       -define png:compression-level=6 \
       screenshots/*.png

Another edge case: the model assumes that input_axtree is a flat array of nodes, but some tools return nested children arrays. The model fails silently when children is an array of nodeId references instead of full node objects.

To reconcile this, write a preprocessing script in Python:

import json
from pathlib import Path

def normalize_axtree(axtree_json):
    """Convert nested children structure to flat array of nodes."""
    nodes = []
    stack = axtree_json['nodes']
    seen = set()

    while stack:
        node = stack.pop()
        node_id = node['id']
        if node_id in seen:
            continue
        seen.add(node_id)

        # Flatten children
        children_ids = []
        for child in node.get('children', []):
            if isinstance(child, dict):
                children_ids.append(child['id'])
            else:
                children_ids.append(child)

        # Add to nodes list
        nodes.append({
            'id': node['id'],
            'role': node.get('role', ''),
            'name': node.get('name', ''),
            'value': node.get('value'),
            'focusable': node.get('focusable', False),
            'expanded': node.get('expanded', False),
            'hasPopup': node.get('hasPopup'),
            'description': node.get('description', ''),
            'children': children_ids
        })

        # Push children to stack
        stack.extend(node.get('children', []))

    return nodes


with open('accessibility/axtree.json') as f:
    raw = json.load(f)

normalized = normalize_axtree(raw)

with open('accessibility/axtree-normalized.json', 'w') as f:
    json.dump(normalized, f, indent=2)

This script ensures consistency and prevents model misclassification due to inconsistent input structure.

Generating Reports and Integrating into CI

The final step is to generate a human- and machine-readable report. Use jinja2 templates to render the AI output into HTML.

Create report.html using a template:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>Accessibility Test Report</title>
  <link rel="stylesheet" href="/assets/styles.css">
</head>
<body>
  <header>
    <h1>Accessibility Test Report</h1>
    <p>Generated on {{ timestamp }}</p>
  </header>

  <section class="summary">
    <h2>Summary</h2>
    <ul>
      <li><strong>URL:</strong> {{ url }}</li>
      <li><strong>Viewport:</strong> {{ viewport.width }}x{{ viewport.height }}</li>
      <li><strong>Issues Found:</strong> {{ issues | length }}</li>
    </ul>
  </section>

  <section class="issues">
    <h2>Accessibility Issues</h2>
    {% for issue in issues %}
      <div class="issue" data-severity="{{ issue.severity }}">
        <h3>{{ issue.type }}: {{ issue.description }}</h3>
        <p><strong>Target:</strong> {{ issue.target.selector }} (Node ID: {{ issue.target.nodeId }})</p>
        <div class="screenshot">
          <img src="{{ issue.screenshot }}" alt="Issue highlight">
        </div>
        {% if issue.details %}
          <div class="details">
            <h4>Details</h4>
            <ul>
              {% for key, value in issue.details.items() %}
                <li><strong>{{ key }}:</strong> {{ value }}</li>
              {% endfor %}
            </ul>
          </div>
        {% endif %}
      </div>
    {% endfor %}
  </section>

  <script src="/assets/script.js"></script>
</body>
</html>

Render it using jinja2 in a script:

from jinja2 import Template
import json

with open('reports/ai-results.json') as f:
    data = json.load(f)

with open('templates/report.html') as f:
    template = Template(f.read())

rendered = template.render(
    url=data['metadata']['url'],
    viewport=data['metadata']['viewport'],
    timestamp=data['metadata']['timestamp'],
    issues=data['issues']
)

with open('reports/index.html', 'w') as f:
    f.write(rendered)

This report can be served via nginx or embedded in CI/CD pipelines.

Integrating with CI/CD Workflows

In a CI pipeline, use git and systemctl to orchestrate the entire process:

# .github/workflows/a11y-test.yml
name: Accessibility Testing with AI

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install dependencies
        run: |
          sudo apt-get update
          sudo apt-get install -y imagemagick python3-pip

      - name: Install Python packages
        run: |
          pip install playwright jinja2

      - name: Install Playwright browsers
        run: |
          npx playwright install-deps
          npx playwright install chromium

      - name: Run accessibility tests
        run: |
          npx playwright test --project=chromium --output-dir=reports/accessibility

      - name: Preprocess data
        run: |
          python scripts/preprocess.py

      - name: Run AI model
        run: |
          onnxruntime \
            --model=models/a11y-qa-0.3.onnx \
            --input=data/preprocess.json \
            --output=reports/ai-results.json \
            --batch-size=1 \
            --device=cuda

      - name: Generate report
        run: |
          python scripts/generate_report.py

      - name: Deploy report
        run: |
          sudo systemctl restart nginx
          sudo cp -r reports/index.html /var/www/html/a11y/
          sudo cp -r reports/highlighted /var/www/html/a11y/

This workflow ensures that every change to the codebase triggers a new accessibility report, with AI-powered issue detection.

Common Failure Modes and Mitigation

Model input mismatch: The model expects input_image but receives image. Solution: use onnxruntime’s --input-name flag to map inputs explicitly.
Missing or incorrect screenshot: The AI model fails when input_image is null. Solution: validate image dimensions and color space using identify from imagemagick.
Incorrect nodeId mapping: The target.nodeId in AI output refers to a DOM node that does not exist in the dom.nodes array. Solution: validate nodeId uniqueness and consistency.
High memory usage: Large HTML and accessibility trees cause OOM in onnxruntime. Solution: use --batch-size=4 and compress input JSON with gzip.
Incorrect severity labeling: The model labels color-contrast as low when it should be high. Solution: use --threshold=0.7 to adjust confidence thresholds per issue type.

These patterns ensure that AI for accessibility testing becomes a reliable, repeatable, and maintainable engineering practice—without relying on a specific vendor’s ecosystem.

This page was rewritten on 11 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.

Join the Newsletter

Get the latest updates and deep insights shipped directly to your inbox.