Shopify Accessibility Baselines: How to Build a Defensible Index

An accessibility index is useful only when readers can understand the sample, reproduce the measurement, and separate automated findings from conformance claims.

AccessComply does not currently publish a representative Shopify-wide failure percentage. Earlier convenience-sample claims were removed because the underlying store list and reproducible aggregate dataset were not published. A selected automated sample should not be marketed as a fact about every Shopify store.

Minimum methodology for a credible index

Sampling

Define the population and selection method before scanning. A credible Shopify benchmark should disclose store geography, industry, catalog size, traffic or ranking source, theme type, app usage, and inclusion or exclusion rules. A hand-picked list cannot support a platform-wide percentage.

Scan scope

Report the number and types of pages reached per store, desktop and mobile viewports, authentication status, crawl limits, timeouts, robots behavior, redirects, password protection, and failed pages. A one-page homepage test is not comparable to a multi-template crawl.

Rule engine

Publish the engine and version, standards tags, configuration, custom rules, impact mapping, and date. Rule changes can alter results even when the storefront does not change.

Counting and deduplication

Separate unique rule patterns, affected nodes, affected pages, and store-level prevalence. Repeated navigation can create hundreds of node occurrences from one theme defect. Presenting raw occurrences alone can exaggerate the amount of distinct remediation work.

Manual review

State whether people reviewed false positives, severity, duplicate patterns, content meaning, keyboard behavior, screen-reader output, and task completion. An automated-only index should be labeled automated-only.

Uncertainty and publication

Publish aggregate data and enough methodology to reproduce it without exposing merchant-sensitive information. Distinguish observations in the sample from inferences about the wider Shopify population. Correct or withdraw claims when source data cannot be audited.

How to interpret an individual AccessComply scan

The public scanner checks up to 10 discoverable pages, subject to crawler safety and time limits, at desktop and mobile viewports. The installed app uses plan-specific scan limits. Results are automated WCAG-mapped findings in the reported scope.

Use the result to prioritize technical work. Do not read a high score as conformance or a low score as legal exposure. Confirm important journeys manually and document pages or states the scan could not reach.

What a future AccessComply index should include

Before publishing new aggregate claims, AccessComply should provide:

• a dated, documented sampling method;
• the exact scanner build and rules configuration;
• crawl-success and exclusion data;
• store-level normalized aggregate tables;
• deduplicated pattern counts alongside raw node counts;
• an independent manual-review sample;
• limitations and confidence intervals where applicable;
• a privacy-preserving reproducibility package.

Until then, merchants should scan and test their own storefront rather than using an unsupported industry percentage as a proxy.

Run a free automated scan and review the reported scope before acting on the results.