Regression protection for accessibility fixes

Each run is a new scan of your shop, compared with what Validiscan already knows about it. A run takes four steps:

1. The shop is scanned again. ValidiscanBot scans the shop with the scan settings stored for it. With the default settings, that means the same kinds of pages as a full scan, at desktop and phone size, including the cart and the first checkout step, without completing a purchase or entering payment data.
2. The findings are matched. Every failure the run finds is matched with your shop’s existing findings, so Validiscan knows which ones were fixed before and which ones are new.
3. The pages are compared. The pages the run reached are compared with those of the comparison scan.
4. The result is recorded. The run appears in the timeline in the customer portal. When it found something, or could not finish, a digest email goes out.

Regression protection runs weekly or monthly, on a fixed schedule. A weekly run takes place on a set weekday and time, a monthly run on a set day of the month and time, both in your organization’s time zone. The service Validiscan offers covers a weekly re-scan.

The customer portal shows the schedule and the date of the next check. A run is skipped while your shop is paused in Validiscan, or while Validiscan has no recorded permission to scan it.

Each run is compared with the history of your shop’s findings and with a comparison scan, the baseline. The comparison scan is the latest completed verification scan, or the latest completed full scan if the shop has no verification scan yet. Validiscan picks it again for every run, so after a new verification scan, the next run is compared with that one.

Findings are recognized from one scan to the next by a fingerprint made from the rule, the element’s selector and the shape of its HTML. That is how a run knows that a failure it finds is one that was fixed before.

A regression is a failure that was fixed and is found again. Precisely: a finding with the status Fixed, Verified or Resolved externally that a run finds again. The run reopens it, so its status is Open again, and reports it as a regression.

Fixed problems can come back, for example after a theme or app update.

Validiscan sends at most one digest email per run. A run sends one when it finds a regression, a new finding, a resolved finding or a coverage change, or when it could not finish. A run that finds none of these sends no email, unless Validiscan has switched on a digest after every run because you asked for one. That setting is off by default.

• A run that could not finish sends a digest too, so that you know the shop was not checked that time.
• The digest goes to the email addresses set up for your shop’s regression protection, in your organization’s language.
• Its subject names your shop and says how many fixed problems came back, or that none did (with the number of new problems, if there are any), or that the check could not finish.
• It lists what came back first, then the new findings with the serious and critical ones first, then the resolved findings and any change in the pages checked. Its links open the findings and the timeline in the customer portal.
• While regression protection is paused, no runs take place and no digest is sent.

The customer portal has a regression protection page for each shop. It shows the schedule, the next check, and the latest checks, newest first, each with its date and its result:

Under each check, the timeline lists the problems that came back, the new ones and how many of them are serious or critical, the ones no longer found, and the kinds of pages that appeared or disappeared. Each problem links to its finding and says where it stands now, for example that it was fixed again. For a check that could not finish, the timeline says so in a plain sentence and shows no technical error text.

A regression reopens the finding, so it is listed again among your shop’s open findings in the customer portal. It can then be fixed in the same steps as the project: a fix delivered as a pull request, an unpublished copy of your Shopify theme or a patch bundle, followed by a verification scan. Fixing what a re-scan finds is not part of regression protection.

It covers a weekly re-scan against the verified baseline, an email when something regresses, and the timeline in the portal. It is billed monthly and can be cancelled to the end of any month.

Regression protection does not repeat the manual checks of the project, and it changes nothing in your shop’s code. A re-scan is automated: it finds what automated checks can find on the pages it reaches.