ValidiscanBot: what it requests and how to block it

1. The home page. The first request is a plain HTTP request for the home page, made outside the browser. It shows whether the shop answers and which platform it runs on. Redirects are followed one hop at a time, at most 10. This request and its redirects come before robots.txt is read, so they are made even when robots.txt disallows everything. If the home page answers with an error status, such as 403, 429 or 503, or does not answer at all, the free check ends there.
2. robots.txt. Next, ValidiscanBot reads robots.txt at the root of the host the home page ended on, for example https://www.example.com/robots.txt. If robots.txt disallows the home page for ValidiscanBot, or cannot be read because the server answers with a server error (5xx), with 429 (too many requests) or not at all, the free check stops and requests nothing else. A missing robots.txt (any other 4xx status) or an empty one allows everything. For a shop that lives below a path, such as /de/, and has no robots.txt or an empty one at the root of its host, it also reads the one in that path.
3. Sitemaps. It reads the shop’s sitemap.xml, the sitemaps robots.txt names, and the sitemaps those list: at most 50 sitemap files. A sitemap file that robots.txt disallows is skipped, unless robots.txt names it.
4. Finding pages. It requests candidate pages as plain HTML, to sort them by type (product lists, product pages, content pages, legal pages and others) and to find links to more pages. The addresses come from the sitemaps and from links on the pages it has read, and only addresses on the shop’s own domain are considered. It skips every address robots.txt disallows and makes at most 45 of these requests, counting the home page request of step 1; redirect hops come on top.
5. Search and login. To find a search results page, it requests the platform’s usual search address with words taken from page titles, then with common words, until one answers. The login page comes from a link on the home page or, without one, from the platform’s usual login addresses. Contact and legal pages come from the home page’s links, with no extra request. If it has found fewer than 15 pages by then, it goes back to finding pages, within the same 45 requests.
6. The pages in the browser. Last, it loads the chosen pages in its browser, at most 15, one at a time, each first at desktop size and then at phone size.

The limit of 15 pages counts the pages ValidiscanBot tests, not its requests. Besides the two loads of each page in the browser, a free check makes the plain requests of steps 1 to 5. Each redirect hop is a request of its own. A page load that fails is tried up to two more times. Some tests load a page again; the cookie banner test, for example, opens it in a fresh browser window without cookies. If a free check breaks off, for example because its browser crashed or the scanning service was stopped, it is started again from the beginning. How many requests this makes in total depends on the shop.

Each request ValidiscanBot starts itself to your shop waits until at least 1 s after the previous one started, and after that one was answered, whichever is later. The rule covers the plain requests, every redirect hop they follow, and every page load in the browser, the fresh window of the cookie banner test included. It is kept per origin: scheme, host name and port. For a page load in the browser, “answered” means that the page’s response has arrived, not that the page has finished loading.

After robots.txt has been read, a Crawl-delay longer than 1 s widens the gap, up to 10 s. A shorter one does not narrow it.

The files a page loads itself, such as styles, scripts, images, fonts, data requests and frames, are part of loading that page. They load as in any browser, without the gap.

One exception is known. When your shop answers a page load in the browser with a redirect, the browser follows it at once, without the gap and without checking robots.txt. Validiscan has recorded this as an open gap in its scanner.

• It never adds a product to a cart and never opens the cart or the checkout. Cart and checkout pages are not part of a free check.
• It sends no forms and does not log in. The login page is loaded as a page to test, nothing more.
• It uses no password and does not scan a staging copy of the shop.

ValidiscanBot operates each page as a visitor would, inside its own browser. It moves through the page with the keyboard. It tests the cookie banner and then accepts it, so the rest of the page can be tested; to try the banner’s reject or close control and the Escape key, it uses a fresh window without cookies. Where a page has a product variant control, it switches to another variant and then back. Where a page has a password field, it checks whether pasting into the field is blocked, with a test paste that leaves the field as it was. Your pages’ own scripts may react as they would for any visitor, for example by recording the cookie choice.

Its browser sends no requests to a list of known analytics, advertising and chat services, and it loads no video or audio files. Consent services are not on that list, because the cookie banner is part of what is tested.

A group for ValidiscanBot replaces the * group: ValidiscanBot then follows only the Disallow and Allow lines of its own group. If you add a group for ValidiscanBot, repeat in it the * rules you want it to keep. With the lines below, ValidiscanBot stays out of /account/ but may load pages under /drafts/:

User-agent: *
Disallow: /drafts/

User-agent: ValidiscanBot
Disallow: /account/

In a free check, robots.txt is checked for sitemap files other than those robots.txt names, for every address taken from sitemaps and links, for the search and login requests, for every redirect hop of the plain requests, and for the pages and frames the browser loads from your shop’s host. A page whose plain request was redirected to a disallowed address is dropped.

It is not checked for:

• the home page request and its redirects, which come before robots.txt is read;
• the files an allowed page loads itself, even from a path robots.txt disallows;
• redirects the browser follows during a page load (the open gap described above);
• pages outside your shop’s origin. ValidiscanBot reads only the robots.txt of the scheme, host name and port your home page ended on. A page on another host name of the same domain that your shop links to, such as blog.example.com next to www.example.com, can be included without that host’s robots.txt being read.