SupplementscannerBot
SupplementscannerBot is the crawler behind Supplementscanner. It reads the public product data of the supplement stores we compare that run on Shopify, so that every product can be shown at its price per unit with a link to the store.
How it identifies itself
It identifies itself with this user agent:
Mozilla/5.0 (compatible; SupplementscannerBot/1.0; +https://supplementscanner.eu/bot)What it reads
Only what a store publishes for everyone: its robots.txt, its product sitemap, the public catalogue listing, and each product's public product data and page (name, price, sizes, stock, label).
It never logs in, never adds anything to a cart, never opens a checkout and never reads customer or account data.
How often
A product is checked again only when it is due: several times a day when its price or stock changes often, about once a day or less when it does not. The catalogue listing is read a few times a day, page by page, and sent again only if it changed since the last read.
The rules it follows
- It reads the robots.txt of each store, and of every other host that store's products live on, once a day. It follows the group that names SupplementscannerBot or, when none does, the group for every crawler, and asks for nothing that group disallows: not a product, not a listing, not a sitemap. If a store stops letting it read the file, it keeps following the copy it read last.
- A Crawl-delay slows it down for that store: it rechecks fewer products at a time, so that the product checks average no more than one request per that many seconds over five minutes, and the catalogue listing waits that long between pages.
- A product page that asks not to be indexed, with a robots meta tag or an X-Robots-Tag header saying noindex for every crawler or for SupplementscannerBot, is not listed. On a store whose product data comes from an API, the page itself is read for this before a product is first listed and about once a month after.
- When a store answers that we are asking too much (a 429 or 430), it pauses that store for the time the Retry-After header names, or for a default pause when there is none.
- A store that keeps refusing it is set aside and only tried again much later.
To keep it out of all or part of your store, name it in your robots.txt:
User-agent: SupplementscannerBot
Disallow: /How it proves who it is
Every request carries a Web Bot Auth signature: an Ed25519 HTTP message signature (RFC 9421) over the host it is sent to and its Signature-Agent header, which points at our published key directory. A request that names SupplementscannerBot without a valid signature did not come from us.
- Signature-Agent
"https://supplementscanner.eu"
Contact
Questions about our crawler, or a request to read your store less or not at all: write to us through the details on our legal notice.