About our crawler
If you found our crawler in your server logs, this page explains who we are, why we visited, and how to limit or block us.
Who we are
The crawler belongs to ThinkTrail Solutions Inc. It powers SiteGrade, which measures the technical quality of websites: security, accessibility, search engine basics, and web best practices. Its requests identify themselves with this user-agent string:
ThinkTrailAudit/1.0 (+https://www.thinktrailsolutions.com/sitegrade/crawler/)
Why we visited
A visit usually means your site is part of an assessment commissioned by an organization you work with (commonly a vendor whose products you market or resell), or that you asked us to assess it. Our crawler reads public pages only. It doesn't log in, submit forms, or attempt to bypass any access control.
How it behaves
- Reads robots.txt first. It doesn't read pages that your robots.txt disallows for our crawler or for all crawlers, and it honours any Crawl-delay you set.
- Paces itself. By default it makes one request per second, or slower if your Crawl-delay asks for it.
- Works within strict limits. By default it reads at most 100 pages, no more than three links deep from your homepage.
- Checks links lightly. To find broken links, it sends a small HEAD request to each linked address (up to 500 per assessment, paced per host) to confirm it responds. It doesn’t read those pages.
- Measures, never stresses. It notes response times as it goes. It never load-tests your site.
- Loads a few pages in a browser. Up to five pages are also loaded in a headless Chrome browser using Google Lighthouse to measure page quality. Those requests carry Lighthouse's standard browser identifier, which contains “Chrome-Lighthouse”.
Before every assessment, the crawler requests your homepage once to confirm the site is reachable. We ignore robots.txt only when the site's owner has explicitly authorized the assessment.
How to limit or block it
Add a rule for our crawler to your robots.txt. To keep it out of your site entirely:
User-agent: ThinkTrailAudit Disallow: /
To slow it down instead, add a Crawl-delay in seconds:
User-agent: ThinkTrailAudit Crawl-delay: 5
Questions or concerns
If our crawler caused a problem, or you'd like your site excluded from future assessments, email support@thinktrailsolutions.com with your domain. A person reads every message, and we respond within two business days.