Crawl Control Cluster
Build the crawl-layer files before you validate indexation
This cluster is for deterministic file generation, not remote inspection. Start here when you need to publish a robots.txt file or produce a sitemap from a known canonical URL inventory.
Robots.txt Generator
Generate a clean robots.txt file for launch surfaces, docs hubs, and product shells without hand-formatting directive blocks.
XML Sitemap Generator
Turn a known list of canonical URLs into copy-ready sitemap XML without crawling, fetching, or remote validation.
Robots.txt Tester
Validate whether one crawler can fetch one URL path from pasted rules or a bounded live fetch of the published robots file.
Guide
Test robots.txt safely before important URLs are blocked
Use the supporting guide to test one URL path against one crawler in draft mode, then recheck the live root file after publish.
Next Up
Validation follows generation
After you publish the generated files, move into the indexation workflow to inspect live search signals. The dedicated sitemap validation surface now exists separately so the generator stays fast and focused in v1.