Crawl Control Cluster
Build the crawl-layer files before you validate indexation
This cluster is for deterministic file generation, not remote inspection. Start here when you need to publish a robots.txt file or produce a sitemap from a known canonical URL inventory.
Robots.txt Generator
Generate a clean robots.txt file for launch surfaces, docs hubs, and product shells without hand-formatting directive blocks.
XML Sitemap Generator
Turn a known list of canonical URLs into copy-ready sitemap XML without crawling, fetching, or remote validation.
Robots.txt Tester
Validate whether one crawler can fetch one URL path from pasted rules or a bounded live fetch of the published robots file.
Next Up
Validation follows generation
After you publish the generated files, move into the indexation workflow to inspect live search signals. The dedicated sitemap validation surface now exists separately so the generator stays fast and focused in v1.