Indexing & Crawls

Robots.txt & Sitemap Builder

Build optimized search engine crawling instructions and index XML sitemaps to maximize search indexing.

Presets

Block AI Training Crawlers (GPTBot, CCBot)Allow major Search Engines (Google, Bing, Yandex)

Disallowed Directories

Sitemap URL