SEO Tools

Robots.txt Generator

Generate a robots.txt file to control search engine crawler access.

Default Access Policy

Crawl Delay (seconds)

Block AI Scrapers & Bots Disallows GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, CommonCrawler, anthropic-ai, cohere-ai.

Block Aggressive SEO Crawlers Disallows SemrushBot, AhrefsBot, DotBot, MJ12bot, Rogerbot to preserve server bandwidth.

Generated robots.txt

Loading robots.txt...

Frequently Asked Questions

What is robots.txt?

robots.txt is a text file placed at the root of a website that instructs search engine crawlers which pages they should or should not access.

Explore our full collection of free, privacy-first developer and SEO tools.