Question 1

What is a robots.txt file?

Accepted Answer

A robots.txt file is a plain text file placed at the root of your website (e.g., https://example.com/robots.txt) that instructs web crawlers which pages or sections of your site they are allowed or not allowed to access.

Question 2

Does robots.txt actually stop bots?

Accepted Answer

Well-behaved bots like Googlebot respect robots.txt rules. However, malicious bots may ignore the file entirely. For sensitive content, you should use proper authentication rather than relying solely on robots.txt.

Question 3

What does the 'Block AI Bots' preset do?

Accepted Answer

The Block AI Bots preset adds Disallow: / rules for known AI training crawlers including GPTBot (OpenAI), CCBot (Common Crawl), anthropic-ai, and Applebot-Extended. This prevents these bots from crawling your content for AI training datasets.

Question 4

Should I include my sitemap in robots.txt?

Accepted Answer

Yes, it is a recommended SEO practice to include your sitemap URL in robots.txt. This helps search engine crawlers discover and index your pages more efficiently.

Question 5

What is the difference between Allow and Disallow?

Accepted Answer

Disallow tells a bot not to access the specified path. Allow explicitly permits access to a path, which is useful when you have a broad Disallow rule but want to allow specific sub-paths. Allow takes precedence over Disallow when both match the same path.

Robots.txt Generator

Preview

Frequently Asked Questions