Security & devFreeNo signup
Free robots.txt generator. Allow or block crawlers, add disallow rules, and include your sitemap URL. Copy-ready output.
If empty, rules apply to User-agent: *
Search engines look for /robots.txt to learn which paths they may crawl. Combine with sitemaps and on-page robots meta tags for clearer control.
At the site root: https://example.com/robots.txt
No. It’s a crawl hint. Use noindex for stronger exclusion from search results.
Only for private/staging sites. Public content sites usually allow crawling and point to a sitemap.
No. Some bots ignore it. Googlebot does not use Crawl-delay.