What is robots.txt?
It gives crawlers guidance for your site.
Technical SEO tool
Free Robots.txt Generator
Create crawler rules
Build robots.txt
Production site
Start with Production for a live site, or choose a safer staging/private preset.
Use * to apply the rules to all crawlers, or enter one specific bot name.
Add paths that crawlers may access, one per line, starting with /.
Add paths to block from crawling, one per line, starting with /.
Leave blank unless a crawler explicitly supports a delay; use whole seconds.
Add the full public URL of each sitemap, one per line.
Generated output
User-agent: *Frequently asked questions
Is it free?
Yes.
Can I download it?
Yes