Robots.txt Generator
Generate a robots.txt file to control how search engines crawl your site.
Choose a preset or customize your own rules.
Helps search engines find all your pages
Delay between crawler requests (some bots ignore this)
Control search engine crawlers
Choose a preset or customize your rules to tell search engines which pages they can and cannot access.
User-agent: *
Disallow: /admin/
Allow: /
Sitemap: /sitemap.xml What is robots.txt?
Robots.txt is a text file that tells search engine crawlers which URLs they can access on your site.
Control access
Block crawlers from accessing private pages, admin areas, or duplicate content.
Optimize crawl budget
Help search engines focus on your important pages by blocking low-value URLs.
Point to sitemap
Include your sitemap location to help crawlers discover all your pages.
Common robots.txt rules
Allow all crawlers
User-agent: *
Allow: /Let all bots access all pages
Block all crawlers
User-agent: *
Disallow: /Prevent all bots from crawling (use carefully!)
Block a specific folder
User-agent: *
Disallow: /admin/
Disallow: /private/Block specific directories from all bots
Block AI crawlers
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: CCBot
Disallow: /Prevent AI training bots from accessing your content
Robots.txt best practices
It's not for security
Robots.txt is a guideline, not a security measure. Malicious bots can ignore it. Don't use it to hide sensitive data.
Always include a sitemap
Adding your sitemap URL helps search engines discover pages that might not be easily found through links.
Test before deploying
Use Google Search Console's robots.txt Tester to verify your rules work as expected before going live.
Be specific with paths
Use trailing slashes for directories (/admin/) and be precise. Wildcards (*) and end-of-URL ($) can help with patterns.
Launching a new website?
Create a waitlist to collect interested visitors while you build. Launch with an audience ready to convert.
I can only say good things about Waitlister. Their landing page is very user friendly, and Waitlister's pricing is more than reasonable.
See our other tools
24 free tools available
Business Name Generator
Generate creative, memorable business names based on your keywords and industry. AI-powered suggestions in seconds.
AI Logo Generator
Generate unique logo concepts for your brand with AI. Get 4 variations in multiple styles instantly.
Domain Name Generator
Generate brandable, memorable domain names for your business. AI-powered suggestions optimized for availability.
Slogan & Tagline Generator
Create catchy slogans and taglines that capture your brand essence. Multiple tones and styles available.
Landing Page Copy Generator
Generate high-converting copy for every section of your landing page—headlines, benefits, FAQs, and CTAs.
Product Description Generator
Generate compelling product descriptions that sell. Get 3 styles, bullet points, and SEO metadata.