Create custom robots.txt files to control search engine crawlers.
Rule Builder
Advanced Editor
Add Rules
Add paths to allow or disallow. Use / to block/allow entire site, or specific paths like /admin/
Time delay between successive crawler requests
Link to your XML sitemap
Edit the robots.txt file directly with full control over syntax
Generated robots.txt:
Found an issue with this tool?
If any part of this tool is not working properly, please let us know.
🤖 Robots.txt Generator
A powerful and user-friendly tool to create and manage your robots.txt file. Control how search engine crawlers interact with your website with ease.
✨ Key Features
🎯 Rule Builder
Intuitive interface to add allow/disallow rules without writing code. Simply select paths and rule types.
✏️ Advanced Editor
Direct text editing mode for experienced users who want full control over the robots.txt syntax.
🤖 User Agent Control
Target specific bots (Googlebot, Bingbot, etc.) or create custom user agent rules.
⚡ Quick Presets
One-click templates: Allow All, Block All, Standard Setup, and Strict Protection.
⏱️ Crawl Delay
Set time delays between crawler requests to manage server load.
🗺️ Sitemap Integration
Easily add your sitemap URL to help search engines discover your content.
💾 Export Options
Copy to clipboard or download as robots.txt file ready for upload.
📝 Comment Toggle
Option to include helpful comments in your generated file.
📖 How to Use
Step 1: Choose Your Mode
Select between Rule Builder (beginner-friendly) or Advanced Editor (for direct editing).
Step 2: Select User Agent
Choose which bots your rules apply to:
* - Applies to all bots (most common)
Googlebot - Google's crawler
Bingbot - Microsoft Bing's crawler
Or select any other specific bot from the dropdown
Choose "Custom" to target a specific bot not in the list
Step 3: Add Rules
Define what crawlers can and cannot access:
Disallow: Block access to specific paths
Allow: Explicitly allow access (useful for exceptions)
Common Examples:
/admin/ - Block admin area
/private/ - Block private content
/ - Block or allow entire site
/images/ - Block image directory
Step 4: Configure Options
Crawl Delay: Set seconds between requests (e.g., 10)
Sitemap URL: Add your sitemap location
Include Comments: Add helpful comments to your file
Allow All Paths: Override all rules and allow everything
Step 5: Generate & Export
Click "Generate robots.txt" to create your file. Then:
Click "Copy" to copy to clipboard
Click "Download" to save as robots.txt file
Upload the file to your website's root directory
🎯 Quick Start Presets
Preset
Description
Best For
Allow All Bots
Allows all crawlers to access everything
Public websites wanting maximum visibility
Block All Bots
Blocks all crawlers from entire site
Private sites, development servers
Standard Setup
Blocks admin/private areas, allows rest
Most websites (recommended)
Strict Protection
Blocks admin, system files, with crawl delay
Security-conscious sites
💡 Understanding Robots.txt
What is robots.txt?
The robots.txt file is a text file placed in your website's root directory that tells search engine crawlers which pages or sections of your site they can or cannot access.
Security Warning: robots.txt is NOT a security mechanism. It's a polite request that well-behaved crawlers follow, but malicious bots can ignore it. Never rely on it to hide sensitive information.
Pro Tip: Always test your robots.txt file using Google Search Console's robots.txt Tester tool to ensure it works as intended.
File Location: The robots.txt file must be placed in your website's root directory (e.g., https://example.com/robots.txt), not in subdirectories.
Best Practice: Start with a permissive robots.txt and only block what's necessary. Over-blocking can hurt your search engine visibility.
🚀 Getting Started
Ready to create your robots.txt file? Use the quick preset buttons for instant configurations, or build custom rules step-by-step. The tool provides real-time preview so you can see exactly what will be generated.
Need help? Start with the "Standard Setup" preset and customize from there!