Robots.txt generator
Create simple crawler rules for your website.
Favorites are saved in this browser. Find them in My favorites.
Robots.txt generator
Your input stays in this browser unless stated otherwise
Clear your input, files and result, and restore the default settings.
0 charactersClear the source text.
Sida loo isticmaalo Robots.txt generator
- Paste the source into the input editor, or load the built-in example.
- Set sitemap url, crawl policy.
- Run robots.txt generator, review the output, then use the available copy or download controls.
Waxa qalabkani taageero
Create simple crawler rules for your website.
- Sitemap URL
- Set this value for your task.
- Crawl policy
- Allow all · Disallow all · Custom disallow paths
Xadka iyo habaynta
Text operations generally accept up to 2,000,000 characters. Individual parsers, generators and formulas apply additional limits shown by their controls.
Isha tusaalaha
/private/ /search/
Su’aalaha badanaa la isweydiiyo
What does robots.txt generator do?
Create simple crawler rules for your website.
Where is my input processed?
This operation processes its source in your browser. Copy and download are explicit actions; source input is not saved in an account or history.
What are the input limits?
Text operations generally accept up to 2,000,000 characters. Individual parsers, generators and formulas apply additional limits shown by their controls.