Robots.txt generator
Create simple crawler rules for your website.
Favorites are saved in this browser. Find them in My favorites.
Robots.txt generator
Your input stays in this browser unless stated otherwise
Clear your input, files and result, and restore the default settings.
0 charactersClear the source text.
Како користити Robots.txt generator
- Paste the source into the input editor, or load the built-in example.
- Set sitemap url, crawl policy.
- Run robots.txt generator, review the output, then use the available copy or download controls.
Могућности алата
Create simple crawler rules for your website.
- Sitemap URL
- Set this value for your task.
- Crawl policy
- Allow all · Disallow all · Custom disallow paths
Ограничења и обрада
Text operations generally accept up to 2,000,000 characters. Individual parsers, generators and formulas apply additional limits shown by their controls.
Пример уноса
/private/ /search/
Честа питања
What does robots.txt generator do?
Create simple crawler rules for your website.
Where is my input processed?
This operation processes its source in your browser. Copy and download are explicit actions; source input is not saved in an account or history.
What are the input limits?
Text operations generally accept up to 2,000,000 characters. Individual parsers, generators and formulas apply additional limits shown by their controls.