Robots.txt tester
Test supplied crawl rules against a path and crawler.
Favorites are saved in this browser. Find them in My favorites.
Robots.txt tester
Lookups send the query to the configured network services
Clear your input, files and result, and restore the default settings.
0 charactersClear the source text.
How to use Robots.txt tester
- Paste the source into the input editor, or load the built-in example.
- Set url path to test, crawler user agent.
- Run robots.txt tester, review the output, then use the available copy or download controls.
What this tool supports
Test supplied crawl rules against a path and crawler.
- URL path to test
- Set this value for your task.
- Crawler user agent
- Set this value for your task.
Limits and processing
Text operations generally accept up to 2,000,000 characters. Individual parsers, generators and formulas apply additional limits shown by their controls.
Example source
User-agent: * Disallow: /private/ Allow: /private/public/
Frequently asked questions
What does robots.txt tester do?
Test supplied crawl rules against a path and crawler.
Where is my input processed?
This operation processes its source in your browser. Copy and download are explicit actions; source input is not saved in an account or history.
What are the input limits?
Text operations generally accept up to 2,000,000 characters. Individual parsers, generators and formulas apply additional limits shown by their controls.