Robots.txt tester
Test supplied crawl rules against a path and crawler.
Robots.txt tester
Como usar Robots.txt tester
The robots.txt tester checks pasted robots.txt rules against a URL path and crawler user agent, then reports whether crawling is allowed and which rule matched. It picks the most specific matching user-agent group, supports * wildcards and $ end anchors, and applies the longest matching path, with Allow winning ties. It does not fetch the live file.
- Paste the source into the input editor, or load the built-in example.
- Set url path to test, crawler user agent.
- Run robots.txt tester, review the output, then use the available copy or download controls.
Que admite esta ferramenta
Test supplied crawl rules against a path and crawler.
- URL path to test
- Set this value for your task.
- Crawler user agent
- Set this value for your task.
Límites e procesamento
Text operations generally accept up to 2,000,000 characters. Individual parsers, generators and formulas apply additional limits shown by their controls.
Fonte de exemplo
User-agent: * Disallow: /private/ Allow: /private/public/
Preguntas frecuentes
How does the robots.txt tester decide between Allow and Disallow?
Within the chosen user-agent group, the longest matching path wins, with asterisks ignored when measuring length. If an Allow and a Disallow rule match with equal length, Allow takes precedence. With no matching rule, the path is allowed.
Can it test my live robots.txt file?
No. It only tests the text you paste and does not download anything from your site. Copy your current robots.txt file into the input, then enter the path and user agent you want to check.
Does an allowed result mean my page will be indexed?
No. The result only says whether the supplied rules permit crawling that path. It does not determine whether a URL is indexed, and robots.txt is not an access-control mechanism for private pages.
Where is my input processed?
This operation processes its source in your browser. Copy and download are explicit actions; source input is not saved in an account or history.
What are the input limits?
Text operations generally accept up to 2,000,000 characters. Individual parsers, generators and formulas apply additional limits shown by their controls.