Robots.txt generator
Create simple crawler rules for your website.
Robots.txt generator
Näin käytät työkalua Robots.txt generator
The robots.txt generator writes a crawler rules file for all user agents using one of three policies: allow all, disallow all, or disallow a custom list of paths entered one per line. You can add a Sitemap line with an absolute URL. Each custom path must start with a slash, and the result downloads as a plain text file.
- Paste the source into the input editor, or load the built-in example.
- Set sitemap url, crawl policy.
- Run robots.txt generator, review the output, then use the available copy or download controls.
Työkalun toiminnot
Create simple crawler rules for your website.
- Sitemap URL
- Set this value for your task.
- Crawl policy
- Allow all · Disallow all · Custom disallow paths
Rajoitukset ja käsittely
Text operations generally accept up to 2,000,000 characters. Individual parsers, generators and formulas apply additional limits shown by their controls.
Esimerkkisyöte
/private/ /search/
Usein kysytyt kysymykset
How do I block a folder in robots.txt?
Choose Custom disallow paths and enter each folder on its own line starting with a slash, such as /private/. Each line becomes a Disallow rule under User-agent: *. Paths that do not begin with a slash are rejected.
Does robots.txt keep pages private or remove them from Google?
No. robots.txt controls crawling, not confidentiality or guaranteed deindexing. Anyone can still open a disallowed URL, so protect private content with access controls such as passwords or authentication.
Can I write rules for a specific crawler like Googlebot?
Not with this generator. It always writes a single User-agent: * group that applies to every crawler. To target a specific bot, edit the downloaded file and add a separate User-agent group for it.
Where is my input processed?
This operation processes its source in your browser. Copy and download are explicit actions; source input is not saved in an account or history.
What are the input limits?
Text operations generally accept up to 2,000,000 characters. Individual parsers, generators and formulas apply additional limits shown by their controls.