# Robots.txt generator

> Create simple crawler rules for your website.

[Open tool](https://www.toolcabana.com/br/robots-generator) · [Generators and marketing](https://www.toolcabana.com/br/category/generators-and-marketing)

Tool ID: robots-generator. Requested language: br. Description language: en. Complete guide translation: no; untranslated sections use English.

## Overview (en)

The robots.txt generator writes a crawler rules file for all user agents using one of three policies: allow all, disallow all, or disallow a custom list of paths entered one per line. You can add a Sitemap line with an absolute URL. Each custom path must start with a slash, and the result downloads as a plain text file.

## Supported tasks (en)

Create simple crawler rules for your website.

## Steps (en)

1. Paste the source into the input editor, or load the built-in example.
2. Set sitemap url, crawl policy.
3. Run robots.txt generator, review the output, then use the available copy or download controls.

## Settings

- Sitemap URL (en; key: sitemap; type: text)
- Crawl policy (en; key: policy; type: select): Allow all [en], Disallow all [en], Custom disallow paths [en]

## Limitations (en)

Text operations generally accept up to 2,000,000 characters. Individual parsers, generators and formulas apply additional limits shown by their controls.

## Example input

```text
/private/
/search/
```

## Privacy and connections (en)

This operation processes its source in your browser. Copy and download are explicit actions; source input is not saved in an account or history.

## Questions (en)

### How do I block a folder in robots.txt?

Choose Custom disallow paths and enter each folder on its own line starting with a slash, such as /private/. Each line becomes a Disallow rule under User-agent: *. Paths that do not begin with a slash are rejected.

### Does robots.txt keep pages private or remove them from Google?

No. robots.txt controls crawling, not confidentiality or guaranteed deindexing. Anyone can still open a disallowed URL, so protect private content with access controls such as passwords or authentication.

### Can I write rules for a specific crawler like Googlebot?

Not with this generator. It always writes a single User-agent: * group that applies to every crawler. To target a specific bot, edit the downloaded file and add a separate User-agent group for it.

### Where is my input processed?

This operation processes its source in your browser. Copy and download are explicit actions; source input is not saved in an account or history.

### What are the input limits?

Text operations generally accept up to 2,000,000 characters. Individual parsers, generators and formulas apply additional limits shown by their controls.

## Related tools

- [Sitemap generator](https://www.toolcabana.com/br/sitemap-generator)
- [Invoice generator](https://www.toolcabana.com/br/invoice-generator)
- [Email mailto generator](https://www.toolcabana.com/br/mailto-generator)
- [Password generator](https://www.toolcabana.com/br/password-generator)