Robots.txt Generator

What this tool does

Calculate robots.txt using your entered campaign figures. robots.txt is a voluntary convention for well-behaved crawlers, not access control — never list sensitive paths you have not protected another way, and test the file after publishing because a single wrong Disallow can block a whole site.

How to use the Robots.txt Generator

1. Enter or select rules: user-agent | disallow path | allow path (one rule per line; repeat a user-agent for more paths).
2. Enter or select sitemap url (optional).
3. Read the calculated result; change any measurement to compare alternatives.

Formula

group the entered rules by user-agent in entry order and emit a User-agent block per agent with its Disallow and Allow lines, then append the sitemap line when a sitemap URL is entered

rules

Rules: user-agent | disallow path | allow path (one rule per line; repeat a user-agent for more paths)

sitemap

Sitemap URL (optional)

robots.txt is a voluntary convention for well-behaved crawlers, not access control — never list sensitive paths you have not protected another way, and test the file after publishing because a single wrong Disallow can block a whole site.

Worked example

For robots.txt generator, the following measurements illustrate the exact method: Rules: user-agent | disallow path | allow path (one rule per line; repeat a user-agent for more paths): * | /admin/ | * | /search/ | Googlebot | | /private-offers/; Sitemap URL (optional): https://example.com/sitemap.xml.

Results explained

robots.txt

robots.txt from the formula above. robots.txt is a voluntary convention for well-behaved crawlers, not access control — never list sensitive paths you have not protected another way, and test the file after publishing because a single wrong Disallow can block a whole site.

User-agent groups

User-agent groups from the formula above. robots.txt is a voluntary convention for well-behaved crawlers, not access control — never list sensitive paths you have not protected another way, and test the file after publishing because a single wrong Disallow can block a whole site.

Frequently asked questions