SEO Generator

robots.txt generator

Визуальный конструктор с готовыми шаблонами

Templates

robots.txt

User-agent: *
Allow: /

robots.txt: building a correct file

Basic principles

robots.txt controls site crawling: it tells crawlers where not to go, saving crawl budget for important pages. The generator assembles the file from ready blocks and templates for popular CMSs.

The mandatory minimum: a User-agent: * section, blocked service sections, and a Sitemap directive with the full sitemap address.

Mistakes to avoid

Do not use robots.txt to “remove” pages from the index — that is what noindex is for. Do not block static files (CSS, JS, images): without them the search engine cannot render the page correctly.

After publishing, check the file with our robots.txt checker and in the webmaster panels — Yandex and Google will show how they interpret your rules.

Frequently asked questions

Where should I upload the robots.txt file?+

Strictly to the site root: the file must open at yoursite.com/robots.txt. Crawlers will not find it in subfolders.

What should be blocked in an online store’s robots.txt?+

Usually the cart, checkout, account area, search results, sorting and filter pages with parameters (or handle them via Clean-param for Yandex).

Do I need separate rules for Yandex and Google?+

Most often a common User-agent: * section is enough. A separate Yandex section is needed if you use specific directives — for example, Clean-param, which Google does not support.

Can robots.txt harm a site?+

Yes, on error: an accidental Disallow: / blocks the whole site from crawling, and blocking CSS/JS worsens how search engines rate pages. Always check the generated file before uploading.