SEO UTILITIES
Robots.txt Generator — Free & Valid robots.txt
Build a valid robots.txt with allow and disallow rules plus a sitemap line. Runs in your browser and outputs clean syntax to upload to your site root.
Vos fichiers ne quittent jamais votre appareil.
A robots.txt file sits at the root of your domain and tells crawlers which paths they may or may not request. This generator assembles those rules — user-agent groups, allow and disallow paths, crawl-delay, and a sitemap reference — into correctly formatted text following the robots exclusion standard.
It runs in your browser and helps you dodge the classic footgun: a stray Disallow: / that hides your entire site from search.
Crawling is not indexing
This is the point people miss most. Disallow blocks a crawler from fetching a path — but a disallowed URL can still appear in search results if other pages link to it, because blocking the crawl doesn't remove the URL from the index. To keep a page out of results, let it be crawled and add a noindex robots meta tag instead.
And robots.txt is a request, not a lock. Well-behaved search engines honour it; malicious scrapers ignore it entirely. Never use it to hide sensitive content — use real access controls for that.
A sample robots.txt
A common setup — allow everything except an admin area, and point crawlers to the sitemap:
User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /
Sitemap: https://example.com/sitemap.xml
How to use it
- Pick which crawlers the rules apply to and add your allow and disallow paths.
- Add your sitemap URL and any crawl-delay, then generate the file.
- Save the output as robots.txt and upload it to your site's root directory.
Frequently asked questions
- Where does the file go?
- At the root of your domain, reachable at https://yourdomain.com/robots.txt. Crawlers only look there; a robots.txt in a subfolder is ignored.
- What's the danger of a wrong rule?
- A single Disallow: / under User-agent: * tells every crawler to stay off your whole site, which can remove you from search. Double-check disallow paths and test the file in a search console before relying on it.