Robots.txt Generator
Build a robots.txt file that allows crawlers on public pages and blocks API or admin paths you do not want crawled.
Runs in your browser. Nothing you enter is uploaded.
Overview
robots.txt is a plain text file at the root of a domain that tells compliant crawlers which paths they may request. This generator assembles a valid file from a site URL, a list of disallowed paths, and an optional blanket Allow rule, then appends the sitemap reference that belongs at the bottom.
The single most misunderstood thing about this file is what it controls. It governs crawling - whether a bot fetches a URL - not indexing. A page blocked in robots.txt can still appear in search results, usually as a bare URL with no description, if other pages link to it. Blocking a URL you want removed from the index is the classic own-goal, because the crawler can no longer see the noindex tag that would have removed it.
Paths worth disallowing, and one you must not
The genuinely useful disallow rules cover URLs that generate crawl activity without producing anything a searcher wants: administrative interfaces, API endpoints returning JSON, internal search result pages that can spawn infinite parameter combinations, cart and checkout steps, and print-only duplicates. On a small site none of this matters much. On a large catalogue with faceted navigation it is the difference between crawlers reaching your product pages and getting lost in filter permutations.
The rule you must not add is one that blocks CSS, JavaScript, or image directories. Google renders pages to evaluate them, and a page it cannot render is a page it evaluates badly. Older tutorials that recommend blocking /wp-includes/ or /assets/ predate rendering-based indexing and actively harm sites that still follow them.
- /admin/ and any CMS login path - no search value, steady crawl cost.
- /api/ endpoints that return data rather than pages.
- Internal search result URLs, which multiply without limit.
- Staging or preview subdirectories that mirror production content.
- Never: stylesheets, scripts, fonts, or the image directories used by your templates.
How the directives are matched
Paths are matched as prefixes, not as exact strings. Disallow: /admin blocks /admin/, /admin-tools/, and /administration/ alike. Adding the trailing slash - Disallow: /admin/ - narrows it to the directory, which is almost always what you meant. The generator normalises any path you type by prepending a slash if you leave it off, but it will not add the trailing slash for you.
When both an Allow and a Disallow rule could apply to the same URL, compliant crawlers follow the more specific rule, measured by path length. That is why an Allow: / line alongside a set of narrow Disallow rules is safe: the specific blocks win for the paths they cover, and everything else stays open.
The Sitemap directive is independent of user-agent grouping. It can sit anywhere in the file and applies globally, which is why this generator places it on its own line after a blank separator at the end.
What this generator writes, line by line
The output always opens with User-agent: * so the rules apply to every crawler that reads the file. If the Allow all checkbox is ticked, an Allow: / line follows immediately. Each non-empty line from the disallow textarea is trimmed, given a leading slash if it lacks one, and emitted as its own Disallow directive in the order you typed it.
After the rules, a blank line is inserted and a Sitemap directive is appended using your site URL with any trailing slash stripped and /sitemap.xml joined on. If your sitemap lives at a different filename or you publish a sitemap index, edit that line after copying - the generator assumes the conventional location.
No per-crawler blocks, crawl-delay directives, or wildcard patterns are produced. Those are valid robots.txt syntax but support varies between crawlers, and a file that quietly does nothing on the crawler you care about is worse than a simple file you can read at a glance.
A typical small-site file
Input
Site URL: https://example.com/ Disallow paths: /admin/ api/ /search? Allow all other paths: checked
Output
User-agent: * Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /search? Sitemap: https://example.com/sitemap.xml
Note that "api/" was typed without a leading slash and came out normalised as /api/. The /search? entry blocks the internal search endpoint and every query string hanging off it, which is the prefix-matching behaviour working in your favour. The trailing slash on the site URL was dropped so the sitemap line does not end up with a doubled slash.
Limitations
- robots.txt controls crawling only. Blocked URLs can still be indexed from external links, and blocking a page prevents crawlers from ever seeing a noindex directive on it.
- Compliance is voluntary. Major search engines respect the file; scrapers and malicious bots routinely ignore it, so it is not a security or privacy mechanism.
- The generator emits a single User-agent: * group. It does not produce crawler-specific rules, crawl-delay, or wildcard patterns.
- The Sitemap line assumes /sitemap.xml at the root. Sites using a sitemap index or a non-standard filename need to edit that line by hand.
- Nothing here validates that the paths you disallow actually exist, so a typo silently blocks nothing at all.
When not to use this tool
- To keep a page out of search results, use a noindex meta tag or an x-robots-tag header on a crawlable URL - robots.txt is the wrong instrument and will make removal harder.
- To protect private data or gated content, use authentication. A disallow line advertises the path to anyone who reads the file.
How to use it
- 1
Enter the site URL with its protocol so the generated Sitemap directive resolves to the correct host.
- 2
List one disallowed path per line, adding a trailing slash when you mean a directory rather than every path that starts with those characters.
- 3
Leave Allow: / enabled unless you are deliberately building a deny-by-default file, then copy the output to a file named robots.txt at your domain root.
- 4
Fetch https://yourdomain.com/robots.txt in a browser to confirm it serves as plain text, then check it in Search Console for parsing errors.
Questions
I blocked a page in robots.txt but it still shows in Google. Why?
Because blocking crawling does not block indexing. If Google knows the URL exists from a link elsewhere, it can list it without ever fetching it, which is why those results appear with no description. To remove it, allow crawling again and serve a noindex tag, or use the removals tool in Search Console for an interim fix.
Do I need a robots.txt file at all?
A site with no file is treated as fully crawlable, which is fine for small sites with nothing worth excluding. It becomes worthwhile once you have administrative paths, an internal search endpoint, or faceted URLs that can multiply - and the Sitemap directive is a small convenience regardless.
Where exactly does the file go?
At the root of the host it applies to: https://example.com/robots.txt. It does not cascade - a file at example.com does not govern shop.example.com, and a file in a subdirectory is ignored entirely. Each subdomain needs its own.
Should I list every page I do not want crawled?
No. Prefix matching means one directory rule usually covers hundreds of URLs, and a long file of individual page rules is hard to audit and easy to break. If you find yourself listing pages individually, the underlying issue is usually URL structure.