Robots.txt Generator

Build a robots.txt file that allows crawlers on public pages and blocks API or admin paths you do not want crawled.

robots.txt
User-agent: * Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml

What this tool does

robots.txt tells search crawlers which paths they may request. It does not block indexing by itself — use noindex meta tags for that — but it saves crawl budget on admin and API routes.

How to use it

  1. 1

    Enter your website URL.

  2. 2

    List paths to disallow (e.g. /admin/, /api/).

  3. 3

    Copy the generated robots.txt and upload to your site root.

Common disallow paths

Block /admin/, /api/, staging paths, and internal search result pages. Do not block CSS or JS — Google needs them to render pages.

Always include your sitemap URL at the bottom of robots.txt.

Tips

  • Place robots.txt at your domain root.
  • Test in Google Search Console robots.txt tester after publishing.

Related tools

All tools →

Meta Tag Generator

Generate SEO title, description, and Open Graph tags

Keyword Density Checker

Analyze keyword frequency and avoid stuffing

URL Slug Generator

Create SEO-friendly URL slugs from any headline

Schema Markup Generator

Create JSON-LD for 20+ schema types — FAQ, Product, HowTo & more

Schema Validator

Validate JSON-LD schema markup and check required fields

Sitemap Generator

Generate sitemap.xml from a list of URLs

Related guides

Frequently asked questions

Where does robots.txt go?▼

At your domain root: https://yoursite.com/robots.txt

Can I block Google entirely?▼

Technically yes with Disallow: /, but that removes you from search. Use noindex on specific pages instead.