Robots.txt Generator
Build a valid robots.txt file to control search engine crawlers and point to your sitemap.
About this tool
robots.txt tells search crawlers which paths they may request. It does not block indexing by itself — use noindex meta tags for that — but it saves crawl budget on admin and API routes.
How to use the Robots.txt Generator
- 1
Enter your website URL.
- 2
List paths to disallow (e.g. /admin/, /api/).
- 3
Copy the generated robots.txt and upload to your site root.
Common disallow paths
Block /admin/, /api/, staging paths, and internal search result pages. Do not block CSS or JS — Google needs them to render pages.
Always include your sitemap URL at the bottom of robots.txt.
Pro tips
- Place robots.txt at your domain root.
- Test in Google Search Console robots.txt tester after publishing.
Social Media Specs & Character Limits
Optimal text formatting, length guidelines, and parameters for all major platforms:
| Platform | Max Length | Ideal Length | Hashtag Policy |
|---|---|---|---|
| Instagram Feed & Reels | 2,200 chars | 125 - 150 chars | 3 - 5 tags (Recomm.) |
| TikTok Posts | 4,000 chars | 100 - 200 chars | 3 - 5 tags (Ideal) |
| YouTube Description | 5,000 chars | 1,500 chars (SEO) | 3 - 10 tags |
| X / Twitter Post | 280 chars | 100 - 140 chars | 1 - 2 tags |
| LinkedIn Articles & Posts | 3,000 chars | 500 - 1,000 chars | 3 - 5 tags |
Frequently Asked Questions
Where does robots.txt go?▼
At your domain root: https://yoursite.com/robots.txt
Can I block Google entirely?▼
Technically yes with Disallow: /, but that removes you from search. Use noindex on specific pages instead.