One misplaced slash in a hand-written robots.txt can quietly de-index an entire site, so most people are better off generating it than typing it from memory.
Free hosting tool
Robots.txt Generator
Build Allow/Disallow rules per user-agent, add your sitemap, and get a ready-to-upload robots.txt. Nothing you enter is sent to a server.
Not a security tool. Robots.txt only asks well-behaved crawlers to skip a path, it doesn't password-protect anything and some bots ignore it entirely. Never rely on it to hide sensitive content.
How to use the file this generates
Copy the text above, or download it directly, then upload it as robots.txt to the root of your domain so it's reachable at yoursite.com/robots.txt. Most WordPress hosts and SEO plugins like Yoast or RankMath can also serve a virtual robots.txt without a manual upload, check your plugin's settings first if you're not sure which method your site already uses.
Robots.txt generator FAQ
What does a robots.txt file actually do?
It tells well-behaved search engine crawlers which parts of your site they should or shouldn't crawl. It is a request, not a security measure: it doesn't password-protect anything, and a crawler that ignores the rules (as some do) can still access those paths. Use it to manage crawl behaviour, not to hide sensitive content.
Where do I upload robots.txt?
In the root of your domain, so it's reachable at yoursite.com/robots.txt exactly. Most WordPress hosts and several SEO plugins (Yoast, RankMath) can also generate and serve it automatically without a manual file upload, check your plugin's settings before adding one by hand.
What is the WordPress default preset for?
A commonly used starting point that disallows crawling of /wp-admin/ (your dashboard, which has no reason to be indexed) while explicitly allowing /wp-admin/admin-ajax.php (used by many plugins and themes for legitimate front-end functionality that needs to keep working for logged-out visitors too).
Should I block my whole site while it's in development?
A "Disallow all" robots.txt keeps well-behaved crawlers out, but it is not a substitute for password-protecting a staging site, since the file itself is public and some bots ignore it anyway. Combine it with actual access restrictions (like HTTP basic auth) for a site you genuinely don't want found.
Do I need a crawl-delay rule?
Rarely for a typical site. It tells a crawler to wait a number of seconds between requests, which matters mostly for very large sites worried about server load from aggressive bots. Most major crawlers, including Googlebot, ignore this directive anyway and use their own crawl-rate logic instead.
Is this generator private?
Yes. It runs entirely in your browser. Nothing you enter is sent to a server.
Free robots.txt generator by ANUPRESS
The one mistake that actually hurts your SEO
A stray Disallow: / left over from a staging site, or a rule copied from another project that blocks your entire /blog/ folder, can sit unnoticed for months while search engines quietly stop crawling real content. Robots.txt has no warning system: there’s no error message when a rule is too broad, it just works exactly as written. Building the file with a generator instead of hand-editing it keeps the syntax correct and makes it much easier to see, at a glance, exactly what you’re blocking before you upload it.
The WordPress default, explained
The WordPress preset above disallows /wp-admin/, since your dashboard has no reason to appear in search results, while explicitly allowing /wp-admin/admin-ajax.php. That second line matters more than it looks: a lot of plugins and themes call admin-ajax.php to load content for logged-out visitors, and a blanket block on the whole /wp-admin/ folder without that exception can quietly break front-end functionality that depends on it.
Sitemap placement, and why it’s usually the last line
The Sitemap: directive can technically go anywhere in the file, but convention places it at the end, after all the user-agent groups, since it applies to the whole file rather than to any one crawler. Pointing to your sitemap here is optional (most SEO plugins also submit it directly to Search Console), but including it costs nothing and gives any crawler reading the file a direct path to your full URL list.
Before you block your whole site
A “Disallow all” file is the right call for a genuine staging or development environment, but it’s not a privacy or security measure. The file itself is public at yoursite.com/robots.txt, and it only works on crawlers that choose to respect it, plenty don’t. If a site genuinely shouldn’t be found while you’re building it, pair this with real access control, like HTTP basic auth or an IP allowlist, rather than relying on robots.txt alone.