robots.txt Generator

How to use the robots.txt generator

1. Choose what to keep crawlers out of, and decide about AI crawlers. The file updates as you go.
2. Add your sitemap’s address.
3. Download the file and upload it to your site’s root, so it opens at https://your-domain/robots.txt. It only works there: crawlers never look for it in a subfolder, and each subdomain needs its own.
4. Test a few important URLs with the tester before you publish. A single Disallow: / in the wrong group can take a site out of search.

The file is built in your browser; nothing is sent to Cloudhim.

How crawlers read robots.txt

The rules are grouped by User-agent. A crawler follows only the group with the most specific name that matches it, and falls back to User-agent: * only if there isn’t one. So a crawler with its own group ignores your * rules completely. Within a group, the longest matching path wins, and Allow wins a tie. This is the standard (RFC 9309) and how Google and Bing behave; the tester above uses the same logic.

Should you block AI crawlers?

AI companies use different crawlers for different jobs, so you can choose. Training crawlers collect pages to train models; blocking them doesn’t affect search. AI search crawlers build the indexes that ChatGPT search, Claude and Perplexity use to answer questions and link to sources; blocking them means you won’t be cited. User-triggered fetchers load one page because someone asked an assistant about it. Google’s AI Overviews use the normal Googlebot, so they can’t be blocked separately without leaving Google Search. You can also publish an llms.txt file that points AI tools to your best pages.

robots.txt is not security

The file is public, and badly behaved bots ignore it, so never use it to hide private pages: listing them there shows everyone where they are. Protect private areas with a password (the .htaccess Generator or htpasswd Generator can help), and use a noindex tag to keep a public page out of search results.