What is this tool?
A robots.txt file gives crawler instructions for a website. It can point to a sitemap, block low-value paths, and define broad crawling preferences. It does not protect private content.
How to use it
Enter your sitemap URL and optional blocked paths such as /admin/ or /tmp/, then copy the generated robots.txt file.
Privacy
This tool is designed to process your input locally in the browser. Avoid entering passwords, private keys, or other sensitive production data into any online tool.
Frequently asked questions
Can robots.txt hide private pages?
No. Robots.txt is a public instruction file and should not be used to protect private or sensitive content.
Where should robots.txt be placed?
It should be available at the root of the site, for example https://example.com/robots.txt.
Should I include my sitemap?
Yes. Adding a Sitemap line helps crawlers discover your sitemap location.