Practical guide
Robots.txt and Sitemap Basics for New Websites
For a new website, robots.txt and sitemap.xml are basic search-engine housekeeping. They do not guarantee ranking, but they help crawlers understand what they may crawl and which URLs are worth discovering.
Open the free toolStep-by-step
- Create a robots.txt file at the site root.
- Allow important public pages and avoid blocking CSS, JavaScript, or tool pages by accident.
- Add a Sitemap line that points to your sitemap.xml URL.
- Make sure sitemap.xml lists canonical URLs only.
- Submit the sitemap in Google Search Console and Bing Webmaster Tools.
- Re-submit or request indexing after adding important new pages.
Important tips
- Robots.txt is public and should not be used to hide private pages.
- Blocking a page can prevent crawling but does not remove every existing search result immediately.
- A sitemap should include useful canonical URLs, not every duplicate variation.
- Use 301 redirects to avoid both www and non-www versions being indexed separately.
Frequently asked questions
Where should robots.txt be placed?
It should be available at the root, such as https://example.com/robots.txt.
Does sitemap.xml guarantee indexing?
No. It helps discovery, but search engines still decide whether and when to index pages.
Should robots.txt include a sitemap URL?
Yes. A Sitemap line is a simple way to point crawlers to your sitemap.