robots.txt and Sitemaps: The Minimum Setup
5 min read
Neither file has a visible page and no visitor ever sees them, yet together they decide whether the site is in search results at all. They are also the two things I most often find set wrong: the whole site blocked during development, never unblocked at launch, and the site quietly gone for months. The good news is that both are simple, and once you understand them, five minutes tells you whether you have a problem.
1. What each file does
robots.txt sits at the root of your domain and is a set of door rules for crawlers, saying which paths to stay out of. It is a request, not an enforcement. Legitimate search engines honour it and malicious programs do not. So it holds back only the well-behaved side, and it is not a security control.
A sitemap is a directory listing the addresses you want indexed, usually as an XML file in the sitemaps.org format. It does not guarantee that every page gets indexed, but it saves search engines the work of finding their own way around. On sites with many pages, or with sparse internal linking, the difference is noticeable.
| robots.txt | Sitemap | |
|---|---|---|
| What it controls | Access: which paths not to crawl | Inventory: the addresses you want indexed |
| Where it lives | The root of your domain, under a fixed filename | Usually the root too, with its address on the last line of robots.txt |
| Cost of getting it wrong | One wrong line and the whole site leaves search results | Slower indexing, or search engines reading a pile of deleted addresses |
| Do you need it | Always, even if it is only three lines | Worth having even on a small site: it is one more checkpoint |
They do different jobs, you need both, and they have to agree with each other. A path blocked in robots.txt cannot be read even when it is listed in the sitemap, so the two files end up contradicting each other. Search Console reports that conflict as an error, but if nobody looks, it can sit there quietly for months.
2. What a minimum working setup looks like
Most small business sites need no complicated rules. A clean robots.txt does two things: it opens the whole site to search engines, and it says where the sitemap is.
The first block is User-agent: * followed by Allow: /, meaning every crawler may read everything. The last line is Sitemap: plus the full address of your sitemap, so a crawler knows where the directory is as soon as it arrives. Three lines and the file is finished, which is enough for most sites.
If there really are paths that should stay out of the index, add Disallow lines for them. Admin login pages, shopping carts and internal search result pages have no value in an index, and blocking them is reasonable. To open the site to AI engines, you can name them and allow them explicitly; the ones you do not name will generally fall back to the asterisk rules.
The sitemap is best generated by your build or publishing process. Versions maintained by hand go stale almost without exception, because a new article does not get added and a changed address does not get updated. In time the file becomes a list that misleads search engines, which is worse than having none.
3. Three mistakes that keep coming up
These three are the ones I see most. They have one thing in common: the site looks completely normal, and unless you open those two files, nothing gives them away. So they typically survive until somebody asks why the site cannot be found.
The worst is a site-wide block that was never lifted. Disallow: / is commonly used during development to keep unfinished work out of the index, and the line survives the launch. The cost of that one line is the entire site's search visibility, and no error message will ever mention it. Plenty of sites with no search traffic months after launch died right here.
Second is blocking stylesheets and script files. Some older guides recommend it on the grounds of saving crawl resources. Search engines need those files to render the page properly, and blocking them affects how the page is judged on mobile devices. The resources saved are not remotely worth it.
Third is a sitemap full of wrong addresses: large numbers of deleted pages, or addresses still on a domain you have left. When search engines keep fetching links that fail, they trust the list less, and new pages you add get processed more slowly as a result.
4. How to verify it yourself
Both files are public, so no tool is needed. The four checks below rule out all three mistakes above. Run them after a redesign, a hosting move or a domain change. The rest of the time you can leave them alone.
Four checks, five minutes
- Open your domain with /robots.txt on the end. Seeing content means the file exists. Look for a line reading Disallow: / on its own.
- Open your domain with /sitemap.xml on the end. Check the addresses are your current ones and the count is close to your real page count.
- Read the Sitemaps report in Search Console. The last read date and the number of addresses found should match what you counted yourself.
- Run a few important pages through URL Inspection. Confirm they are not blocked and that they are indexed.
5. Should low-value pages go in the sitemap?
A sitemap is not a dump of every address you have. It also says "these are the pages I think are worth indexing", and what goes in should live up to that. Tag pages, paginated listings and thank-you pages usually do not belong, since nobody searches for them even when they are indexed.
Addresses that have been deleted or redirected should come out, because leaving them in only produces failed fetches. If you publish in more than one language, the same file can mark which pages correspond to each other, so search engines treat them as versions of one document rather than duplicates.
One field that gets overlooked is the last-modified date, and it has to be honest. Stamp today's date on every page each time the file is generated while the content has not changed, and after a few comparisons the field simply gets ignored. Then when you do update something important, nothing comes back to look at it.
These two files share a quality: invisible when they are right, fatal when they are wrong. Check them once before launch, then again only when you redesign or change domain. If your site has been live a while and still cannot be found, this is the first place worth looking.
Common questions
- My site only has five pages. Do I still need a sitemap?
- If the site is small and every page is reachable from the homepage, it will be found without one and the difference is small. It costs almost nothing, though, and having one lets you see fetch status in Search Console, which is one more checkpoint. If any page cannot be reached from the homepage, you do need it.
- Can I use robots.txt to keep pages private?
- No, and it backfires. robots.txt is a public file that anyone can read, so putting a sensitive path in it announces that something is there. Anything genuinely private needs a login or server-level access control. For a page that can be public but should not be indexed, a noindex tag is the correct tool.
- How long does a change to robots.txt take to apply?
- Search engines re-read the file regularly and usually pick up a new version within a day. Search Console shows the last version it read, so you can confirm it landed. After you lift a block, the pages still take time to be indexed again and they do not come back immediately.