SEO & performance
robots.txt
robots.txt is a file at the root of a domain that tells crawlers which paths they may request. It controls crawling, not indexing — a URL blocked in robots.txt can still appear in search results if other pages link to it, and blocking it actually prevents Google from seeing the noindex tag that would remove it. To keep a page out of the index, allow the crawl and use noindex.
Why it matters
The distinction between crawling and indexing is the single most misunderstood thing in technical SEO, and robots.txt sits exactly on the fault line.
Blocking a URL in robots.txt stops crawlers requesting it. It does not remove it from search results. Worse, it guarantees the page stays there if anything links to it, because Google can see the link but cannot fetch the page to discover the noindex directive that would have removed it. The block prevents the very instruction that was meant to do the job.
This produces the classic outcome: a URL listed in results with no description, marked as blocked, which the site owner cannot get rid of because the fix they applied is what is preventing the fix from working.
How it works on Shopify
Shopify serves a sensible default robots.txt covering cart, checkout, account, and internal search paths. It can be customised through a robots.txt.liquid template, which allows adding rules — and allows breaking things.
The correct use is preventing crawlers from spending time on URLs that were never going to rank and do not need to be assessed: internal search results, cart URLs, parameter-driven paths that have never been indexed.
The wrong use is deindexing. To remove a page from search results, allow the crawl and serve a noindex directive. Once the page has dropped out of the index — which takes weeks and needs confirming in Search Console — blocking it becomes safe if desired.
For faceted navigation this ordering matters a great deal, because filter URLs are usually already indexed by the time anyone notices them. Blocking first freezes them in place.
It is also worth remembering that robots.txt is public and advisory. Well-behaved crawlers respect it; others do not, and it is not a security measure.
Common mistakes
- Blocking to deindex. The central error, and it produces the opposite result.
- Disallow: / on a staging site that later becomes production. Catastrophic and surprisingly common.
- Blocking CSS or JavaScript. Google renders pages. Blocking assets means it renders them badly.
- Blocking AI crawlers unintentionally. A broad rule can remove the site from AI search sources — relevant to GEO.
- Assuming it hides anything. The file is public and lists the paths you consider sensitive.
- Editing without testing. Search Console's tester exists; a rule that matches more than intended is silent.
When you need help
The trigger is a set of URLs in search results that will not go away, or an indexing problem that started after someone edited the file.
Untangling that means establishing the correct sequence — unblock, serve noindex, wait for removal, then block if still wanted — and holding it, which requires patience more than expertise. The broader case is a large catalogue where robots.txt, canonical tags, noindex, and the XML sitemap need to agree with each other, since contradictory signals are what most indexing problems actually are.
Need this done on your store?
Site structureRelated terms
- Crawl budgetCrawl budget is the number of URLs a search engine will crawl on your site in a given period, set by your site's authority and how fast your server responds. It only becomes a real constraint on large sites — but a Shopify store with unmanaged faceted URLs reaches that scale quickly, and spends its budget on filter permutations instead of products.
- XML sitemapAn XML sitemap is a machine-readable list of the URLs you want search engines to crawl, along with when each last changed. Shopify generates one automatically at /sitemap.xml and it cannot be edited directly. A sitemap helps discovery on large or poorly-linked sites; it does not force indexing, and listing a page there does not override a noindex tag.