SEO & performance
Crawl budget
Crawl budget is the number of URLs a search engine will crawl on your site in a given period, set by your site's authority and how fast your server responds. It only becomes a real constraint on large sites — but a Shopify store with unmanaged faceted URLs reaches that scale quickly, and spends its budget on filter permutations instead of products.
Why it matters
Crawl budget is the SEO concept most often invoked where it does not apply. A store with two hundred products does not have a crawl budget problem, and time spent worrying about one is time not spent on the things that would help.
Where it becomes real is at scale, and Shopify stores reach that scale faster than their product count suggests. A catalogue of five thousand products with unmanaged faceted navigation can expose several hundred thousand crawlable URLs. At that point the crawler is spending its allocation on filter permutations, and genuinely new products wait days or weeks to be discovered.
The symptom is a lag between publishing and indexing. New products not appearing, updated pages showing stale information in results, and a large gap between submitted and indexed counts in Search Console.
How it works on Shopify
Two factors set the budget. Crawl capacity is how much your server can take without slowing down — rarely a constraint on Shopify's infrastructure. Crawl demand is how much Google wants to crawl, which follows the site's authority and how often its content genuinely changes.
Since capacity is not the lever on Shopify, the work is entirely about not wasting the demand you have.
The waste comes from a predictable list: filter and sort URL combinations, internal redirect chains that consume two fetches instead of one, soft 404s on removed products, tracking parameters creating duplicate addresses, and pagination running deeper than anything needs to.
The tools are internal linking discipline — crawlers follow links, so not linking to permutations removes most exposure — canonical tags for consolidation, noindex for pages that must remain accessible but should not rank, and a clean XML sitemap listing only URLs you actually want crawled.
robots.txt prevents crawling but does not remove anything already indexed, which is the distinction that trips most implementations.
Common mistakes
- Worrying about it on a small site. Under a few thousand URLs, it is not the constraint.
- Blocking to deindex. A blocked URL is never fetched, so its noindex is never seen.
- Sitemaps full of noindexed URLs. Contradictory signals waste crawls and confuse consolidation.
- Redirect chains. Each hop costs a fetch. Point redirects at the final destination.
- Deep pagination. Page 40 of a collection is not worth crawling and rarely worth linking to.
- Ignoring the crawl stats report. Search Console shows exactly what Google is spending its time on, and almost nobody looks.
When you need help
The trigger is measurable: new products taking a long time to appear, or a discovered-URL count many multiples of the real page count.
The diagnosis starts in the crawl stats report and log files, which show what is actually being fetched rather than what should be. The fix is usually a faceted navigation policy plus internal linking changes, and on a large catalogue it is one of the few technical interventions with a visible effect on how quickly the store's content reaches search results.
Need this done on your store?
Site structureRelated terms
- Faceted navigationFaceted navigation is the filtering and sorting UI on a collection page — by size, colour, price, brand. It is excellent for shoppers and hazardous for SEO: each combination of filters can generate a crawlable URL, and a handful of facets can multiply into thousands of near-duplicate pages that drain crawl budget. The fix is deciding deliberately which facets are indexable and blocking the rest.
- XML sitemapAn XML sitemap is a machine-readable list of the URLs you want search engines to crawl, along with when each last changed. Shopify generates one automatically at /sitemap.xml and it cannot be edited directly. A sitemap helps discovery on large or poorly-linked sites; it does not force indexing, and listing a page there does not override a noindex tag.
- robots.txtrobots.txt is a file at the root of a domain that tells crawlers which paths they may request. It controls crawling, not indexing — a URL blocked in robots.txt can still appear in search results if other pages link to it, and blocking it actually prevents Google from seeing the noindex tag that would remove it. To keep a page out of the index, allow the crawl and use noindex.