Glossary

Crawl budget

Crawl budget is how many pages Googlebot can and wants to crawl on your site in a given period. A SaaS site with 300 pages rarely needs to think about it. A docs site whose filters spin out 80,000 URLs does.

What is crawl budget?

Crawl budget is the number of URLs Googlebot can and wants to crawl on your site over a period of time. Google splits it into two parts. The crawl capacity limit is how hard Googlebot can hit your server without slowing it down. Crawl demand is how much Google wants to crawl, based on how popular your pages are and how often they change.

Let's say you sell an API monitoring tool. Your marketing site has 250 pages, and your docs cover four versions of the API. Every docs page has filters for language and version, and each combination makes a new URL. Before long, Google sees 60,000 URLs where your team thinks it has 2,000 pages.

Now Googlebot spends most of its visits on near copies of the same page. Your new pricing page or launch post may wait days to get crawled, and some docs pages may never make it into the index at all.

Why crawl budget matters for B2B SaaS

For most B2B SaaS sites, it doesn't. Google's guide to crawl budget says it's written for sites with about a million unique pages that change weekly, or about 10,000 pages that change daily. If your site has a few hundred pages, spend your time on content and links instead.

It starts to matter when your site grows URLs faster than people write pages. Here's where I see that happen in SaaS:

  • Programmatic pages: An integrations directory, a template gallery or "X for Y" pages can add thousands of URLs at once. If many are thin, Google may crawl them slowly and index only some. Our glossary entry on programmatic SEO covers how to plan them.
  • Docs and help centers: Version switchers, language tabs and search pages create endless URL variants. Docs are often the biggest part of a SaaS site by page count.
  • Faceted filters: A customer stories page with filters for industry, size and region can create a URL for every combination, and Googlebot will try to follow them all.
  • A site move: After a redesign or domain change, Google has to recrawl everything. If half your crawls go to redirect chains, the new pages get picked up more slowly.

How to manage crawl budget

The goal is simple: spend Googlebot's visits on pages you want in search. Here's my process:

  • Check whether you have a problem: Look at the Page indexing report in Search Console. A big pile of URLs under "Discovered, currently not indexed" on a large site is the clearest sign Google isn't getting to everything.
  • Cut duplicate URLs: Point filter, sort and tracking parameter URLs to the main version with a canonical tag, and stop linking to the variants in your navigation. Fewer URLs means each one gets crawled more often.
  • Block what never needs crawling: Use robots.txt for internal search results and endless filter combinations. Google notes that noindex still costs a crawl, since Googlebot has to fetch the page to see the tag.
  • Return the right status codes: Send a 404 or 410 for pages you removed for good, and fix redirect chains so each old URL goes straight to its new home in one hop.
  • Keep the server fast: Google raises the crawl limit when your server responds quickly and lowers it when it slows down or returns 5xx or 429 errors. A slow docs host can hold back crawling of the whole domain.
  • Keep sitemaps clean: List only live, indexable pages with accurate last modified dates. A sitemap full of redirects and 404s sends Googlebot to the wrong places.

A crawl budget example

Segment is a good real example of a SaaS site with a lot of URLs. Its integration pages and public docs cover hundreds of tools, and we broke down how they rank in our Segment technical SEO teardown. A site that size has to keep its docs and catalog URLs tidy for all of them to get crawled.

Here's a hypothetical version of the fix for the API monitoring company above:

ProblemURLs beforeFixURLs after
Language and version filters on docs48,000Canonical to the default language and latest version2,400
Internal docs search pages9,000Blocked in robots.txt0 crawled
Old blog tag pages1,200Removed, returning 4100
Redirect chains from a 2024 move3,500Each redirect points straight to the final URL3,500, one hop each

The numbers are made up for the example. The pattern is common: most wasted crawling comes from a handful of URL patterns, so fixing three or four rules can shrink the URL count by a lot, as it does here.

How to measure crawl budget

You can't see the budget itself, but you can see how Google spends it. Here's what I'd watch:

  • Crawl requests per day: In Search Console, go to Settings and open Crawl Stats. A steady line is normal. A sharp drop often follows server errors or slow responses.
  • Average response time: The same report shows how fast your server answers Googlebot. If it climbs, the crawl rate usually falls soon after.
  • Crawls by response and purpose: Check what share of requests hit 200 pages, redirects and 404s, and how much is refresh crawling of known pages compared with discovery of new ones.
  • Time to index new pages: Note when you publish an important page and when it shows as indexed with the URL Inspection tool. If it takes weeks on a large site, crawl budget may be the reason.
  • Server logs: For big sites, your logs show exactly which URLs Googlebot requests. Tools like Screaming Frog Log File Analyser make them easier to read.

Common crawl budget mistakes

  • Worrying about it on a small site: If you have 400 pages and some aren't indexed, the cause is almost always quality or internal links. Crawl budget is rarely the reason.
  • Using noindex to save crawls: Google still has to fetch a noindexed page. Use robots.txt for URLs you never want crawled, and canonical tags for duplicates you still want consolidated.
  • Blocking pages you want indexed: A robots.txt rule written too broadly can block your docs or blog folder. Test every new rule before you ship it.
  • Launching thousands of thin pages: A batch of 5,000 integration pages with one sentence each gives Google little reason to crawl them often. Launch a smaller set with real content and grow it.
  • Leaving parameters in internal links: If your own navigation links to URLs with tracking or sort parameters, Googlebot will crawl every one of them.

If you run a large docs site, here's a first step. Open Crawl Stats and check what share of Googlebot's requests went to pages you actually want people to find. What does that number look like for you?

Read more

FAQ

Does crawl budget matter for small websites?

Usually no. Google's own guide is written for sites with about a million pages, or about 10,000 pages that change every day. A SaaS site with a few hundred pages gets crawled fine as long as the pages are linked and return a 200 status.

How do I check my crawl budget?

Open the Crawl Stats report under Settings in Google Search Console. It shows crawl requests per day, average response time and the status codes Googlebot got, so you can spot drops or a flood of wasted requests.

Does blocking pages in robots.txt free up crawl budget?

Blocking URLs you never want crawled, like internal search results, stops Google from spending time on them. Google says it won't hand that time to other pages unless your site is already hitting its crawl limit, so don't use it as a quick fix.

Do AI crawlers use crawl budget too?

AI bots like GPTBot and PerplexityBot crawl on their own schedules, apart from Googlebot. They still put load on your server, so watch your server logs if a large docs site suddenly slows down.

Want your brand in the answers your buyers read?

We build the Organic Content Engine for B2B SaaS teams: search content that ranks and gets cited in AI answers, plus LinkedIn thought leadership from your founders.