robots.txt
robots.txt is a plain text file at yoursite.com/robots.txt that tells crawlers which parts of your site they may visit. A SaaS team might use it to keep bots out of internal search results, then find out one leftover line from staging blocked Google and ChatGPT from the whole site.
What is robots.txt?
robots.txt is a plain text file at the root of your domain that tells crawlers which pages they may and may not visit. Search bots like Googlebot read it before they crawl, and so do the bots behind ChatGPT, Claude and Perplexity.
Let's say you sell project management software. Your site has a marketing section, a blog, a help center and an in-app search page that creates a new URL for every query. You don't want Google spending time on thousands of "search?q=" pages, so you add one line to robots.txt that tells all bots to skip /search/. The rest of the site stays open.
The file is public. Anyone can type yoursite.com/robots.txt into a browser and read it, so it's a list of polite requests to well-behaved bots. It doesn't protect anything private, and a bad bot can ignore it.
Why robots.txt matters for B2B SaaS
One file decides whether search engines and AI tools can read your site at all. A single wrong line can undo years of SEO work, and nothing on the page itself looks broken.
I see the same story after site launches. A staging site gets a "Disallow: /" rule to keep it out of Google, then the rule ships to production with everything else. Rankings slide for weeks before anyone opens the file. Our website revamp SEO checklist puts this check on launch day for that reason.
Here's why it deserves a few minutes every quarter:
- It controls AI visibility too: OpenAI says sites that block OAI-SearchBot don't show up in ChatGPT search answers. If your file or CDN blocks the AI search bots, you can't be cited, however good your pages are.
- CDNs now make choices for you: Since July 2025, Cloudflare asks every new domain whether to allow AI crawlers, with blocking as the default. Your robots.txt can say "allow" while a firewall rule says "block".
- It protects your crawl time: Big SaaS sites create endless URLs from filters, search and tracking parameters. Blocking those leaves more of Google's visits for the pages that sell, which is what crawl budget is about.
How to set up robots.txt
A good robots.txt for a B2B SaaS site is short. Most of mine fit on one screen. Here's my process:
- Start from "allow everything": List only the folders that should never be crawled, like internal search, cart or checkout steps, admin paths and URLs with session parameters. Everything you leave off the list stays open.
- Name the AI bots on purpose: Decide bot by bot. OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages for AI answers, while GPTBot and ClaudeBot collect training data. Google-Extended decides whether Google can use your pages to train and ground Gemini, and Google says it doesn't affect your place in Google Search.
- Add your sitemap: A "Sitemap:" line with the full URL of your XML sitemap helps every crawler find your important pages, including ones that aren't linked well yet.
- Use noindex for pages that must stay out of Google: A blocked page can still get indexed if other sites link to it, and Google can't see a noindex tag on a page it isn't allowed to crawl. So for thank-you pages or old campaign pages, allow crawling and add noindex.
- Cover every subdomain: docs.yoursite.com, app.yoursite.com and help.yoursite.com each need their own file. Public docs often bring more technical buyers than the blog, so check those first.
- Keep it small and plain: Google reads the first 500 KiB and ignores the rest, and it doesn't support the crawl-delay rule. Plain allow and disallow lines are all most sites need.
A robots.txt example
Here's what a robots.txt could look like for a hypothetical HR software company with a marketing site, a blog and an in-app search page:
| Line | What it does |
|---|---|
| User-agent: * | The rules below apply to every bot |
| Disallow: /search/ | Keeps bots out of internal search result pages |
| Disallow: /*?utm_ | Skips copies of pages created by campaign tracking links |
| User-agent: GPTBot | Starts a separate group of rules for OpenAI's training bot |
| Disallow: / | Opts the whole site out of training, a business choice some teams make |
| Sitemap: https://example.com/sitemap.xml | Points every crawler to the full list of pages |
In this setup, OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot all fall under the first group. They can read the blog, pricing and comparison pages, so the company can still be cited in AI answers while it opts out of training.
You can learn a lot by reading other companies' files. Type any competitor's domain plus /robots.txt into your browser and you'll see which bots they allow and which folders they hide.
How to measure robots.txt
You don't measure robots.txt the way you measure a page. You check that it does what you meant. Here's what I look at:
- Search Console robots.txt report: Under Settings, Search Console shows the robots.txt files Google found for your site, when it last fetched them and any errors it hit.
- Blocked by robots.txt in the Pages report: Search Console's Pages report lists URLs Google skipped because of your file. Every URL there should be one you meant to block.
- URL Inspection: Test one important page, like your pricing page, and confirm Google says crawling is allowed.
- Server or CDN logs: Look for visits from OAI-SearchBot, Claude-SearchBot and PerplexityBot. If they never show up, check your firewall and bot settings before you blame the file.
- AI answers that cite you: Run a few buying prompts in ChatGPT and Perplexity after any change. If your pages never appear as sources, access is one of the first things to rule out.
Common robots.txt mistakes
- Shipping the staging file: The "Disallow: /" rule that kept staging out of Google goes live with the new site. Open yoursite.com/robots.txt the same hour you launch.
- Using it to hide private pages: The file is public, so listing /internal-pricing/ tells everyone where to look. Put private pages behind a login instead.
- Blocking CSS and JavaScript: If Google can't load your scripts, it may not see the content they render. Leave asset folders open.
- Blocking a page you want deindexed: Google can't read a noindex tag on a page robots.txt blocks, so the page can stay in results. Remove the block until Google has seen the tag.
- Treating all AI bots the same: Blocking every AI bot to stop training also stops AI search citations. Make the training choice and the search choice separately.
Here's a first step that takes one minute. Open yoursite.com/robots.txt in a new tab right now. Can you explain every line in it?
Read more
FAQ
Where is the robots.txt file?
It always sits at the root of the domain, like yoursite.com/robots.txt. Each subdomain needs its own file, so docs.yoursite.com and app.yoursite.com are read separately from your main site.
Does robots.txt stop a page from being indexed?
No. It stops crawling, and Google can still index a blocked URL without its content if other pages link to it. To keep a page out of search, let Google crawl it and add a noindex tag, or put it behind a login.
Should a B2B SaaS site block AI crawlers in robots.txt?
Most should allow the AI search bots, like OAI-SearchBot, Claude-SearchBot and PerplexityBot, because those decide whether you can be cited in answers. Training bots like GPTBot are a separate business choice you can make on their own.
How long does a robots.txt change take to work?
Google usually caches the file for up to 24 hours, and OpenAI says its bots take about the same. Plan for a day before a fix shows up in crawl logs.



