Key takeaways
- AI tools can only cite what their crawlers can read. Check robots.txt and any firewall first.
- Allow the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Blocking training crawlers like GPTBot is a separate choice.
- Write pages that answer one question clearly near the top, with facts about your business that match everywhere online.
- Measure it: AI referrals in GA4, AI features inside Search Console, and regular manual checks. No one can guarantee a citation.
How do ChatGPT and AI Overviews choose which sites to cite?
Both search the web before they answer. Google's AI Overviews draw from pages in Google's index that are eligible for a snippet. ChatGPT search relies on OpenAI's own crawler, OAI-SearchBot. They then pick pages that answer the question clearly and seem trustworthy. The exact selection is not public, so nobody controls it.
That last point matters when you hire help. Google says in its guide to AI features that no special optimization is needed beyond normal SEO. What you can control is access (can the crawler read your page?), clarity (does the page answer the question?) and reputation (do other sites confirm what you say about yourself?). The rest of this guide covers those three, in that order.
Which AI crawlers should you allow in robots.txt?
Allow the crawlers that power AI search answers: Googlebot, OAI-SearchBot for ChatGPT, Claude-SearchBot for Claude and PerplexityBot for Perplexity. Crawlers used mainly for model training, like GPTBot, ClaudeBot and Google-Extended, are a separate decision. Blocking them does not remove you from those search features, according to each company's documentation.
Each company publishes what its bots do. This is what their own pages say, in plain words:
| Crawler | Company | What it does | If you block it |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Finds sites to show in ChatGPT search | Your site is not shown in ChatGPT search answers, though it can still appear as a navigational link (OpenAI) |
| GPTBot | OpenAI | Collects content that may be used to train OpenAI's models | Signals your content should not be used for training; OpenAI says you can block it and still allow OAI-SearchBot |
| Claude-SearchBot | Anthropic | Indexes content to improve Claude's search results | May reduce your visibility in Claude's search answers (Anthropic) |
| ClaudeBot | Anthropic | Collects content that could be used for model training | Signals your future content should be excluded from training |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity results; Perplexity says it is not used to train models | Perplexity recommends allowing it so your site appears in its results (Perplexity) |
| Google-Extended | A control token for whether Google may use your content to train future Gemini models | Google says it does not affect inclusion or ranking in Google Search (Google) |
A simple robots.txt that welcomes search crawlers looks like this. Adapt it to your site, and keep any rules you already need:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Disallow: /wp-admin/
Sitemap: https://www.example.com/sitemap.xml
Robots.txt is only half the check. Firewalls, CDN bot protection and security plugins can block crawlers even when robots.txt allows them. I saw exactly this on a clinic website, where a verification screen hid the whole site from Google and AI crawlers. Once it was removed, the site earned 948 impressions in Google AI results in 28 days (see the case study). A technical SEO review tests your site the way a crawler sees it.
What kind of content gets quoted by AI tools?
Pages that answer a specific question in the first two or three sentences, then back the answer with details, examples and sources. AI tools need a passage they can lift and trust. Google's advice on helpful, reliable, people-first content applies directly: show real experience, say who wrote it and keep facts current.
Google's guide to helpful, reliable, people-first content asks questions like: does the content provide original information, does it show first-hand expertise, would someone trust it? Those same qualities make a page easy to quote. In practice:
- Use question headings that match how customers ask, and answer each one directly below it.
- Include specifics an AI can repeat: prices or price ranges, service areas, hours, how long a job takes.
- Add tables and lists for comparisons, steps and options.
- Show who wrote the page and why they know the topic. Add a date and update it when facts change.
- Link each guide to the service page where a reader can act.
Why do consistent business facts matter for AI answers?
AI tools build a picture of your business from many sources: your website, your Google Business Profile, directories, reviews and news. When the name, address, phone, services and service area match everywhere, the picture is clear and the AI can recommend you with confidence. Conflicting facts make you easier to skip or describe wrongly.
Start with an About page that states plainly who you are, where you work and what you do. Then add structured data. Google explains that structured data helps it understand the content of a page, and Organization or LocalBusiness markup is a clean way to state your name, address, phone, logo and profiles. To be precise: Google says no special schema is needed for AI Overviews. Schema is about clarity, not a shortcut.
Do reviews and mentions on other sites help?
Yes, because AI answers often lean on third-party sources. Pew found that Wikipedia, YouTube and Reddit were the most cited sources in Google's AI summaries. For a local business the equivalent is reviews, local directories, industry associations and local press. Earn genuine mentions there; never buy or fake them.
When someone asks an assistant for "the best dentist in my area", it tends to summarize what reviews and trusted sites say, not only your own claims. A steady flow of real reviews, complete directory listings and occasional mentions in local media give it something to work with. The Pew study makes the pattern visible: in AI summaries, community and reference sites carry a lot of weight.
How do you measure traffic from ChatGPT and AI Overviews?
Use three views. In Google Analytics 4, look at session source for referrals like chatgpt.com, perplexity.ai and gemini.google.com. In Search Console, AI Overview and AI Mode traffic is included in the Performance report under Web. Then run regular manual checks of the questions your customers ask in each assistant.
Google confirms in its documentation that sites appearing in AI features are counted in Search Console's overall search traffic, inside the Web search type. It does not split them out in that report, so manual checks and third-party tools fill the gap. If you work with a consultant, give them access as a user rather than sharing your password; Search Console lets property owners add users with full or restricted permissions.
On a large recovery-directory site, I built an in-house SEO platform to do this at scale. Among other things, it tracks traffic from ChatGPT, Perplexity, Gemini and other assistants, and how often Google AI Overviews cite each site, with weekly alerts when something changes. You can see how it fits into the project in the recovery directory case study. For a small business, a simple GA4 report plus a monthly manual check is usually enough to start.
Can anyone guarantee you will be cited by ChatGPT?
No. OpenAI, Google, Anthropic and Perplexity decide what their assistants cite, and answers change from one conversation to the next. An honest provider can remove technical obstacles, improve your content and reputation, and measure the results over time. Anyone who promises a guaranteed citation or a fixed position inside ChatGPT is overselling.
What you can expect from good work is a higher chance of being cited, more accurate descriptions of your business, and a clear view of what AI search actually sends you. My AI search optimization service covers exactly that, and it builds on the method on the SEO consultant home page. For the bigger picture on Google's side, read what AI Overviews mean for a small business.
Where should you start?
Start with an AI-search audit, which costs $400. I check crawler access, how assistants describe your business today and which pages are worth rewriting first. If you would rather talk it through before deciding, book a free 30-minute diagnosis and I will show you the three changes I would make first on your site.
See the pricing page or book your free diagnosis.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT?
Not from ChatGPT search, according to OpenAI. GPTBot is for model training; OAI-SearchBot is the crawler that finds sites for ChatGPT search, and you can allow one while blocking the other.
Do I need an llms.txt file?
Google says no special files are needed for its AI features. It does no harm, but crawler access, clear content and consistent business facts matter far more.
How long does it take to appear in AI answers?
It depends on how often crawlers visit and how competitive the question is. After fixing access problems, some sites show up within weeks; content and reputation work takes months.
Is AI search optimization different from SEO?
It builds on the same foundations: crawlable pages, helpful content and a good reputation. The extra work is crawler access for each assistant, quotable answers and separate measurement.
Can I see which questions ChatGPT cites me for?
Not directly. OpenAI does not offer a Search Console equivalent, so you combine GA4 referral data with regular manual checks of your key questions.
Sources
- Google Search Central, AI features and your website
- Google Search Central, Creating helpful, reliable, people-first content
- Google Search Central, Introduction to structured data
- Search Console Help, Managing owners, users, and permissions
- OpenAI, Overview of OpenAI crawlers
- Anthropic Help Center, Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity, Perplexity crawlers
- Google Search Central, Google's common crawlers (Google-Extended)
- Pew Research Center, Google users are less likely to click on links when an AI summary appears (July 22, 2025)