What does ChatGPT look at when citing a source?

ChatGPT looks at three things when it cites a source: whether OAI-SearchBot is allowed to crawl your site, whether you are in the search index it reaches for, and whether you appear on the reference and community surfaces it already leans on. Only the first is a hard gate — OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.

One line in a file most owners have never opened can quietly remove you from ChatGPT's answers. That file is robots.txt, the plain-text file at the root of your domain that tells each crawler what it may fetch. In Answer Engine Optimization (AEO) — engineering a site so AI assistants quote it — this is the one check that is a switch rather than a craft. Your writing can be excellent and your markup perfect; if the crawler never arrives, none of it is in the pool the answer is drawn from.

Watch the walkthrough

Which OpenAI bot actually gates your citations?

OAI-SearchBot, and it takes two minutes to check. Open your own site with /robots.txt on the end and look for that name. OAI-SearchBot is OpenAI's search crawler, and OpenAI's own crawler documentation says sites opted out of it will not be shown in ChatGPT search answers. Now the part almost everybody gets wrong: GPTBot is a different bot with a different job. OpenAI describes GPTBot as crawling content that may be used to train their foundation models. Blocking GPTBot changes nothing about whether ChatGPT cites you today — it changes whether your writing can enter a future model. Two bots, two questions. Read the row, not the label.

Why does a named group cancel the rules you wrote for everyone?

Because that is what the specification says. Google's documentation on how robots.txt is interpreted states that only one group is valid for a particular crawler: the crawler finds the group whose user-agent matches it most specifically, and every other group is ignored. A few lines further down comes the sentence that quietly costs people their visibility — user-agent-specific groups and global groups are not combined. Whatever sits inside a named group is the entire law for that crawler. If you have ever pasted an "AI bots" block from a blog post into a file that already had careful rules in it, go and read what that group actually contains, because it is the only thing that crawler will see.

What does our own robots.txt get wrong?

Our named AI-crawler groups inherit nothing from the rules above them — and we show that on camera rather than grading someone else. Our file passes check one: it carries a group for OAI-SearchBot with Allow. Look up, though, and the wildcard group at the top carries our actual rules — do not crawl the 404 page, do not crawl the portal. Not one of those reaches OAI-SearchBot. The moment we gave that crawler its own named group it stopped inheriting anything. That is not a bug in our file, it is the specification working as written. Here it costs us little, because those paths sit behind a login — and a login is what protects private pages, never a line in this file.

How many crawlers in your file actually matter?

Three, out of the seventeen we name. The ones that can cost you a citation today are OpenAI's search crawler, Perplexity's crawler and Anthropic's search crawler. The rest are not junk — GPTBot governs whether your writing can enter a future training corpus, which is a real question, just a slower one. But if somebody charges you to unblock the training bots so that AI can cite you, you are paying for the wrong row. Notice also the one gating crawler we deliberately did not name: Googlebot stays in the wildcard group, so it still follows our rules — and Googlebot is how you reach the index the second check is about.

Bing or Google — which index does ChatGPT reach for?

Both halves are true, and most advice gives you only one. Half one: OpenAI's own help page on ChatGPT search says it partners with other search providers, and it names Bing and Shopify. We read that page directly on 1 August 2026 and Bing was named, so being indexed in Bing is not wasted effort. Half two: Microsoft retired the Bing search APIs in August 2025, and across that year, on a thousand-prompt panel run by Profound, ChatGPT's overlap with Bing's results fell from 26% to 8% while its overlap with Google's climbed from around 12% to 33%. In one public test ChatGPT even quoted a page indexed only in Google. Those are other people's measurements and we hand them to you as measurements, not as a law. The instruction is neither obey Bing nor ignore Bing: put the weight on Google first — real server-rendered pages, a sitemap that is true, Googlebot not blocked — then spend the twenty free minutes it takes to submit that sitemap in Bing Webmaster Tools, because OpenAI still names Bing and it costs you nothing.

Which sources does ChatGPT already lean on?

Not the web evenly. In a study of 600,000 citation events in the United States published this year, Wikipedia and Reddit together accounted for more than a quarter of everything ChatGPT cited. For a small business the move is therefore not to be everywhere; it is to be correctly described on the two or three reference and community surfaces where your category is genuinely discussed. And now the caveat, because you should hear it from us: in another measurement across more than 100 million citations, Reddit's share collapsed from about 60% to about 10% in roughly six weeks. Treat the source mix as weather. The crawler line is architecture. Build on the architecture and dress for the weather.

What do free AI-visibility tools say about crawler access?

Nothing, and that is a gap in what they are built to answer rather than a criticism of the tools. Ahrefs' AI visibility checker is genuinely free — no signup, results in seconds — and the report behind it is real: total AI mentions, mentions by platform, top topics, top cited domains, top cited pages. Search that page for the word "robots" and you get zero hits: nothing about crawler access, nothing about OAI-SearchBot, nothing about GPTBot. Semrush is the same shape and generous with access, three runs a day without registering, with a well-built report — visibility score, mentions, citations, platform coverage, top cited pages, competitor comparison — and not one of those dimensions is crawler access. HubSpot's grader needs no account for a single run and returns a composite score across sentiment, presence quality, brand recognition, share of voice and market competition. Five dimensions, and not one of them is access. If your one gating line were wrong, all three would hand you a low number and none of them would tell you why.

What does our own open-source tracker do instead?

It ties each bot to what blocking it actually costs. aeo-platform is our own tracker, free and open source on npm, and its crawl audit does not report a count of blocked bots. It labels each one by tier: search tier means blocking it removes you from that engine's answers; training tier means training corpus only, with no measured effect on today's citations; user tier means the fetch is user-triggered and generally ignores your file anyway. It also carries a guard that refuses to write you a task for unblocking a training bot. That refusal is the product — a tool that hands you five urgent-looking access tasks when only one of them can change a citation is selling activity, not visibility.

Where does our own site fail this checklist?

On check three, and badly. Search Wikipedia for Webappski today and you get zero results — we verified it again on 25 August 2026. We are absent from the surface ChatGPT leans on hardest, and we cannot fix it by writing a page about ourselves, because Wikipedia asks for independent coverage first. That check is red on our own board and it stays red for a while. We would rather show you that than sell you a shortcut, and it is the honest shape of this whole checklist: two of the three things ChatGPT looks at are earned slowly, and only the first one is a switch you can flip this afternoon.

What is the full ChatGPT checklist?

Five checks, in order, and we run them for every client. One: OAI-SearchBot allowed in robots.txt — the only one of the five that is a switch. Two: read the named group you gave it, because it inherits nothing from the group above. Three: server-rendered pages in Google's index, and Googlebot not blocked. Four: a sitemap submitted in Bing Webmaster Tools, because OpenAI still names Bing. Five: an honest presence on the two or three reference and community surfaces where your category is discussed — and no faith in that mix staying as it is. The first takes two minutes, and most sites have never looked.

Frequently asked questions

Should I block GPTBot?

That is a decision about training, not about citations. OpenAI describes GPTBot as crawling content that may be used to train their foundation models, so blocking it does not remove you from today's ChatGPT answers — OAI-SearchBot is the bot that does that. Block GPTBot if you do not want your writing in a future model, and know that it buys you nothing in visibility either way.

I pasted an AI-bots block into my robots.txt. Is that safe?

Only if you read what the block contains. Naming a crawler creates its own group, and per the specification that group is the only one that crawler obeys — user-agent-specific groups and global groups are not combined. Any Disallow rule you wrote in the wildcard group stops applying to every bot you just named. Re-read the pasted block as if it were your whole file for those crawlers, because it is.

Do I still need to be in Bing to be cited by ChatGPT?

It still helps and it is cheap, but Google is where the weight has moved. OpenAI's help page still names Bing as a search partner, which is why submitting a sitemap in Bing Webmaster Tools is worth twenty free minutes. Independent panel measurements across 2025 show ChatGPT's overlap with Bing results falling and its overlap with Google's rising, so put the real effort into being indexable by Google first.

Will a free AI-visibility tool tell me if a crawler is blocked?

Not the three we checked. Ahrefs, Semrush and HubSpot report mentions, cited pages, platform coverage and composite scores — useful for knowing whether you appear, silent on why you do not. A blocked search crawler and a boring page produce the same low number in those reports. Check robots.txt yourself, or use a tracker whose crawl audit names the tier of each bot.

How do I get onto the sources ChatGPT leans on?

Slowly, and by being described accurately rather than by publishing about yourself. Wikipedia requires independent coverage first, so press and genuine third-party writing come before any encyclopedia entry. On community surfaces, be present where your category is actually discussed and be correct there. Treat the mix as unstable — one measured collapse took Reddit's share from roughly 60% to 10% in about six weeks.

Want these five checks run across your site?

We run all five — crawler access, group inheritance, index coverage, Bing submission and source presence — on real client sites, and the first AI-visibility audit is free. If you want to know whether ChatGPT can currently reach you at all, request a free audit at webappski.com.