Skip to content
Free check

Short answer · website settings

Should I block AI crawlers from my website?

Short answer

For most local businesses that want to be recommended, no. AI crawlers are the programs AI companies use to read websites; if your site blocks them, an assistant may not be able to read your services, area or contact details, and may describe you from older or second-hand information instead.

By Word of Chat · Published 27 October 2026

Claymation friendly little robot with a magnifying glass reading the window of a small shop
Illustration

What an AI crawler is

A crawler is a program that visits web pages and reads them. Search engines have used them for decades. AI companies run their own: OpenAI, for example, documents one crawler used to collect training data (GPTBot) and another used for search inside ChatGPT (OAI-SearchBot). Google has a separate setting, Google-Extended, which controls whether your content can be used for its Gemini models without affecting normal Google Search. Names and rules change, so check each company’s own documentation.

How to check your own site

  • Open yourwebsite.co.uk/robots.txt in a browser. It is a short text file.
  • Look for lines that say "User-agent:" followed by an AI crawler name, with "Disallow: /" underneath. That blocks that crawler from the whole site.
  • A line saying "User-agent: *" with "Disallow: /" blocks everything, including search engines. That is almost always a mistake on a live business site.
  • Some website platforms and security plugins have an "block AI bots" switch. Ask whoever runs your site whether it is on.

When blocking might make sense

Some businesses choose to block training crawlers while allowing search crawlers, because they do not want their writing used to train models. That is a reasonable choice. Just make it deliberately, knowing the trade-off, rather than by accident.

The honest limits

Allowing crawlers does not mean an assistant will recommend you; it only removes one obstacle. Each AI company decides how its crawlers and assistants work, and the details change over time.

Want to know where you stand?

Our free check asks ChatGPT and Google’s Gemini the kind of questions your customers ask and emails you, in plain English, whether you were named and who was named instead. Four tiny steps; your details come last.

Step 1 of 4: Your trade

What’s your trade?

One tap. We use it to write the questions your customers would ask.

If you want more, every service has a fixed price.

Questions

Will allowing AI crawlers slow my website down?

For a typical small business site, crawler visits are a tiny amount of traffic. If you ever see a problem, your web host can help.

Does blocking AI crawlers affect my Google ranking?

Blocking Google-Extended is designed not to affect Google Search. Blocking Googlebot itself would remove you from search, so never do that by accident.

Does the free check look at my robots.txt?

The free check focuses on what ChatGPT and Google’s Gemini say about you. The paid audit looks at technical blockers like this as part of the fix list.