Start free trial Go to workspace
Service AI visibility diagnosisSite readabilityContent and sourcesMonitoringPricingCasesFAQHow an engagement runs
Platforms DeepSeekERNIE BotKimiDoubaoTencent YuanbaoTongyi QianwenWhy report them separately
Industries Legal servicesHealthcareEducationIndustrial equipmentBusiness services / B2BLocal servicesConsumer electronicsSkincare and cosmeticsWhat actually differs by industry
Resources Learning centreBlogGEO glossaryComparisonsFAQ
Free toolsPartners
About About DaleGEOContact us
Start free trial
HomeGlossaryrobots.txt
Glossary

robots.txt

All terms Get a free diagnosis

Definition

robots.txt is the file that tells crawlers which parts of a site they may fetch. For GEO the relevant part is whether AI crawlers are allowed.

Why it matters

Allowing crawlers in robots.txt is necessary but not sufficient — CDN rules and firewalls frequently block the same user agents, and that block is invisible unless it is tested directly.

Related terms

Concepts that only make sense together

AI crawler

An AI crawler is a bot operated by a model vendor to fetch web content — for training, for live retrieval, or for both. Each identifies itself with its own user agent.

 Crawlability

Crawlability is whether a crawler can actually retrieve a page — covering robots rules, firewall and CDN behaviour, rate limiting, redirects and status codes.

 llms.txt

llms.txt is a plain-text file at the site root that indexes the site for large language models — what the organisation is, which pages matter, and how they relate. It is an emerging convention rather than a standard.

Find out where your brand actually stands in AI answers.

An AI visibility diagnosis across the platforms your customers use, on your real questions. No charge for the first look.

Get a free diagnosis 17722562362