Website owners trying to keep AI crawlers away from their content face a deceptively simple choice: put a disallow rule in robots.txt or enforce the decision deeper in the infrastructure. The two approaches may express the same policy, but technically they do very different things.
A robots.txt directive asks a crawler not to visit. A server, CDN or Web Application Firewall can refuse the request altogether. That distinction is the central point in Helen Pollitt's September 3 Search Engine Journal Ask an SEO analysis, and it has become increasingly important as publishers decide how much access AI companies should have to their sites.
For cooperative crawlers from established companies, robots.txt can be an effective and convenient policy mechanism. For a crawler that ignores the protocol, misidentifies itself or deliberately spoofs another user-agent, robots.txt is not an access-control system at all. If preventing access genuinely matters, enforcement has to happen in the request path.
Robots.txt is a rulebook, not a security barrier
The Robots Exclusion Protocol was designed to communicate crawling preferences. A publisher can name a user-agent and disallow the entire site or selected directories. An instruction such as User-agent: GPTBot followed by Disallow: / tells OpenAI's GPTBot not to crawl the site.
Major AI companies publish crawler identities and document robots.txt controls for at least some of their bots. Pollitt cites OpenAI's GPTBot and OAI-SearchBot, Anthropic's Claude crawlers, Google's Google-Extended control and PerplexityBot as examples of reputable agents for which publishers can express preferences this way.
The important technical limitation is that the file does not physically stop an HTTP request. Compliance is implemented by the crawler. A bot that chooses not to honor the rule can still request the URL, and the origin will normally return the page unless another layer denies access.
That makes robots.txt closer to a published access policy than an authentication or firewall mechanism. For responsible bots, the distinction may not matter operationally because they comply. For an adversarial scraper, it matters completely.
Server-level blocking changes the answer from “please don't” to “access denied”
A server can inspect incoming requests and apply rules before returning content. Known bot user-agents, IP ranges, headers or other request characteristics can be used to deny traffic rather than merely asking the crawler to stay away.
This is an actual enforcement decision. If the request matches the blocking rule, the crawler does not receive the protected content through that request regardless of whether its software recognizes robots.txt.
Server rules therefore solve the fundamental compliance problem, but they introduce operational complexity. Web servers are typically controlled by infrastructure or development teams, and mistakes at this layer can affect legitimate users and search crawlers as well as unwanted bots.
A rule based only on a user-agent string also remains easy to evade. User-agent headers are supplied by the requester. A scraper calling itself a normal browser or impersonating another crawler may bypass a simplistic string match.
A CDN can stop the request before it consumes origin resources
Moving enforcement to the CDN shifts the block closer to the edge of the network. Instead of allowing the crawler's request to travel to the origin server and then rejecting it, the CDN can intercept the traffic first.
That has a practical cost advantage for sites experiencing large volumes of automated crawling. Requests blocked at the edge do not consume the same origin bandwidth and application resources that they would if every request reached the server.
Modern CDN providers increasingly offer AI-specific bot controls rather than forcing site owners to maintain long lists of user-agents manually. Cloudflare's current AI bot documentation, for example, describes policies that classify crawlers by activities such as AI training and can block verified bots as well as additional unverified traffic classified in those categories.
Managed classifications can reduce one of the maintenance weaknesses of hand-written robots.txt lists: new crawler identities appear frequently, and a static file does not automatically know that a newly launched bot should inherit an old policy.
The WAF is the strongest layer against simple identity spoofing
A Web Application Firewall goes beyond reading the visible user-agent string. Depending on the provider and configuration, it can evaluate request patterns, bot signals, IP reputation, behavioral characteristics and other indicators to decide whether traffic is automated or suspicious.
This is why Pollitt identifies the WAF as the strongest option in most technology stacks for dealing with more sophisticated crawlers, particularly bots that attempt to evade user-agent-based rules. The system can ask not only what the requester claims to be, but whether its behavior is consistent with that identity.
Cloudflare's bot-management documentation illustrates the principle: requests can be classified using bot scores and verified-bot status, allowing rules to distinguish known legitimate crawlers from traffic that appears automated.
Its AI Crawl Control documentation also explicitly places WAF custom rules in the enforcement path for crawler blocking. The WAF filters incoming requests before later AI Crawl Control functions are applied.
That does not make a WAF infallible. Sophisticated scrapers can distribute traffic, imitate browser behavior or find signals that a particular detection system does not recognize. The accurate claim is that WAF-based controls provide stronger enforcement and spoofing resistance than robots.txt or simple user-agent rules, not that they guarantee perfect bot exclusion.
Fake user-agents expose the weakness of crawler-name blocklists
Many crawler-control examples start with a named bot: identify GPTBot, ClaudeBot or another agent and deny it. That works only while the requester truthfully identifies itself.
The HTTP user-agent header is not a cryptographic identity. A crawler can claim to be a common desktop browser, another bot or virtually any arbitrary string. If the entire security policy consists of checking whether the header contains a known AI crawler name, changing that string can be enough to pass the rule.
This is the strongest argument for layered bot management when content protection is a genuine requirement. Network information, verified-bot lists, request behavior and rate patterns can provide evidence that is harder to falsify than a single text header.
Even then, detection is probabilistic. A well-designed scraper may look sufficiently human to avoid automated classification. Security teams therefore need to think in terms of reducing unauthorized access rather than imagining that one rule creates an impenetrable perimeter.
Not every AI crawler has the same purpose
Blocking “AI” as one category can also be too crude. Some crawlers collect material for model training. Others support AI search and citation. Some requests are triggered directly by a user asking an assistant to open or summarize a particular page.
Those purposes have different implications for publishers. A site may want to prevent training use while remaining discoverable in AI search. Another publisher may want to block automated retrieval entirely. An ecommerce company might actively want its public product information available to shopping agents while protecting other sections of the site.
Pollitt's earlier analysis of whether publishers should block AI crawlers or measure their value first highlights an additional complication: some user-triggered fetchers do not make the same robots.txt commitments as their companies' training or search crawlers. A policy that assumes every bot associated with one AI company behaves identically can therefore fail.
The first decision is not technical. It is deciding what forms of access the business wants to allow.
robots.txt still has an important role
The fact that robots.txt is voluntary does not make it useless. It remains simple, transparent and widely understood. SEO teams can usually change it without redesigning infrastructure, and reputable crawler operators can use it as a clear expression of publisher intent.
It also supports granular rules. A site can allow a crawler generally while excluding a sensitive directory, or block one named crawler without affecting traditional search engines.
For a publisher whose goal is simply to tell well-behaved AI companies not to crawl for a particular purpose, robots.txt may be sufficient. Adding a complex WAF policy can create unnecessary engineering overhead if the relevant bots already respect the file.
The mistake is treating that convenience as proof of enforcement. robots.txt works because the other party agrees to follow it.
Enforcement creates its own risks
Infrastructure-level blocking is more powerful precisely because mistakes have immediate consequences. A malformed robots.txt rule can accidentally discourage crawling, but an overly broad firewall rule can actively prevent legitimate services or users from receiving content.
Bot classification can also produce false positives. Automated monitoring, accessibility tools, partner integrations, legitimate search crawlers and security services may share characteristics with unwanted automation. Blocking based on aggressive behavioral thresholds can catch traffic the business actually needs.
Changes at the WAF or CDN layer therefore need testing, logging and an owner who understands the site's infrastructure. Pollitt notes that these systems often sit outside an SEO team's direct control, creating a need for coordination with developers or security teams.
This is not a reason to avoid enforcement. It is a reason to treat crawler blocking as infrastructure policy rather than a quick SEO checkbox.
Logs tell you whether your policy is doing anything
One advantage of server, CDN and WAF controls is observability. These layers can record which requests were blocked, challenged or allowed. That creates evidence about the volume and identity of automated traffic reaching the site.
Server logs are equally useful when a publisher relies primarily on robots.txt. If a named crawler has been disallowed but requests from that agent continue to reach protected URLs, the logs can reveal the discrepancy.
Monitoring is particularly important because bot ecosystems change. New user-agents appear, providers modify crawler behavior and attackers change evasion techniques. A blocklist created once and never reviewed will gradually become less representative of the traffic hitting the site.
The policy should therefore have a feedback loop: define which access is acceptable, enforce it at an appropriate layer, observe the resulting traffic and revise the controls when reality diverges from the policy.
Blocking AI crawlers can also reduce AI visibility
The decision is not purely defensive. Preventing an AI search crawler from retrieving a site may reduce the likelihood that current information from that site can be surfaced, summarized or cited in AI experiences.
For publishers worried about uncompensated training, that tradeoff may be acceptable. For brands trying to increase visibility in ChatGPT, Claude, Perplexity or other answer engines, blocking every AI-related agent could work directly against the marketing objective.
Training, search retrieval and user-triggered access should therefore be evaluated separately where the provider and infrastructure allow that distinction. A universal “block AI” rule is easy to understand but can discard potentially valuable discovery channels along with unwanted crawling.
There is no technically correct policy independent of the business model. The technical stack only determines how reliably the chosen policy is enforced.
The best architecture is layered
If a publisher has a strong requirement to prevent certain crawlers from accessing content, Pollitt recommends moving as high in the server stack as practical: WAF where available, CDN where appropriate and server controls when those higher layers are unavailable.
That does not require abandoning robots.txt. The layers can complement each other. robots.txt communicates intent to cooperative crawlers. Managed CDN controls stop known unwanted traffic before it reaches the origin. A WAF adds behavioral inspection and stronger protection against basic spoofing. Server rules provide another enforcement point.
The exact combination depends on the site's architecture, budget and tolerance for maintenance. A small site may reasonably rely on robots.txt for reputable bots. A publisher facing aggressive scraping has a very different threat model.
The key is to stop confusing communication with access control. A disallow directive tells a crawler what the publisher wants. A firewall rule determines what the crawler is actually allowed to receive.
Robots.txt asks; infrastructure enforces
The robots.txt debate is sometimes framed as though one method has replaced the other. In reality, they solve different problems. robots.txt is an established protocol for cooperative crawler governance. Server, CDN and WAF controls are enforcement mechanisms.
For responsible AI crawlers, the lighter mechanism can work perfectly well. For bots that ignore directives or disguise their identity, it cannot provide a technical barrier because it was never designed to do so.
A WAF is currently the strongest of the commonly available layers discussed here because it can inspect more than a self-declared crawler name. But even a WAF is not an absolute guarantee against sophisticated automation, and every stronger block introduces operational and visibility tradeoffs.
The practical rule is therefore simple: use robots.txt when you need to state a preference to bots you trust to cooperate. When access itself must be prevented, enforce the decision at the network or server stack — and monitor the logs to make sure the door is actually staying closed.