Cloudflare’s Free-Plan AI Crawler Labels Come From a Spoofable Header

Last updated: August 18, 2026

Cloudflare AI Crawl Control reports how requests grouped as AI crawler traffic interact with a zone. On the Free plan, Cloudflare identifies those crawlers from the user-agent string each request declares about itself. A user-agent is an HTTP header. Cloudflare’s own API reference annotates user-agent filtering with the words “Can be spoofed.”

Verdict in one line: the Free-plan crawler table is a reliable way to find traffic worth investigating, an unreliable way to name who sent it, and no way at all to establish why a request failed.

What the Free-plan label establishes

A user-agent is a header the client writes. Many crawlers publish stable strings so site owners can recognize them, and Cloudflare uses those strings to fill the crawler and operator columns on the Free plan.

That mechanism answers one narrow question. Which requests matched a user-agent pattern associated with a known crawler?

It leaves three larger questions open. Did the request come from infrastructure the named operator controls. Did that operator, rather than a person using its product, choose the URL. Did the request cause the status code sitting next to it in the table.

Cloudflare’s GraphQL reference for AI Crawl Control marks the two filters differently. Filtering by user agent is available on all plans and carries the note “Can be spoofed.” Filtering by detection ID is described as reliably verified by Cloudflare and is limited to Bot Management.

Detection IDs are a stronger signal, and they are not a certificate of corporate intent either. Cloudflare’s documentation describes detection IDs as covering verified-bot detections, heuristics, and anomaly detections, and a single request can match more than one. The upgrade buys better classification. It does not buy proof of who decided to send a request or why a server answered the way it did.

Sources: Cloudflare, AI Crawl Control GraphQL API reference · Cloudflare, Detection IDs · Cloudflare, Bot reference, 23 April 2026

What one zone displayed in 24 hours

We read our own Free-plan zone on 18 August 2026 between 13:19 and 19:00 JST. The date control read “Last 24 hours (JST)” for every capture, which means each reading covered a rolling window ending at the moment we opened the page.

In the view captured at 17:00 JST, AI Crawl Control displayed roughly 4.05k requests grouped as AI crawler traffic. The status breakdown showed about 1.98k in the 2xx range, 1.67k in the 5xx range, 349 in the 4xx range, and 51 in the 3xx range. FSR arithmetic on those figures puts the 5xx share near 41 percent. The dashboard abbreviates its larger numbers, so that percentage is approximate and we have not exported exact integers.

The 5xx line was not flat. Two visible spikes rose from a near-zero floor, one shortly before 22:00 on 17 August and one shortly after 02:00 on 18 August, each reaching roughly 450 on the hourly chart.

The path view, filtered to 5xx, listed file names that do not exist on this WordPress install. Five representative entries: /backend/.env, /admin/.env, /serviceAccountKey.json, /application.properties, and /actuator/env. The list ran to sixteen pages at ten rows per page. Every path on the first page showed the same request count of 24.

Sources: Future Stack Reviews, Cloudflare AI Crawl Control, captures timestamped 18 August 2026 13:19 to 19:00 JST

Why identical path counts are a lead and not a finding

Identical counts across unrelated paths are unusual enough to investigate. They are not enough to conclude what produced them.

At least five explanations fit the same display. A tool walking a fixed list at a fixed rate. A scheduled batch that ran a set number of times. Client retries after each failure, converging on a similar total. Pagination or aggregation in the dashboard query grouping several sequences into one row. Sampling, which Cloudflare applies to its adaptive analytics datasets and which produces estimates rather than exact counts.

Distinguishing those requires request-level rows joined by an identifier, which the Free-plan table does not display. We are recording the pattern as a lead and naming what would settle it, rather than assigning it to a scanner we did not identify.

Sources: Future Stack Reviews, path view capture, 18 August 2026 · Cloudflare, GraphQL Analytics sampling

Where the failures were grouped

Switching the grouping control from crawler to hostname produced the most useful observation of the session.

The hostname that serves the articles does not appear among those rows. It appears instead at the top of the data transfer view, carrying 111.72 MB of roughly 129 MB moved to AI crawlers, which FSR arithmetic puts near 86 percent.

Two zone settings are relevant, and we are stating them as observations rather than as an explanation. The zone carries a proxied wildcard A record, which can answer for a hostname when no more specific record exists. The encryption mode is Full (Strict), unchanged for 108 days. Our certificate inventory at the origin listed the apex hostname and one staging hostname, and did not list the subdomains above.

Those facts make an origin TLS path worth testing. They do not identify what happened. Section 5 explains why.

Sources: Future Stack Reviews, AI Crawl Control hostname and data transfer views, DNS records, SSL/TLS overview, and origin certificate inventory, all 18 August 2026 · Cloudflare, Wildcard DNS records

Why the exact status code decides the question

The dashboard grouped these responses into a 5xx bucket. It did not show which 5xx code was returned, and we did not retrieve per-request status fields. That gap is the reason this briefing does not name a cause.

Cloudflare separates two failures that a missing certificate could plausibly produce, and they are not the same event.

Error 525 means the SSL handshake between Cloudflare and the origin failed. Error 526 means Cloudflare could not validate the certificate the origin presented while Full (Strict) is set, and Cloudflare’s support documentation lists a requested hostname absent from the certificate’s Common Name or Subject Alternative Name among the conditions to check.

Read plainly, a hostname that a certificate does not cover points toward 526 rather than 525. We are not asserting that either. Both codes require the origin to have been reached, and several other paths produce a 5xx without the certificate mattering at all.

We have published the test rather than the conclusion because the test is cheap and the conclusion is not recoverable after the fact. Anyone reading this on their own zone can run it in ten minutes. We could not, because the requests had already passed out of the window we could query at request level.

Sources: Cloudflare, Error 525 · Cloudflare, Error 526, 23 April 2026

What the second surface added

We opened Security Analytics for the same zone at 17:18 JST. It showed 20.03k total requests, 3.35k served by Cloudflare, 16.66k served by the origin, 9 requests mitigated, and a suspicious activity count of zero.

The tempting reading is that two Cloudflare dashboards disagreed about the same day. That reading is wrong, and the numbers on the page show why.

The 20.03k figure is close to the total request count on the traffic analytics page for a comparable period. Security Analytics is not a filtered view of what Cloudflare acted on. It reports incoming HTTP requests, including requests no security product touched. The surface dedicated to requests that Cloudflare flagged or actioned is Security Events, which is a different page.

The suspicious activity counter is narrower still. It reports requests identified by the detections that are enabled on the zone. A zero there means those specific detections did not identify traffic in that category. It is not a verdict that the traffic was benign, and it is not independent confirmation that the traffic was hostile.

The operational lesson is about surface semantics rather than vendor error. A reader who opens one page and draws a conclusion about the other is joining two datasets that count different populations over overlapping windows. Our two captures were taken eighteen minutes apart, both on rolling windows, so they do not even cover an identical interval.

Sources: Future Stack Reviews, Security Analytics capture, 18 August 2026 17:18 JST · Cloudflare, Security Analytics · Cloudflare, Security Events

Where Cloudflare’s current bot mapping lives

Cloudflare currently describes AI bots with more than one set of names, and a site owner meets several of them inside the same product.

The security settings screen offers three behaviors: Search, Agent, and Training. The AI Crawl Control bot reference table uses four category labels: AI Crawler, AI Search, AI Assistant, and Search Engine. The July 2026 announcement describes a wider BotBase taxonomy of eleven behaviors.

The mapping between these is published. Cloudflare explains that Search, Agent, and Training form the updated behavior taxonomy, that a single bot can carry more than one behavior, and that the older category names are retained for backward compatibility. The bot reference table itself points to the Cloudflare Radar Bots Directory for an up-to-date list.

What we can say is narrower than “there is no table.” The static reference page a site owner lands on carries legacy categories, while current per-bot behavior lives on a separate public surface. The information exists. The path to it is split across pages, and the page most operators will open first is not the current one.

If you are deciding whether a Training block affects a particular crawler, the static table is the wrong place to look. Check the Radar directory record for that bot, and record the date you checked, because behavior assignments can change.

Sources: Cloudflare, Bot reference, 23 April 2026 · Cloudflare, Bots concepts, 1 July 2026 · Cloudflare Radar, Bots Directory

What Cloudflare has announced for September 15

On 1 July 2026 Cloudflare announced new defaults taking effect on 15 September 2026. Bots classified as Training or as Agent are to be blocked on pages that display ads, while Search remains allowed.

Two Cloudflare sources describe the scope differently, and we are preserving both rather than choosing. The developer changelog frames the change around new domains onboarding to Cloudflare. The press release adds new customers, new sites belonging to existing customers, and existing Free customers who have not changed their dashboard settings by that date.

Cloudflare defines the Training category to include mixed-purpose crawlers used for both Training and Search, and states that these will be blocked by every configuration that blocks AI training, including the legacy Block AI bots option. Whether the ad-page default alone produces that outcome for a site owner who takes no action is not stated in the documentation we read. We are recording it as an open question rather than answering it.

The announcement is also not final. The press release states that Cloudflare will engage with the ecosystem, listen to feedback, and run tests to finalize new defaults and classifications with a deadline of 15 September 2026, and its forward-looking statements section names bot defaults explicitly. Anything written today about how that date will behave describes an announcement rather than a shipped product.

Sources: Cloudflare, Your site your rules, 1 July 2026 · Cloudflare, Press release, 1 July 2026 · Cloudflare, Block AI bots, 1 July 2026

What each surface and plan can answer

Most confusion in this area comes from asking a surface a question it was never built to answer. Two tables separate the questions.

One clarification about the referral row, because it is easy to over-read. Referral metrics count visits arriving from AI operator domains. They do not count citations, and a page can be cited in an AI answer without producing a visit. Buying a paid plan buys a visit count, not a citation count.

Sources: Cloudflare, Analyze AI traffic · Cloudflare, GraphQL API reference · Cloudflare, Security Analytics · Future Stack Reviews, Free-plan dashboard, 18 August 2026

What to export before the window moves

The most useful thing in this briefing is not a finding. It is the order of operations we should have followed and did not.

A rolling 24-hour view is not a dataset. Two captures taken two hours apart cover different intervals, and joining their numbers produces a total that never existed. Before analyzing anything, pin a fixed interval and pull the same interval on every surface.

We lost the ability to answer the interesting question because we investigated before we exported. That is the transferable lesson, and it costs nothing to apply.

Sources: Cloudflare, Confidence intervals · Cloudflare, Sampling

What we did not test

Every number in this briefing came from a dashboard that abbreviates its larger values. Percentages derived from those values are FSR arithmetic and approximate. Anyone reproducing this work should export exact integers rather than reading ratios off a rounded display.

Sources: Future Stack Reviews, session record and screenshot set, 18 August 2026

FAQ

Methodology

Future Stack Reviews operates the Cloudflare zone described here on the Free plan. Every figure was read from that account on 18 August 2026 between 13:19 and 19:00 JST and captured as timestamped screenshots.

Each capture used the dashboard’s rolling “Last 24 hours” window, which ends at the moment the page loads. Readings taken at different clock times therefore cover overlapping rather than identical intervals. Our AI Crawl Control status capture and our Security Analytics capture were eighteen minutes apart, and other comparisons in this briefing span up to two hours. Where we place two figures side by side, that gap applies.

Cloudflare states that Security Analytics uses sampled data except when displaying raw logs, and that its adaptive analytics datasets may return estimates. We did not retrieve confidence intervals, sample sizes, or exact integers. Figures presented in thousands are the dashboard’s own abbreviations, and every percentage here is FSR arithmetic derived from them.

We did not change any setting on the zone during or after the observation. DNS records, encryption mode, AI bot policies, and robots.txt configuration are as they were before this work began.

Documentation was read at the URLs linked in each section on 18 August 2026. Two pages, the Error 526 support article and the Radar Bots Directory, were confirmed through search results and vendor-hosted index files rather than by opening the rendered pages ourselves; readers should open both directly.

This briefing depends on measurement surfaces built and operated by the vendor being audited. Cloudflare supplies the instrument, defines the categories, sets the retention window, and controls the sampling. We disclose that dependency because we did not preserve independent request-level telemetry for the window described, and after the fact it could not be recovered.

Future Stack Reviews accepts no payment for coverage and places no affiliate links in audit briefings. No commercial relationship exists between Future Stack Reviews and Cloudflare.

Sources: Future Stack Reviews session record, 18 August 2026 · Cloudflare, Sampling · Cloudflare, Security Analytics

Verdict

Read the Free-plan crawler table for what it reports: which requests carried which self-declared name, in what volume, against which paths and hosts. That is a genuine capability and it is free.

Do not read it as an answer to who or why. The name is a header. The status column is a bucket. Neither field carries the evidence that would support an accusation or a root-cause claim, and on the Free plan the fields that would are either absent or expire before most people think to look.

Our own session is the demonstration. We spent a day building an explanation for a failure pattern and finished without the two values that would have settled it, because we analyzed the dashboard before we exported from it.

For 15 September, the preparation that survives every open question in this briefing is not a setting change. Pin a fixed interval, export every grouping from it, capture both status fields, pull the matching origin logs, and store the result outside the platform. Whatever Cloudflare ships on that date, you will be comparing it against something real.