LexLint

The law for AI, agents, crawlers, and scrapers

From robots.txt to agent identity: the controls sites use, and when the law is behind them

18 August 202610 minute readLaw cited as of 18 August 2026

A small blank wooden sign on a stake, standing at the edge of a wide mown lawn under a flat overcast sky.
A sign, on a lawn. In December 2025 a United States court reached for exactly this comparison to decide what a robots.txt file is. Illustration generated with AI.

A crawler control is a request. For thirty years the whole mechanism rested on the crawler choosing to read it, and mostly that was enough. What changed is not the file. It is who is reading it, and what a court will do about it.

We have a working interest in the answer. UnGovr documents governments across 200+ countries, which means crawling public sector sites at a scale of millions of pages in a busy month, under every legal regime those governments sit in. So we kept notes on what each jurisdiction asks of a crawler, and of a model trained on what it collects. The notes became a researched corpus, and the corpus became LexLint. This is the first article drawn from it.

A convention, and it said so

Martijn Koster proposed robots.txt on the www-talk mailing list in February 1994, after a badly behaved crawler caused trouble on a server he ran. The resulting document, A Standard for Robot Exclusion, recorded a consensus reached on a mailing list on 30 June 1994, and was unusually direct about what that was worth.

It is not an official standard backed by a standards body, or owned by any commercial organisation. It is not enforced by anybody, and there is no guarantee that all current and future robots will use it.

A Standard for Robot Exclusion, 1994

Every property of today's crawler controls is already there. It coordinates operators who want to coordinate, it has no enforcement of its own, and it applies to whoever decides to apply it.

The standard that took 28 years

robots.txt was finally written down as a standards-track document in September 2022, as RFC 9309, by Koster with three Google engineers. The IETF publishing it did not make it binding, and the document is careful about the distinction. Participation is a request: it specifies rules "that crawlers are requested to honor". The rules themselves, once a crawler has chosen to participate, are mandatory: "if the crawler successfully downloads the robots.txt file, the crawler MUST follow the parseable rules". The standard is strict about how to obey and silent about whether to. It also forecloses the reading that would have made it a security control.

These rules are not a form of access authorization.

RFC 9309, Robots Exclusion Protocol

That sentence does a great deal of work in the legal section below, and the people who know the protocol best wrote it.

One file became a shelf of signals

A single allow-or-deny decision stopped being expressive enough some time ago, and the response has been specialisation rather than replacement. Crawling split from indexing: a robots.txt rule stops the fetch, while noindex and the X-Robots-Tag header stop the listing, and only work if the crawler was allowed to fetch the page and read them. Purpose split by user-agent token: OpenAI, Anthropic and Google each publish three, for a bulk training crawler, a search indexer, and a live agent fetching because a person just asked something. Blocking one does not block the others, which is the point of having three.

The third split moved from access to use, which is where copyright arrives. The TDM Reservation Protocol (a 2024 W3C Community Group report, expressly not a W3C standard) defines a tdmrep.json file for reserving mining rights; Cloudflare's Content Signals Policy adds a Content-Signal line inside robots.txt carrying search, ai-input and ai-train; Really Simple Licensing (RSL) attaches licence terms, including prices. Then there are ai.txt and llms.txt, single-organisation proposals no standards body has adopted. Widely discussed and standardised are different states, worth keeping apart when deciding what to implement.

Then the crawler stopped being a crawler

robots.txt assumes a crawl: one operator's bot, moving at its own pace, building something durable. An agent fetching one page because a person asked a question thirty seconds ago fits that badly, and the operators say so themselves. OpenAI's documentation, on its user-initiated ChatGPT-User agent:

Because these actions are initiated by a user, robots.txt rules may not apply.

OpenAI, bots documentation

Cloudflare now sorts the space into Search, Training and Agent on the same reasoning, and from 15 September 2026 will block the last two by default on ad-monetised pages for new domains. It has also revived HTTP 402, so a price rather than only a refusal can be an answer.

The deeper shift is from assertion to proof. A user-agent string is a claim anyone can make, as the Web Bot Auth architecture draft says: "User-Agent, when used alone, can be spoofed meaning anyone may attempt to act as that agent". The alternative is a signature. RFC 9421 made HTTP message signatures a proposed standard in February 2024, and the IETF chartered a working group in October 2025 to profile it so a bot can prove which operator it belongs to. A parallel group is consolidating the per-vendor tokens into one vocabulary, and its drafts are worth checking before citing: the vocabulary narrowed from five terms to two between revisions, the current text says it lacks consensus, and the draft defining how to attach it has expired.

Six stages, in order

Strip the vocabulary away and a well-behaved fetch accounts for the same six things it did in 1994, with more places to look at each stage.

What a well-behaved fetch accounts for Six stages a crawler passes through on one request: say who you are, a named agent, a contact URL, a signature; read the rules, robots.txt, sitemaps, per-purpose tokens; look for a reservation, mining opt-outs, content signals, licence; ask cheaply, conditional requests, a rate a site absorbs; back off when told, 429 and 503 with Retry-After are instructions; stop at a wall, a login, a paywall or a challenge is a stop. Six stages What a well-behaved fetch accounts for 1 Say who you are A named agent, a contact URL, a signature 2 Read the rules robots.txt, sitemaps, per-purpose tokens 3 Look for a reservation Mining opt-outs, content signals, licence 4 Ask cheaply Conditional requests, a rate a site absorbs 5 Back off when told 429 and 503 with Retry-After are instructions 6 Stop at a wall A login, a paywall or a challenge is a stop
The order matters: identity and rules come before the request, the reservation governs what may be done with what comes back, and the last stage is the one with the clearest legal consequences.

Why none of it binds by itself

Standards bodies write interoperation, not obligation, and they say so when the subject invites confusion. The AI preferences vocabulary draft: "Preferences do not themselves create rights, obligations, or prohibitions." Cloudflare, on its own policy: "content signals express preferences; they are not technical countermeasures against scraping." Neither is hedging. Both describe the mechanism accurately, which is the publication of an intention that some other body of law may or may not pick up.

Three routes by which it can bind anyway

A signal that creates no obligation of its own can still be the fact that decides a case. Three routes exist, they behave differently, and only one currently works on the passive signal alone.

Contract, and the letter that follows it

Terms of use are ordinary contract, and reach where intellectual property does not: in Ryanair v. PR Aviation (2015) the EU's Court of Justice held that where a database has neither copyright nor database-right protection, the Database Directive's user protections do not apply, so nothing stops the owner restricting use by contract instead. Facebook v. Power Ventures and Craigslist v. 3Taps are cited constantly as United States scraping cases, and what triggered both was an individually addressed cease-and-desist letter plus an address block, then continued access. Not robots.txt. No United States decision holds that robots.txt alone forms a contract or revokes authorisation, and the nearest case runs the other way: Field v. Google (2006) found an implied licence for caching partly because the site owner knew how to exclude the crawler and chose not to.

Computer-misuse statutes, which want a wall

This is the route people reach for first, because these statutes speak of circumventing protective measures. It is also the route that has now been tested on a passive signal and failed. The CFAA turns on access "without authorization", read narrowly by the Supreme Court in Van Buren (2021) as a "gates-up-or-down inquiry" and applied to scraping in hiQ v. LinkedIn (2022): "when a computer network generally permits public access to its data, a user's accessing that publicly available data will not constitute access without authorization" under that statute. LinkedIn still won on its contract claim, which is route one doing work route two would not.

The DMCA anti-circumvention provision looks more promising, and a court reached it directly in December 2025. In Ziff Davis v. OpenAI the Southern District of New York dismissed a section 1201 claim built on a crawler disregarding robots.txt, holding that such files

do not "effectively control" access to that content any more than a sign requesting that visitors "keep off the grass" effectively controls access to a lawn.

Ziff Davis v. OpenAI, S.D.N.Y., December 2025

That is one district court order in live litigation, not settled appellate law, but its reasoning tracks the statutory text elsewhere. Germany's section 202a reaches data "specially protected against unauthorised access, by circumventing the access protection", and the EU directive on attacks against information systems reaches access "committed by infringing a security measure". Each phrasing needs a wall, and a request is not one. Reddit's 2025 case against Perplexity is the one to watch, because it pleads section 1201 against alleged evasion of active blocks rather than of robots.txt, and that is probably where the line gets drawn.

Copyright and mining, where the signal is the hinge

One body of law has deliberately given a machine-readable signal legal force. Article 4 of the EU's DSM copyright directive permits text and data mining, on a condition.

The exception or limitation provided for in paragraph 1 shall apply on condition that the use of works and other subject matter referred to in that paragraph has not been expressly reserved by their rightholders in an appropriate manner, such as machine-readable means in the case of content made publicly available online.

Directive (EU) 2019/790, Article 4(3)

Recital 18 narrows "appropriate" for online content to machine-readable means, "including metadata and terms and conditions of a website or a service". The signal is not evidence of intent sitting beside the law here. It is the element the exception turns on. The AI Act then puts a duty on the other side of it: since 2 August 2025, providers of general-purpose AI (GPAI) models must have a policy to "identify and comply with, including through state-of-the-art technologies, a reservation of rights expressed pursuant to Article 4(3)".

Which raises what counts as machine-readable, and the first appellate answer is narrow. In Kneschke v. LAION (Higher Regional Court of Hamburg, 10 December 2025) a reservation written in ordinary prose in a site's terms did not satisfy Article 4(3), judged against what technology could reasonably parse at the time of the 2021 use; the court noted the answer could differ today, given what language models can now read, and allowed a further appeal. Outside the EU the picture inverts. Japan's Article 30-4 permits mining with no opt-out at all, Singapore's exception cannot be overridden by contract, and the United Kingdom's stays narrow while its government consults on adopting an opt-out model.

What this leaves you with

Ignoring a control is not automatically lawful, and honouring one is not automatically sufficient. Both questions are jurisdictional. Within the EU's reach, a machine-readable reservation is closer to a rule than a request. Under a computer-misuse statute, what matters is whether there was a wall and whether you went around it, which is a question about your conduct rather than about the file. Terms of use bind on their own terms nearly everywhere, whatever the crawler controls say.

Which protective measures carry legal restrictions in which jurisdictions is the subject of the next article in this series, and it is a longer answer than this one. Meanwhile the Law pages carry the per-jurisdiction posture we hold today, and LexLint itself will read your app's declared profile and report which of these attach to it.

Sources and further reading

Implementations that read the rules

Where the edges are being tested

This article is a research summary of published law and published specifications, with every source linked above. It is not legal advice, it creates no attorney-client relationship, and it is not authorisation to access any system. Law cited as of 18 August 2026; the litigation described is live and some of it will move.