There is no robots.txt for people · personal data in a scraped corpus

19 August 20267 minute readLaw cited as of 19 August 2026

Lex, a small robot with a large LEX name plate across his front, stands on a path in an empty public park with one hand on the arm of a weathered wooden bench, leaning toward the small engraved memorial plaque fixed to its backrest. — The park is open to everyone. The name on the bench belongs to somebody.

Every article in this series so far has treated a page as a work: who may copy it, who may mine it, what a sign on it means. That frame has a hole in it. A page can be lawfully crawled, lawfully mined, and still unlawful to process, because some of what is on it is a person, and copyright law has never had anything to say about people.

We ran into this the moment we tried to publish anything drawn from agency records logs. UnGovr puts out aggregate reports on who files public records requests, built from the logs agencies publish, and those logs are full of people. The publisher that produces a report will not run if a subject names a private individual, and an organisation has to be on an approved list before it can appear at all. Categories and named organisations go out. Nobody else does. That gate is code rather than policy, because a policy is not a thing that refuses.

Two regimes, one page, neither answering the other

A crawl of a European page can raise both questions at once, and they run in parallel. The copyright analysis asks whether an exception covers the copy you made and whether rights were reserved. The data protection analysis asks whether you had a lawful basis to process personal data at all. A yes to the first is not evidence toward the second.

Answering everything on the top half tells you nothing about the bottom half. That is the whole reason this article exists.

The most expensive misunderstanding in this area is the belief that "publicly available" is itself a permission. Under the GDPR it is not a lawful basis, not an exemption, and not a defence. It is a fact about how you obtained the data, and it goes into the balancing test rather than around it.

What the regulator said six weeks ago

On 7 July 2026 the EDPB adopted Guidelines 03/2026 on web scraping in the context of generative AI, the first document to address this practice directly rather than by implication. It is version 1.0 and open for public consultation until 30 October 2026, with a final text not expected before the end of the year, so it is guidance rather than settled law and it may move. It also builds on Opinion 28/2024, which the Board issued in December 2024 at the request of the Irish supervisory authority.

Three points from it matter to anyone who operates a crawler.

Consent is not available to you, and that is not a loophole
A scraper has no relationship with the people whose data it collects and cannot obtain consent at scale, so consent is effectively ruled out. That leaves legitimate interests as the only realistic basis, which sounds like good news and is not: it is the basis with a test attached.
The test has three parts and the third one is the hard one
A genuine and clearly articulated interest. Necessity, including that no less intrusive alternative would do. And then the balance against the impact on people, weighing the nature of the data, the context in which it was published, the effects, and what those people could reasonably expect. "They put it on the internet" is an argument about the third limb, not a way of skipping it.
Being excused from telling people individually does not excuse telling them
Article 14 requires informing people whose data you collected from somewhere other than them, and Article 14(5)(b) excuses that where the effort would be disproportionate, which is where every scraper hopes to live. The Board's position is that the carve-out relieves you of individual notice and nothing else: you still publish a detailed notice, you still make rights exercisable, and you are expected to offer a way to opt out before collection.

That last item is the one worth reading twice, because it is the first time anything in this series has asked a crawler operator to publish a mechanism rather than to read one. The reservation formats in the previous article are things a publisher puts up for you to find. This is a thing you put up for people to find.

The enforcement record is unusually clear

Most of the law in this series is thin, recent and contested. This corner is not. The Clearview AI decisions are a consistent line across several European regulators, all reaching the same conclusion about the same argument: collecting images from public web pages and processing them is not made lawful by the pages having been public. Italy's regulator fined the company 20 million euros in 2022, France's imposed 5.2 million, and the Dutch authority added 30.5 million in 2024, taking the European total past 100 million.

The reach question was answered separately and recently. In October 2025 the United Kingdom's Upper Tribunal upheld three of the Information Commissioner's four grounds of appeal, concluding that Clearview's processing related to monitoring the behaviour of United Kingdom residents and therefore did not fall outside United Kingdom data protection law. That is the holding to notice if you are outside Europe and assumed this did not apply to you.

Where other regimes genuinely diverge

This is not a global standard wearing different names, and one divergence is sharp enough to change engineering decisions.

California's Consumer Privacy Act excludes publicly available information from the definition of personal information altogether, covering both information lawfully made available from government records and information a person has made available to the general public. The exact thing that is merely a factor in the European balancing test is a threshold exclusion in California. But read the qualification, because it is aimed squarely at this use: information is not publicly available if it is used for a purpose incompatible with the one for which the government made it available. A crawler collecting official records for a purpose the publishing agency did not have in mind is precisely the case that qualification describes.

Brazil's general data protection law and South Korea's personal information law both sit closer to the European model than the Californian one, with lawful-basis analysis rather than a public-information exclusion. If you are designing one pipeline for all of them, design to the European test and treat California as the place where you may do more, never as the baseline.

Government pages are the hard case, not the easy one

There is an intuition that public sector data is exempt because it is a public record. It is not, and the overlap is where the genuinely difficult questions live. A councillor's name attached to a vote is personal data that they can have no expectation of controlling. A complainant's name in the same minutes, or an applicant's address in a planning file, is personal data published for a specific purpose by an authority under a specific duty, and the fact that the duty made it public does not transfer that purpose to you.

What we take from this for our own crawling is not a clever exemption but a design constraint: the fact that a page is reachable settles nothing about the names on it, the purpose the authority published them for is a fact worth recording alongside the data, and the answer changes at the border. It is the reason our per-jurisdiction notes exist at all.

That closes the season. Five articles, and the shape they make is that almost nothing about crawling is decided by the crawl. It is decided by what you do next, who you did it to, and where they live. The Law pages carry the per-jurisdiction posture we hold today, and LexLint itself will read your app's declared profile and report which of these attach to it.

Sources and further reading

EDPB, Guidelines 03/2026 on web scraping in the context of generative AI Adopted 7 July 2026, version 1.0, open for consultation until 30 October 2026. The first direct treatment of this practice, and the document to re-read when the final version lands.
EDPB Opinion 28/2024 on data protection and AI models December 2024, and the framework the 2026 guidelines build on. Also the source of the anonymisation analysis, which decides whether a trained model is itself personal data.
GDPR Article 6(1)(f), legitimate interests The basis a scraper will end up relying on, and the three-limb test the guidelines apply to it.
GDPR Article 14, information to be provided where data is not obtained from the data subject Including the disproportionate-effort carve-out in Article 14(5)(b), and the conditions attached to it in the same paragraph.
Information Commissioner's Office, Upper Tribunal judgment on Clearview AI October 2025, on whether a company outside the United Kingdom scraping data about United Kingdom residents is within reach of the law. It is.
Dutch Data Protection Authority, fine imposed on Clearview AI September 2024. The clearest single statement that scraping public pages is not a lawful basis, from the regulator that imposed the largest of the fines.
California Consumer Privacy Act Where the publicly-available exclusion sits, and the incompatible-purpose qualification that limits it for government records.
EDPB weighs in on key questions on personal data in AI models A readable summary of the 2024 opinion, useful if the primary documents are more than you need.

This article is a research summary of published law and regulatory guidance, with every source linked above. It is not legal advice and it creates no attorney-client relationship. Law cited as of 19 August 2026; the EDPB guidelines described are in draft and open for consultation, and their final text may differ.