TDM Reservation Protocol: opting out of AI training
Blocking a crawler and opting out of AI training are different problems. Here is what the TDM Reservation Protocol is, how it works, and whether AI companies honor it yet.
Blocking GPTBot in robots.txt stops that crawler from fetching your pages at all. But blocking access and opting out of AI training are two different requests, and until recently there was no standard way to make the second one without also making the first. The TDM Reservation Protocol is an attempt to close that gap.
What the TDM Reservation Protocol actually is
TDMRep, short for the Text and Data Mining Reservation Protocol, is a W3C Community Group specification. It is not a law, and no major browser or AI vendor is required to follow it. What it does is give a site owner a machine-readable way to state whether content can be used for text and data mining, the process behind training most AI models, separately from whether a crawler is allowed to fetch the page.
The idea traces back to the EU's Copyright Directive (2019/790), which lets rights holders opt out of commercial text and data mining, but only if that opt-out is expressed in a machine-readable format. TDMRep is one proposed answer to what that format should look like. It has real backing from publishers and standards groups, but it is still early, and adoption by the companies actually doing the mining is thin.
How it differs from robots.txt and llms.txt
robots.txt controls whether a crawler can request a URL in the first place. llms.txt, where sites use it, points AI systems at a curated summary of the site. Neither one says what happens to the content after it has been fetched. TDMRep sits above both: it lets you allow a crawler to read a page while separately declaring that the content is not licensed for training.
- robots.txt answers: can this bot request this URL at all
- llms.txt answers: here is a curated map of my site for AI systems to read
- TDMRep answers: here is my policy on mining this content, no matter who fetches it
How the signal gets published
TDMRep defines two ways to publish a reservation. One is a link, placed in an HTTP header or an HTML tag, pointing to a policy document. The other is a policy file placed at a well-known location, such as /.well-known/tdmrep.json. The policy itself is a small, machine-readable document stating whether mining is allowed, and it can set a general default while carving out named exceptions for specific uses or partners.
Whether AI companies actually honor it
This is the part to be honest about. The major AI crawlers, including OpenAI's GPTBot and Google's Google-Extended, currently document robots.txt disallow rules as their opt-out mechanism, not TDMRep. Publishing a TDMRep policy today does not stop those crawlers from training on your content unless you have also blocked them in robots.txt. The protocol is real and published, but enforcement by the companies doing the training is not there yet.
Where it does carry weight is as a documented legal position. If a dispute ever turns on whether you expressed a machine-readable opt-out under EU law, a dated, versioned TDMRep policy is evidence that you did. That is a legal and record-keeping value, not a technical block.
What to actually do about it
- Keep robots.txt as your real control for crawl access. Disallow the AI bots you do not want reading your site at all.
- If you operate in or serve users in the EU, or you work with legal counsel on copyright exposure, add a TDMRep policy file as a documented opt-out signal, and treat it as a compliance record rather than a blocking tool.
- Do not loosen or remove robots.txt rules just because you have added a TDMRep policy. They do different jobs and neither substitutes for the other.
- Check back on vendor adoption every few months. The moment a major AI company starts honoring TDMRep in practice is the moment this shifts from a paper trail to an actual control.
TDMRep names a real gap: a way to say 'you can crawl this, just do not train on it.' It is not yet a lever that stops training on its own.
Until adoption catches up, do your access control in robots.txt first. Add a TDMRep policy alongside it if you have a genuine copyright or compliance reason to, and treat it as documentation of your position, not a wall around your content.