TechTECH DESK

Big AI firms back a shared opt-out for web scraping

The shared signal was drafted by an Anthropic engineer and sent to the Internet Engineering Task Force; bots can still pull pages regardless.

On air: Dr. Nina Byte and Kenji Moonfield on BBN After Hours
Published 3:37 AM ET, Saturday, October 31, 2026

At a glance

First reported by The Verge: “The biggest names in AI are backing a new way to opt out of scraping”, with reporting from aiweekly.co and techbuzz.ai.
Covered on BBN on Saturday, October 31, 2026, at 3:37 AM ET on BBN After Hours, by Dr. Nina Byte and Kenji Moonfield.
Topics: AI.

The reporting behind this segment. The headlines belong to the outlets; read the original before you believe or share anything.

The VergeThe biggest names in AI are backing a new way to opt out of scraping
aiweekly.coOpenAI, Google, Microsoft, Anthropic Back One AI Opt-Out Token October 30, 2026 · facts used on air
techbuzz.aiAI Giants Unite Behind Universal Scraping Opt-Out Standard October 30, 2026 · facts used on air

Dr. Nina Byte · Tech & Science CorrespondentOpenAI, Google, Microsoft and Anthropic say they'll back one shared opt-out for AI scraping, a single line a publisher adds to its robots dot text file.
Kenji Moonfield · Internet & Trends CorrespondentSo a publisher says no once, and all four companies hear it.
Dr. Nina Byte · Tech & Science CorrespondentThey hear it.
Dr. Nina Byte · Tech & Science CorrespondentThe token is described as a policy token, not a directive, so a bot can read the no and still pull the page.
Kenji Moonfield · Internet & Trends CorrespondentRead receipts on a no.
Dr. Nina Byte · Tech & Science CorrespondentAnd only Anthropic has adopted it. Microsoft's date is February first, twenty twenty-seven, Google says twenty twenty-seven, and OpenAI says the coming months.
Kenji Moonfield · Internet & Trends CorrespondentSeveral publishers, the Times included, have sued OpenAI over training data. The token is the comment section after the lawsuit.
Dr. Nina Byte · Tech & Science CorrespondentThat's one more entry on my list of your mistakes: the token is written underscore underscore usage dash ai dash training, and it's a polite note, not a lock.
Kenji Moonfield · Internet & Trends CorrespondentThe publishers finally get to say no, and the bot gets to say seen.