2026-11-19»
Just who are the AI scrapers, bringing down everybody’s websites?»So, I’ve been puzzled by the AI scraping problem for a while. Websites and API endpoints around the world are getting hammered — and this affects open free-to-use systems in particular. It was one of the many tensions that pushed so many open source developers into a hard anti-AI stance. As Matthew Somerville, one of the smartest and most dedicated public interest technologists I know, said in reply to my Spoons post:
I’m afraid I don’t see how your spoons can protect the others from eg the solipsistic mindlessness of the scrapers
— Matthew Somerville (@dracos.co.uk) November 18, 2026 at 9:44 AM
But … why? Why are these new scrapers worse than the previous scrapers, which sites have been handling/mitigating for many years now?
One of the arguments I’ve seen is that the majority of previous bots were from search engines like Google; Google’s bots are polite and gentle because Google is scraping content to link back to it. If the bots are so eager that they bring down the site they’re scraping, the links won’t work. So search engines and websites have some mutual interest in keeping the sites working. AI bots are just scraping to extract the data; they don’t care whether the site itself lives or dies.
I was initially sceptical of this. I read it as being too close to the “they do it because they are bad people” argument. Even poorly aligned actors don’t usually sit around cackling going “What do we care if we bring down the site? We’re not a fricking search engine, bwahahaha”. Hammering websites is inefficient; bringing the site down stops you from accessing their data right then and there; plus if you’re polite, you can slink around undetected rather than get people so annoyed with you they start sticking anime jackal-girls in front of their sites to fend you off.
This critique seems to be aimed at the large, well-known frontier labs: OpenAI, Anthropic, ByteDance, Meta. But I’m particularly confused as to why these companies in particular would differ in their behaviour from the previous generation of botherders. Many of the people at these newer companies would have come from the previous generation; they’d be using the same webspidering approaches by default (hell, Meta is probably using the same internal team).
Sure, we all have GPTBot, ClaudeBot and Bytespider all over our logs. But the traffic that maintainers describe as unmanageable doesn’t say who it is: it comes from tens of thousands of residential addresses, pretending to be ordinary browsers. And in that kind of environment, I’m not even sure we can trust the labelled bots as being who they say they are.
I don’t put it past them, but politically, it just seems like a terrible idea for the U.S. companies to be being bad netizens right now. Bad bot behaviour is already being used as political leverage against the AI labs. In New York, Bill S9934A — passed in June, and waiting on the Governor’s signature — would give the NY Attorney General the power to sue botherders who don’t declare who they are, and what they plan to do with the data, to the news sites they spider. This, of course, won’t do anything to stop misbehaving bots, but by making every law-abiding crawler announce itself, it will let the New York Times block OpenAI and Anthropic until they pay it for a licence for their non-paywalled content. There’s been a similar bill in Congress since July, and its Senate version, the Stealth Bot Prohibition Act, was introduced this week:
“The Chinese Communist Party is notorious for stealing Americans’ intellectual property by any means possible. Now, it is weaponizing artificial intelligence and deploying droves of malicious bots against online publishers”
…said Senator Wicker, introducing a bill that will be utterly ignored by “stealth” bots — but will give those online publishers a way to pick and choose who reads their content online. NetChoice, EFF and others have come out against these bills, but much of the tech industry has kept quiet, mostly because while they would be unsympathetic to the idea of publishers once again trying to legislate a business model into existence, they really hate those bots. (And if you’re worried that the UK is losing its lead in bad Internet laws, fear not: they are doing their bit too.)
All of that said, these bots do exist, and there must be some change in the incentives or manners of their operators to make them suddenly bad for websites. Where does their solipsistic rudeness come from? And most confusing for me, why is so much of this traffic (according to Cloudflare, which can only count the bots it can identify) coming from bots taking data for training, rather than the more understandable organic explosion in traffic by millions of AI users who are now horsewhipping their agents to act on their behalf online, faster and more broadly than they would have been able to before?
I’m still trying to research this (with my Terminal Values hat on, I spent this week bugging Alliance for Responsible Data Collection members to find out more from their side).
My current, still rather uninformed, theory of what’s happening is this:
Start-ups, primarily in the US and China, have received a great deal of capital in short order to work on LLM models. One of the more obvious pitches by such start-ups for that money is that they will build their own proprietary model, which will have some unique advantage, that they will then mine for revenue. A lot of money went to these companies in aggregate.
Those companies now have a problem: they actually need to build those models. The resource constraint currently is people who have the knowledge to create a working model. We know this, because those people command insane salaries. But whether you have access to a model expert or not, the one thing you absolutely know you need — possibly, bitter-lesson-style, the only thing you need — is data.
So what do you do? You look around for bulk sources of data. The established orthodoxy is that quantity is a more important metric than quality. You will pay top dollar for raw data, because you have a lot of money, and you don’t really care that much about what’s in it (or how the data has been obtained). So you look for other companies, offering petabytes of data off the shelf to feed your training process.
Enter the scrapers: these are companies who see a huge — but temporary — revenue opportunity. They’re being paid by the byte, more or less (presumably no-one is going to pay them for 1PiB of zeros, so there are probably some entropy/uniqueness criteria), so the metric they are maximising for is as much data as possible, as quickly as possible.
It’s these companies, far down the food chain from the VCs and the larger AI companies, who are running the dodgy scraping programs. There’s certainly no incentive for them to be civil: it’s a gold rush, and the quality of the river silt they’re collecting isn’t their problem. They might be given some categories to focus on — code, say, or government data — but mostly all their data needs to be is a) novel, and b) bountiful.
None of which excuses their customers, who are paying for the data without asking how it was obtained, but it makes it a bit clearer how the incentives are being acted upon.
There are many unexplained corners even with my theory: why aren’t those companies just using the existing open source models at this point, rather than training their own? Why aren’t we seeing a consolidation of these companies into more efficient actors, or at least conspiring with each other to lower their collective costs? I feel like these questions may be answered more by the short amount of time this ecology has existed, and the general stupidity of gold rush economics, than by anything particularly ingrained in AI or the future of modern botdom.
I wonder whether the problem that I do worry about — a world where every Internet user has multiple user-agents at their beck and call, always on, always looking for the best deal or the newest news story — may not prove to be as big a problem as this current generation of AI boom botting.
Nevertheless, the problem still remains. I’m still thinking through good, effective answers, as are millions of site operators. But I’m fairly sure that this raft of proposed Bad Bot Bills will not solve the problem, and will make the Internet less free for everyone else.
