Robots Devoured the Internet While We Slept. And That's Not a Metaphor

A developer of the charity database PatronView found that 99% of the site's visitors are not people: of 1.28 million served pages, less than six thousand were viewed by humans. He calculated that for every page seen by a person, there are about 214 downloads that nobody ever sees. The site experienced a storm of 3.6 million requests in one day from 361,844 unique IPs, mostly from China, with many bots passing Cloudflare's captcha. His solution was to block entire countries at the edge, which reduced the influx. The article introduces a key metric: how many pages a bot downloads per visitor it sends back. For Anthropic's Claude-SearchBot, the ratio was 42,680 pages to 12 visitors in a week. Similar issues affected The Numbers, a movie database founded by Bruce Nash, which went offline for a week and returned significantly reduced after targeted scraping attempts. The site was a target for prediction-market traders seeking early box-office data. The article also notes that Google's AI Overviews appear in 43% of searches, reducing click-through rates by a third and pushing the share of zero-click searches above 60%. Publishers are reconsidering deals with Google, and a German court ruled Google responsible for false statements in its AI overviews. Content provenance is becoming important, with standards like C2PA, but implementation is inconsistent. Websites are fighting back with firewall rules and even a font called ShieldFont that shows different text to humans and bots. The article concludes with a warning about AI-driven cyberattacks, citing an Anthropic report that a state-sponsored group used its coding tool to attack about 30 organizations, with the AI performing 80-90% of the work.