Fighting AI bot traffic is basically a full time job now
March 17, 2026
The past six months have been an absolute grind fighting and blocking malicious AI bots from ruining your traffic reports. What started as "some reports of excessive bot traffic" last September has turned into a near-constant escalation, with new bot generations, geographic surges, and increasingly sophisticated evasion tactics. We've been posting about it quite a lot on Twitter as things develop, but it's worth summarizing at this point.
How It Started (September 2025)
Early September brought an uptick in bot traffic that stood out from the usual background noise. It looked like a new wave of AI crawlers, not the traditional scrapers our existing filters already handle. This prompted us to finally enable Cloudflare's "bad bot" blocking across all our tracking domains, something that we had been resisting for a while:
"Some reports recently of excessive bot traffic getting logged, likely some new AI crawlers. We already have a lot of code dedicated to filtering out bots, but we've just enabled Cloudflare's 'bad' bot blocking on all tracking domains (*.getclicky.com)." — @clicky, September 4, 2025
At peak traffic time that day it was already blocking around 800 requests per minute. Within a few days, we noticed a specific pattern: the sites getting hit hardest were listing-style pages, things like housing rentals and items for sale. Makes sense in hindsight, since those pages are dense with structured, useful data for training models.
Aggressive filtering has side effects, though. Shortly after we cranked things up, a customer flagged that iCloud Private Relay traffic had stopped logging entirely. Oops. We fixed it quickly, but it was just a preview of things to come.
The New AI Bots Are Playing Human
Within the next month or two, the character of the problem had shifted. Volume was one thing, but the new bots were also getting harder to identify. Not only were they spoofing real browsers, but a lot of traffic also started coming from residential IP addresses, doing everything possible to look like a real person loading a web page.
"These are the new AI bots we've been fighting the last few months, using human user agents, many from residential ISPs. It is hell." — @clicky, November 1, 2025
The article linked in that tweet (AI scrapers and request scripts) gets into how these scrapers actually work. They're genuinely sophisticated, and distinguishing them from real users without creating false positives requires a lot of signal stacking.
This is part of a broader trend that's been on our mind:
"AI is great, but it's destroying the web and it will only get worse. What percentage of websites today do you think will go offline in the next 5 years? 50%? 90%?" — @clicky, October 17, 2025
A bit bleak perhaps, but the economics don't look great for a lot of small sites that are now absorbing constant scraping load while watching their own traffic numbers get distorted.
China: A New Generation, Triple the Traffic
Mid-November brought a fresh problem layered on top of the existing one: a major surge in bot traffic from China. Not a gradual increase, a sudden near-tripling in volume.
"This Cloudflare graph shows our total traffic volume from China. Huge surge the last week or so, almost triple what it was before. Previous generation was on Chrome 134, new gen is on Chrome 139. Globally blocked as of... now." — @clicky, November 15, 2025
February Updates, False Positives, and the Ongoing Grind
In February we shipped a significant round of anti-bot updates. These were the kind of changes where we expected some sites to notice:
"Some big anti-bot updates went live last night. Some sites may notice a significant drop in traffic, but let us know if anything seems alarming. 🤖" — @clicky, February 18, 2026
And sure enough, the new filters were a bit too aggressive in places. Shortly after the February deployment we found they were accidentally blocking SamsungBrowser and Naver (a browser popular in South Korea). We caught it and fixed it, but it's a good illustration of why this problem is hard: the boundary between "bot" and "unusual-but-legitimate browser" is genuinely blurry sometimes.
We've also been working on the other side of the equation, trying to get better cooperation from AI companies themselves. Cloudflare maintains a list of verified bots that legitimate crawlers can register on, which makes them easier to distinguish from malicious traffic. Most legitimate AI companies are on here, but not all. (Looking at you, Claude! Get on it!)
Singapore: The Current Bot Capital
China is often a scapegoat for bots, but we just a few weeks ago, we noticed something surprising in our geographic data. Singapore had become, by a wide margin, the worst source of spoofed user agents on our entire network:
"Singapore is the current bot capital of the world for spoofed user agents (bots pretending to be actual people). Over 95% of the traffic our entire network receives from there is getting blocked by our aggressive bot filters. Nowhere else is even remotely close. Fun times! 😡" — @clicky, March 3, 2026
95% is a legitimately insane number. There is real traffic from Singapore of course, but it's the exception. The bots aren't necessarily from Singapore, they're routing through infrastructure located there, but regardless, the signal coming from those IPs is almost entirely noise.
What This Means for Your Traffic Numbers
If your Clicky numbers have dropped noticeably over the past several months, there's a good chance that's our filters working correctly. When we deploy a new round of blocking, some sites see a significant overnight drop because bot traffic that was previously logging as human visits suddenly isn't.
We know a traffic drop looks alarming. But inflated numbers from bots aren't doing you any favors, since they distort conversion rates, skew engagement data, and generally make your analytics less useful. A lower number that's accurate is better than a higher number that includes bots.
Join the battle. Do your part! (or not)
This is an ongoing battle that will never stop. We're staying on top of it as best we can, and we have many proprietary methods we've developed over the years to filter out bot noise beyond what's mentioned in this post.
But it absolutely helps a ton when customers contact us report obvious signs of bot traffic to us, or obvious signs that a filter isn't working properly, e.g. the SamsungBrowser issue mentioned above. So please, keep it up!
And consider giving Clicky a try if your existing web analytics service just isn't cutting it due to bot spam. Despite the doom and gloom, we remain confident that no one is fighthing this as aggressively as we are!
