What happened to TheNumbers.com

A couple years ago, I was running a website which allowed the public to view all the US government handouts to small businesses during the COVID-19 pandemic. It also tracked all the fraudulent loans being prosecuted by the DOJ, and allowed anyone to run structured queries over the public dataset. There was a "donate" button which took in ~$2k in donations over the lifetime of the site, and you could download the entire underlying dataset (around 10 GB uncompressed) for free directly on the site.

Despite the "download all data" link being prominently placed on the front page, the AI scrapers decided it would be more efficient to download terabytes upon terabytes of raw HTML by paginating through every possible facet on the search endpoint. Even with CloudFront caching results and a fairly efficient backend setup, the monthly bill ended up with around $1k just going toward network ingress/egress, so I shut down the site the following month.

At the risk of oversimplifying things from a distance, this site -- especially the free, public-facing part of it -- seems like it would be an ideal candidate for a rewrite using static site generator/framework. That, coupled with a bot-aware CDN should keep them online at a reasonable cost for many years to come.

Otherwise, I'm very curious to know more about their old and new architecture and what sorts of mitigation/scaling strategies they've started using to keep the site online.

Just FYI, the bigger companies all allow you to block crawlers via robots.txt:

    # Block Anthropic (Claude)
    User-agent: ClaudeBot
    Disallow: /
    User-agent: Claude-SearchBot
    Disallow: /
    User-agent: Claude-User
    Disallow: /

    # Block OpenAI (ChatGPT)
    User-agent: GPTBot
    Disallow: /
    User-agent: OAI-SearchBot
    Disallow: /

    # Block Perplexity
    User-agent: PerplexityBot
    Disallow: /

    # Block Google's AI Training
    User-agent: Google-Extended
    Disallow: /
    User-agent: Google-Extended-Factual
    Disallow: /

    # Block Microsoft's Search & AI Crawler
    User-agent: Bingbot
    Disallow: /

I think people are missing one of the points of the article. It's not just that agents are hammering the site, it's that there might be lurking vulnerabilities that allow malicious usage, which is why it went down, then came back up with a fraction of the data and a reduced design.

The article says (speculates?) that malicious users are trying to get privileged access for an edge in prediction market betting. From the article:

> If you could see The Numbers data before everyone else, every single week, you would have a significant edge over all the other traders - learning the answers slightly ahead of publication would allow you to front-run the trades.

"One Reddit theory even suggested it was a deliberate rug pull designed to cripple the free site to push people towards paid products."

I know this isn't really the point of the article, but I've been thinking about this a lot. I wonder if we're going to see more resources go this way.

I used to publish little doo-dads as opensource software. Not because it was something that was legitimately ground breaking or anything (it absolutely wasn't) but because I had a problem, and I thought "heh wouldn't it be cool if someone else had a similar problem and could use my resource for it."

But now I'm really reluctant to give more stuff to the free web. Because the fact that it gets scraped and added to a pile of training data to later be monetized really rubs me the wrong way.

And I can see why maintainers of sites like this, or other free but incredibly useful resources might start to get irate at that.

I think I probably use AI like a lot of consumers out there. Search engines have gotten bad and AI really good at answering fairly specific questions. Often pointing at sites like wikipedia. Pretty clear changing my behavior will have zero impact but it is certainly part of the problem. Feels a lot like my CO2 consumption.

*Edit - CO2 creation

What a throwback. Back in 2015 I have started Applaudience, which at the time was the only provider of real-time cinema ticket sales data. I still remember comparing our numbers against TheNumbers.com as part of calibration. I have since moved on to other businesses, but this remains one of my favorite pieces of technology that I have developed. Would love to bring it back one day.

It feels like there is fundamentally missing infrastructure here that is needed to make these problems go away.

Basically bots need to be (somehow) paying for the traffic they create, or prevented from creating it, or told to go away and then fined if they violate the request. No idea how to do these or even at what level in the stack they should happen, but they need to happen eventually somehow.

I know people have opinions about Cloudflare but why not use it here, at least as a stop gap? Stopping bot traffic is one thing it does very well.

> The world we have built thus far is so incredibly ill-prepared for the power and scale of the AI models we all have access to.

Exactly, so use the AI to secure your servers. Ask the AI to audit your site for any security holes. If unable to rewrite, at least harden the existing code. AI’s are really really cheap (and fast) security consultants now.

If the AI companies destroy the open web, eventually they'll need to start curating knowledge sources just like netflix makes movies and amazon has physical stores...

This article, and possibly the owners of the site, seem to muddle together several problems:

1. bot traffic causing infrastructure cost

2. scraping circumventing paying for licenses

3. the risk of hacking

Good read, I was so curious how this happened a few months ago

Ironic that polymarkets were being advertised as helping society make better predictions.

If the scrapers are going to get it anyway, put the data up as a zip somewhere.

They should sue Anthropic for this distillation attack

The real story here is that prediction markets were banned for a reason and loosening the rules is causing chaos just as was expected. AI plays little role in this story, hacking by humans would also be motivated by financial returns, unless the element is that AI hacking is cheaper and the returns are not so big.

I always liked this site, but reading this and seeing the anger about expecting the site maintainer to do things for you is repulsive. Frankly, if he wanted to pull his site down with no notice that is perfectly within his right. It was/is his site. He doesn't owe anyone a .tar.gz either. His work.

What does a cyber attack have to do with AI scraping?

Its just too easy for a technically minded bored person to produce slop that hammers websites. There needs to be a penalty for this.

Sorry! Just yesterday many of us decided that AI scraping isn't a real problem, and anytime it is blamed, it's a cover for something else.

https://news.ycombinator.com/item?id=49005747

[dead]

This is an odd example to present this argument through.

I'm not defending AI bots overwhelming websites or hackers motivated by Polymarket, etc., but I don't really believe a 30 year old website with "approximately 160,000 source files serving around 2 million pages" is a good litmus test for the state of the online world. What's worse is the absurdity that this basic site offering niche data would be targeted because Polymarket depends on it for some of their bets, something that most websites don't have to deal with. Frankly, that seems like a much more interesting angle to explore than "this old website that should have been re-written multiple times over the last three decades now doesn't have a choice but to be re-written."

Oddly, it seems like the solution, at least in this specific case, is relatively simple (and ironic): pay for an LLM to re-write this site in a modern, scalable, secure way and have that LLM monitor the site to ensure it is behaving correctly. Put this thing in a modern platform behind a proper cache (i.e., throw the whole thing up in Cloudflare) and many of these problems just don't exist anymore. If this site is as basic as it seems, this could be a weekend project.

It's certainly an interesting story, and I don't blame the owner of The Numbers for handling his site the way he has, but framing this as "AI is destroying our beloved internet" just seems obtuse, at least through this lens.