AWS: Inaccurate Estimated Billing Data – $1.7 billion

Services emit metering values that arent directly tied to prices. Every SKU/line item is defined in a “pricing plan”, with a unit type, regions, and price per unit. The metering records are joined to a pricing plan based on account id, region, sku, etc. mess up the unit type in the pricing plan and the metering data conversion doesnt work, and you get crazy bills.

Years ago I found an actual hidden error in my bill. (This was early 2010s). The system was calculating the EC2 reservation savings incorrectly for some of my servers. I was crunching all their detailed usage data on a regular basis in an 18 tab spreadsheet and couldn't get it to fully reconcile. I spent months trying to track down the discrepancy. Once I found it, I had to convince AWS their system was wrong, which took another big chunk of time. Meanwhile the discrepancy continued to accumulate.

After 14 months I got a $7,000 refund. I was told it had to be approved by the head of AWS. I've never seen a calculation error on their part since.

> You're right to question my calculation. The MCP server failed to connect when I tried to look up the field definition. I guessed instead of validating. This is on me. But look at all the revenue!

https://www.techtimes.com/articles/320266/20260712/anthropic...

The customer service people I talked to in the grocery store said this changed sometime in the last year. My guess is that it’s an unintended side effect of removing the pay-by-palm feature.

This is obviously unrelated but I joked about what else Amazon wasn’t reliably calculating….

Seems totally irresponsible not to send an immediate follow up email to make customers aware.

This is peanuts compared to a major cybersecurity catastrophe that’s surely in the making.

To give credit to the technology and the people using it - and I’m not being facetious - it’s actually incredible that at the current levels of usage the unprecedented catastrophic event has not yet happened.

"Operational issue - AWS Billing Console (Global) Service - AWS Billing Console Severity Impacted - Inaccurate Estimated Billing Data"

Forecasted month end $18,729,381,032,152.4

Apparently my company owes the combined GDP of France, Germany, and UK to AWs.

"CEO Reveals How He Used AI To Build One-Person Company That's $1.3 Billion In Debt"

https://www.youtube.com/shorts/YERfTT4McsU

Makes you wonder - what if there really would be an incident where some massive amount of traffic got routed to your infrastructure by some heavyweight player? Say Wikipedia accidentally switches their IP to your CloudFront? Would you really be on the hook for $500k?

Host your own people. Host your own.

Maybe it's not just vibe-coded, maybe the numbers themselves are being hallucinated by an LLM.

Edit: I was just about to credit the user when my internet dropped. The source was here: https://news.ycombinator.com/item?id=48945606 - thanks mirzap!

> Amazon Web Services customers receive bills for up to $1.5tn after global glitch

1. Current month's usage * applicable rates; + 2. Estimated future usage for the month * applicable rates.

And Item 1 obviously requires proper data persistence, but Item 2 is just a projection. If they don't have Item 1 correct, AWS's whole system is in question, but I don't think that's the issue. I'm going to guess now -- looking forward to reading the root cause analysis -- that the problem is that someone decided to get too fancy with the estimates, and built a latent requirement that all prior estimates for the month must be available to compute the current estimate. Without estimates working, no estimates are available, and some denominator in an averaging or smoothing or normalizing function goes to 0; then everyone's estimated bill explodes without bound (subject to floating-point arithmetic) resulting in trillion-dollar estimates.

Folks can track it directly on AWS Health: https://health.aws.amazon.com/health/status

In 2026, the gigabyte should probably be the default/minimum unit for something like AWS.

It sounds ridiculous, but something very similar happened to me with Amazon WorkSpaces. During the WorkSpaces setup, an AWS Active Directory (Directory Service) instance was provisioned as part of the deployment. When I later canceled WorkSpaces, I had no idea the Directory Service had to be deleted separately. I kept getting billed for it, and it ultimately cost more per month than the WorkSpace itself had.

Nowadays you just have to risk accidentally billing your parents CC the tune of multi-generation wealth to get that real-world experience.

https://news.ycombinator.com/item?id=48950534

https://health.aws.amazon.com/health/status

But with AWS costs rising anyway (not by that much but OK), I'm probably not the only one to start reconsidering their cloud strategy. I think this might have just pushed me over the edge.

If you owe the bank $1.7B, thats the banks problem. """

What I would be curious about (and I'm sure AWS will never share) is where the incorrect number came from. If the number is somewhat consistent between some groups of accounts, my first guess would be they started summarizing billing across all accounts in whatever cell/grouping/heirarchy AWS architected internally.

Which is just funny.

Do the right thing for the players, Matt!

The crypto network you hosted should pay for itself in 10-20 years just like LLMs. Don’t worry. Consider Bank of America until then if you are good on credit score.

Phew.

I literally almost had a heart attack today.

Looks like this is a bug w/ S3

My sympathies -- I know I would be overcome with panic in such a situation.

I too was shocked when I saw the $1.7billion bill, instead of the usual $1.5billion.

Insane

Yeah, this most certainly is bad code wrapping around a value. AWS will post a notice soon if they haven’t already.

$103,515,940,301.79

ACTUAL Amount: $1,046,294,123,330.95

This cannot happen if you do not do this renting at variable rates.

A thing you own doesn't suddenly bill you trillions of dollars in error. It doesn't hyperscale either, but neither do you.

This is what we received this morning

I might be a winner.

I’m currently dealing with Verizon Wireless and their “Jabronibot” claiming I have a fictional account balance due. It has been sent to collections, but still is being asked for by their legacy system.

The case studies of “Agents in Billing Departments” and potential shareholder lawsuits / E&O claims / reputational damage will be interesting to me. I worked in “risk management” products years ago and this kind of liability is not easily dollar traded away via contract. Will accountability stick to the Decision Makers or will they try to surrogate to the Service Providers? Hmm.

"NEVER represent currency with floating point, multiply by 100 and store in an int before doing any math"

Then flee the country just to be sure.

I think I would have just waited to see what happened when AWS tried to hit my credit card for $1,700,000,000.

When do you ever get that opportunity?

$1,299,988,247,332.56!

That was a fun set of emails to wake up to, figured they had to be phishing for how outrageous of a number it was. But nope! Fun little incident they've got going over there.

https://news.ycombinator.com/item?id=48945681

All hail the new generations of our uberployers.

Honestly, I would worry more about estimated billing that seems plausible in general, but is way to high for you personally. These ridiculous amounts? Not so much.

What an absolute joke. All just so that line goes up. As if their fees weren't high enough vs. alternatives (especially egress). And I'm sure the pro-AI crowd will keep saying we're luddites for not loving this clearly revolutionary and disruptive tech.