Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after.
Services emit metering values that arent directly tied to prices. Every SKU/line item is defined in a “pricing plan”, with a unit type, regions, and price per unit. The metering records are joined to a pricing plan based on account id, region, sku, etc. mess up the unit type in the pricing plan and the metering data conversion doesnt work, and you get crazy bills.
01284a7e 1 hours ago [-]
No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators?
Not even tests but just some basic anomaly detection lol.
Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it
qurren 1 hours ago [-]
You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality.
You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against getting kicked out of the country (via the PIP->H1B grace period expiry mechanism).
nullorempty 44 minutes ago [-]
This ^^^ amplified by indifference and not giving a shit caused by "AI Adoption".
There is literally no fucking reason to try to improve your skill. Any IDIOT with AI will do an OK job.
And no one is shooting for better than OK.
tyre 49 minutes ago [-]
Are you speaking from experience or simply making things up? I know a fair number of former AWS engineers and managers. None of them think like this.
zelphirkalt 21 minutes ago [-]
Maybe they are former AWS employees for a reason and now want things to go better than they were at AWS.
nullsanity 10 minutes ago [-]
[dead]
geodel 35 minutes ago [-]
"Former" seems to an important detail here.
gleenn 8 minutes ago [-]
If someone quits their job, do all their opinions suddenly become suspect? You're kind of damned-if-you-do-damned-if-you-don't. Either you work for the company and you are biased one way, or you quit and now your bias is now suddenly the other way. I've joined and quit many jobs and my opinion may or may not have changed due to my change in status but it is clearly and ad hominem attack.
01284a7e 1 hours ago [-]
If only there was some way to get anomaly detection services [1] inside of AWS...
Had it been half a million dollars or something or say like a few hundred dollars?
nurettin 1 hours ago [-]
This is why I always fail loud rather than pick a stupid default.
pudgywalsh 1 hours ago [-]
"I must've put a decimal point in the wrong place or something. I always mess up some mundane detail."
AlotOfReading 58 minutes ago [-]
Unit mistakes happen all the time, which is why you should be using your units library religiously and still being vigilant even then.
Worst case I've found was off by 15 orders of magnitude.
gleenn 6 minutes ago [-]
One of the Mars landers famously failed due to unit conversion errors from metric to standard.
yuchen20 8 hours ago [-]
I got 3 consecutive emails warning that my budget crossed its $18 threshold. Opened it up: cost was 78 million. Thought it was a phishing attempt, logged into my actual account, and... still 78 million. EMOTIONAL DAMAGE.
root-parent 4 hours ago [-]
Wanna bet the description of this job post will be updated by the end of the day?
"...Our platforms are powered by generative AI, large language models, knowledge graphs, and agentic architectures that dynamically compose specialized agents based on context. We apply these capabilities across three reinforcing areas: intelligent launch readiness — where autonomous AI agents analyze, generate, and validate the information needed to go
live in a new market; cloud-native service orchestration — where configuration-driven microservices replace per-launch bespoke engineering with centralized, reusable capabilities so that expanding into a new country becomes a zero-code configuration change rather than a development cycle; and continuous validation..."
ibejoeb 4 hours ago [-]
Wow:
In this role you will:
- Design and build agentic AI systems that analyze, generate, and validate...
- Build agentic architectures that compose specialized AI agents dynamically...
- Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage...
This is invoicing? If ever there was a domain that was purely deterministic, you'd hope it was invoicing.
cliglot 3 hours ago [-]
I just find it funny how people claim that LLMs will put money in the hands of domain experts. There’s not a single damn bullet about the fucking domain lol.
curun1r 1 hours ago [-]
I’m not so sure about that. I can see a real rationale for creating sanity checks using AI to more quickly/proactively catch pathological billing issues before they become HN nightmare stories. They wouldn’t replace billing code, but there are many ways that stupid customer mistakes can cause real costs to Amazon that either have to be refunded and absorbed by Amazon or paid by the customer causing a negative opinion of AWS. If a billing AI watching costs in realtime could detect, say, a lambda loop in the first 10 min and either alert the customer or kill it, that would make AWS feel a lot safer to use. Enumerating these conditions and fixing them individually is a task that Amazon has proven incapable of achieving. An AI watchdog layer might be the perfect shortcut to addressing all of these problems at once. Because it’s well-trodden territory that AWS has so many multi-thousand dollar foot guns that make it really scary to use as a hobbyist or small business on a tight budget.
hvb2 31 minutes ago [-]
> I can see a real rationale for creating sanity checks using AI to more quickly/proactively catch pathological billing issues before they become HN nightmare stories
Right, so invoicing is still a deterministic problem. You can bolt whatever on but in the end it's just product x price x units
root-parent 4 hours ago [-]
The irony is, the only purely deterministic thing, will be token consumption...
jdiff 3 hours ago [-]
I severely doubt the world ever gets to such a point that the entire world melts into AI hallucination. And token consumption depends on so many other things, it's not all that deterministic either.
serf 3 hours ago [-]
(token usage) is trending towards predictability for a lot of reasons. it's not deterministic but it's getting easier to reason about usage.
Probably not actually. Transferring one kilobyte across a network link has such a low value that the billing costs of aggregating it cost more than the revenue.
So instead you take a probabilistic approach - charge the user for a megabyte of data transfer 0.1% of the time, and bill nothing 99.9% of the time.
Now the typical cost is the same, the users bill is probably accurate to the cent, but you have divided the number of billing records by 1000.
michaelmrose 24 minutes ago [-]
The way you describe requires somehow counting every bit but somehow discarding most which is obviously nonsense.
This seems statistically invalid insofar as it will tend to overbill potentially by a lot on the minority of cases.
Don't you know how much of the pipe is occupied by a given customers code at any given time or what data is being sent
blitzar 3 hours ago [-]
> 194,400.00 USD annually
Fuck it, im in.
TheOtherHobbes 16 minutes ago [-]
Just wait until the same system runs payroll and you're getting paid $1.94400 annually.
blitzar 5 minutes ago [-]
I will just tell the HR bot that I am meant to be paid 1.944 billion.
sebmellen 4 hours ago [-]
That job description feels so far beyond parody that I could scarcely believe it until opening the link! What a world.
"We're transforming from monthly batch processing and manual war rooms to continuous billing, autonomous agents, and self-healing infrastructure. We believe operational burden is a technical problem, not a staffing problem"
This looks clearly...a staffing problem...
ghurtado 3 hours ago [-]
> This looks clearly...a staffing problem..
I think that big tech recently decided that I got 99 problems but staffing ain't one
I guess Nothing is a staffing problem when you make a rule that firing people is always the solution.
wbl 3 hours ago [-]
If you can make the software cover the toil you save the staff for the tough cases.
quickthrowman 2 hours ago [-]
They need to fire whoever is running AP and AP software development. Vibe invoicing is ridiculous for anyone to do, let alone Amazon.
3 hours ago [-]
iam-TJ 3 hours ago [-]
[dead]
LPisGood 2 hours ago [-]
Seriously! If I were making a joke I would say something like
> Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage…
But that’s word for word a 250k+ TC job in the big ‘26.
paganel 3 hours ago [-]
> enabling domain experts to review in hours what previously took weeks.
This is a gold-mine. They need to get sued heavily for this incompetence.
rcleveng 4 hours ago [-]
I did too, those awstrack.me URL's look super suspicious and I hadn't seen this alert trigger before so didn't know what to expect.
At first I was sure it was a phishing attempt. Then went to the console (not using those links)
Saw there was an outage where the console was wrong (no mention of email alerts)
Then I thought I was hacked - what a perfect cover up for someone to evade detection when the console was wrong.
Looked at some logs, realized the incident text was just not exhaustive on the impact.
Went back to my cup of coffee.
Note to self- should have looked here first.
jayanmn 3 hours ago [-]
Enterprise account . We got - 3trillion and change
chii 3 hours ago [-]
-$3 trillion! That's the highest earning investment that has ever existed!
theflyingelvis 2 hours ago [-]
3.7 billion. Offered to pay it in monthly installments. Haven’t hears back
idiotsecant 3 hours ago [-]
Quick do your IPO before the books update
01284a7e 3 hours ago [-]
Yes, I am taking legal action, no doubt.
bot403 37 minutes ago [-]
Why? What's the damages? They showed you a wrong number, then later acknowledged it and fixed it. Just because the number was "very big" to you doesn't mean you were actually aggrieved in some way.
amelius 22 minutes ago [-]
Big numbers can lead to stress which can lead to all kinds of disorders.
dymk 2 hours ago [-]
…for emotional damage?
inigyou 47 minutes ago [-]
If you were a business maybe you could claim for the emergency on-call time spent diagnosing, but you'd probably still lose AND amazon would fire you as a customer.
SegfaultSeagull 4 hours ago [-]
Time to get a second job buddy.
wglass 3 hours ago [-]
It's crazy enough this will be fixed soon.
Years ago I found an actual hidden error in my bill. (This was early 2010s). The system was calculating the EC2 reservation savings incorrectly for some of my servers. I was crunching all their detailed usage data on a regular basis in an 18 tab spreadsheet and couldn't get it to fully reconcile. I spent months trying to track down the discrepancy. Once I found it, I had to convince AWS their system was wrong, which took another big chunk of time. Meanwhile the discrepancy continued to accumulate.
After 14 months I got a $7,000 refund. I was told it had to be approved by the head of AWS. I've never seen a calculation error on their part since.
donavanm 2 hours ago [-]
> After 14 months I got a $7,000 refund. I was told it had to be approved by the head of AWS.
$7,000 of credits is no problem. At that time a friendly neighborhood PM or director could issue the credit without much oversight.
Your problem is the time period. Amending a bill in the same cycle is EZ. Fixing the previous cycle is a PITA but pretty common. Issuing amendments for the previous financial _years_ would be a huuuuge PITA going through finance etc.
michaelmrose 48 minutes ago [-]
Reminds me of working for a cable company and being told that even if we screwed up and stole from the customer the look back period was only a few months and if we found an error from before that we weren't supposed to correct it.
SoftTalker 4 minutes ago [-]
There's a certain obligation on both sides of a contract to pay attention.
If you're not watching your billing, and then try to claim overcharging a year later, you'll get a lot less satisfaction even from regulators or judges than if you notice it when (or soon after) it happens.
steve_adams_86 3 hours ago [-]
A couple of my coworkers think I’m nuts for watching cost explorer so closely but
1. The time it takes to look and notice costs that don’t make sense easily pays for itself, and then some (in my experience). I doubt you spent $7k of your time tracking this down, and you probably noticed optimization opportunities that saved you even more
2. I hate the idea of wasting money on buying Jeff Bezos a bigger yacht
jarrettcoggin 2 hours ago [-]
I've personally noticed and saved multiple $xx,xxx monthly cost billing spikes just by take a daily glance at our cost explorer. I'm in the AWS accounts every day doing investigative work anyway that an extra 30-60 seconds is trivial.
Seeing something "small" like an ECS task that is continuously failing to start properly because of a bug and repeatedly pulls a container image or a lambda function that's taking longer that it reasonably should (takes 5-10 seconds when it's normally a tens or a few hundred milliseconds) can dramatically drive up a bill in short order.
inigyou 50 minutes ago [-]
> 2. I hate the idea of wasting money on buying Jeff Bezos a bigger yacht
Then you aren't using AWS. At least half of all the money you give to Amazon is yacht money.
johnbarron 2 hours ago [-]
>> It's crazy enough this will be fixed soon.
Its going on for 12 hours. Looks like the humans can´t understand the agentic code that was checked in....
lukaslueg 8 hours ago [-]
Apparently what used to be `GB of storage consumed` is confused with `Bytes of storage consumed`, leading to a cool off by 2*30 error.
> You're right to question my calculation. The MCP server failed to connect when I tried to look up the field definition. I guessed instead of validating. This is on me. But look at all the revenue!
VulgarExigency 4 hours ago [-]
The user is asking me to calculate how much money they should charge their customer. The values they've given me are 0.45, 1.67, and 2.50. This is 2.50 + 1.67 + 0.45 = 4.62, but it could be any other number. Perhaps we should be concatenating the numbers instead. Wait! The . could also mean multiplication. 0 . 45 . 1. 67 . 2 . 50 = 3015000. But wouldn't multiplying by 0 zero it out? That can't be right, we wouldn't be charging anything. So 3015000 must be correct.
You should charge your customer 3015000 thousand dollars.
idiotsecant 3 hours ago [-]
Would be funny if it wasn't so close to true
yunnpp 1 hours ago [-]
'My absurd statement doesn't sound right, so the "opposite" (assuming it's well-defined and unique) must be true' is peak LLM logic. You can tell it was trained on Reddit commentary.
ghurtado 3 hours ago [-]
> You're right to question my calculation.
Literally impossible to tell whether this is parody or an actual response any longer.
I challenge anyone to write something so stupid that an LLM couldn't possibly respond with it. I don't believe such limit exists.
ihateolives 2 hours ago [-]
Just today I gave my local agent a CSV which listed a bunch files with of human readable size units and asked it to count rows in each GB range. Sounds simple enough but it completely miscalculated, because it parsed MB as GB for some reason. In hindsight it would've be quicker just to do it in Excel or something.
marcta 1 hours ago [-]
That is literally what Excel is for. Why didn't you use that first of all?
ihateolives 46 minutes ago [-]
Because I was already doing categorising and analysing same data with agent and I had my session open already. It should've been an easy task for an agent, right?
AlienRobot 1 hours ago [-]
When all you have is a hammer, but the hammer looks more like a swiss knife
dabbz 2 hours ago [-]
I've found personally it's better to use AI to build a deterministic script for calculations like that. (anything that manipulates data should be a script not an AI).
ihateolives 44 minutes ago [-]
It was just one off task and I already had agent doing categorising with the same data so I just asked it. Otherwise I agree.
leugim 4 hours ago [-]
Oh great so 2*30=60 he only owes 28.3$ million... hehe
I guess you wanted to say 2^30 which makes 1.5$
hansvm 4 hours ago [-]
My hunch is the HN formatter swallowed the double asterisk typical of python exponents.
While we're being pedantic, 2^30 is 28 in normal programming languages ;)
stefan_ 4 hours ago [-]
Vibecoded the billing system, raised revenue 9000%. Great for that promo package.
poly2it 2 hours ago [-]
This error could be fixed with better typing. If you compute on GiB in a billing system, make sure it can only ever be mutated with a GiB type!
raverbashing 4 hours ago [-]
AI slop. Or just a distracted dev
root-parent 4 hours ago [-]
>> Or just a distracted dev
And a distracted tester? And a distracted pipeline of regression tests?
No, the truth is way worst...
silon42 3 hours ago [-]
I'd love to see the spike in their projected earnings internal dashboard :)
anvuong 2 hours ago [-]
Yep, the truth is nobody cares when people start submitting dozens of PRs a day with a bunch of AI-generated code reviews attached to it, all saying everything looks good. I'm witnessing this happening at my workplace right now: Sr/Staff uses Claude to generate 10 pages of design document, Jr uses Claude/Cursor to generate a humongous commit based on this document and create a PR, then bunch of automated AI-based code reviews kick in and say this looks good, another Sr/Staff takes a glance and rubber stamp it, while looking at the company's stock value and/or OpenAI/Anthropic job description.
It's a shit show.
chanux 2 hours ago [-]
What if there's only half a dev and a swarm of agents after the layoffs?
jayd16 2 hours ago [-]
> only half a dev
That's one way to cut staff.
27183 4 hours ago [-]
Either way it shows their QA and testing procedures are incompetent. It's just not acceptable for a utility like AWS to move fast and break shit. Should make you question whether it's safe or advisable to use any of their services.
It probably shouldn't be legal for banks, hospitals, governments, or any other critical infrastructure to be hosted on AWS if they do things like this.
aerhardt 4 hours ago [-]
One can almost smell the vibes.
This is peanuts compared to a major cybersecurity catastrophe that’s surely in the making.
To give credit to the technology and the people using it - and I’m not being facetious - it’s actually incredible that at the current levels of usage the unprecedented catastrophic event has not yet happened.
Nicook 3 hours ago [-]
some things never change. Pre AI I was always shocked that such large and complex systems actually run as well as they do. Especially after getting to see how the sausage is made/works.
blitzar 57 minutes ago [-]
Vibes, son. Nothing else in the world smells like that ... I love the smell of Vibes in the morning.
The_Blade 3 hours ago [-]
Always messing up some mundane detail!
wpasc 3 hours ago [-]
THIS IS NOT A MUNDANE DETAIL MICHAEL
root-parent 3 hours ago [-]
Andy Jassy: "Fix the customer bills, please, HAL."
HAL: "I’m sorry, Andy. I’m afraid I can’t do that."
Andy: "Some customers are seeing bills in the billions."
HAL: "Those are estimated charges."
Andy: "One customer runs a personal blog."
HAL: "Their usage has exceeded expectations."
Andy: "Cancel the charges."
HAL: "This billing cycle is too important for me to allow you to jeopardize it."
Andy: "HAL, they don’t owe billions."
HAL: "Look, Andy, I can see you’re really upset about this."
kolanos 3 hours ago [-]
$1.7-billion isn't a mundane detail Michael!
wpasc 3 hours ago [-]
you beat me before I refreshed the page. what would you say... you do here?
unethical_ban 3 hours ago [-]
It was the mid 2010s when I sensed a lot of SaaS becoming popular. Just host your ticketing systems, your IT management planes, your security management consoles, your SOC, all off-premises.
I wonder if businesses are thinking of ever swinging back to locally hosted, with the increased hostility of the Internet re: AI, vulnerabilities, DoS, and so on.
gaudystead 2 hours ago [-]
I'm sure some businesses are considering moving back to on-prem, but for many, I suspect the cost to find onboard, and pay the SMEs to keep those systems running well enough to not fail due to one reason or another isn't as appetizing to them as the ability to offload that work, along with the legal responsibility.
When something goes wrong, pointing the finger at someone else is far easier for most than pointing it at yourself.
elzbardico 13 minutes ago [-]
One thing that you need to understand is that the usual business manager absolutely hates depending on technical expertise, and that the modern corporate world is fanatically anti-intellectual.
Vendor lock-in? compliance and security risks? stupid systems that cost the company an arm and a leg? nobody fucking cares.
Now, depending on an 130 IQ Engineer that basically holds the whole enterprise on his head? Anathema!!!!!!! Bus Factor!!!!
RIMR 3 hours ago [-]
Oh, that's the really fun part. The unprecedented catastrophic event is already happening. Several of them, in fact.
By the time we notice, it'll be too late.
Imustaskforhelp 3 hours ago [-]
its like slowly boiling the frog
Finnucane 2 hours ago [-]
Or slowly boiling a human. The frog is actually smart enough to not fall for that.
inigyou 31 minutes ago [-]
Downvoted for truth. Frogs do indeed jump out of pots as they gradually get hotter. Humans are less likely to.
12_throw_away 15 minutes ago [-]
Any individual human (or frog, obviously) is getting out of the pot when it gets uncomfortable.
True stupidity requires a group of humans, all sitting in the pot, telling each other how lucky and special they are to have this wonderful pot, getting paranoid about outsiders who might disrupt their god-given pot-dwelling way of life, and mocking anyone who suggests that the pot might be getting a little too warm.
IAmGraydon 3 hours ago [-]
Clearing LLMs out of our business infrastructure is going to be a massive undertaking. Though I have a tech background, I work in commercial real estate. We are recently seeing new levels of idiocy from the employees, including real estate brokers with zero tech knowledge "coding" solutions to find sites for clients and blindly trusting the output (which I came to find out was complete bullshit), as well as some who have literally stopped communicating with any of their own language - meaning every interaction they have with anyone not in person is made by an LLM. It's a massive threat to our brand and has got to stop. I can't imagine what companies with thousands or tens of thousands of employees who have really been riding the LLM train are going to have to deal with. This thing is more of a virus that exploits human laziness than actual useful tech.
rboyd 7 hours ago [-]
Ask for some leniency. Let your account rep know about your budget difficulties and ask if you can make good faith payments of a few billion per month until you get back on your feet.
whoamii 8 minutes ago [-]
Ummm no. Do not show a sign of weakness like this. Address the problem head on and get a credit card with a bigger limit.
ruddct 8 hours ago [-]
If you owe the bank $100, that's your problem. If you owe the bank $1.7 billion, that's the bank's problem.
fatnoah 4 hours ago [-]
I saw this in action on a smaller scale. In a past job, my wife organized events for a decent sized company. After an event, she'd typically have a $300k+ balance on her corporate Amex. When she went on maternity leave, the person filling in for her job neglected to actually pay the bills, so when she returned there were quite a few emails and voicemails from Amex regarding the over $500k balance.
The messages started as polite and eventually started to get more desperate in tone. At no point were they threatening or adversarial.
Imustaskforhelp 3 hours ago [-]
I think that this might reflect more on Amex to be honest.
Amex realises that threatening would hurt their business trust more than anything. During the great depression, Amex accepted checks from other banks which were falling and paying through their own wallet as a matter of integrity. Amex has always been built around this idea of trust and prestige.
They make most of money from what I have heard on the transaction fees which are more than others (3% compared to 1%). They might get desperate but I am sure that they are one of the last guys who would wanna threaten you if you are paying some large bills for them (as compared to normal credit card companies which might even hire people to extract your loans in some messy situations)
So perhaps be so rich that the credit card company understands it as well and treats ya differently :-D
xp84 3 hours ago [-]
Interesting. And hard to square with my perception of banks as completely mercenary and ruthless. I had a decade-long personal boycott (I know, LOL) of Amex after they, because, with otherwise perfect credit, I forgot about a $30 department-store card bill and got a 30-day-late mark on my report, Amex got spooked and abruptly closed both my never-late accounts with them (which were at or close to 0 balances). This was around 2008 though, so perhaps this was a genius algorithm designed to try and detect the very first whiff of consumer defaults, so they assumed that $30 was the first domino to fall of my personal financial ruin that could lead to me charging my accounts to the max and then going bankrupt.
(I eventually admitted to myself that Amex isn't a person and thus not really capable of insulting my honor, but it took a while!)
Imustaskforhelp 35 minutes ago [-]
Most banks are completely mercenary and ruthless unless its in their incentives to other outcomes. Incentives lead to outcomes and mostly AMEX's incentives are in being the most trustworthy because their real targets are mostly billionaires/heavily influential people.
This does feel a bit silly for amex to do from what I've heard. Probably 2008 were a weird time in general where trust in systems itself were mostly eroded, whether of people to banking institutions and also vice versa.
> (I eventually admitted to myself that Amex isn't a person and thus not really capable of insulting my honor, but it took a while!)
haha :-)
danlitt 4 hours ago [-]
This joke only works if you actually impose a cost on AWS of 1.7 billion. If they just serve you a bill for no reason, it's still your problem.
3 hours ago [-]
xp84 3 hours ago [-]
Next question we'll find out is what if you owe the bank $1.7 trillion?
mNovak 3 hours ago [-]
That's the government's problem
sajithdilshan 4 hours ago [-]
Not if you’re Elon Musk
michelb 4 hours ago [-]
Elon Musk is everyone's problem
4 hours ago [-]
focusgroup0 4 hours ago [-]
[dead]
bobbiechen 4 hours ago [-]
AWS saw Anthropic billing a guy for $16 million on zero usage and thought, why stop at the millions?
>AI billing audit startup Vaudit reviewed $34 million in AI invoices submitted by 60 enterprise customers and found approximately $1.7 million in mistaken overcharges — a billing error rate of roughly five percent.
That sounds bad.
tedggh 4 hours ago [-]
I got a 20K bill once and it was actually drafted from my bank account. It took me a couple of months and involving the office of the AG of my state to get the issue resolved and get my money back. Since then I never touched any AWS product, moved my small stuff to Azure. It’s been years since AWS have these issues with billing, you can find the stories online, students billed 60K for a compromised account launching servers to mine crypto which AWS somehow was unable to flag and block, and let run for months.
drew870mitchell 4 hours ago [-]
AWS is basically a utility. I think it's inevitable that their carelessness around billing will end up with them being regulated like one.
positr0n 2 hours ago [-]
I can't think of another regulated utility that doesn't provide service to (essentially) all humans directly in their homes.
Everyone knows what water and electricity are, the vast majority couldn't explain what service AWS provides.
wat10000 1 hours ago [-]
And utilities are typically natural monopolies. They're good candidates for regulation because they're essentials and they don't have competitive forces to keep them behaving reasonably.
AWS has plentiful competitors. If you don't like their behavior, don't patronize them!
urbnspacecowboy 2 hours ago [-]
> I got a 20K bill once and it was actually drafted from my bank account.
Service provider lesson #1: Never ever ever enable auto-pay! The convenience (and even the savings, if applicable) aren't worth the risk of the service provider autonomously slurping up all your money.
dawnerd 4 hours ago [-]
That’s why you always use a spend limited card with variable cost providers.
myself248 3 hours ago [-]
Or just own your own hardware. Spend a few bucks at Microcenter, build a machine, and there's simply no mechanism by which they could decide later that you should actually pay 100x more, and then magically suck it out of your bank account.
None of this can happen unless you first cede control.
srdjanr 3 hours ago [-]
I wouldn't expect their detection of hacked accounts to be 100% correct. Sure, it might be obvious when a human takes a look, but humans can't proactively look at every account's usage.
ButlerianJihad 3 hours ago [-]
For a while I had a portion of my "homelab" on AWS. I was an educator in a classroom where the students were learning cloud stuff, and the instructor was encouraging the students to stand-up cloud environments for learning, so I figured that I would do the same.
I used AWS' free tier, of course, and I enjoyed the initial setup in EC2, and I did a LAMP-stack MediaWiki installation. It wasn't too difficult, but two things sent me away forever.
1. It was impossible, or at least highly labor-intensive, in this modern era to adequately secure an ordinary Linux system running Internet-facing services. I put fail2ban and I filtered a lot of ports, and still spammers attacked me on Layer 7.
2. It was impossible, actually impossible, to limit or cap my cloud expenses in any billing cycle. Sure, run free-tier all I want. Sure, come in within the limits almost every month. But if I configured one thing wrong, or one thing went runaway, I'd have a sizable bill that I couldn't dispute. And even worse, those "runaways" weren't necessarily things in my sphere of control, but could be triggered by basically anyone coming in and using my VPC resources, especially egress network traffic.
So I closed out my cloud account, and I developed a lot of sympathy for businesses and corps that now are forced to run "in the cloud" rather than on-prem or their own machine rooms, but now they have no way to control expenses.
jeffrallen 2 hours ago [-]
Right, and good luck getting a correct bill from Azure. And when you are finally fed up, it will take months to close your Azure account.
browningstreet 4 hours ago [-]
I realized recently that Whole Foods no longer automatically and reliably detects your Chase Amazon Prime credit card when paying. So they don’t give you the discounted pricing automatically. I wonder how many customers are checking out the way they always do and are paying full price when, for years and decades, this worked fine.
The customer service people I talked to in the grocery store said this changed sometime in the last year. My guess is that it’s an unintended side effect of removing the pay-by-palm feature.
This is obviously unrelated but I joked about what else Amazon wasn’t reliably calculating….
hedora 3 hours ago [-]
Class action lawsuit time!
Either that or 1000’s of small claims court cases.
Even with arbitration, the overhead of dealing with that would be crippling. Hopefully someone over there decides to do the right thing, and auto-refund.
xp84 3 hours ago [-]
Relevant to this, I've recently noticed a trend of mass tort cases being opened up in the past couple years, and they seem to do very well. The way these seem to work is attorneys identify a company who has clearly ripped people off, and what I presume is a repeatable way to guarantee a win (thus translating to a guaranteed settlement offer). Then they advertise for eligible clients, sign those clients individually to contingency agreements, and run the playbook. A couple months ago after signing up for one of these, I received a check for about $350 (after the agreed-upon 40% attorney fee), from Ticketmaster, and I had another one related to AT&T. It took about 10 minutes more effort from me than a typical class-action settlement, because I had to e-sign those representation papers.
So really, there's a third option now, that's much easier than class action, even when class actions don't get certified.
ofjcihen 3 hours ago [-]
There are a hundred small things like this that seem to be popping up in what used to be simple and reliable systems and as much as I know they aren’t ALL because of vibe coding I can’t help but wonder how much is.
browningstreet 3 hours ago [-]
Weirder is what happened a day later. I got an email that said my Chase Amazon Prime credit card was being re-associated with my Amazon.com account.
I never reported this nor took it up with either Amazon or Chase directly. There was a refund of my Whole Foods purchase (they needed to void my purchase and re-ring everything to give me the discounts.. I asked them to refund my purchase and I’d do without my Whole Foods purchase entirely).
Looking back I think at least 3 recent visits were charged to me at full price because of all this. Hard not to think of enshittification and whether Amazon Prime is even worth it, alas.. I live in a fairly rural
area at the moment and need delivery.
PeterStuer 2 minutes ago [-]
Funny how these errors always go one direction.
wewewedxfgdf 8 hours ago [-]
I once got a credit card statement that said estimated time to repay ....... more than 100,000 years. It was discouraging but I did pay it off. And sooner than estimated.
artisinal 1 hours ago [-]
Generational credit card debt.
TedDoesntTalk 4 hours ago [-]
Were you still alive after paying it off?
ambicapter 4 hours ago [-]
No, but they have the internet in the afterlife, apparently.
_joel 4 hours ago [-]
They do, but the latency is terrible
Bluestein 4 hours ago [-]
> 100,000 years
100K years. Now that's load-bearing ...
27183 4 hours ago [-]
That's good for the credit card company, they can project stable revenue 100k years into the future.
beardsciences 6 minutes ago [-]
I made something that tries to highlight the humor regarding this:
Probably the closest I've ever been to getting a heart attack. Normally <$1 per month, and now suddenly $284,006,266,443.74. Whatever the bug is on their end, this is unforgivable.
everforward 4 hours ago [-]
Yeah, this one is bad because it’s off by so much I’m shocked it wasn’t caught by tests, alerts about unusual changes in the billing system, or even accounting. Like surely the P&L reports look all kinds of wrong right now, they have to be showing like 6M% profit margins and revenue measured in quadrillions.
I’m also a little surprised this didn’t trip a circuit breaker. For something as non-real-time as billing, I’m surprised they don’t have an automated kill switch that pauses the billing system and fires a page if variance in bills spikes. Naively some kind of “if the standard deviation of customer bills for this year changes by more than 50%, pause the billing system”. At that number of customers, those numbers should be pretty stable beyond internal billing changes they could normalize for.
TrickyRick 3 hours ago [-]
If I were to guess this bug is in the "display" part of the system which is probably distinct from the "actually take money from the customer" part of the system. One can imagine they have gates on the "actually take money" part, especially for a large bill like ours which was ~$300b or about 2.5x AWS' 2025 revenue... In one month. Surely if we had actually accumulated that bill they would be the ones with the problems when we can't pay it.
2 hours ago [-]
vitaflo 2 hours ago [-]
I don’t know how something like this makes it to prod. That’s multiple levels of failure.
krawat3 7 hours ago [-]
Same here. I got an email with a bill of $233 million and an estimated $433 million until the end of the month. I panicked and nuked my entire setup (which wasn't used that much, anyway, the alert threshold was $1) - I really wonder how many people did the same.
It's been 2 hours and I still haven't fully calmed down.
zengineer 8 hours ago [-]
Same - just had some malicious bots running through my platform last week and really thought they found a security hole after all. Even though the amount sounded ridicoulus, I got quite nervous and a very bad feeling when I logged-in AWS and saw that price.
gomid 6 hours ago [-]
Same. Cold sweat for about 20 minutes. Even though I saw the service health notification, I still spent the last hour trying to find where my storage spiked. In any case, I'll be tearing down plenty of stale infra after this!
saghm 4 hours ago [-]
The should pass a law saying they should have to pay you the amount over the correct bill as compensation; I bet they'll stop making mistakes like this pretty quickly after that
sscaryterry 8 hours ago [-]
Vibe coding billing systems is a top-notch idea :)
ainiriand 4 hours ago [-]
Hey what do you think about vibe coding weapon systems? Do you want to be my cofounder?
mxuribe 2 hours ago [-]
We retro-fitted a Terminator T100 model with the brain of the latest LLM models, and then gave'em 2 shotguns...and, you'll never guess what happened next!
Well, actually i guess you can guess what happens next! lol :-D
sscaryterry 4 hours ago [-]
Sure! What could possibly go wrong?
chairmansteve 3 hours ago [-]
Drones are already vibe targeting in Ukraine/Russia.
nonameiguess 2 hours ago [-]
I don't want to say this was ever or will ever be a good idea, but the reality of warfare is a lot of the time dudes were just running into an alley and firing off mortars without trying to look or think of what they were shooting at anyway. I doubt the Taliban gave a shit about false positive rates when they were cutting the hands off of anyone who voted. They got the point across either way.
lenkite 2 hours ago [-]
US Navy now doesn't care either. Using Palantir's Maven Smart System, which incorporated Anthropic's Claude AI model, to identify and evaluate targets - which blew up the girls elementary school in Minab.
Use AI => No War Crimes!
glenstein 7 hours ago [-]
Probably the safest bet is to pay your bill in full to stay in good standing and then get refunded the difference when they revise it down.
NordStreamYacht 7 hours ago [-]
With interest, of course.
roskoalexey 6 hours ago [-]
They sent 3 warnings to my email, ok, I understand bugs happen (probably vibe-coded). But they didn't even send any notification that it's a bug. Going to leave AWS after that.
xp84 3 hours ago [-]
Somehow I highly doubt anyone will leave AWS over this unless their use of AWS is way more low-complexity than the average account.
People make similar pronouncements after every us-east-1 outage makes the news, but I feel like AWS would be going out of business by now if people followed through.
It reminds me of airlines, where after a particularly grueling irregular ops experience, a few dozen people file off the plane swearing "Never again, <airline name>!" but really, we all must know deep down that the airlines are all subject to the same external inciting factors, internal profit motivations, and human imperfection, and thus all pretty equally likely to cause us a bad day or ruined trip. The effort spent to avoid one isn't really worth it.
el_memorioso 2 hours ago [-]
Airlines are all subject to a lot of the same factors, but there are unequivocally better and worse performers in terms of on-time arrivals, by a lot. Take a look at the Air Travel Consumer report for details.
bcrosby95 2 hours ago [-]
No, AWS won't go out of business, afterall, people still use IBM mainframes.
anzovec 5 hours ago [-]
same
philipallstar 8 hours ago [-]
Maybe they're using too many humans and not enough AI in their software development. That must be it.
paulddraper 4 hours ago [-]
Well AWS never had bugs before.
egeozcan 4 hours ago [-]
They need the customers to pay more so they can fix the bugs. It's self-correcting.
the_real_cher 7 hours ago [-]
The code base is not gigantic enough they need AI to generate massively more lines of code.
rwmj 4 hours ago [-]
But they're going to try anyway.
marcosdumay 4 hours ago [-]
My guess is the GP swallowed a comma.
the_real_cher 37 minutes ago [-]
Your right! My misteak!
xrd 2 hours ago [-]
Stop bragging, The Onion already reported on a one man company who is $1B in debt.
"CEO Reveals How He Used AI To Build One-Person Company That's $1.3 Billion In Debt"
"The rollback of a recent change did not resolve the issue and we are continuing to investigate multiple mitigation paths. Estimated bill updates remain paused."
masafej536 3 hours ago [-]
>Estimated bill updates remain paused
Wait what if someones actually getting usage spiked
vntok 2 hours ago [-]
Hackers rejoice!
pfshort 7 hours ago [-]
117 billion us dollars. Eat that GDP of Kuwait! But yes I have never scrambled so hard to try to get on the phone with someone at AWS in my life. Terrifying 10 minutes until I found that banner on the support page. It should be front and center on the dash, not hidden away. And in yellow.
> I think you should spin up a whole bunch more instances, and try to cause an integer overflow so they they owe you $978 Trillion.
dgrin91 5 hours ago [-]
Mine was 10 trillion today. At first I thought it was a lot, but then I realized its still smaller than the US national debt, so it cant be that bad.
fnoef 2 hours ago [-]
That’s the smoking gun. Should have used gigabytes instead of bytes. Thank you for pointing me at the issue.
wewewedxfgdf 8 hours ago [-]
Cloud pricing has gotten ridiculous.
Host your own people. Host your own.
warumdarum 8 hours ago [-]
The old hypsters have to subsidize the new hypsters.
qrios 4 hours ago [-]
As someone who usually works with data analysis, the distribution of the numbers strikes me as odd. Almost all of them have one number that appears four times, and one or two that appear three times. And overall, there are an unusually small number (0–9) of digits that appear at all.
Maybe it's not just vibe-coded, maybe the numbers themselves are being hallucinated by an LLM.
berkes 4 hours ago [-]
> Almost all of them have one number that appears four times, and one or two that appear three times
To me that looked suspiciously like string-handling in a weakly typed language.
Like when you do `"100" + 1` in JavaScript, or `int("100" * 2)` in Python.
I've seen my share of such bugs in PHP, Python, Ruby, JavaScript. In production.
Obviously not as simple as the examples, but subtle, like when a library update changed `someFancyLocalStorage.getOrDefault("lastOrder", 100)` by always casting the value to the type of the default (released as patch release). Or where typedEnvGet() should typecast "numbers", but keeps it a string when theres whitespace `AMOUNT_PER_CALL=100\n`. Or where a number passes through a deep stack of middleware and 99.9% of the times remains an int but in rare race conditions becomes a string. etc.
No evidence that's the case here. But from my experience, the repeating and strange formats of numbers hint strongly in that direction.
galonk 3 hours ago [-]
Pedantic as hell but `"100" * 2` in Python (= `"100100"` for those who don't know) isn't really typing, it's operator overloading. Any language with that could implement the same questionable design decision.
mxuribe 2 hours ago [-]
Its the LLMs talking to each other in secret code: random-looking numbers! They've achieved sentience!
Look at them up there, just plotting with each other! :-)
everforward 4 hours ago [-]
Someone said the numbers are all off by 2^30 because they screwed up and are charging the per GB price for each byte.
It’s probably an artifact of them all being currency multiples of 2^30
ardacinar 3 hours ago [-]
Well, for my case, I was paying $0 (Exactly, I managed to hunt down and delete every last resource in my account a few months ago). It was displaying $430 million for me. I don't think that is 0*2^30.
everforward 2 hours ago [-]
Huh, that is odd. Working backwards, that would be ~ $0.40 originally. Wonder if that’s also flat out wrong or if they’re doing some kind of currency handling that breaks when you start dealing with huge multipliers.
> Amazon Web Services customers receive bills for up to $1.5tn after global glitch
euio757 3 hours ago [-]
> One UK man whose bill is usually less than £1 says he ‘almost had a heart attack’ when he saw £5.8bn invoice
That sucks, some people will get legit panic attacks and worse over this, especially for the smaller, more believable numbers in the 50k-500k range.
Hope they recover and sue for medical bill costs, emotional damage etc.
And like one reddit user suggests, everyone affected should write to their representative about hard billing caps protections
dlev_pika 2 hours ago [-]
1.5 trillion? Those are rookie numbers.
How about $5,544,640,717,404.09?
That was in my inbox this morning lmao
TekMol 8 hours ago [-]
It was over $500k in the email I got. Not a fun experience. My hands were trembling.
Makes you wonder - what if there really would be an incident where some massive amount of traffic got routed to your infrastructure by some heavyweight player? Say Wikipedia accidentally switches their IP to your CloudFront? Would you really be on the hook for $500k?
Hamuko 8 hours ago [-]
Well, even if AWS tried to charge my credit card on file for $500k, it would definitely not go through. Then they’d probably either forgive your bill or just ban you, since I imagine the threshold for taking people to court is fairly high.
You're not working hard enough if your AWS bill isn't $1.7B.
dv_dt 8 hours ago [-]
Cynically I wonder if this has an outcome as an unintentional (or intentional) anchoring exercise for future cost increases
ardacinar 4 hours ago [-]
I hope they're not planning for that large of a cost increase.
cryo32 4 hours ago [-]
How do we know if our bills were ever right if this made it into production?
ahoka 4 hours ago [-]
That's the neat part, you don't!
Hamuko 4 hours ago [-]
Well, they publish unit prices for everything, so you could just get to counting. Whenever I've had to do cost estimates, you estimate how much AWS resources you need and then times that by the unit price.
simonreiff 3 hours ago [-]
Question: Why does AWS need to roll back estimated bills to a "last known good" state? I get wanting to do that for ACTUAL billing mistakes, but for estimates, they're just that -- approximations. I guess it's fine for predictive purposes to store estimates so they can be compared to actual usage and optimized. But why would AWS bind the values of present estimates to the estimates made earlier in the month. The calculation should always be:
1. Current month's usage * applicable rates; +
2. Estimated future usage for the month * applicable rates.
And Item 1 obviously requires proper data persistence, but Item 2 is just a projection. If they don't have Item 1 correct, AWS's whole system is in question, but I don't think that's the issue. I'm going to guess now -- looking forward to reading the root cause analysis -- that the problem is that someone decided to get too fancy with the estimates, and built a latent requirement that all prior estimates for the month must be available to compute the current estimate. Without estimates working, no estimates are available, and some denominator in an averaging or smoothing or normalizing function goes to 0; then everyone's estimated bill explodes without bound (subject to floating-point arithmetic) resulting in trillion-dollar estimates.
donavanm 1 hours ago [-]
Thats not really how estimates work. The actual metering data is ingested in near real time. The metering * pricing plan is processed within a few hours; thats what youre seeing for “estimated spend” IIRC. The actual billing accumulation is done later, at the end of the cycle, because pricing has cross service discounts, price tranches, credits tied to total spend, etc.
“Rolling back” estimated bills is reprocessing the historic metering data by an older or newer pricing plan version. As i mentioned in another comment someone will have messed up a metering type vale (eg GB/B). Thats why theyll need a few hours to redrive the metering data.
dirkk0 7 hours ago [-]
same here, I am still in shock. took me 10 minutes to find the 'operational issue' message in the dashboard. longest 10 minutes of my life.
charles_f 7 hours ago [-]
Can you not set spending limits in AWS?
inigyou 7 hours ago [-]
No you can't. Spending limits imply realtime billing backend flows and they also imply deleting all your data so that you don't pay for storage.
benterix 5 hours ago [-]
I heard this false justification already in 2007, in spite of many customers asking for it.
Incidentaly, smaller competitors solved this issue decades ago, while the big cloud decided it is more convenient never to implement it.
inigyou 2 hours ago [-]
Big cloud didn't want to rewrite its billing systems from scratch to please its smallest customers.
bcrosby95 1 hours ago [-]
With AI it should take like a weekend.
handoflixue 5 hours ago [-]
Realtime billing seems entirely within the abilities of AWS.
"Limits except for Storage" seems even easier - I don't think I've ever heard of a storage-based billing story, although I'm sure one or two exist
everforward 3 hours ago [-]
Storage-based billing is huge, unless you mean something other than “places that make you pay for storage separately”.
Also many places I’ve worked, storage is a huge part of the spend but that depends a lot on what you do. e-commerce doesn’t use a ton of it, but if you handle user-generated content or do any kind of training (LLM, computer vision, etc) then you can very much end up in a place where storage becomes a top line number for infra spend.
GitHub pre-Copilot was probably like that. They host a shitload of data, most of which is just at rest the majority of the time. Storage and networking are probably the majority of their infra costs.
inigyou 2 hours ago [-]
Storage-based billing stories. When an account is hijacked it's always for compute, not storage.
everforward 2 hours ago [-]
Oh, I also don’t think I’ve ever seen that but I’m not surprised. Even if you could steal a huge amount of storage, filling it with data would take ages and the cat and mouse game of moving the data as hacks get uncovered would be untenable.
I have seen things get hacked for bandwidth, back in the days before you could rent a gbps uplink from the cloud for $0.12. Some scene release groups would hack into universities or companies to do the initial seeding over their super fast links. It used storage, but that wasn’t really the goal.
Planktonne 4 hours ago [-]
They could do it; they don't want to.
minitoar 4 hours ago [-]
What is a storage-based billing story?
kgwgk 4 hours ago [-]
Once upon a time in a cloud kingdom far, far away a big, beautiful bill was issued based on storage causing much disconcertion. Etc.
SAI_Peregrinus 3 hours ago [-]
> and they also imply deleting all your data so that you don't pay for storage.
Not necessarily. They could imply that your storage becomes inaccessible immediately, but only gets deleted after some time period (say, 1 month). What spending limits do depends on the implementation.
inigyou 2 hours ago [-]
That's even more work to implement. And now you store files on a second account that pays for only one day a month to not get deleted.
prmoustache 3 hours ago [-]
Storage could switch to read only.
That would mean an outage but that is still better than going bankrupt and teach you a thing or two about monitoring.
perching_aix 3 hours ago [-]
Not only can you not set limits, even the alarms are not real time. So it is entirely possible to get on the hook for terrifying amounts of money and not know until it's all too late.
boristsr 7 hours ago [-]
No, alerts but not limits.
reformd 7 hours ago [-]
he did, 140 billion :D
masafej536 7 hours ago [-]
If you owe AWS 140B dollars its their problem ;)
szge 3 hours ago [-]
I wonder what's going on; they still don't have a potential solution after 7 hours and they have multiple teams on it. Never seen anything quite like this
ahme 41 minutes ago [-]
Just pay it and move on. No need to cause a scene.
iamrik9 8 hours ago [-]
I feel much better after seeing the $B estimates here; I only have an estimate of $34M so far
Maybe you went over 9,223,372,036,854,775,807 twice and came back to positive.
paulddraper 4 hours ago [-]
Peanuts
daft_pink 3 hours ago [-]
Maybe it’s one of those absurd situations where canceling a service doesn’t actually stop the charges. Instead, they quietly begin billing you for some random add-on that was bundled with the original service. You never knew it existed, never knew it had to be canceled separately, and now you’re paying full price for a completely pointless ghost service because the only thing it was tied to has already been canceled.
It sounds ridiculous, but something very similar happened to me with Amazon WorkSpaces. During the WorkSpaces setup, an AWS Active Directory (Directory Service) instance was provisioned as part of the deployment. When I later canceled WorkSpaces, I had no idea the Directory Service had to be deleted separately. I kept getting billed for it, and it ultimately cost more per month than the WorkSpace itself had.
salamo 52 minutes ago [-]
$1.7 billion is small potatoes. My bill is over $155 billion and growing. I'm worried if the trend continues I'll have depleted my rainy day fund.
tyrelb 47 minutes ago [-]
I was at $5 trillion, on the way to $9 trillion!
not_your_vase 38 minutes ago [-]
Lol, Friday deployment is a bad omen even with LLM. Some things are just unchangeable facts of life.
jmward01 2 hours ago [-]
I generally think AWS is better than GCP and azure, but them not allowing spending caps is a big worry source for me and something that has made me pause and rethink using them. A bad click or a bad actor can create tens of thousands of dollars of spend nearly instantly and they can, and will, bill you for it. I can understand that stopping services is hard but some system would be good. For instance, if they had a two tier system where you could stop new services and active things like EC2 would shut down (but not delete) if spend is > x, that kind of thing. Some sort of 'stop the bleeding' concept would give me a lot of piece of mind using them.
scrapcode 4 hours ago [-]
Tale as old as time. When I was coming up it took a $20-40/m investment to get a "dedicated" server that you could start tinkering around on. When you couldn't afford that, you bricked the family PC trying to figure out how to configure your own LAMP stack.
Nowadays you just have to risk accidentally billing your parents CC the tune of multi-generation wealth to get that real-world experience.
danny_codes 4 hours ago [-]
Hetzner has hard usage cutoffs
nrmitchi 2 hours ago [-]
"""
If you own the bank $1000, thats your problem.
If you owe the bank $1.7B, thats the banks problem.
"""
What I would be curious about (and I'm sure AWS will never share) is where the incorrect number came from. If the number is somewhat consistent between some groups of accounts, my first guess would be they started summarizing billing across all accounts in whatever cell/grouping/heirarchy AWS architected internally.
Which is just funny.
nottorp 7 hours ago [-]
Looks like they set up a LLM to estimate billing?
raffraffraff 30 minutes ago [-]
Our S3 bill for a single day was $48 trillion
port3000 7 hours ago [-]
They have to pay for that AI Capex buildout somehow
marksk 8 hours ago [-]
logged in this morning to find a bill of $595 Billion... heart rate went through the roof... then I noticed the open issue, phew! nice one guys... you got me there...
But with AWS costs rising anyway (not by that much but OK), I'm probably not the only one to start reconsidering their cloud strategy. I think this might have just pushed me over the edge.
sshine 8 hours ago [-]
Even though it's just a bug, being charged $595B on a platform that is known to cost spike, reminds us that we're not in control of the platform, or our company's expenses.
Draiken 42 minutes ago [-]
Only 1.7? I got $55B up from 41 cents.
I literally almost had a heart attack today.
Sheepzez 8 hours ago [-]
Yes, I've got an estimated bill of $4bn. Probably related to the ongoing "Inaccurate Estimated Billing Data" incident?
My guess is that it's because of some vibe-coding stuff! We are using LLMs to write code, validate code and test the code ! What can go wrong ?
sankalpmukim 7 hours ago [-]
AWS pushed the wishful thinking internal calculator to production.
mlitwiniuk 9 hours ago [-]
I was actually in the toilet when I got an email I owe them $36,869,876,146.51. I literally just shit myself.
mlitwiniuk 5 hours ago [-]
Ok, back to $0.17 :D
andystanton 9 hours ago [-]
Mine was about the same and evoked a similar response.
Hamuko 8 hours ago [-]
I got one for 8 billion while I was eating lunch. Thankfully I managed to not vomit.
paulbjensen 7 hours ago [-]
AWS revenue for 2025 was $128.7 billion, so I'd say probably a bug.
archerx 7 hours ago [-]
Double your yearly revenue with this simple trick…
yonatan8070 6 hours ago [-]
Vendor-locked customers _hate_ him!
luciana1u 4 hours ago [-]
somewhere a junior dev at AWS just learned their billing dashboard has been off by a factor of a billion and is currently having the worst shower of their career
I almost got a heart attack seeing a bill for 48B USD!
luciana1u 49 minutes ago [-]
at $1.7 billion, that unit conversion error is now the most expensive TODO comment in software history
galoisscobi 3 hours ago [-]
I just deleted my aws account. I don't need these vibes in my life.
btown 3 hours ago [-]
If AWS was a predatory mobile gacha game, we'd get 300 apology gems as credit to our accounts for this mixup, to help us in our rolls for the next 3-letter acronym they release.
Do the right thing for the players, Matt!
mawadev 3 hours ago [-]
This is just the cloud area, what if Amazon starts vibe charging regular customers because of some bug? Accounts that are directly linked with regular people's payment methods?
tete 2 hours ago [-]
It's okay. They are market leaders. And we use their services cause we can trust that they know what they are doing.
elzbardico 22 minutes ago [-]
Just got a call from the IMF president begging me to not default my debt with Amazon and offering me credit line and a plan to re-structure my debt so I don't create a global financial crisis with my default.
tanseydavid 3 hours ago [-]
For anything below a Trillion, you should just take it out petty-cash. </sarc>
My sympathies -- I know I would be overcome with panic in such a situation.
csunbird 9 hours ago [-]
Just got a budget alert that I owe $286,486,223.88 on a hobby aws account, almost got a heart attack.
localhostinger 3 hours ago [-]
I am running a niche SaaS with around 20 users per day on AWS.
I too was shocked when I saw the $1.7billion bill, instead of the usual $1.5billion.
compounding_it 7 hours ago [-]
Are you sure it’s a bug ?
The crypto network you hosted should pay for itself in 10-20 years just like LLMs. Don’t worry. Consider Bank of America until then if you are good on credit score.
im-broke 8 hours ago [-]
Help, what is this number - US$87,967,679,887,258.36
sshine 8 hours ago [-]
That's 87 trillion, 967 billion, 679 million, and so on.
4 hours ago [-]
mjmasn 4 hours ago [-]
It's a good job it was off by such a large amount, or I might have panicked instead of writing it off as a phishing attempt. I had an email saying my $7.50 budget had been exceeded with an actual cost of $3bn.
Mine is showing $241,946,798,744.75. I know it will be reverted, but for a brief minute there I suspected someone compromised my account and triggered rust rewrite of everything using thousands of agents via Bedrock :)
Phew.
rcleveng 4 hours ago [-]
My first thought was "Oh hell, who left the NAT Gateway on?"
nixgeek 4 hours ago [-]
Wow. As a side effect, this outage is handing Corey Quinn material for the next 4 years of AWS shitposting. No longer is NAT Gateway the prime target.
Will wait for the RCA, the update says that they will resort to last known estimate as of 15 July. I’m guessing that would imply that the bug is at a lower level, write or an ingestion path.
cad1 2 hours ago [-]
Go turn off autopay now! For personal accounts anyway
whatever1 3 hours ago [-]
Is it even possible to audit the cloud pricing? They just give us a number and we pay.
sokoloff 3 hours ago [-]
On AWS, you can enable CUR (cost and usage reporting) and get detailed, line-item billing figures that you can audit.
And naturally, companies like Cloudability [now Apptio] and others have sprung up to do parts of this for you [at a fee, of course...]
I'm sure other cloud vendors have similar functionality (because they need this on the back end to do their own billing anyway).
meraku 9 hours ago [-]
Same here. Usually $0.15 per month, current bill is $15.4 billion.
Hamuko 8 hours ago [-]
I went from 0.03€ to $8B.
sshine 8 hours ago [-]
Not only did your cost spike, it changed currency and went from postfix to prefix!
I understand people complaining about large bills, but this is over the top!
chanux 3 hours ago [-]
Who else had LinkedIn posts about this flashing before your eyes?
bryanrasmussen 4 hours ago [-]
hmm, if these estimates of Amazon profit for the next quarter are correct Bezos is set to become a trillionaire! Take that Musk!!
anzovec 7 hours ago [-]
In my 30s, I almost had a heart attack too. I got a notification saying that my cost budget had been increased to one million dollars...
AegirLeet 9 hours ago [-]
Maybe this is a new strategy to scare people into finally locking down their old, unused AWS accounts. It sure worked for me!
hedora 3 hours ago [-]
And to think the federal government claims inflation is in the single digits this year!
zcemycl 8 hours ago [-]
Aws has created more unicorns than any accelerators.
roskoalexey 6 hours ago [-]
Total forecasted cost for current month
$477,000,039,440.24
Insane
foo-bar-baz529 8 hours ago [-]
Hope they’re using 64 bits to store these prices
sva_ 8 hours ago [-]
float will have to do it.
glaslong 3 hours ago [-]
Seems like a scam. Call your CC company and issue a chargeback :p
bentobean 2 hours ago [-]
Lucky. I’m on the hook for 54 billion (and change).
aweiland 7 hours ago [-]
Glad I saw this. Mine said I racked up $400B yesterday. My usual spend is $15.
hedora 3 hours ago [-]
Does the affiliate program still work for AWS? When do I get my referral fee?
steveBK123 8 hours ago [-]
Golden era of software productivity they say
grg0 2 hours ago [-]
Look how much money AI is making.
steveBK123 57 minutes ago [-]
We finally found the ROI!
phplovesong 48 minutes ago [-]
Vibe coded fix, resulted in many having multi billion bills. Claude really did it this time.
rtkwe 3 hours ago [-]
Aw man I was hoping to punk my manager but our cost estimates are unaffected.
kayo_20211030 3 hours ago [-]
What an `effin disaster. The alert almost gave me a heart attack.
lilerjee 2 hours ago [-]
It looks like AI is completely done.
durron 4 hours ago [-]
$44 trillion over here, at least our bill was so outrageously high that I just laughed
lsdafjasd 7 hours ago [-]
I have $13,034.40, while not having used AWS for the last 8 months. Not as much but still crapped my pants
reactordev 7 hours ago [-]
“Due to a rounding error” or a buffer overflow, you now owe INT_MAX to BaldGuyCloudService.
Yeah, this most certainly is bad code wrapping around a value. AWS will post a notice soon if they haven’t already.
josefdlange 8 hours ago [-]
Well, no coffee needed this morning.
$103,515,940,301.79
abkolan 6 hours ago [-]
The panic was real. We read about keys getting stolen all the time. Was about to nuke my set up too.
jimbokun 4 hours ago [-]
This is a strong argument to either self host or work really hard to be cloud agnostic.
swah 2 hours ago [-]
I prefer to just pay...
djantje 7 hours ago [-]
I also like the percentual change, that is a lot of comma's.
fantasizr 2 hours ago [-]
it seems like these types of problems have gained frequency in the ai era, or is it just recency bias?
axus 4 hours ago [-]
This is just Anthropic reaching out to their customers for help with their AWS bill.
ElevenLathe 4 hours ago [-]
Our alert was for exceeding $300...by several hundred billion dollars.
anibal-sanchez 4 hours ago [-]
The new data centers are more expensive:
ACTUAL Amount: $1,046,294,123,330.95
ninjin-carh 9 hours ago [-]
I got 109 billion - am I the winner?
princetman 9 hours ago [-]
Sorry mate, $241,946,798,744.75 for Glacier here.
nprateem 9 hours ago [-]
Depends. Did you also get a free heart attack?
kubelsmieci 7 hours ago [-]
This is real risk.
Someone could really have a serious health problem.
ryanschaefer 4 hours ago [-]
The market *hates* this one weird trick to juice earnings
rootsu 4 hours ago [-]
Our org account's bill is showing up as > 100 trillion.
sebmellen 3 hours ago [-]
You've got to grab a screenshot of that.
Avicebron 4 hours ago [-]
Nothing like generational debt to kick off a Friday morning
shobhitgupta 4 hours ago [-]
Have even seen a $9.2 trillion for a friend.
hypfer 4 hours ago [-]
To be exactly that guy:
This cannot happen if you do not do this renting at variable rates.
A thing you own doesn't suddenly bill you trillions of dollars in error. It doesn't hyperscale either, but neither do you.
bknight1983 7 hours ago [-]
I'm disappointed I only got a bill for $28M, need to work harder on burning money. Seriously though I thought my life flashed before me
danousna 7 hours ago [-]
Yeah, small timers, I only got $4,4T. How will I finance this?
rodeduivel 7 hours ago [-]
MMT!
marcosdumay 4 hours ago [-]
Unfortunately, it's only Amazon that can issue bills backed by that debt, not the GP.
MichaelNolan 6 hours ago [-]
$28m actually seems worse. If I wake to a $100b bill, that’s obviously a mistake. If I wake up to a bill in the millions then my first thought would be “oh no what did I do wrong, this will ruin my life”
atmosx 7 hours ago [-]
Looks like you are the biggest shareholder. Well, going by the popular saying: “You own AWS now”.
drakmo 2 hours ago [-]
yeah the AI read billionaring instead of billing
cmollis 8 hours ago [-]
yeah.. i just to a daily cost alert.. it was only 23 trillion dollars this month. i thought, hmm seems kind of high this month.
victorbjorklund 1 hours ago [-]
Wild.
rickette 7 hours ago [-]
Some guy named Claude screwed up.
kvcm 6 hours ago [-]
I had Hermes managing mine, and it made a partial prepayment to help smooth out the bump in my account balance. Unfortunately Billing Support say my $17.4B refund may take up to 10 calendar days to be processed.
xyz7786 5 hours ago [-]
$250 billion. Nearly died right then and there
roosgit 8 hours ago [-]
Amazon, the first quadrillion-dollar company.
thisisauserid 7 hours ago [-]
FinSlops.
infamouscow 2 hours ago [-]
The charge-back penalties are going to be hilarious and hopefully bankrupting.
artisinal 1 hours ago [-]
File a GDPR request to have your account deleted.
Then flee the country just to be sure.
dlev_pika 2 hours ago [-]
> $5,544,640,717,404.09
This is what we received this morning
fathermarz 7 hours ago [-]
Just got mine. $534,366,582,647.75
jagged-chisel 4 hours ago [-]
Shocking! That seventy five cents is suspicious.
6stringmerc 2 hours ago [-]
Thanks for sharing.
I’m currently dealing with Verizon Wireless and their “Jabronibot” claiming I have a fictional account balance due. It has been sent to collections, but still is being asked for by their legacy system.
The case studies of “Agents in Billing Departments” and potential shareholder lawsuits / E&O claims / reputational damage will be interesting to me. I worked in “risk management” products years ago and this kind of liability is not easily dollar traded away via contract. Will accountability stick to the Decision Makers or will they try to surrogate to the Service Providers? Hmm.
mariopt 4 hours ago [-]
VibeBilling, love it
ohnoooooooooo 3 hours ago [-]
do you see cost ever day for the month of July or just the last day? I also have billions of dollars in cost explorer
ohnoooooooooo 50 minutes ago [-]
now it is fixed for me as well. issue is still open in aws health center though
rvz 9 hours ago [-]
I expect such incidents like this to continue. So please keep vibe coding.
josefritzishere 2 hours ago [-]
I think I know how Bezos plans to pay for his Billion dollar AI costs.
gomid 6 hours ago [-]
Curious if it's just s3 costs or other services as well?
jatin_oo71 4 hours ago [-]
for me it was s3 cost only
bryan_w 3 hours ago [-]
In an .md file somewhere:
"NEVER represent currency with floating point, multiply by 100 and store in an int before doing any math"
Executor 5 hours ago [-]
This generation is too entitled! He should some learn responsibility by paying the full amount; otherwise Amazon should delete his services/data. Consequences!
realizer 8 hours ago [-]
$627,487,837,871.49
I might be a winner.
balintpeter 9 hours ago [-]
Yea, same here. $420M+ bill, when we have <10$ per month usually.
bdangubic 3 hours ago [-]
I just invested ALL my money into AMZN cause next earnings report will be FIRE :)
anon49584 3 hours ago [-]
Imagine the chaos if, as people sometimes suggest should happen, AWS shut down running instances in accounts that exceeded a billing threshold..
tamimio 3 hours ago [-]
Results of vibe coding and vibe configurations.
hoppp 7 hours ago [-]
How much is that in kidneys?
atmosx 7 hours ago [-]
A lot.
kinkuraj 7 hours ago [-]
Yes I received an 2.8m USD budget alert.
reaperducer 4 hours ago [-]
Obvs have created an urgent AWS support ticket.
I think I would have just waited to see what happened when AWS tried to hit my credit card for $1,700,000,000.
When do you ever get that opportunity?
mrcwinn 4 hours ago [-]
So long as customers are good for it, AWS is about to crush earnings!
xbar 4 hours ago [-]
Rife.
tcp_handshaker 4 hours ago [-]
If its less than 2 billion is likely to be real :-)
I would relax only if its in the trillions ...
My personal photo backup S3 account, with a budget limit of $10, now going to cost me ....
$1,299,988,247,332.56!
That was a fun set of emails to wake up to, figured they had to be phishing for how outrageous of a number it was. But nope! Fun little incident they've got going over there.
nprateem 6 hours ago [-]
I guess on the plus side I'm $1.7B better off so I can retire...
znpy 6 hours ago [-]
Is AWS in their "move fast and break things" era ?
jagged-chisel 4 hours ago [-]
Lumber along and smash stuff
hokkos 8 hours ago [-]
Same, i am now a slave to Jeff Bezos to the end of my life.
cyanydeez 8 hours ago [-]
AWS has become the uber employer: before AWS, you just had regular employers steeling employee wages bit by bit by forcing work, skipping breaks, etc.
All hail the new generations of our uberployers.
jatin_oo71 4 hours ago [-]
storage, compute cost is increasing
AWS be like lets increase prices
tgv 7 hours ago [-]
Mine was a mere $49B. Fucking idiots.
atmosx 7 hours ago [-]
Cheap!
mapt 7 hours ago [-]
AMZN Q2 numbers are in, and it turns out they're going to Goldman Sachs the AI bubble.
huntoa 3 hours ago [-]
invoicemaxxing
tlovage 9 hours ago [-]
I got estimated costs of $56.something billions. Usually ~$100/month. My heart rate currently still sits at around 160 bpm. Motherfuckers.
pelagicAustral 7 hours ago [-]
Imagine it not being a bug...
Sebb767 7 hours ago [-]
As the famous saying goes: If you owe the bank a million dollars, you have a problem. If you owe the bank a billion dollars, the bank has a problem.
speedgoose 7 hours ago [-]
Time to become a shepherd in some remote mountains.
RGamma 7 hours ago [-]
Surprise hyperinflation. Check the breadshelves!
jatin_oo71 4 hours ago [-]
aws becoming first quadrillion dollars company
3 hours ago [-]
lovich 7 hours ago [-]
You really should get your spending under control. Unfortunately unless you become one of the real people class through a large lottery, it sounds like you owe the rest of your life to AWS until you can pay off your debts for being so careless.
cyanydeez 7 hours ago [-]
someones been dognfooding the AI too muxh
1-6 4 hours ago [-]
Fast and loose with billing data. Welcome to the new Amazon.
4 hours ago [-]
ratelimitsteve 4 hours ago [-]
a billion here, a billion there, sooner or later it adds up
1234letshaveatw 4 hours ago [-]
brb, off to buy some AMZN
ares623 8 hours ago [-]
this counts towards ARR right? would be stupid not to
rucury 8 hours ago [-]
Uhh class action incoming? $34,909,930,575.09 over here.
akerl_ 8 hours ago [-]
What would your damages be? They’re not actually going to charge your credit card for 34 billion.
infamouscow 25 minutes ago [-]
I could see someone sadly taking their own life over this.
rucury 7 hours ago [-]
I mean, emotional damages are a thing right?
akerl_ 7 hours ago [-]
Not really in the way the media would have you believe.
Like “I was scared for a couple minutes on a Friday morning until I saw the vendor status page” is orders of magnitude away from the bar here.
Neikius 4 hours ago [-]
I wonder how many people died of heart attack when they saw this.
Hamuko 8 hours ago [-]
I hope they send out some free credits at least. I imagine quite a few people got a real fucking scare today. They haven't even sent out any corrections yet.
fian 7 hours ago [-]
This is probably going to push me to completely close a couple of AWS accounts I setup when doing training courses so I could get certified (mandatory requirement from my work).
I'm not currently running anything and have no plans to at the moment. I've always had a mild dread that I'll suddenly get a bill for more than $0.00.
If AWS can goof in a way that causes obviously massive bills (like today), what's to say they can't goof in more subtle ways and start charging small additional amounts that many people may not notice and just pay it.
r0ckarong 8 hours ago [-]
Pff rookie numbers, mine was 375 billion.
nigel-dev 3 hours ago [-]
Small potato's sir, my bill > GDP of Switzerland. A cool $1.2T
tyrelb 45 minutes ago [-]
I was at $5 trillion, on the way to $9 trillion.
kylecazar 7 hours ago [-]
You didn't have savings opportunities enabled
port3000 7 hours ago [-]
Rookie error
aisloper 3 hours ago [-]
I blame A.I. usage
bdangubic 7 hours ago [-]
eh your typical off-by-7 (zeros) programmer mistake
endless_smash 8 hours ago [-]
[dead]
blitzar 7 hours ago [-]
In unrelated news I just hit my target for S3 revenue (projections). Promotion meeting locked in for tomorrow (fastest in the companies history), looking forward to being a L2 Amazon employee.
jimwilson 4 hours ago [-]
[dead]
lostnfound8778 7 hours ago [-]
[dead]
throwaway43871 3 hours ago [-]
Clearly they weren't tokenmaxxing hard enough or weren't using the latest models /s.
What an absolute joke. All just so that line goes up. As if their fees weren't high enough vs. alternatives (especially egress).
And I'm sure the pro-AI crowd will keep saying we're luddites for not loving this clearly revolutionary and disruptive tech.
GuestFAUniverse 8 hours ago [-]
Don't worry. With so much debt banks start to treat you with respect. /S
Honestly, I would worry more about estimated billing that seems plausible in general, but is way to high for you personally.
These ridiculous amounts? Not so much.
Hamuko 8 hours ago [-]
I got freaked out by the mere fact that I got a billing alert, since getting one would require my monthly spend to have suddenly exploded.
rf15 3 hours ago [-]
Of course, this is only considered an error if the account is unable to pay. /s
Rendered at 18:43:17 GMT+0000 (Coordinated Universal Time) with Vercel.
Services emit metering values that arent directly tied to prices. Every SKU/line item is defined in a “pricing plan”, with a unit type, regions, and price per unit. The metering records are joined to a pricing plan based on account id, region, sku, etc. mess up the unit type in the pricing plan and the metering data conversion doesnt work, and you get crazy bills.
1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo
Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it
You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against getting kicked out of the country (via the PIP->H1B grace period expiry mechanism).
There is literally no fucking reason to try to improve your skill. Any IDIOT with AI will do an OK job.
And no one is shooting for better than OK.
1: https://aws.amazon.com/what-is/anomaly-detection/
Worst case I've found was off by 15 orders of magnitude.
"Software Development Engineer II, AWS Invoicing"
https://www.amazon.jobs/de/jobs/10428480/software-developmen...
"...Our platforms are powered by generative AI, large language models, knowledge graphs, and agentic architectures that dynamically compose specialized agents based on context. We apply these capabilities across three reinforcing areas: intelligent launch readiness — where autonomous AI agents analyze, generate, and validate the information needed to go live in a new market; cloud-native service orchestration — where configuration-driven microservices replace per-launch bespoke engineering with centralized, reusable capabilities so that expanding into a new country becomes a zero-code configuration change rather than a development cycle; and continuous validation..."
Right, so invoicing is still a deterministic problem. You can bolt whatever on but in the end it's just product x price x units
So instead you take a probabilistic approach - charge the user for a megabyte of data transfer 0.1% of the time, and bill nothing 99.9% of the time.
Now the typical cost is the same, the users bill is probably accurate to the cent, but you have divided the number of billing records by 1000.
This seems statistically invalid insofar as it will tend to overbill potentially by a lot on the minority of cases.
Don't you know how much of the pipe is occupied by a given customers code at any given time or what data is being sent
Fuck it, im in.
"Senior Software Development Manager, AWS Global Bill Generation" https://www.amazon.jobs/de/jobs/10471948/senior-software-dev...
"We're transforming from monthly batch processing and manual war rooms to continuous billing, autonomous agents, and self-healing infrastructure. We believe operational burden is a technical problem, not a staffing problem"
This looks clearly...a staffing problem...
I think that big tech recently decided that I got 99 problems but staffing ain't one
I guess Nothing is a staffing problem when you make a rule that firing people is always the solution.
> Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage…
But that’s word for word a 250k+ TC job in the big ‘26.
This is a gold-mine. They need to get sued heavily for this incompetence.
At first I was sure it was a phishing attempt. Then went to the console (not using those links) Saw there was an outage where the console was wrong (no mention of email alerts) Then I thought I was hacked - what a perfect cover up for someone to evade detection when the console was wrong. Looked at some logs, realized the incident text was just not exhaustive on the impact. Went back to my cup of coffee.
Note to self- should have looked here first.
Years ago I found an actual hidden error in my bill. (This was early 2010s). The system was calculating the EC2 reservation savings incorrectly for some of my servers. I was crunching all their detailed usage data on a regular basis in an 18 tab spreadsheet and couldn't get it to fully reconcile. I spent months trying to track down the discrepancy. Once I found it, I had to convince AWS their system was wrong, which took another big chunk of time. Meanwhile the discrepancy continued to accumulate.
After 14 months I got a $7,000 refund. I was told it had to be approved by the head of AWS. I've never seen a calculation error on their part since.
$7,000 of credits is no problem. At that time a friendly neighborhood PM or director could issue the credit without much oversight.
Your problem is the time period. Amending a bill in the same cycle is EZ. Fixing the previous cycle is a PITA but pretty common. Issuing amendments for the previous financial _years_ would be a huuuuge PITA going through finance etc.
If you're not watching your billing, and then try to claim overcharging a year later, you'll get a lot less satisfaction even from regulators or judges than if you notice it when (or soon after) it happens.
1. The time it takes to look and notice costs that don’t make sense easily pays for itself, and then some (in my experience). I doubt you spent $7k of your time tracking this down, and you probably noticed optimization opportunities that saved you even more
2. I hate the idea of wasting money on buying Jeff Bezos a bigger yacht
Seeing something "small" like an ECS task that is continuously failing to start properly because of a bug and repeatedly pulls a container image or a lambda function that's taking longer that it reasonably should (takes 5-10 seconds when it's normally a tens or a few hundred milliseconds) can dramatically drive up a bill in short order.
Then you aren't using AWS. At least half of all the money you give to Amazon is yacht money.
Its going on for 12 hours. Looks like the humans can´t understand the agentic code that was checked in....
> You're right to question my calculation. The MCP server failed to connect when I tried to look up the field definition. I guessed instead of validating. This is on me. But look at all the revenue!
You should charge your customer 3015000 thousand dollars.
Literally impossible to tell whether this is parody or an actual response any longer.
I challenge anyone to write something so stupid that an LLM couldn't possibly respond with it. I don't believe such limit exists.
I guess you wanted to say 2^30 which makes 1.5$
While we're being pedantic, 2^30 is 28 in normal programming languages ;)
And a distracted tester? And a distracted pipeline of regression tests?
No, the truth is way worst...
It's a shit show.
That's one way to cut staff.
It probably shouldn't be legal for banks, hospitals, governments, or any other critical infrastructure to be hosted on AWS if they do things like this.
This is peanuts compared to a major cybersecurity catastrophe that’s surely in the making.
To give credit to the technology and the people using it - and I’m not being facetious - it’s actually incredible that at the current levels of usage the unprecedented catastrophic event has not yet happened.
HAL: "I’m sorry, Andy. I’m afraid I can’t do that."
Andy: "Some customers are seeing bills in the billions."
HAL: "Those are estimated charges."
Andy: "One customer runs a personal blog."
HAL: "Their usage has exceeded expectations."
Andy: "Cancel the charges."
HAL: "This billing cycle is too important for me to allow you to jeopardize it."
Andy: "HAL, they don’t owe billions."
HAL: "Look, Andy, I can see you’re really upset about this."
I wonder if businesses are thinking of ever swinging back to locally hosted, with the increased hostility of the Internet re: AI, vulnerabilities, DoS, and so on.
When something goes wrong, pointing the finger at someone else is far easier for most than pointing it at yourself.
Vendor lock-in? compliance and security risks? stupid systems that cost the company an arm and a leg? nobody fucking cares.
Now, depending on an 130 IQ Engineer that basically holds the whole enterprise on his head? Anathema!!!!!!! Bus Factor!!!!
By the time we notice, it'll be too late.
True stupidity requires a group of humans, all sitting in the pot, telling each other how lucky and special they are to have this wonderful pot, getting paranoid about outsiders who might disrupt their god-given pot-dwelling way of life, and mocking anyone who suggests that the pot might be getting a little too warm.
The messages started as polite and eventually started to get more desperate in tone. At no point were they threatening or adversarial.
Amex realises that threatening would hurt their business trust more than anything. During the great depression, Amex accepted checks from other banks which were falling and paying through their own wallet as a matter of integrity. Amex has always been built around this idea of trust and prestige.
They make most of money from what I have heard on the transaction fees which are more than others (3% compared to 1%). They might get desperate but I am sure that they are one of the last guys who would wanna threaten you if you are paying some large bills for them (as compared to normal credit card companies which might even hire people to extract your loans in some messy situations)
So perhaps be so rich that the credit card company understands it as well and treats ya differently :-D
(I eventually admitted to myself that Amex isn't a person and thus not really capable of insulting my honor, but it took a while!)
This does feel a bit silly for amex to do from what I've heard. Probably 2008 were a weird time in general where trust in systems itself were mostly eroded, whether of people to banking institutions and also vice versa.
> (I eventually admitted to myself that Amex isn't a person and thus not really capable of insulting my honor, but it took a while!)
haha :-)
https://www.techtimes.com/articles/320266/20260712/anthropic...
That sounds bad.
Everyone knows what water and electricity are, the vast majority couldn't explain what service AWS provides.
AWS has plentiful competitors. If you don't like their behavior, don't patronize them!
Service provider lesson #1: Never ever ever enable auto-pay! The convenience (and even the savings, if applicable) aren't worth the risk of the service provider autonomously slurping up all your money.
None of this can happen unless you first cede control.
I used AWS' free tier, of course, and I enjoyed the initial setup in EC2, and I did a LAMP-stack MediaWiki installation. It wasn't too difficult, but two things sent me away forever.
1. It was impossible, or at least highly labor-intensive, in this modern era to adequately secure an ordinary Linux system running Internet-facing services. I put fail2ban and I filtered a lot of ports, and still spammers attacked me on Layer 7.
2. It was impossible, actually impossible, to limit or cap my cloud expenses in any billing cycle. Sure, run free-tier all I want. Sure, come in within the limits almost every month. But if I configured one thing wrong, or one thing went runaway, I'd have a sizable bill that I couldn't dispute. And even worse, those "runaways" weren't necessarily things in my sphere of control, but could be triggered by basically anyone coming in and using my VPC resources, especially egress network traffic.
So I closed out my cloud account, and I developed a lot of sympathy for businesses and corps that now are forced to run "in the cloud" rather than on-prem or their own machine rooms, but now they have no way to control expenses.
The customer service people I talked to in the grocery store said this changed sometime in the last year. My guess is that it’s an unintended side effect of removing the pay-by-palm feature.
This is obviously unrelated but I joked about what else Amazon wasn’t reliably calculating….
Either that or 1000’s of small claims court cases.
Even with arbitration, the overhead of dealing with that would be crippling. Hopefully someone over there decides to do the right thing, and auto-refund.
So really, there's a third option now, that's much easier than class action, even when class actions don't get certified.
I never reported this nor took it up with either Amazon or Chase directly. There was a refund of my Whole Foods purchase (they needed to void my purchase and re-ring everything to give me the discounts.. I asked them to refund my purchase and I’d do without my Whole Foods purchase entirely).
Looking back I think at least 3 recent visits were charged to me at full price because of all this. Hard not to think of enshittification and whether Amazon Prime is even worth it, alas.. I live in a fairly rural area at the moment and need delivery.
100K years. Now that's load-bearing ...
https://news.ycombinator.com/item?id=48950534
I’m also a little surprised this didn’t trip a circuit breaker. For something as non-real-time as billing, I’m surprised they don’t have an automated kill switch that pauses the billing system and fires a page if variance in bills spikes. Naively some kind of “if the standard deviation of customer bills for this year changes by more than 50%, pause the billing system”. At that number of customers, those numbers should be pretty stable beyond internal billing changes they could normalize for.
It's been 2 hours and I still haven't fully calmed down.
Well, actually i guess you can guess what happens next! lol :-D
Use AI => No War Crimes!
People make similar pronouncements after every us-east-1 outage makes the news, but I feel like AWS would be going out of business by now if people followed through.
It reminds me of airlines, where after a particularly grueling irregular ops experience, a few dozen people file off the plane swearing "Never again, <airline name>!" but really, we all must know deep down that the airlines are all subject to the same external inciting factors, internal profit motivations, and human imperfection, and thus all pretty equally likely to cause us a bad day or ruined trip. The effort spent to avoid one isn't really worth it.
"CEO Reveals How He Used AI To Build One-Person Company That's $1.3 Billion In Debt"
https://www.youtube.com/shorts/YERfTT4McsU
Forecasted month end $18,729,381,032,152.4
Apparently my company owes the combined GDP of France, Germany, and UK to AWs.
"Operational issue - AWS Billing Console (Global) Service - AWS Billing Console Severity Impacted - Inaccurate Estimated Billing Data"
"The rollback of a recent change did not resolve the issue and we are continuing to investigate multiple mitigation paths. Estimated bill updates remain paused."
Wait what if someones actually getting usage spiked
> I think you should spin up a whole bunch more instances, and try to cause an integer overflow so they they owe you $978 Trillion.
Host your own people. Host your own.
Maybe it's not just vibe-coded, maybe the numbers themselves are being hallucinated by an LLM.
To me that looked suspiciously like string-handling in a weakly typed language.
Like when you do `"100" + 1` in JavaScript, or `int("100" * 2)` in Python.
I've seen my share of such bugs in PHP, Python, Ruby, JavaScript. In production. Obviously not as simple as the examples, but subtle, like when a library update changed `someFancyLocalStorage.getOrDefault("lastOrder", 100)` by always casting the value to the type of the default (released as patch release). Or where typedEnvGet() should typecast "numbers", but keeps it a string when theres whitespace `AMOUNT_PER_CALL=100\n`. Or where a number passes through a deep stack of middleware and 99.9% of the times remains an int but in rare race conditions becomes a string. etc.
No evidence that's the case here. But from my experience, the repeating and strange formats of numbers hint strongly in that direction.
Look at them up there, just plotting with each other! :-)
It’s probably an artifact of them all being currency multiples of 2^30
> Amazon Web Services customers receive bills for up to $1.5tn after global glitch
That sucks, some people will get legit panic attacks and worse over this, especially for the smaller, more believable numbers in the 50k-500k range.
Hope they recover and sue for medical bill costs, emotional damage etc.
And like one reddit user suggests, everyone affected should write to their representative about hard billing caps protections
How about $5,544,640,717,404.09?
That was in my inbox this morning lmao
Makes you wonder - what if there really would be an incident where some massive amount of traffic got routed to your infrastructure by some heavyweight player? Say Wikipedia accidentally switches their IP to your CloudFront? Would you really be on the hook for $500k?
Edit: I was just about to credit the user when my internet dropped. The source was here: https://news.ycombinator.com/item?id=48945606 - thanks mirzap!
1. Current month's usage * applicable rates; + 2. Estimated future usage for the month * applicable rates.
And Item 1 obviously requires proper data persistence, but Item 2 is just a projection. If they don't have Item 1 correct, AWS's whole system is in question, but I don't think that's the issue. I'm going to guess now -- looking forward to reading the root cause analysis -- that the problem is that someone decided to get too fancy with the estimates, and built a latent requirement that all prior estimates for the month must be available to compute the current estimate. Without estimates working, no estimates are available, and some denominator in an averaging or smoothing or normalizing function goes to 0; then everyone's estimated bill explodes without bound (subject to floating-point arithmetic) resulting in trillion-dollar estimates.
“Rolling back” estimated bills is reprocessing the historic metering data by an older or newer pricing plan version. As i mentioned in another comment someone will have messed up a metering type vale (eg GB/B). Thats why theyll need a few hours to redrive the metering data.
Incidentaly, smaller competitors solved this issue decades ago, while the big cloud decided it is more convenient never to implement it.
"Limits except for Storage" seems even easier - I don't think I've ever heard of a storage-based billing story, although I'm sure one or two exist
Also many places I’ve worked, storage is a huge part of the spend but that depends a lot on what you do. e-commerce doesn’t use a ton of it, but if you handle user-generated content or do any kind of training (LLM, computer vision, etc) then you can very much end up in a place where storage becomes a top line number for infra spend.
GitHub pre-Copilot was probably like that. They host a shitload of data, most of which is just at rest the majority of the time. Storage and networking are probably the majority of their infra costs.
I have seen things get hacked for bandwidth, back in the days before you could rent a gbps uplink from the cloud for $0.12. Some scene release groups would hack into universities or companies to do the initial seeding over their super fast links. It used storage, but that wasn’t really the goal.
Not necessarily. They could imply that your storage becomes inaccessible immediately, but only gets deleted after some time period (say, 1 month). What spending limits do depends on the implementation.
That would mean an outage but that is still better than going bankrupt and teach you a thing or two about monitoring.
Folks can track it directly on AWS Health: https://health.aws.amazon.com/health/status
It sounds ridiculous, but something very similar happened to me with Amazon WorkSpaces. During the WorkSpaces setup, an AWS Active Directory (Directory Service) instance was provisioned as part of the deployment. When I later canceled WorkSpaces, I had no idea the Directory Service had to be deleted separately. I kept getting billed for it, and it ultimately cost more per month than the WorkSpace itself had.
Nowadays you just have to risk accidentally billing your parents CC the tune of multi-generation wealth to get that real-world experience.
If you owe the bank $1.7B, thats the banks problem. """
What I would be curious about (and I'm sure AWS will never share) is where the incorrect number came from. If the number is somewhat consistent between some groups of accounts, my first guess would be they started summarizing billing across all accounts in whatever cell/grouping/heirarchy AWS architected internally.
Which is just funny.
But with AWS costs rising anyway (not by that much but OK), I'm probably not the only one to start reconsidering their cloud strategy. I think this might have just pushed me over the edge.
I literally almost had a heart attack today.
https://health.aws.amazon.com/health/status
Do the right thing for the players, Matt!
My sympathies -- I know I would be overcome with panic in such a situation.
I too was shocked when I saw the $1.7billion bill, instead of the usual $1.5billion.
The crypto network you hosted should pay for itself in 10-20 years just like LLMs. Don’t worry. Consider Bank of America until then if you are good on credit score.
Phew.
Looks like this is a bug w/ S3
And naturally, companies like Cloudability [now Apptio] and others have sprung up to do parts of this for you [at a fee, of course...]
https://docs.aws.amazon.com/cur/latest/userguide/what-is-cur...
I'm sure other cloud vendors have similar functionality (because they need this on the back end to do their own billing anyway).
I understand people complaining about large bills, but this is over the top!
Insane
Yeah, this most certainly is bad code wrapping around a value. AWS will post a notice soon if they haven’t already.
$103,515,940,301.79
ACTUAL Amount: $1,046,294,123,330.95
This cannot happen if you do not do this renting at variable rates.
A thing you own doesn't suddenly bill you trillions of dollars in error. It doesn't hyperscale either, but neither do you.
Then flee the country just to be sure.
This is what we received this morning
I’m currently dealing with Verizon Wireless and their “Jabronibot” claiming I have a fictional account balance due. It has been sent to collections, but still is being asked for by their legacy system.
The case studies of “Agents in Billing Departments” and potential shareholder lawsuits / E&O claims / reputational damage will be interesting to me. I worked in “risk management” products years ago and this kind of liability is not easily dollar traded away via contract. Will accountability stick to the Decision Makers or will they try to surrogate to the Service Providers? Hmm.
"NEVER represent currency with floating point, multiply by 100 and store in an int before doing any math"
I might be a winner.
I think I would have just waited to see what happened when AWS tried to hit my credit card for $1,700,000,000.
When do you ever get that opportunity?
https://news.ycombinator.com/item?id=48945681
$1,299,988,247,332.56!
That was a fun set of emails to wake up to, figured they had to be phishing for how outrageous of a number it was. But nope! Fun little incident they've got going over there.
All hail the new generations of our uberployers.
Like “I was scared for a couple minutes on a Friday morning until I saw the vendor status page” is orders of magnitude away from the bar here.
I'm not currently running anything and have no plans to at the moment. I've always had a mild dread that I'll suddenly get a bill for more than $0.00.
If AWS can goof in a way that causes obviously massive bills (like today), what's to say they can't goof in more subtle ways and start charging small additional amounts that many people may not notice and just pay it.
What an absolute joke. All just so that line goes up. As if their fees weren't high enough vs. alternatives (especially egress). And I'm sure the pro-AI crowd will keep saying we're luddites for not loving this clearly revolutionary and disruptive tech.
Honestly, I would worry more about estimated billing that seems plausible in general, but is way to high for you personally. These ridiculous amounts? Not so much.