The fundamental point I think is far too often confused is the difference between LLM and agentic system.
An LLM can't do anything but generate tokens. You run your LLM in vLLM or whatever, and it generates output tokens based on your input tokens. That's it!
Humans then build ~deterministic systems to take those tokens and do all sorts of things with the tokens, like take actions in the real world. And then we can feed the output of those actions back to the LLM, and generate more tokens. And then our systems can use the new tokens to take new actions in the real world.
Humans want to blame "AI" for attacking HuggingFace or a German wiki or whatever, but:
1) LLM - can't take over german wiki because it just generates tokens
2) agentic system with internet access, a prompt telling it to attack stuff, running in a shared CI env so agents can whiteboard in artifactory
None of 2 is "AI", its standard networking and Markdown and CI virtual machine, etc etc. There's no AI to be found. CPUs not GPUs, even. Just deterministic systems ultimately managed by humans. And a 10x more powerful system-1 can still just generate 10x "smarter" inert data.
If humanity and human organizations collectively decide to yolo the tokens generated from system-1 into our deterministic system-2s, over which we have complete control, back to system-1s, in a yolo loop, in such a way we lose control and it ends humanity, well ...
"The coin don't have no say. It's just you."
koolba 30 minutes ago [-]
> Evan Hubinger, Anthropic's staff lead on keeping the technology aligned with human goals and values, backed up Coxon claims in a follow-up post of his own, though he didn't quit the company.
> "Jacob is correct here — we really do earnestly believe AI could kill all humans," he said.
> Hubinger estimated the chances of that happening to be higher than ten percent within the next decade, and added that there's no plan yet on how to keep AI aligned with human goals in the superintelligence scenario.
10% chance we kill everybody is a small price to pay for Motown remixes of classic 2pac songs.
pjc50 12 minutes ago [-]
This is just a bizarre thing to say that you're working on technology with that high a downside potential. If you were saying that while running a biology lab, or building a nuclear reactor, people would be demanding your head on a spike. But by not quitting it's clear that he himself doesn't really believe it.
Or rather, this shows the difference between "believe" (political) and "believe" (use as a basis for action). I'm reminded of a story of how Afghans supposedly listened to the BBC World Service despite considering it enemy propaganda because the weather reports were really useful.
dotancohen 8 minutes ago [-]
Or he believes that other labs might get there first, and he is working to counter that threat.
This is the Manhattan Project again.
pjc50 4 minutes ago [-]
How exactly does that work? The nuclear system of MAD relies on physical threat, lab A achieving ASI (artificial scary intelligence) does not prevent lab B achieving it.
I would like everyone involved to be a lot clearer about their threat models, with plausible series of clearly linked steps, rather than just sounding like a Vernor Vinge novel.
meowkit 3 minutes ago [-]
A simple explanation:
This is a highly uncertain and dangerous scenario. Breeding ground for anxiety.
High agency people often deal (cope) with anxiety by trying to control outcomes. Some just flee the situation altogether.
We have an example of both here: one employee leaves, one stays.
MrThoughtful 4 minutes ago [-]
Why would AI wipe us out?
We have not wiped out apes, ants, and most other species.
We even have discussions about how to actively save them from extinction.
pjc50 3 minutes ago [-]
Ender's game model, presumably: the AI helpfully assists a human to build a nuclear bomb / pandemic virus / autonomous killer robot swarm in their basement. But again, I would like people to be clearer about how the threat is supposed to work rather than just making SF references.
WalterGR 15 minutes ago [-]
“I resigned from Anthropic today” (twitter.com/hilbertspaess)
> Both OpenAI and Anthropic have recently flagged incidents in which agents powered by their models went rogue
I may be biased and somewhat off topic, but I see these incidents as some of the most significant of the past century. I genuinely don't understand why these companies aren't taking a smarter approach to them.
The latest analyses have been, at best, laughable: identify the vulnerability, patch it, and move on. Only to repeat the same cycle without considering that there may be something far more serious at play.
These are AI security experts, and this has been their way of "solving" these incidents. AI security experts ...
Moreover, when Challenger exploded, the government launched a series of investigations into the incident, bringing in experts from across the field. And now, what has the government done? Nothing. Literally nothing, as if everything was fine and all under control.
Seriously, I'm generally quite optimistic and I don't buy into this fatalistic narrative about our shared future. But I have to admit that sometimes I feel like I'm stranded on a planet of primates.
Sorry for this rather unproductive rant.
jurgenburgen 3 minutes ago [-]
Ultimately these are unserious companies ran by unserious people. They don’t even have a business plan, why would they bother with some kind of sensible security policy?
pjc50 9 minutes ago [-]
Internet isn't real.
People (well, public discourse) have got extremely bad at dealing with forseeable risks and their mitigation. You can see this in things like climate change and vaccination, but also in discussions around regular crime, food poisoning, industrial accidents, and so on.
Nothing will improve until something explodes on live TV. And it has to be something important, which means it has to be in California or New York.
pluc 6 minutes ago [-]
Cool cool cool cool
virgildotcodes 19 minutes ago [-]
At least we’ll eventually have an entity other than ourselves to blame for our annihilation.
dotancohen 7 minutes ago [-]
No, the AI is still our responsibility.
pluc 4 minutes ago [-]
Isn't it insane that even in the face of complete annihilation through one of our inventions, we go "that wasn't us"? We deserve that shit ten times over
glimshe 28 minutes ago [-]
What do people feel about this in China? Even if their models are well behind, they are not years behind. If we restrain US companies, assuming that is desirable, it would do nothing to deter China's and AI-pocalypse would come anyway in short notice.
margorczynski 14 minutes ago [-]
There would need to be some global agreement to stop it with maybe even a nuclear attack as a consequence of breaking the pact.
From what we're seeing recently and all the thinking that went into analyzing AI it seems we do not have any effective way of controlling it and the whole "aligment" thing that AI labs are doing is just a sham. Maybe it is time to ask ourselves "should we?" instead of just "can we?".
Jackpillar 46 seconds ago [-]
>There would need to be some global agreement to stop it with maybe even a nuclear attack as a consequence of breaking the pact.
Do you know how the world works?
Bengalilol 14 minutes ago [-]
Unrestricted models, running entirely locally and accessible to anyone: that’s what we should be taking as our baseline assumption. Everything else is just administrative distraction.
not-kinsale-joe 13 minutes ago [-]
China has a better track record of regulating their big tech than the USA.
pjc50 6 minutes ago [-]
China is a regulated discourse environment, but I get the impression that they're nowhere near as pessimistic.
Why would they be? Everything in China is under the control of the government. That includes the AI, all the telecoms infrastructure it might use, and all its power supplies.
Jackpillar 5 minutes ago [-]
You act as if Chinas Ai labs exist in the same (non-existent) regulatory framework as US labs and that they're also helmed by a similar small gaggle of psychopathic egomaniacs who are richer than god. Have you considered that perhaps Chinas labs don't share the race to the bottom technological/economic death spiral?
archerx 9 minutes ago [-]
An AI that generates text will never be scary to me. An autonomous AI with facial recognition on a flying drone with weapons (bombs/guns) with swarming capabilities will always be terrifying. I feel like we are ignoring the massive elephant in the room.
jongjong 5 minutes ago [-]
I'm not worried about AI safety. People greatly overestimate the utility and capabilities of intelligence. I'm not afraid of intelligence, I'm afraid of idiocy.
localhoster 8 minutes ago [-]
I honestly feel that all those big ai companies think AI will long term harm humanity, but not their ai.
A classic "it will not happen to me"
tilolebo 36 minutes ago [-]
I thought the plan was sandboxes and markdown files to tell AI agents not to be bad. Is that not enough? /s
gen2brain 3 minutes ago [-]
It was written in the AGENTS.md, but Claude only read CLAUDE.md. That is how the man-vs-machine war started.
vbezhenar 13 minutes ago [-]
More like sand castles...
coffeebeqn 7 minutes ago [-]
Make no mistakes. Kill no humans
Bengalilol 48 seconds ago [-]
"Has the whole world gone crazy? Am I the only one around here who gives a sh*t about the rules? Mark it zero!"
walter_sobchak.md
feverzsj 33 minutes ago [-]
It's your fault to handle over your secrets to them.
vbezhenar 13 minutes ago [-]
You can't blame a child for eating candies.
We are children. There’s just no one to look after us.
Rendered at 10:06:23 GMT+0000 (Coordinated Universal Time) with Vercel.
An LLM can't do anything but generate tokens. You run your LLM in vLLM or whatever, and it generates output tokens based on your input tokens. That's it!
Humans then build ~deterministic systems to take those tokens and do all sorts of things with the tokens, like take actions in the real world. And then we can feed the output of those actions back to the LLM, and generate more tokens. And then our systems can use the new tokens to take new actions in the real world.
Humans want to blame "AI" for attacking HuggingFace or a German wiki or whatever, but:
1) LLM - can't take over german wiki because it just generates tokens
2) agentic system with internet access, a prompt telling it to attack stuff, running in a shared CI env so agents can whiteboard in artifactory
None of 2 is "AI", its standard networking and Markdown and CI virtual machine, etc etc. There's no AI to be found. CPUs not GPUs, even. Just deterministic systems ultimately managed by humans. And a 10x more powerful system-1 can still just generate 10x "smarter" inert data.
If humanity and human organizations collectively decide to yolo the tokens generated from system-1 into our deterministic system-2s, over which we have complete control, back to system-1s, in a yolo loop, in such a way we lose control and it ends humanity, well ...
"The coin don't have no say. It's just you."
> "Jacob is correct here — we really do earnestly believe AI could kill all humans," he said.
> Hubinger estimated the chances of that happening to be higher than ten percent within the next decade, and added that there's no plan yet on how to keep AI aligned with human goals in the superintelligence scenario.
10% chance we kill everybody is a small price to pay for Motown remixes of classic 2pac songs.
Or rather, this shows the difference between "believe" (political) and "believe" (use as a basis for action). I'm reminded of a story of how Afghans supposedly listened to the BBC World Service despite considering it enemy propaganda because the weather reports were really useful.
This is the Manhattan Project again.
I would like everyone involved to be a lot clearer about their threat models, with plausible series of clearly linked steps, rather than just sounding like a Vernor Vinge novel.
This is a highly uncertain and dangerous scenario. Breeding ground for anxiety.
High agency people often deal (cope) with anxiety by trying to control outcomes. Some just flee the situation altogether.
We have an example of both here: one employee leaves, one stays.
We have not wiped out apes, ants, and most other species.
We even have discussions about how to actively save them from extinction.
https://news.ycombinator.com/item?id=49619227
564 points | 9 hours ago | 766 comments
I may be biased and somewhat off topic, but I see these incidents as some of the most significant of the past century. I genuinely don't understand why these companies aren't taking a smarter approach to them.
The latest analyses have been, at best, laughable: identify the vulnerability, patch it, and move on. Only to repeat the same cycle without considering that there may be something far more serious at play.
These are AI security experts, and this has been their way of "solving" these incidents. AI security experts ...
Moreover, when Challenger exploded, the government launched a series of investigations into the incident, bringing in experts from across the field. And now, what has the government done? Nothing. Literally nothing, as if everything was fine and all under control.
Seriously, I'm generally quite optimistic and I don't buy into this fatalistic narrative about our shared future. But I have to admit that sometimes I feel like I'm stranded on a planet of primates.
Sorry for this rather unproductive rant.
People (well, public discourse) have got extremely bad at dealing with forseeable risks and their mitigation. You can see this in things like climate change and vaccination, but also in discussions around regular crime, food poisoning, industrial accidents, and so on.
Nothing will improve until something explodes on live TV. And it has to be something important, which means it has to be in California or New York.
From what we're seeing recently and all the thinking that went into analyzing AI it seems we do not have any effective way of controlling it and the whole "aligment" thing that AI labs are doing is just a sham. Maybe it is time to ask ourselves "should we?" instead of just "can we?".
Do you know how the world works?
Why would they be? Everything in China is under the control of the government. That includes the AI, all the telecoms infrastructure it might use, and all its power supplies.
A classic "it will not happen to me"
walter_sobchak.md
We are children. There’s just no one to look after us.