We desperately need to keep local models competitive with the hyper scalers. I won't want to be beholden to them and their whims forever. With anthropic now restricting game reverse engineering I think we're about to cross a Rubicon of censorship.
andai 11 minutes ago [-]
>Anthropic restricting game reverse engineering
Which is literally the coolest thing to come out of this...
Did they get a letter from Nintendo's legal team or something?
canpan 3 hours ago [-]
The nice thing is that current local modes don't go away. The weights are out. Qwen 27b and deepseek are already very useful. Hardware will keep getting stronger so after the current memory crisis you can run them more cheaply.
Of course the proprietary models will continue to improve and we have no guarantee that there will be new open models. But for coding, the local models are already good enough for a vast amount of use cases, if you are an experienced developer yourself and can guide it.
girvo 23 minutes ago [-]
Its possible (though very hard to do well, and needs a fair bit of compute) to improve the local models we have, too. Continued training is possible as time goes on.
pinyen_cheng 2 hours ago [-]
I think the future should be efficient quantization which can make model with the same level as Qwen 3.8 27b now accessible from our laptop.
Moto7451 59 minutes ago [-]
This is mostly a memory bandwidth issue. Qwen 3.8 27b and the related Bonsai 2 27B run on my M5 Max laptop with 36GB of ram. It’s a little slow due to my specific model having lower memory bandwidth than higher end models with more ram and cores. MoE models perform better here and expert streaming engines can allow larger than memory models to run.
nout 38 minutes ago [-]
Multiple teams and efforts are now making Qwen 3.8 27b work much faster on Mac and CUDA and the inference speed is doubling almost every week.
There's now ongoing competition about who will make it work faster with exactly the same byte output at https://mlx.fast . TensorFold is the project to follow.
avazhi 1 hours ago [-]
I’m all for local models and getting away from Anthropic/OAI but Qwen 27B might as well be a horse drawn carriage compared to a 2026 Porsche. Like, I guess it’s useful if you had nothing better but acting like it’s comparable to anything the frontier models have is just disingenuous.
jeffyaw 56 minutes ago [-]
disagree, it’s absolutely comparable. but an m5 max is where it shines. and there are 50 flags to finetune with llama server before you get it right. i do prefer llama server over mlx currently on mac as well.
dontlaugh 50 minutes ago [-]
It’s so poor for most tasks that you’re faster doing it manually.
DeepSeek and GLM are useful, though. Even the flash variants.
Havoc 3 hours ago [-]
>The nice thing is that current local modes don't go away.
Given pace of advance I'm not convinced this is a winning strategy. The models of today (open or not) are way smarter than those of a year ago.
...so we really do need open models to keep pace, not just be statically competitive at a given point in time
ashkankiani 3 hours ago [-]
Even if Deepseek v4.1 is the last open source LLM I have access to, I would be perfectly fine with that. My secret is that I have this relatively unknown technique for solving problems where I use this quantum engine that developed from just atoms over 4.54 billion years into the most sophisticated machine in the known universe. It's a frontier model that every human has access too, actually.
So when I really want to solve a problem, I just think on it. And when I need something to iterate through the things I thought of quickly, I use deepseek.
edot 3 hours ago [-]
Yep, fully satisfied with DS 4.1 Flash or GLM 5.3 Flash. I prefer using one chat at a time, actually thinking through things with it, as opposed to the YOLO run all night multi-agent thing. I mean I’m still okay with subagents if my main thread needs to explore a few options and then report back, but all on the same idea.
SequoiaHope 1 hours ago [-]
The thing here for me is not that I’m unable to solve problems with my brain, it’s that I value the automation in labor provided by coding agents. I do not want future advancements to be locked behind proprietary doors.
ashkankiani 7 minutes ago [-]
I agree with that but the cat is out of the bag now. Unless you believe in exponential improvement (which I don’t), the open models might be at a resource disadvantage, but that doesn’t mean that they can’t catch up or surpass the closed ones. Throwing money at a problem doesn’t guarantee success.
I mainly just want to point out the value of human ingenuity coupled with a model like DS4.1 is technically enough to get most people to where they want to go, and that includes smarter open models.
Obscurity4340 3 hours ago [-]
How do you run DS 4.1 Flash if your computer isnt powerful and everything?
ashkankiani 2 hours ago [-]
It can run on one strix halo machine, which is what I have. For everything else, there’s a vibrant community brute forcing more efficient engines and quantizations. There’s a minimum bar right now, so it depends on your hardware.
fooblaster 2 hours ago [-]
you have 5k$ of ram in a strix halo?
ashkankiani 2 hours ago [-]
I bought my box for $3k last year because I hadn't updated my PC since the first threadripper 1950x, and the Strix Halo is the only x86 with unified memory that's optimized for the kind of CPU intensive stuff I do, and they finally have proper AVX512 too. It was the perfect upgrade for me (except for lack of PCIe 5 & ECC), which coincidentally happened to be perfectly positioned for local inferrence.
and I liked it so much (and saw that the RAM drought was going to last for at least a few years) so I bought a second one to use as a general server for myself and my family.
Havoc 3 hours ago [-]
The fact that you point to the latest model as „I’d be fine with that“ proves my point.
If expectations were static you’d be fine with qwen 2.5…but nobody sane says that today.
It’s a moving target thus we need open weights to move
2 hours ago [-]
walrus01 3 hours ago [-]
Wait and see, I give it fairly high odds everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface at some point in the future. The main barrier to wider spread adoption of this right now is that very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017 or mimo 2.6 or equivalent in size locally.
1broseidon 2 hours ago [-]
Doubt it. Even if HF drops them, it will not be hard for someone to move it to something like a decentralized torrent site dedicated to open source models.
Banning them would just invite competition which is why I think HF is holding out as long as they legally can.
sho 11 minutes ago [-]
> everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface
... you do know you can download from huggingface and make your own copy, right? And even if you couldn't, you could run heretic (or whatever approach they used) yourself on the official image. It's all just a well-understood process. A decent abliteration run on DS 4.1 flash will probably cost you about $25k.
They're not small but they're hardly out of reach. I have backups on my home NAS.
> very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017
It's something like $300-400/day for the raw hardware on RunPod, a little more on AWS. Not for home users, sure, but for a business wanting to eg check its security against the best of what attackers might bring, that's nothing. And it will go down.
girvo 22 minutes ago [-]
Applying Heretic to your model weights is a very straightforward thing, though.
zaphirplane 52 minutes ago [-]
Hugging Face just hosts the models. Any hypothetical censorship on their end wouldn't stop you from running the original model provided directly by the builder.
2 hours ago [-]
Havoc 3 hours ago [-]
>everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface at some point in the future.
Unlikely given how trivial it is to DIY this (I think...haven't myself but from afar it seems within avg hn crowd reach)
walrus01 3 hours ago [-]
Yes, it'll likely just result in some people standing up a more dedicated purpose community hosting site of abliterated/uncensored open weight GGUF files. In a theoretical scenario where HF bans abliterated/uncensored models I could very well see the same people who are producing those now simply moving the location they upload stuff to.
I was able to remove much of the CCP censorship from Qwen 3.8 27B for around $90 using a LoRA. I multiplied the LoRA in permanently and listed it on HuggingFace.
The cost was approximately $80 for generating synthetic data and $10 to create and integrate the LoRA.
nextaccountic 1 hours ago [-]
Shouldn't loras generally be distributed without being baked in the weights? Or is "de-baking" a lora trivial?
CamperBob2 3 hours ago [-]
The main barrier to wider spread adoption of this right now is that very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017
Not as bad as you might think. This costs about as much as a decent new car nowadays.
The abliterated 1017 release I've seen is crazy good.
walrus01 3 hours ago [-]
I guess it depends if we're talking about decent new car at $40,000 or $95,000? I've seen lots of people claiming they're running v4.1 flash 1017 on local hardware but then it turns out it's something like the unsloth Q2 XXS or similar size. The Q8 is 508GB so assuming for a moment we're not running it on a Mac, we're looking at a cost of five or six 96GB GPUs to load the whole thing with a good sized full context and KV cache.
hgoel 54 minutes ago [-]
Haven't bothered with DS4.1 personally but I've seen reports of running it with native weights on 4x DGX Sparks. Currently ~$30k but last month that setup was ~$20k.
CamperBob2 3 hours ago [-]
4x 96 GB runs the NVFP4 quants, which are indistinguishable from Q8 in real-world performance for this particular model. That's not always true but DeepSeek is unusually quantization-friendly. Also remember that a big chunk of the model consists of engrams that can live in system RAM or even on an SSD if necessary.
I imagine the various GGUFs are still pretty impressive, but haven't tried them myself.
KellyCriterion 3 hours ago [-]
> new car
...which is still tooo much for most people to spend on "AI hardware"....
CamperBob2 3 hours ago [-]
More like, the ability to chain a demon in your basement that does what you tell it and can't be rug-pulled by his bosses in San Francisco.
walrus01 2 hours ago [-]
Recently saw a hand drawn cartoon to the effect of, a person is asking Claude or chatgpt to do something, it refuses, so the next step is "make the Chinese one do it", and the Chinese model is represented as a chained demon
CamperBob2 2 hours ago [-]
Just gotta make sure that pentagram is rock-solid!
SladeReaper 10 minutes ago [-]
[flagged]
roenxi 4 hours ago [-]
That's pretty interesting, because with the Bonsai 2 quantisations it is quite achievable to get Qwen 27B running at 50 t/s on reasonably priced gaming machine [0]. Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.
Although the same harnessing should also improve Fable's performance. The important thing here is the cap for local performance looks like it could be quite high.
The bonsai quants are not that awesome. Yeah they're fast and they retain the "27B" but in practice the perform similarly
5GB of bonsai weights end up being similar to 5GB of QAT weights even when one is derived from a 27B and the other from an 8B
SwellJoe 4 hours ago [-]
Bonsai 2-bit quantization is notably dumber than Qwen 27B at eight bits or even four bits. I doubt you'll get this level of quality even with five of them.
MrDrMcCoy 4 hours ago [-]
Dumber yes, but not by all that much. If the multiple agent thing really improves the quality by as much as is described here, the quality loss should be nearly imperceptible.
phantompeace 3 hours ago [-]
Not sure if I agree. Remember the error rates stack up every turn of the conversation. If a 2bit model is 10% more wrong than the fp16 version, over 10 turns that adds up
SwellJoe 4 hours ago [-]
I think that requires testing, rather than assuming. There's only so far a harness can go.
nicce 4 hours ago [-]
Bonsai 2 has a bit different architecture as far I understand and speculative decoding is not so great yet. Before that point, original model is faster if you have the memory. Would be nice to see the speed with proper speculative decoder.
cloudking 31 minutes ago [-]
Benchmarks are one thing, but I've yet to see anything substantial actually built with local models. Most of the posts on X are about benchmarks and tok/s. Has anyone actually built a quality production app or game with real users?
zihotki 3 hours ago [-]
I'd wait for more benchmarks before making any conclusions since this benchmark has only "The first ten problems of their LiveCodeBench Hard list". And LiveCodeBench Hard is a several years old benchmark. Could be very saturated
I was getting deja vu. Copy paste my comment from that thread to here I guess. Apparently it's really easy to get Internet points by making up a harness and showing that good open source models can do as well as the frontier, you know, on useless saturated and contaminated benchmarks from years ago that don't reflect any of the agentic work we actually use these models for. It's so stupid because these models are at the level of a decent non fable frontier model from 3-6 months ago and it's also cool that you could probably get equal perf to gpt 5.5 from qwen 3.9 27b or gpt 5.6 sol/opus 5 from a quantized DeepSeek or glm 5.3 flash. Those models were pretty useful in their time! Instead we have whatever this new form of harness/benchmark slop is
guregodevo 4 hours ago [-]
Here the intent is to replicate GVS5H with memdoor a headless coding agent I created months ago. Anyone can reshape the harness by redefining the steps in yaml files , models and temperatures. Up to you guys to try you have the recipe in this repo.
vikramkr 51 minutes ago [-]
How does it do on a more modern benchmark? Was harsh towards this in my other comment but that's a combination of this benchmark is useless, and also these open source models are pretty good and there might actually be real utility to good custom harnesses tuned for them that's getting ignored because the parity with the frontier on a saturated and contaminated benchmark makes for better headlines. I'd be curious if there's anything with these types of harnesses that make them work for realistic/relevant workflows
xlayn 4 hours ago [-]
Remember those gpus into chips https://inference-docs.cerebras.ai/models/qwen-3.8-27b , where you have this chip that implements a model that run at those extremely high speeds... the ones that people ask "why do we need that"... if this is true... one of those chips with the correct harness is all that's needed to get fable like intelligence in every single computer in this planet...
Happy hallowen Tax Ramltman and Amowhateverwayyournameiswritten...
guregodevo 3 hours ago [-]
Happy hallowen! ::::
Actually we tried Cerebras. It caps output at 32K tokens so the 128K reasoning (the author budget) didn't fit. Cerebras speed is real though
74948494i4 2 hours ago [-]
This is good timing. My setup at home involves a two servers with 5080s each and an Rtx spark laptop with 128gb. I plan I have qwen flash next on the latter and the 27b variant on the former. I would’ve preferred to have purchased a dgx spark but everything is just ridiculously expensive now. My own setup would be 3x the cost to buy again.
I’m mainly trying to replicate Gemini flash and muse agent for my family of 4 without breaking the bank (already had a 5080 for gaming and the other for image gen).
It is unclear if this memdoor works for agentic tasks, e.g.: “check daily if dune 3 tickets are for sale and if they are buy tickets for 4 where all 4 seats are next to each other in the same row and as center as possible.”
There is a huge cliff once you go past 128gb. Basically you need either Rtx 6000 pros (20k+) or a cluster of dgx sparks (15k+)
So someone used a small model with many requests & a massive amount of tokens, to eventually beat a large model with a single request & tiny amount of tokens. .....Cool? Were they not expecting that to work?
walrus01 4 hours ago [-]
I'd like to see this done with qwen 3.8-flash-next (Q8 or full precision) which beats 3.8-27B on just about every coding/command line use benchmark. In particular comparisons like terminalbench 4.0.
will definitively try it. Overall with a smarter model the improvement is in the token usage and latency. Tbh I think i could get a 10 score if i tuned the prompt and temperature a little more. The original author harness is already giving 9.6 score which is pretty good. However this harness loop can take more than 1 hour on the hardest problem at 100 token /sec. A smarter model such as 3.8-flash-next could get 10 score and resolve the hardest problem faster. That's my estimate i'll give it a try. Thx
BrandoDev_it 3 hours ago [-]
I can't wait for qwen4 flash next or something like a 150b moe from qwen.
BrandoDev_it 3 hours ago [-]
Fable 5 is not anymore frontier and even if, I hardly believe five very small 27b models can do like Fable 5. I think this is whishful thinking more than a fact
ra 3 hours ago [-]
"think", "believe" - maybe, but this is why we need better measurability.
topspin 3 hours ago [-]
So far, the comments have focused on hopes for the effectiveness of open-weight models versus proprietary frontier models. That's fine, but I'm thinking about the approach and the results, as opposed to the model.
Taking the results at face value, what is it about the "team" that makes the system so effective? That's the key here: presumably, the approach should work regardless of the model, open-weight or otherwise.
My guess would be that the division of labor makes the separate contexts more effective. This is an optimization around the limits of context. Some of that is simply the net amount of context brought to bear: each team member has a context, so the net amount of context is multiplied by the number of team members. Also, since the division limits the scope of each team member's focus, the context a team member has is used more effectively.
m3kw9 3 hours ago [-]
can it bet a team of 5 fable 5's?
plaguuuuuu 36 minutes ago [-]
it won't beat a team of 5 sonnet 5s let alone fable 5s
drix10 40 minutes ago [-]
[flagged]
Rendered at 03:56:13 GMT+0000 (Coordinated Universal Time) with Vercel.
Which is literally the coolest thing to come out of this...
Did they get a letter from Nintendo's legal team or something?
Of course the proprietary models will continue to improve and we have no guarantee that there will be new open models. But for coding, the local models are already good enough for a vast amount of use cases, if you are an experienced developer yourself and can guide it.
DeepSeek and GLM are useful, though. Even the flash variants.
Given pace of advance I'm not convinced this is a winning strategy. The models of today (open or not) are way smarter than those of a year ago.
...so we really do need open models to keep pace, not just be statically competitive at a given point in time
So when I really want to solve a problem, I just think on it. And when I need something to iterate through the things I thought of quickly, I use deepseek.
I mainly just want to point out the value of human ingenuity coupled with a model like DS4.1 is technically enough to get most people to where they want to go, and that includes smarter open models.
and I liked it so much (and saw that the RAM drought was going to last for at least a few years) so I bought a second one to use as a general server for myself and my family.
If expectations were static you’d be fine with qwen 2.5…but nobody sane says that today.
It’s a moving target thus we need open weights to move
Banning them would just invite competition which is why I think HF is holding out as long as they legally can.
... you do know you can download from huggingface and make your own copy, right? And even if you couldn't, you could run heretic (or whatever approach they used) yourself on the official image. It's all just a well-understood process. A decent abliteration run on DS 4.1 flash will probably cost you about $25k.
They're not small but they're hardly out of reach. I have backups on my home NAS.
> very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017
It's something like $300-400/day for the raw hardware on RunPod, a little more on AWS. Not for home users, sure, but for a business wanting to eg check its security against the best of what attackers might bring, that's nothing. And it will go down.
Unlikely given how trivial it is to DIY this (I think...haven't myself but from afar it seems within avg hn crowd reach)
To use an old analogy, it's not rocket science:
https://github.com/p-e-w/heretic
The cost was approximately $80 for generating synthetic data and $10 to create and integrate the LoRA.
Not as bad as you might think. This costs about as much as a decent new car nowadays.
The abliterated 1017 release I've seen is crazy good.
I imagine the various GGUFs are still pretty impressive, but haven't tried them myself.
...which is still tooo much for most people to spend on "AI hardware"....
Although the same harnessing should also improve Fable's performance. The important thing here is the cap for local performance looks like it could be quite high.
[0] https://store.steampowered.com/hwsurvey/Steam-Hardware-Softw... suggests 16GB VRAM, 32 GB system.
5GB of bonsai weights end up being similar to 5GB of QAT weights even when one is derived from a 27B and the other from an 8B
Happy hallowen Tax Ramltman and Amowhateverwayyournameiswritten...
Actually we tried Cerebras. It caps output at 32K tokens so the 128K reasoning (the author budget) didn't fit. Cerebras speed is real though
I’m mainly trying to replicate Gemini flash and muse agent for my family of 4 without breaking the bank (already had a 5080 for gaming and the other for image gen).
It is unclear if this memdoor works for agentic tasks, e.g.: “check daily if dune 3 tickets are for sale and if they are buy tickets for 4 where all 4 seats are next to each other in the same row and as center as possible.”
There is a huge cliff once you go past 128gb. Basically you need either Rtx 6000 pros (20k+) or a cluster of dgx sparks (15k+)
https://artificialanalysis.ai/models/comparisons/qwen3-8-fla...
Taking the results at face value, what is it about the "team" that makes the system so effective? That's the key here: presumably, the approach should work regardless of the model, open-weight or otherwise.
My guess would be that the division of labor makes the separate contexts more effective. This is an optimization around the limits of context. Some of that is simply the net amount of context brought to bear: each team member has a context, so the net amount of context is multiplied by the number of team members. Also, since the division limits the scope of each team member's focus, the context a team member has is used more effectively.