> And the comparisons in this post are not going to be running some 2.58-bit-gguf-in-ollama with a couple test prompts.
Genuine question : is there something fundamentally wrong with Ollama ?
I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly).
I thought the main advantage of VLLM was better concurrency management (better batching).
But if the quality of the interference itself is an issue, then maybe I should reconsider my choice.
jonplackett 1 hours ago [-]
I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.
prettyblocks 57 minutes ago [-]
My problem is how hot they run. I'm on an m4 pro. Do you have the same issue?
jonplackett 23 minutes ago [-]
It’s hot and also LOUD and runs the battery down quick.
But I’m having a lot of luck just running things when I’m away from the computer and can leave it plugged in.
It starts going weird (unreliable and slow) with context over 80k so you have to pick tasks one at a time and baby sit a lot more than Claude. But it really is very capable and feels like there’s an intelligence there to talk to. Maybe gpt-4 level clever?
I have an m5 max 64gb and I think anything slower would be quite painful.
lukan 51 minutes ago [-]
I don't have the hardware but a often mentioned advice is to put your mac into energy saving mode - it still will work, a bit slower, but stays cool.
28 minutes ago [-]
downrightmike 55 minutes ago [-]
Mineral oil bath?
kees99 21 minutes ago [-]
Or, set it up in another room, and connect from the other laptop.
alexchantavy 49 minutes ago [-]
How many tok/s are you getting? What gen mbp?
dominotw 3 minutes ago [-]
interested in this too. i suspect ppl dropping generic "its awesome" comments are not actually using in just managed to get it running for a prompt or two.
StarlaAtNight 55 minutes ago [-]
how quick does it respond? what are specs of your laptop?
Gareth321 24 minutes ago [-]
I tried it on my M1 MacBook Pro. It's slow but surprisingly smart as a general purpose LLM. Maybe GPT-5.3 level. I gave it a bunch of tools and it can search the internet, make product recommendations, document, code, etc.
alexpotato 9 minutes ago [-]
Had the same reaction so had Grok create a script to:
- find a free GPU droplet on digital ocean
- fire it up
- pull in a snapshot of the model + extra files/packages etc
- set up a ssh tunnel so that the localhost:8000 routes to the above
Then I just configured OpenCode to use the above and was off to the races.
Works out to be about ~$2/hr all said and done which isn't bad as I only pay when I'm using it (but could get expensive with 24/7 running)
chorlton2080 49 minutes ago [-]
Does it need to respond fast? For important applications, I'm sure we'd all be fine waiting 20 minutes for a high quality, usable answer. Or is it the need for interative refinements that make speed relevant?
jonplackett 21 minutes ago [-]
It requires patience but it’s more like waiting 5 mins for it to do tasks. You need to be much more involved though and do things slower than Claude where you can trust it to do a lot of tasks at once. It doesn’t have the context for that
applicative 43 minutes ago [-]
Did you read even the title?
anotherCodder 1 hours ago [-]
[flagged]
Rendered at 21:25:38 GMT+0000 (Coordinated Universal Time) with Vercel.
Genuine question : is there something fundamentally wrong with Ollama ?
I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly).
I thought the main advantage of VLLM was better concurrency management (better batching).
But if the quality of the interference itself is an issue, then maybe I should reconsider my choice.
But I’m having a lot of luck just running things when I’m away from the computer and can leave it plugged in.
It starts going weird (unreliable and slow) with context over 80k so you have to pick tasks one at a time and baby sit a lot more than Claude. But it really is very capable and feels like there’s an intelligence there to talk to. Maybe gpt-4 level clever?
I have an m5 max 64gb and I think anything slower would be quite painful.
- find a free GPU droplet on digital ocean
- fire it up
- pull in a snapshot of the model + extra files/packages etc
- set up a ssh tunnel so that the localhost:8000 routes to the above
Then I just configured OpenCode to use the above and was off to the races.
Works out to be about ~$2/hr all said and done which isn't bad as I only pay when I'm using it (but could get expensive with 24/7 running)