Coincidentally, I just started on Build a Large Language Model (From Scratch), a repo/book/course by Sebastian Raschka [0][1][2]. Maybe it is a good problem to have to have to decide which learning resource to use.
If you're interested in this resource, I highly recommend checking out Stanford's CS336 class. It covers all this curriculum in a lot more depth, introduces you into a lot of theoretical aspects (scaling laws, intuitions) and systems thinking (kernel optimization/profiling). For this, you have to do the assignments, of course... https://cs336.stanford.edu/
the_real_cher 2 hours ago [-]
how does one get the lectures? I don't see the option for any lectures.
I did it back in the day when fast.ai was relatively new with ULMFiT. This must have been when Bert was sota. The architecture allows you to train a base and specialize with a head. I used the entire Wikipedia for the base and then some GBs of tweets I had collected through the firehouse. I had access to a lab with 20 game dev computers. Must have been roughly GTX 2080s. One training cycle took about half a day for the tokenized Wikipedia so I hyper parameter tuned by running one different setting on each computer and then moving on with the winner as the starting point for the next day. It was always fun to come to work the next morning and check the results.
The engineering was horrible and very ad-hoc but I learned a lot. Results were ok-ish (I classified tweets) but it gave me a good perspective on the sheer GPU power (and engineering challenges) one would need to do this seriously. I didn't fully grasp the potential of generating output but spent quite some time chuckling at generated tweets (was just curious to try it).
NSUserDefaults 2 hours ago [-]
Been doing it since the day I was born. The beginnings were hard but I’m getting there.
hliyan 17 minutes ago [-]
You've actually been primarily training a physics model, with an LLM attached to it.
antirez 43 minutes ago [-]
Context: he is one of the MLX developers, a skilled ML researcher.
ofsen 1 hours ago [-]
This looks like exact copy of this video of andrej karpathy ( https://youtu.be/kCc8FmEb1nY ) but in a writing format, am i wrong ?
steveharing1 44 minutes ago [-]
The documentation is really helpful enough to get started
16 minutes ago [-]
hiroakiaizawa 2 hours ago [-]
Nice. What scale does this realistically reach on a single machine?
55 minutes ago [-]
lynx97 1 hours ago [-]
Model: 36L/36H/576D, 144.2M params
runs on a Blackwell 6000 Max-Q, using 86GB VRAM. Training supposedly takes 3h40m
iamnotarobotman 3 hours ago [-]
This looks great for a first introduction to training LLMs, and it looks simple enough to try this locally. Great job!
baalimago 3 hours ago [-]
Train your LM from scratch*
I doubt you have a machine big enough to make it "Large".
mips_avatar 2 hours ago [-]
You can fully train a 1.6b model on a single 3090. That’s a reasonably big model.
electroglyph 1 hours ago [-]
you can train it, but not fully
nucleardog 2 hours ago [-]
Hey now! I've got a half terabyte of RAM at my disposal! I mean, it's DDR4 but... it's RAM!
And it's paired with 48 processor cores! I mean, they don't even support AVX512 but they can do math!
I could totally train a LLM! Or at least my family could... might need my kid to pick up and carry on the project.
But in all seriousness... you either missed the point, are being needlessly pedantic, or are... wrong?
This is about learning concepts, and the rest of this is mostly moot.
On the pedantic or wrong notes--What is the documented cut-off for a "large" language model? Because GPT-2 was and is described as a "large" language model. It had 1.5B parameters. You can just about get a consumer GPU capable of training that for about $400 these days.
Malcolmlisk 52 minutes ago [-]
Then rewrite the title and call it "learn how to do a non usable llm from scratch"
improbableinf 34 minutes ago [-]
Opus 4.7 is non-usable for the tasks I have — but it’s considered an LLM.
And no one is stopping anyone from tweaking few parameters in this repo to go above 10M parameters.
flowdesktech 17 minutes ago [-]
[flagged]
Rendered at 07:59:30 GMT+0000 (Coordinated Universal Time) with Vercel.
[0] https://github.com/rasbt/LLMs-from-scratch
[1] https://www.manning.com/books/build-a-large-language-model-f...
[2] https://magazine.sebastianraschka.com/p/coding-llms-from-the...
The engineering was horrible and very ad-hoc but I learned a lot. Results were ok-ish (I classified tweets) but it gave me a good perspective on the sheer GPU power (and engineering challenges) one would need to do this seriously. I didn't fully grasp the potential of generating output but spent quite some time chuckling at generated tweets (was just curious to try it).
runs on a Blackwell 6000 Max-Q, using 86GB VRAM. Training supposedly takes 3h40m
I doubt you have a machine big enough to make it "Large".
And it's paired with 48 processor cores! I mean, they don't even support AVX512 but they can do math!
I could totally train a LLM! Or at least my family could... might need my kid to pick up and carry on the project.
But in all seriousness... you either missed the point, are being needlessly pedantic, or are... wrong?
This is about learning concepts, and the rest of this is mostly moot.
On the pedantic or wrong notes--What is the documented cut-off for a "large" language model? Because GPT-2 was and is described as a "large" language model. It had 1.5B parameters. You can just about get a consumer GPU capable of training that for about $400 these days.
And no one is stopping anyone from tweaking few parameters in this repo to go above 10M parameters.