A Stanford lecture stripped of jargon. Here's what actually goes into building ChatGPT, Claude, or Llama , and the part that surprises almost everyone: the architecture is the least important piece.
When a Stanford professor walks into a room full of computer science students and says "the thing everyone thinks matters most actually matters least," you pay attention.
That's more or less how this lecture opens. ↗ 0:00
To build a large language model , the technology behind ChatGPT, Claude, Gemini, and Llama , you need five things working together:
Now here's the surprise: almost all of academic AI research focuses on number one , architecture. New transformer designs, new attention mechanisms, new model shapes. It looks important. Researchers love it.
In practice, architecture is the least important of the five. What actually determines whether a model is good or bad is data, evaluation, and systems. That's what industry focuses on , and it's what this lecture is really about. ↗ 2:00
The lecture covers two phases of building an LLM. The first is pretraining , training a model to understand language by reading enormous amounts of text. The second is post-training , taking that language-understanding model and turning it into an assistant that's actually helpful to talk to. GPT-3 was mostly pretraining. ChatGPT was what happened when post-training got serious.
At its core, a language model is doing one thing: predicting what word comes next. ↗ 3:24
That sounds almost too simple to produce something as capable as ChatGPT. But think about what you actually need to know to predict the next word well.
If the sentence is "The mouse ate the ___", you need to know grammar, you need to know that mice eat things, and you need to know that cheese is more likely than, say, a bicycle. That's grammar, world knowledge, and common sense , all baked into one task.
The technical term is an autoregressive language model. "Autoregressive" just means it generates one word at a time, using everything it's already generated as context for the next word. ↗ 5:15
This is why longer responses take longer to generate , the model is literally running through a loop, producing one token, then using that token to produce the next, then the next. There's no way to shortcut it.
During training, the model is shown real text and asked: what comes next? Its answer gets compared to what actually came next. If it was wrong, the weights , the internal numbers that define how the model thinks , get nudged in the right direction. Do this enough times across enough text, and the model starts developing a real model of how language works.
When you hear people talk about a model's "weights" or "parameters," they're talking about billions of numbers stored inside the model. These are what change during training. By the end, those numbers encode everything the model knows , about grammar, facts, reasoning, tone. A 7-billion-parameter model has 7 billion of these numbers.
Before a model can predict the next word, it needs to convert text into a format it can process. This is called tokenization , and it's one of those things that sounds like a minor implementation detail but turns out to matter a lot. ↗ 10:43
You might think: just split on spaces, one word per token. But this breaks immediately. What happens when someone has a typo? What about a language like Thai, which doesn't use spaces between words? And what about extremely rare words that almost never appear , should the model not be able to handle them?
The other extreme is treating every individual character as its own token. That works, but then every sentence becomes a very long sequence, and AI models get computationally expensive as sequence length increases.
So the solution lands in the middle: subword tokenization. Common words get their own token. Rare words get split into recognizable chunks. On average, each token covers about 3-4 characters of text.
The word "tokenizer" doesn't become one token , it gets split into "token" and "-izer." These are both common enough subwords that the model has seen them before in other contexts, so it can handle the word even if it rarely saw "tokenizer" as a whole. ↗ 12:43
The most common method for building a tokenizer is called Byte Pair Encoding (BPE). Start by treating every character as its own token. Then find the most common pair of tokens that appear side by side, and merge them into a new single token. Repeat. After enough merges, you end up with a vocabulary of common subwords that covers the language efficiently.
This is why when you use ChatGPT or Claude, you're billed per "token" and not per word , and why a token isn't quite a word. It's a compressed chunk of text that the model has learned to treat as a single unit.
You've probably heard that LLMs are trained on "the internet." That's roughly true , but what it actually means in practice is more chaotic than it sounds. ↗ 28:40
The starting point is something called Common Crawl , a web crawler that continuously indexes publicly accessible web pages. Right now, that's around 250 billion pages, roughly 1 petabyte of data. That's one million gigabytes.
Here's the problem: a random page from the internet is not Wikipedia. It might be a broken HTML file with half-finished sentences. It might be spam. It might be a forum post full of slurs. It might be a boilerplate legal disclaimer copy-pasted across 10,000 pages. Training a model on all of that, unfiltered, would produce something that generates broken, repetitive, toxic nonsense.
So before any training happens, teams spend enormous effort cleaning and filtering the data:
This is the part where the "architecture matters least" claim starts to make sense. If you give a model with a brilliant architecture terrible data, you get a terrible model. If you give a model with an unremarkable architecture excellent, carefully curated data, you get something genuinely good. Data quality changes the slope of improvement. Architecture mostly just shifts the starting point.
One of the most remarkable findings in modern AI is that the improvement of language models follows a predictable mathematical pattern. ↗ 41:57
If you plot model performance against the amount of compute used to train it , on a log-log scale , you get a straight line. That means if you double your compute, you can predict roughly how much better your model will get. Same for data: add more, improve by a predictable amount. Same for model size.
Imagine being able to predict, before spending a cent, how good your model will be after you spend $10 million on training. Scaling laws let AI labs do something close to that , which means they can plan two and three years ahead with reasonable confidence.
As of this lecture, these laws were still holding. Nobody has found the ceiling yet. ↗ 44:34
Scaling laws also changed how AI labs work. The old approach: train a big expensive model, tune its settings on that big expensive model, ship it. The new approach: train dozens of small cheap models, tune settings on those, use the patterns to predict what a large model would do, then train the large one once , with the right settings already dialed in. This is why scaling laws are a practical tool, not just an academic curiosity.
One more insight from scaling laws: if you're deciding between two architectures, you don't have to bet your entire compute budget on one. Train both at small scale, fit scaling law curves to each, and extrapolate. The better architecture at scale usually reveals itself in the slope of the curve , and that slope is visible even on cheap small experiments.
After pretraining, a model knows an enormous amount about language and the world. But if you ask it a question, it might continue your question rather than answer it , because that's what "predicting the next token" does. It's completing patterns, not having a conversation. ↗ 1:01:58
This is where post-training comes in , the step that turns a language model into an assistant. It has two main phases.
The first is called Supervised Fine-Tuning (SFT). The idea is simple: collect examples of questions paired with good answers, then continue training the model on that data. The model already knows language and facts , now you're just teaching it the right format and style for being helpful. ↗ 1:02:24
You don't need much of it. Research has shown that scaling from 2,000 to 32,000 training examples barely changes the result. The reason: the model already knows the content. SFT mostly just teaches it how to respond , the format, the tone, the style. The knowledge was already there from pretraining.
This is actually one of the most counterintuitive insights in the field: SFT isn't teaching the model new information. It's showing the model which version of itself , which "user type" from all the internet text it absorbed , to optimize for. The helpful, clear, direct one.
SFT is also what made the leap from GPT-3 to ChatGPT. GPT-3 was a remarkable pretrained model that mostly stayed inside research labs. Adding supervised fine-tuning on question-answer pairs made it something anyone could use.
SFT has a fundamental limitation: it can only be as good as the humans writing the training examples. If you ask someone to write the ideal answer to a question, they'll do their best , but most people are better at recognizing a good answer than producing one. ↗ 1:10:05
This is the problem that Reinforcement Learning from Human Feedback (RLHF) was designed to solve. Instead of asking humans to write perfect answers, you ask them to compare two answers and pick the better one. Comparing is much easier than generating.
There's also a newer, simpler version of this called DPO (Direct Preference Optimization). Instead of training a separate reward model and using reinforcement learning , which is complex , DPO skips straight to training the model directly on the human preference pairs. It turns out to perform just as well, with far less complexity. ↗ 1:20:00
This is a side effect of RLHF. When humans compare two responses, they tend to prefer longer ones , even when the shorter one is more accurate. The model learns that pattern. The more RLHF training, the longer the outputs become. If you've ever been mildly annoyed that an AI response is twice as long as it needs to be, that's RLHF optimizing for what human labelers clicked on. ↗ 1:19:17
One more twist: collecting human feedback at scale is expensive and slow. Humans also disagree with each other about 34% of the time , even when given clear guidelines. Increasingly, AI models are being used to simulate human preferences, and they often agree with the human consensus more consistently than individual humans do. Not because they're better , they have less variance, which helps on a binary comparison task. This is an active and genuinely unresolved area of the field.
You're probably not going to train a language model from scratch. That requires teams of engineers, millions of dollars in compute, and months of work. But the things this lecture covers show up in every AI tool you're already using , and understanding them makes you better at using those tools.
1. These models don't "know" things the way you do. They learned statistical patterns from text. When they're wrong, it's often because the pattern they learned doesn't apply to your specific question , not because they "misunderstood."
2. The model you're using was shaped by human preferences. RLHF means every AI assistant has been tuned based on what reviewers preferred. That makes them generally helpful, but also sometimes prone to telling you what you want to hear, or giving longer answers than necessary.
3. Data quality is everything. If you want better outputs, give better inputs. Clear, specific prompts outperform vague ones for the same reason good training data outperforms messy data , the model has more to work with.
If you want to go further, the Stanford professor recommends three courses: CS224N for historical context and adjacent material, CS324 (Large Language Models) for an in-depth treatment of everything covered here, and CS336 for actually building your own LLM from scratch. He describes that last one as "an amazing class , very heavy workload, so be careful." ↗ 1:44:47
And if you want to just start using open-source models yourself , running something like Llama locally on your own computer, for free, with no API costs , that's something we'll cover in a future breakdown. It's more accessible than it sounds.