How LLMs Work, Explained gave the overview. This page builds the machine instead, one step at a time. It starts from the smallest piece, one number multiplied by one weight, and each step adds exactly one idea to the step before. By Step 9 the pieces have become a transformer; by Step 13 they have become ChatGPT. Every step has its formula and, where it helps, real numbers: five example inputs for the first steps, then the sentence “I cut the apple with a ___” followed through GPT-2 small.
Steps 1–5 · From One Weight to a Neural Network
Step 1 · Multiply a Number by a Weight
Start with one input, x = 0.3, and one weight, w = 0.5. The result is 0.15. On its own this does almost nothing: the weight only says how much this one input should count. Every step that follows is built from this operation.
Step 2 · Add Many of Them Up
| Input xi | Weight wi | wixi |
|---|---|---|
| 0.3 | 0.5 | 0.15 |
| −0.6 | −0.2 | 0.12 |
| 0.8 | 0.4 | 0.32 |
| 0.5 | 0.1 | 0.05 |
| 0.9 | −0.3 | −0.27 |
| Weighted sum | 0.37 |
Five inputs, each counted by its own weight. A positive weight lets an input push the total up, a negative one pushes it down, and a weight near zero means the input is mostly ignored.
Step 3 · Add a Bias and a Curve
where σ(z) = 1 ÷ (1 + e−z)
With a bias b = 0.1 the sum becomes 0.47, and the S-shaped curve σ turns that into 0.615. The bias shifts the point at which the neuron switches on. The curve bends the output, and that bend is what matters: without it, stacking neurons could only ever produce straight lines, however many you used. GPT-2 uses a similar curve called GELU.
Step 4 · Put Neurons Side by Side
Give the same five inputs to a hundred neurons, each with its own weights, and you have a layer. The weights now form a matrix W, with one row per neuron (100 rows × 5 columns), and the biases form a vector b. The output h is a list of a hundred numbers, each neuron having picked out a different pattern in the input.
Step 5 · Stack the Layers
Feed each layer’s output into the next. Layer ℓ turns hℓ into hℓ+1 with its own weights and biases. Early layers find simple patterns; later layers find patterns in those patterns. This is a neural network, and it can learn almost any mapping from inputs to outputs. What it cannot do yet is read a sentence: so far its inputs have been plain numbers.
Steps 6–9 · From Words to a Transformer
Step 6 · Turn Words into Numbers
(GPT-2 small: E is 50,257 × 768, P is 1,024 × 768)
From here on, the input is “I cut the apple with a ___”. The text is cut into tokens and each is looked up in a fixed vocabulary: “apple” becomes ID 17180. That ID is only a label and says nothing about fruit, so each ID picks a row of a learned table E, 768 numbers placing related tokens near each other. A second table P adds the token’s position t, so that word order counts. The sentence is now a grid of numbers, one row of 768 per token, which the network from Step 5 can take as input.
Step 7 · Let Every Word Look at Every Other Word
with Q = XWQ, K = XWK, V = XWV
A plain network looks at each token alone. Attention lets every token score how relevant every earlier token is to it. Each token makes a query (Q: what am I looking for?), a key (K: what do I offer?) and a value (V: what do I pass on?), all by multiplying by weight matrices, as in Step 4. Query-key scores become weights through softmax, and each token takes that weighted mix of values. In the apple sentence, the last token “a” now carries information from “cut” and “apple”. GPT-2 small runs 12 of these attention “heads” side by side, each with dk = 64.
Step 8 · Combine Attention with the Network
x ← x + FFN( LN(x) )
where FFN(x) = GELU( xW1 + b1 ) W2 + b2
Attention mixes information between tokens; a small network from Step 5, the feed-forward network (FFN), then works on each token on its own, widening its 768 numbers to 3,072 and back. Two extra details make it trainable at depth: LN, layer normalisation, keeps the numbers in a steady range, and the “x +” adds each result to its input instead of replacing it. In the apple sentence, the representation of “a” moves closer to “something you cut an apple with”.
Step 9 · Stack the Blocks
ℓ = 0, 1, …, L−1
(GPT-2 small: L = 12; GPT-3: L = 96)
The same pattern as Step 5, one level up: each block refines what the one before produced. This stack is the transformer, the “T” in GPT. GPT-2 small has about 124 million weights in total, and every one of them is a w from Step 1.
Steps 10–12 · From Scores to Text
Step 10 · Score Every Possible Next Token
pi = exp(zi ÷ T) ÷ Σj exp(zj ÷ T)
The final 768 numbers for the last token are compared with the embedding of every token in the vocabulary (GPT-2 reuses the table E from Step 6), giving 50,257 scores, called logits. Softmax turns them into probabilities that add up to 100%; T is the temperature, 1 by default. For the apple sentence: knife 16%, fork 14%, and thousands of other tokens sharing the rest. The model has produced not a word but a probability for every possible word.
Step 11 · Pick One Token
top-k: pi ÷ Σj∈top-k pj
top-p: smallest set S with Σi∈S pi ≥ p
With knife at 16%, the model is far from certain. How one token is chosen is a separate decision, made by decoding settings that change which guess is used, never the model itself.
| Method | What it does | In the apple sentence | Failure mode |
|---|---|---|---|
| Greedy | Always take the most probable token | Always “knife” | Repetitive. Over a long text GPT-2 falls into loops, repeating the same phrase |
| Sampling | Draw at random, weighted by probability | Knife 16% of the time, fork 14%, and so on | Can pick an unlikely token: 33% of the probability sits outside the top 50 tokens |
| Top-k | Keep only the k most probable tokens, rescale, then sample | With k = 3: knife 40%, fork 36%, sharp 24% | A fixed k is too narrow when many words fit and too wide when one clearly does |
| Top-p (nucleus) | Keep the smallest set of tokens whose probabilities add up to p, such as 90%, then sample | The kept set grows or shrinks with the model’s confidence | With a flat distribution the set can still be large |
| Temperature | Divide every score by T before softmax (Step 10). Below 1 sharpens the distribution, above 1 flattens it | Knife is 44% at T 0.5 and 3% at T 1.5 | Low is focused but bland; high is varied and more likely to go off track |
▸ The figures fit together. Top-k only rescales the survivors. Knife rises from 16% to 40%, so the top three must hold 16% ÷ 0.40 = 40% of the probability together. Fork then works out at 14% ÷ 0.40 = 35%, which is the 36% shown once rounding is allowed for, and “sharp” must start at about 24% × 0.40 ≈ 10%. The other 50,254 tokens share the remaining 60%.
Step 12 · Append It and Repeat
p(x1, …, xn) = Πt p( xt | x<t )
“Knife” is added to the sentence and Steps 6 to 11 run again on “I cut the apple with a knife”, one token at a time, until the model produces a stop token (in GPT-2, <|endoftext|>, ID 50256) or hits a length limit. That loop is text generation.
- Why greedy text loops: once a phrase has appeared, the model sees it in its own input, which makes the same phrase more probable again. Always taking the top token never breaks the cycle; sampling usually does.
- Why answers can change: with sampling on, the same prompt follows a different path each time, and one different early token changes everything after it.
- Why it is fast enough: a KV cache keeps the keys and values from Step 7 for earlier tokens, so each new token costs only its own pass.
- Why it has a limit: every pass reads the whole sentence so far, up to the context window: 1,024 tokens in GPT-2 small, the size of table P in Step 6.
Step 13 · From GPT to ChatGPT
w ← w − η · ∂Loss ÷ ∂w
Up to here every weight could be random. Training sets them. Show the model real text, measure how much probability it gave to the token that actually came next (the loss), and nudge every w a little in the direction that would have raised it (gradient descent, with step size η). Repeat across a large share of the public internet and you get a pre-trained GPT. Train it further on examples of instructions and good answers, then on which answers people prefer (RLHF), and you get a chat model like ChatGPT.
▸ Rewind all 13 steps and you are back at Step 1: every answer ChatGPT gives is a very long chain of one number times one weight, added up and bent by a curve. Nothing is learned except the weights. OpenAI has not published ChatGPT’s own internals; the steps here are the published GPT design that it is built on, illustrated with GPT-2.
What the Settings Mean at Work
Most AI tools expose some of the Step 11 settings, and many set them for you. They are worth understanding for one reason: they change how an answer is worded, never what the model knows. If the right answer is not among the model’s likely tokens, no setting will find it.
| Setting | Use it for | Watch for |
|---|---|---|
| Temperature 0 / greedy | Extraction, classification, structured output, anything that must be repeatable or reviewed | Consistent is not correct: a wrong answer comes back the same way every time. Long outputs can loop |
| Low (about 0.2–0.5) | Summaries and answers grounded in supplied documents | Still sampling, so still a small chance of a stray token |
| Moderate (about 0.7–1) with top-p | Drafting emails, reports, code comments | Facts and figures in the draft need checking |
| High (above 1) | Brainstorming, naming, alternatives | More tokens from the tail, which is where fluent but invented details come from |
▸ For audit and regulated work, record the model and version, the decoding settings and the output itself. Even at temperature 0, many hosted APIs do not promise identical output on every call, and some newer reasoning models do not let you set temperature at all. Reproducibility comes from keeping the record, not from re-running the prompt. Related risks are covered in AI Security: New Attack Surface.
Bottom line
- Steps 1–5 — weights times numbers, summed, bent by a curve, side by side, stacked: a neural network.
- Steps 6–9 — words become numbers, attention lets them inform each other, and stacked blocks make a transformer.
- Steps 10–12 — the output is a probability for every token; a decoding rule picks one, and the loop repeats.
- Step 13 — training sets the weights; instruction tuning and RLHF turn a GPT into ChatGPT.
- Settings change style, not knowledge — pick them for the job, and log them where it matters.
Sources & method. The step-by-step structure and the notation of Steps 3 and 5 (σ(Σwixi + b), hℓ+1 = σ(Wℓhℓ + bℓ)) and the five example inputs follow the YouTube Short “Ok, So Basically ChatGPT” by Graduate ML; the weights and bias in Steps 1–3 are illustrative values chosen for this page. The apple sentence follows “How an LLM Works in 90 Seconds” by Deepak Jose; its GPT-2 small probabilities (knife 16%, fork 14%, the top-50 tail, top-k and temperature figures) and the token ID for “apple” are quoted from that video’s description and were not re-measured. Attention and the transformer block are from Vaswani et al., “Attention Is All You Need” (2017); GPT-2’s sizes, pre-normalisation and shared embedding table are from its published configuration; GPT-3’s block count is from Brown et al. (2020); top-p sampling and the analysis of greedy repetition are from Holtzman et al., “The Curious Case of Neural Text Degeneration” (2019). The settings guidance is the author’s. Original text; no third-party graphics or text reproduced.
← Back to Cyber security