For someone who has never thought about how an AI model actually runs. It teaches what inferenceInference Running a trained model to produce a reply. Training happens once per model; inference happens every time anyone uses it, which is why most of the money in AI now goes there. is, why it's hard, and the handful of ideas that explain most of the field.
The companion article, Voice AI inference 101, takes one real system, an AI you can talk to out loud, and breaks it into equations of input levers and output levers.
1 · What "inference" means
Every AI model has two lives.
In the first life, called training, a company spends months and a lot of money showing the model enormous amounts of text until it becomes good at guessing what comes next.
What training produces is, quite literally, one very large file of numbers. Those numbers are called the weightsWeights The numbers that make up a trained model. A big model's weights fill about 140 GB, and all of them have to be read for every token it writes., and for a big model the file is around 140 gigabytes, which is roughly thirty full-length HD movies.
In the second life, called inferenceInference Running a trained model to produce a reply. Training happens once per model; inference happens every time anyone uses it, which is why most of the money in AI now goes there., that file is loaded onto powerful computers and actually used. Every time you send a message to ChatGPT or Claude, or talk to a voice assistant, the model runs on your message and produces a reply. That act of running the model is inference.
Training happens once per model. Inference happens billions of times a day, for every user, for as long as the product exists, which is why most of the money in AI is now spent on inference rather than training.
Inference engineering is the work of making each of those replies faster, cheaper and more reliable, without making the model any dumber.
That last part matters, because it tells you what inference engineers do not touch: the model's intelligence is fixed by training.
What they control is everything around it. Which chips the model runs on, how many people share one chip, what the model remembers and what gets thrown away, and where in the world the computers sit.
The one fact that explains almost everything
A language model doesn't write its reply all at once. It writes one small piece, called a tokenToken The small pieces a model reads and writes. For English text a token is about three-quarters of a word, so a 100-word reply is roughly 130 tokens. (in English, a token is about three-quarters of a word). Then it looks at everything written so far and writes the next token, and it repeats that loop until it decides the reply is finished.
So a 100-word reply means the model ran about 130 times in a row, and each run had to wait for the one before it.
Keep this in your head, because nearly every problem in the rest of this article traces back to it.
2 · Reading and writing: the two halves of every reply
When your message arrives, the model does two jobs that behave very differently.
First, it reads. It takes in your message, plus the instructions the company wrote for it (called the system promptSystem prompt The standing instructions the company writes for the model: its role, its rules, its tone. They are sent with every request, so a long one costs time on every turn unless it is cached.), plus the whole conversation so far. Engineers call this prefillPrefill The reading half of every reply: the model takes in your message, its instructions and the conversation so far all at once, because every word is already there. It decides how long you wait before the first word appears..
What makes reading different is that every one of those words is already there. The model doesn't have to wait for any of them, so instead of going through them one by one, it takes them all in at the same time, the way you take in a short sentence at a glance rather than letter by letter. Engineers say it reads in parallel.
How long the reading takes decides how long you stare at a blank screen before the first word appears. That wait is called TTFTTTFT Time to first token: how long you wait before the first word of a reply appears. It is set mostly by the reading (prefill) step., "time to first tokenToken The small pieces a model reads and writes. For English text a token is about three-quarters of a word, so a 100-word reply is roughly 130 tokens.".
Then, it writes. This half can't happen all at once, because the reply doesn't exist yet: the model has to write the first token before it can work out the second, and the second before the third. Engineers call this decodeDecode The writing half of every reply: the model produces one token at a time, each one waiting for the last. It decides how fast the words flow once they have started..
How fast the writing goes decides how quickly words stream out once they've started, measured as TPOTTPOT Time per output token: how long each new word takes once the reply has started. It is set mostly by the writing (decode) step., "time per output token" (or flipped around, tokens per second).
3 · The pantry door and the notebook
Here is the surprising part, and the whole field is built around it.
The chip that runs AI models is called a GPUGPU The chip that runs AI models. It does maths extremely fast, but keeps everything in its own limited memory and spends much of its time waiting for data to arrive.. Picture it as a chef who can chop impossibly fast, but who keeps every ingredient in a pantry behind one narrow door. The model's weightsWeights The numbers that make up a trained model. A big model's weights fill about 140 GB, and all of them have to be read for every token it writes., all 140 gigabytes of them, live in the pantry (the GPU's memory, sometimes called VRAMVRAM / HBM The GPU's own fast memory, where the model's weights and every conversation's notebook have to fit. In the pantry picture, it is the pantry. or HBM).
To write one single tokenToken The small pieces a model reads and writes. For English text a token is about three-quarters of a word, so a 100-word reply is roughly 130 tokens., the chef has to carry every one of those 140 gigabytes through the door, do a small amount of chopping with them, and then carry them all through again for the next token.
On a top GPU like NVIDIA's H100H100 An NVIDIA data-centre GPU, and the one this article's numbers use: 80 GB of memory that can move about 3.35 TB a second. Newer chips (H200, Blackwell) are faster, but the ideas are the same., the chopping for one token (the actual maths) takes about a tenth of a millisecond. Carrying the weights through the door takes tens of milliseconds.
So while the model is writing, the chip spends well over 99% of its time waiting at the door and almost none of it doing maths.
Engineers say writing is memory-boundMemory-bound Limited by how fast data can move out of memory, not by how fast the chip can calculate. Writing a reply is memory-bound: the chip spends most of its time waiting for the weights to arrive.: its speed is set by how fast memory can move data (the memory bandwidthBandwidth How fast data can move from one place to another, such as out of the GPU's memory. In the pantry picture, it is how wide the door is., which is how wide the door is), not by how fast the chip can calculate.
Reading works the other way round. The chef carries the weights through the door once and then uses them on all 2,000 words of your prompt at the same time, so there's a lot of chopping for every trip.
Reading is compute-boundCompute-bound Limited by how fast the chip can do maths, not by memory. Reading a prompt is compute-bound, because each trip through memory feeds thousands of tokens' worth of work.: it's limited by maths speed. That gives you a pairing worth memorising, because every trick you'll hear about is aimed at one side of it:
You can even estimate writing speed on the back of an envelope. Time per token ≈ Bytes the chip must read ÷ how fast memory moves. An H100 moves about 3.35 terabytes a second.
Imagine one chip big enough to hold a 140 GB model: it would take about 42 milliseconds per token, around 24 tokens a second, for a single user.
(In reality one H100 holds only 80 GB, so a model that size is split across several GPUs, each carrying its share through its own door, which gets it down to roughly 10–20 ms.)
Shrink the model to 70 GB by storing each number with half the bits, and it roughly doubles in speed. That's the whole reason a technique called quantizationQuantization Storing the model's numbers with fewer bits, for example 8 or 4 instead of 16. There are fewer bytes to move, so writing gets faster, with a small loss in quality if pushed too far. exists.
The notebook: the KV cache
When the model writes token number 500, it has to look back at the 499 tokens before it to decide what fits next. That looking back is called attentionAttention The step where the model looks back over all the tokens before it to decide what fits next. It reads from the notebook (the KV cache) so it doesn't redo old work..
Working everything out about those 499 tokens from scratch at every single step would be absurdly wasteful. So the model keeps a notebook: for every token it has already seen, it stores two small lists of numbers, called a key and a value.
This notebook is the KV cacheKV cache The model's notebook: notes it keeps for every token it has already seen, so it never has to redo that work. It lives in the GPU's scarce memory and grows with every word of the conversation., short for key-value cache.
With it, each new step only has to work out notes for the one new token and then glance at the notebook for everything else.
The catch is size. For a 70-billion-parameter model (70 billion weights), the notes for a single token take about 320 kilobytes. That sounds tiny until you multiply it out.
A 4,000-token chat needs about 1.3 GB of notes, and those notes have to sit in the GPU's memory, the same scarce pantry that holds the weights, for as long as the conversation is alive.
They're also read on every writing step, so a longer conversation makes each token a little slower as well.
4 · Serving many people at once
Go back to the chef. If carrying the weightsWeights The numbers that make up a trained model. A big model's weights fill about 140 GB, and all of them have to be read for every token it writes. through the door is the expensive part, the obvious move is to cook for lots of customers while the ingredients are already on the counter. That's batchingBatching Serving many people's requests in the same step, so each trip through memory feeds many conversations at once. It is the main reason AI replies are affordable at all..
The GPUGPU The chip that runs AI models. It does maths extremely fast, but keeps everything in its own limited memory and spends much of its time waiting for data to arrive. reads the weights once and uses them to produce the next tokenToken The small pieces a model reads and writes. For English text a token is about three-quarters of a word, so a 100-word reply is roughly 130 tokens. for, say, 64 different conversations in the same step. Reading the weights once and serving 64 people costs far less than reading them 64 times, so the cost of each token falls dramatically.
Batching is the single reason AI chat is affordable at all.
So why not batch ten thousand people? There are two limits, and they are the same two limits that come back again and again in every real system.
Memory. The weights are shared, but every conversation brings its own notebook, and all of those notebooks have to fit in the pantry at the same time.
The pantry fills up with notebooks long before the chef runs out of chopping speed. This is usually what decides how many people one GPU can serve.
Delay. The weights are carried through the door once per step, but every person's notebook has to be carried through too. So each extra person adds their notebook to every trip, and with long conversations the notebooks, not the weights, end up setting the pace.
Each step gets slower for everyone in the batch. A bigger batch is cheaper per person but slower for each person.
This is the most basic trade-off in inferenceInference Running a trained model to produce a reply. Training happens once per model; inference happens every time anyone uses it, which is why most of the money in AI now goes there.: throughputThroughput The total work the machine gets done, counted in tokens per second across everyone it serves. Bigger batches raise throughput but make each person wait a little longer. (total tokens the machine produces) against latencyLatency How long one person waits for their reply. It trades off against throughput: serving more people at once makes each of them wait a little longer. (how fast any one person gets theirs).
Two ideas from the research you'll hear named whenever people talk about serving models.
Continuous batchingContinuous batching Instead of waiting for a whole group of requests to finish together, the engine swaps a finished conversation out and a new one in at every step, so no seat at the table stays empty.: instead of waiting for a whole group to finish together, the engine swaps a finished conversation out and a new one in at every single step, so no seat at the table stays empty.
PagedAttentionPagedAttention A way of storing each conversation's notebook in small fixed-size pages instead of one big reserved block, so no memory sits wasted. It is the idea behind vLLM. (the 2023 paper behind vLLMvLLM Free, open-source software that runs a model on GPUs and serves many people at once. It is the most widely used inference engine.): store each notebook in small fixed-size pages instead of one big reserved block, the way a computer's operating system handles memory.
That stops memory being wasted on half-empty reservations, so more conversations fit on the same GPU.
5 · The toolbox, and a check on what you've learned
Almost every technique in inferenceInference Running a trained model to produce a reply. Training happens once per model; inference happens every time anyone uses it, which is why most of the money in AI now goes there. engineering is one of these seven. You don't need the details yet; you need to know which lever each one pulls and what it costs, because that's how people who build these systems talk about them.
| Technique | In one line | What it improves | What it costs |
|---|---|---|---|
| Batching | Serve many people from each trip through the door | Cost per token | Each person gets a bit slower; every notebook must fit |
| Prefix caching | Reuse notebook pages for words already read | Wait before the first word; cost | Memory to keep old notebooks around |
| Quantization | Store numbers with fewer bits (16 → 8 → 4) | Writing speed; room for more notebooks | Small quality loss if pushed too far |
| Speculative decoding | A small fast model guesses the next few tokens, the big one checks them all in one trip | Writing speed | Wasted work when guesses are wrong (the answer itself never changes) |
| Parallelism | Split one model across several GPUs | Big models fit, and run faster | The GPUs have to talk to each other constantly |
| Disaggregation | Do the reading and the writing on separate GPUs | Each side tuned for its own job | Notebooks have to be shipped between machines |
| Eviction and offloading | Throw away old notes, or move them to slower, cheaper memory (normal computer RAM, SSD) | More conversations fit | The model may forget, or you pay time to bring notes back |
Check yourselfTap to reveal
Why does a model write so much slower than it reads?
Reading uses each trip through the pantry door for thousands of tokensToken The small pieces a model reads and writes. For English text a token is about three-quarters of a word, so a 100-word reply is roughly 130 tokens. at once. Writing produces one token per trip, and the trip (moving 140 GB of weightsWeights The numbers that make up a trained model. A big model's weights fill about 140 GB, and all of them have to be read for every token it writes.) is what takes the time, not the maths.
What is in the KV cache, and why does it matter?
Notes the model keeps for every token it has seen, so it doesn't redo work. They grow with every word of the conversation and sit in the GPUGPU The chip that runs AI models. It does maths extremely fast, but keeps everything in its own limited memory and spends much of its time waiting for data to arrive.'s scarce memory, so they decide how many conversations a GPU can hold.
If you store the weights in half as many bytes, what happens to writing speed?
It roughly doubles, because time per token ≈ Bytes read ÷ memory speed. That's quantizationQuantization Storing the model's numbers with fewer bits, for example 8 or 4 instead of 16. There are fewer bytes to move, so writing gets faster, with a small loss in quality if pushed too far..
Batching makes tokens cheap. Why can't you batch everyone?
Every conversation's notebook must fit in memory at the same time, and every step gets slower as the batch grows. Memory usually runs out first.
Where to go next
You now have the four ideas that explain most of inferenceInference Running a trained model to produce a reply. Training happens once per model; inference happens every time anyone uses it, which is why most of the money in AI now goes there.: the model writes one tokenToken The small pieces a model reads and writes. For English text a token is about three-quarters of a word, so a 100-word reply is roughly 130 tokens. at a time, reading is limited by maths while writing is limited by memory, the notebook (the KV cacheKV cache The model's notebook: notes it keeps for every token it has already seen, so it never has to redo that work. It lives in the GPU's scarce memory and grows with every word of the conversation.) fills the scarce memory, and batchingBatching Serving many people's requests in the same step, so each trip through memory feeds many conversations at once. It is the main reason AI replies are affordable at all. is what makes it affordable until the notebooks run out of room.
The companion article, Voice AI inference 101, takes one real system, an AI you can talk to out loud, and turns these ideas into equations of input levers and output levers: what sets the delay, the cost and the quality of every reply.
If you want to go deeper
- Inference Engineering by Philip Kiely (Baseten)free online. Chapters 2 (models) and 5 (techniques) go deeper on everything here.
Glossary
- TokenA piece of a word, about three-quarters of a word in English. Models read and write tokens.
- Weights / parametersThe numbers that make up the model. "70B" means 70 billion of them.
- GPU, VRAM / HBMThe chip, and its own fast memory.
- Prefill / decodeReading the input / writing the output.
- TTFT / TPOTTime to first token / time per output token.
- KV cacheThe model's notebook of per-token notes, kept in GPU memory.
- Prefix cachingReusing notebook pages when a new request starts with words already read.
- Batching, continuous batchingServing many requests per step, and swapping requests in and out at every step.
- QuantizationStoring numbers with fewer bits.
- Speculative decodingA small model drafts, the big model verifies several tokens in one trip.
- DisaggregationReading and writing on separate GPUs.
- Eviction / offloadingDropping notes / moving them to slower, cheaper memory.
Figures marked "about" are back-of-envelope estimates meant to build intuition, not benchmarks. The 70B figures assume a Llama-3-70B-style model in 16-bit numbers on NVIDIA H100sH100 An NVIDIA data-centre GPU, and the one this article's numbers use: 80 GB of memory that can move about 3.35 TB a second. Newer chips (H200, Blackwell) are faster, but the ideas are the same. (80 GB, about 3.35 TB/s).
Next read
Voice AI inference 101 →
Part 2 of the series: what sets the delay, the cost and the quality of every reply.