Voice AI inference 101

This article builds on Inference engineering 101. If you haven’t read it, start there.

I spent the last full month reading 10 research papers, watching 25 YouTube videos, going through lessons and speaking with AI researchers working on voice AI inferenceInference Running a trained model to produce a reply. Training happens once per model; inference happens every time anyone uses it, which is why most of the money in AI now goes there.. I got so into it that I decided to organise an event around it.

We called it Research on Tap. The idea is simple: bring researchers into a room, pick a problem and discuss the research papers that could help solve it. This time, 12 of us got together to work through voice AI inference.

A lot of voice AI inference comes back to how inference works in general. Throughout the article, I’ll bring in practical notes from the people working on these systems, connecting what the research says with what’s actually happening on the ground.

By the end, you should be able to pick up any voice agent, find where the caller is left waiting, and know which levers shorten that wait and what each one costs.

1 · Output levers and input levers

The clearest way to understand a system like this is in input levers and output levers, so let's set it up that way.

It's how I break almost everything down, whether it's growth, marketing, product or tech. It's the GrowthX format, and it's how I wrote the whole 100 Days of Careers series: the output levers are what you want, and the input levers are what you can change to get there.

Here, the output levers are the three things anyone running a voice agent cares about.

The input levers are everything you can change to move them, and almost every input that helps one output hurts another. Deciding which output to give up, and by how much, is what building voice AI is really about.

1. Delay. This is the pause between you finishing a sentence and the agent starting to answer, and it's called voice-to-voice latencyVoice-to-voice latency The time from when the caller stops talking to when they hear the first sound of the reply. Under about 800 ms feels natural; past about 1.5 seconds it starts to feel broken.👇Hover any dotted word to see what it meansTap any dotted word to see what it means. People talking to each other leave about 200 milliseconds between turns. We give an AI a little more slack than that: one that answers in under about 800 ms feels natural, and past roughly 1.5 seconds it starts to feel broken.

   An average can hide the problem, because a caller never hears an average; they hear every single reply. So engineers track two numbers: the typical reply (the median, written P50P50 The median, or typical, reply: half of all replies are faster than this and half are slower.) and the slowest reply in every twenty (the slow tail, written P95P95 The slow tail: the slowest reply in twenty. One awkward pause in every twenty replies can ruin a call even when the average looks fine.). One awkward three-second silence in every twenty replies ruins a call, even when the average looks fine.

2. Quality. This is whether the conversation actually works. Did it hear you right, and understand you? Does it remember what you said earlier? Does it take turns well, never cutting you off and stopping when you talk over it? And does it sound like a person?

   These parts multiply rather than add, because any one of them at zero ruins the call. A perfect voice answering a question you never asked is still a failed call.

3. Cost. How much each minute of conversation costs to run.

×

From Research on Tap: Nobody agreed on which of the three matters most. Some use cases want a reply now and will forgive a weaker answer. Others will wait a second or two but can't afford a wrong one.

One person's experience was that callers care more about delay than quality: if the agent takes eight seconds, the answer had better be extraordinary.

So I proposed turning the three into one goal and a set of limits, the way you'd frame a project at work.

That gives every lever in the rest of this article the same simple test: does it bring the delay down without breaking a limit?

The goal · delay500 msFrom the moment the caller stops talking to the first sound they hear, the figure the room called golden

• Limit · costUnder ₹2 a minutePeople quoted ₹1–3 a minute for systems running today, with ₹1.5 as the number to beat. Most of it goes on the language model and the text-to-speech voice
• Limit · hardwareFixedA faster GPU only pays off if you can pack proportionally more callers onto it, and that rarely scales in a straight line
• Limit · qualityNo dropFrom the inference side, speed is the only part of quality you touch

We can't control the model's quality. The only part of it we touch is speed. As simple as that.

2 · How a voice agent works

When you phone a voice agent, you aren't talking to one model. You're talking to a relay race of several models, each one passing the baton to the next, and each one adding a little delay before you hear anything back.

Follow one reply through the relay. Your voice travels over the internet to a server, where a tiny model listens for one thing: is anyone talking right now? (It's called VADVAD Voice activity detection: a tiny model that answers one question, “is someone talking right now?” It is the first step in knowing when the caller has finished., voice activity detection.) When you go quiet, a second check decides whether you've finished your thought or only paused for breath, and that's turn detectionTurn detection Deciding whether the caller has finished their thought or has only paused. Too eager and the agent interrupts people; too cautious and every reply starts late..

Then a speech-to-text model (STTSTT Speech to text: the model that turns the caller's audio into words, also called ASR. A wrong word here carries into everything after it., also called ASR) writes down your words, and the LLMLLM Large language model: the kind of AI that reads and writes text, like ChatGPT or Claude. In a voice agent it is the part that decides what to say., a large language model like the one behind ChatGPT, reads them and writes a reply. A text-to-speech model (TTSTTS Text to speech: the model that turns the written reply into a voice. Modern ones are small language models that write audio instead of words.) turns that reply into a voice, and the audio travels back to your phone.

If every runner waited for the one before it to finish completely, the delays would simply stack up, one whole runner after another. The idea that makes the relay workable is that nobody waits for the whole baton. Speech-to-text sends words as it hears them, the LLM sends each tokenToken The small pieces a model reads and writes. For English text a token is about three-quarters of a word, so a 100-word reply is roughly 130 tokens. the moment it's written, and TTS starts speaking the first sentence while the LLM is still writing the second. Engineers call this streaming.

That changes what you should measure. For each runner, what matters is not how long it takes to finish, but how long until its first useful piece comes out, because that's the moment the next runner can start.

Where the time goes

×

Now you can write the delay down as a sum, with one part for every gap in the relay. Each part is moved by a few input levers, and the rest of this article is those levers, one at a time.

Delay=

Audio in

+ Waiting to be sure they've finished

+ Finishing the transcript

+ LLMLLM Large language model: the kind of AI that reads and writes text, like ChatGPT or Claude. In a voice agent it is the part that decides what to say.: waiting in line

+ LLM: reading (prefillPrefill The reading half of every reply: the model takes in your message, its instructions and the conversation so far all at once, because every word is already there. It decides how long you wait before the first word appears.)

+ LLM: writing the first phrase

+ TTSTTS Text to speech: the model that turns the written reply into a voice. Modern ones are small language models that write audio instead of words.: first sound

+ Audio out

The three LLM parts happen in the chef's kitchen from Inference engineering 101: waiting in line for a seat at the table (the GPUGPU The chip that runs AI models. It does maths extremely fast, but keeps everything in its own limited memory and spends much of its time waiting for data to arrive. is busy with other callers), reading your message, and writing the first few words.

To see how big each part is in practice, here is a typical budget for a well-built system. It comes from a free primer, "Voice AI & Voice Agents", by Daily, the company behind Pipecat, an open-source toolkit for building voice agents. Their 650 ms for the LLM is a hosted AI service's time to its first response, and it includes writing the first phrase, so it won't line up exactly with the parts above.

A typical voice reply: 1,293 ms from "they stopped talking" to "they hear a voice"

Audio in: mic, encoding, network, smoothing buffer 114 ms

Deciding they're done + finishing the transcript 300 ms

LLM until its first token 650 ms

Gathering a phrase for TTS 20 ms

TTS first sound 120 ms

Audio out, to the speaker 89 ms

From Research on Tap: Nobody in voice measures delay the same way, so the room agreed on one number before anything else: time to first audio (TTFATTFA Time to first audio: from the moment the caller stops talking to the first sound they hear back. The room picked it as the one number to chase.), from the moment the caller stops talking to the first sound they hear back.

It sounds simple, but it's the easiest number to fool yourself with. One person had built the whole relay on rented A100s (an older NVIDIA GPU) and found TTS was the slowest runner, at 1.5 to 2 seconds to its first sound, while the LLM took only 300–500 ms.

The room pulled that apart. The stopwatch for TTS had started when the LLM started writing, but TTS waits for a whole sentence before it speaks, so it was being charged for all the time the LLM spent writing that sentence.

Where you start the stopwatch decides which box looks slow.

That's why the room counted only blocking time: the waiting the caller actually hears. Counted that way, systems like the ones in the room took about 1.2–3 seconds, against a goal of 0.5. The rest of this article is about closing that gap.

The plumbing

Four parts of the sum have nothing to do with thinking: sound travelling in, sound travelling out, finishing the transcript, and the first sound from TTS. They're easy to forget, and yet in Daily's budget the sound travelling in and out was about a sixth of the whole delay on its own.

Sound travelling in and out takes about 200 ms together, and the biggest lever is simply distance. A caller in the UK talking to a server in California spends about 140 ms on the round trip before anything else happens, while the same caller talking to a server in London spends about 15 ms, so where you put your GPUs is a delay decision, not just a cost one.

How the sound travels matters too. Audio crosses the internet in small packets, and on a patchy mobile connection some of them get lost on the way.

WebRTCWebRTC The standard way to send live audio over the internet. It skips a lost packet instead of freezing while it waits for it, so calls don't stall., the standard built for live calls, skips a lost packet and carries on, which you'd usually hear as a tiny glitch. An ordinary web connection (WebSockets, which run on TCP) stops everything and waits for the lost packet to be sent again, which on a bad connection shows up as random long pauses, exactly the kind that ruin your slowest replies.

Two smaller things add up as well. Bluetooth headsets can quietly add several hundred milliseconds, and because packets arrive unevenly, systems hold a few of them in a small waiting room called a jitter bufferJitter buffer A small waiting room for incoming audio, because packets arrive unevenly over the internet. A bigger buffer sounds smoother but adds delay before anything is heard. before playing them, which smooths the sound out at the price of a little delay.

Finishing the transcript should be quick if speech-to-text is streaming, because it has been writing down the words all the while the caller was talking, so when they stop, only the last fraction of a second is left. A good target is about 200 ms.

This is also where quality starts. The LLM can only answer what it was told, so one wrong word here poisons everything after it, and a useful trick is to tell the LLM to quietly correct words that were probably misheard.

The first sound from TTS takes about 195 ms to 340 ms for a typical reply from published services. Modern TTS models are themselves small language models that write audio tokens, tiny slices of sound, instead of words.

One detail from Baseten's book matters later. A TTS model only needs to write about 80–100 audio tokens a second to keep up with someone speaking, so once it's that fast, extra writing speed goes into serving more callers on one GPU rather than shortening anyone's wait. That makes it a cost lever, not a delay lever. Hold on to that idea, because the LLM turns out to work the same way.

The room took it as given that callers are served from their own region. The real fight is in the other four parts: waiting to be sure they've finished, waiting in line, reading and writing.

3 · Which input levers the goal leaves you

Before trying anything clever, it helps to cross off the levers the goal and its limits don't allow, because some of the best-known tricks in inferenceInference Running a trained model to produce a reply. Training happens once per model; inference happens every time anyone uses it, which is why most of the money in AI now goes there. are off the table here.

QuantizationQuantization Storing the model's numbers with fewer bits, for example 8 or 4 instead of 16. There are fewer bytes to move, so writing gets faster, with a small loss in quality if pushed too far. is out. Storing the model's numbers with fewer bits makes it faster, as Inference engineering 101 showed, but it can change what the model says, and quality isn't allowed to drop. A faster GPUGPU The chip that runs AI models. It does maths extremely fast, but keeps everything in its own limited memory and spends much of its time waiting for data to arrive. is out too, because the hardware is fixed.

And which model to use is the product's call, not an inference lever. From the inference side, your job is to make a given model answer faster.

What's left is exactly the list the room worked through, in this order:

1. Waiting to be sure they've finished
2. Starting before they finish
3. The LLMLLM Large language model: the kind of AI that reads and writes text, like ChatGPT or Claude. In a voice agent it is the part that decides what to say., reading and writing
4. Many callers at once: the notebook, scheduling and routing (this is where the cost is decided)
5. Tool callsTool call The model asking the system to run a lookup or an action, like checking a booking, and waiting for the result before it carries on. Each one adds a whole extra round trip before the caller hears anything.
6. Hour-long calls

Each one gets the same treatment: what the lever is in plain terms, how it works, where the obvious version breaks, what the papers and the room found, and finally its trade, which is what you give up to get it.

Many of the strongest papers here aren't about voice at all. They're about the machinery under every language model, like memory, caching and how GPUs talk to each other, and they still apply, because a voice agent's LLM runs on that same machinery.

4 · Lever 1: waiting to be sure they've finished

On a phone call you can't see the other person, and yet you rarely talk over each other. You're reading three clues at once without noticing: the silence, the tone of their voice as it rises or falls, and whether their sentence sounds finished.

A voice agent has to make the same call, and it's harder than it sounds. The simplest rule is to wait for silence, say 0.8 seconds of it, and treat that as the end of the turn.

That rule breaks in both directions. Waiting 0.8 seconds adds 800 ms before the agent has even started thinking, which on its own is more than the room's whole goal. Waiting less means the agent starts answering while people are only pausing mid-sentence, which feels rude and counts as a quality failure.

Smarter systems use a small turn-detectionTurn detection Deciding whether the caller has finished their thought or has only paused. Too eager and the agent interrupts people; too cautious and every reply starts late. model that listens to the words and the tone, the way you do. "So what I was thinking was…" is obviously unfinished and "yes, that's right." obviously isn't, so a model like this can decide in around 250 ms instead of waiting out a fixed silence.

From Research on Tap: The room split on which clue to trust.

Listening to the sound is fast, about 80–100 ms, because people tend to trail off before they stop. But sound alone hears "I want to order a burger from…", then a pause, and decides the caller is done.

Reading the words catches that, because "from…" is obviously unfinished, but it has to wait for the transcript, so it's four to five times slower.

The answer was to use both, in order. The sound is the fast first check, and only when it thinks the caller has stopped do you read the last few words before committing.

That turns the sound check into a dial for how many false alarms you'll accept. Turn it towards speed and you might act when you're only 50% sure, but every time the caller keeps talking, you throw that work away and start again.

The best idea was to set that dial per use case rather than per person. A support call and a career counselling call need different amounts of patience, and a therapy call can't keep cutting people off.

It also fits the budget. The sound check takes only 80–100 ms, so any extra logic has to cost less than that, and a setting chosen before the call starts costs nothing at all.

Turn-taking has a second half, called barge-inBarge-in The caller starting to talk while the agent is still speaking. The agent has to stop and remember only the words it actually said out loud, not the ones it had written but never spoke.: the caller talking over the agent. When that happens, the agent has to stop talking and remember only the words it actually said out loud, not the ones it had written but never spoke, or it will carry on as if the caller had heard things they never did.

Trade: delay against interrupting people. Every millisecond you save here is paid for in callers you interrupt.

5 · Lever 2: starting before they finish

Think about how you manage a 200 millisecond gap in an ordinary conversation. You don't wait for the other person to finish and then start thinking about your answer. You're already working it out while they're still talking, and you're quietly guessing when they're about to stop.

This lever asks a voice agent to do the same thing. It can save a lot of time, but there's one difference from you: the agent pays for every guess in GPUGPU The chip that runs AI models. It does maths extremely fast, but keeps everything in its own limited memory and spends much of its time waiting for data to arrive. time.

Here's why it saves time. Remember from Inference engineering 101 that the model reads your message before it writes anything, and keeps notes on everything it has read in its notebook (the KV cacheKV cache The model's notebook: notes it keeps for every token it has already seen, so it never has to redo that work. It lives in the GPU's scarce memory and grows with every word of the conversation.).

If you feed the model the transcript while the caller is still talking, say "I'd like to book a table for", then by the time they finish with "…four at eight", it has already read most of the sentence and written notes on it. The finished sentence begins with exactly the words it has already read, so those notes are reused as they are, and only the last few words are left to read once the caller stops.

The catch is that a transcript isn't final while someone is still speaking. Speech-to-text keeps changing its mind as it hears more, and the words it's least sure about are always the last few.

If "…fly to Boston" becomes "…fly to Austin", every note from "Boston" onward is wrong and has to be thrown away and read again. The notes before the changed word are still good, which is why a correction near the end costs very little and a correction early in the sentence costs a lot.

And you can't simply ignore corrections: if "burger" was first heard as "pizza", the model has to see the fix, or it answers the wrong order.

You can push the bet one step further and start writing the reply before you're sure the caller has finished, then throw the draft away if they keep talking. That sounds risky, but it's safe as long as you never speak before the turn is confirmed: a wrong guess wastes GPU time, and never puts wrong words in the caller's ear.

So is all that thrown-away work a problem? It depends entirely on how busy the GPU is, and that's exactly where it went wrong for one builder at Research on Tap.

From Research on Tap: One person had built the reading half of this, feeding the half-finished transcript in to be read but not yet written, and then switched it off. On real calls, speech-to-text revised itself so often that the GPU kept redoing reading it had already done, and the cost stopped making sense.

Someone in the room argued that the waste is free, because the GPU would be sitting idle while the caller talks anyway. That's true when there's one caller. In production there's always someone else's work waiting, so every note you throw away is time taken from another caller.

Another person had the fix, and it's a good example of how simple these rules have to be. Only pass a word on to the model once it has survived two transcript updates in a row: if "book a table for" is still there when the next word arrives, it has probably settled, so it's safe to read.

That one rule cut the re-reading to 14% of the time. Anything cleverer, like a check that tries to work out whether the meaning has changed, takes longer to run than the time it saves.

The papers say the bet can pay off. The best fit on the whole list, Endpoint Anticipation (2026), predicts that the caller is about to finish up to 2.56 seconds before they actually do, and starts the LLMLLM Large language model: the kind of AI that reads and writes text, like ChatGPT or Claude. In a voice agent it is the part that decides what to say. and the text-to-speech voice on the unfinished sentence.

Replies arrived about 505 ms sooner on average, a real bite out of the 1.2–3 seconds the room's systems were taking, and the price was 28.4% more computation. That's the delay-against-cost trade with real numbers on both sides.

The other two papers place the same bet at different moments. PredGen (2025) spends the GPU work before the turn ends: it drafts possible replies while the caller is still talking, so the first sentence is ready for the voice the moment they stop. RelayS2S (2026) spends it just after: a small, fast speech model says the first few words straight away, buying time while a slower, smarter pipeline writes the rest.

If this shape feels familiar, it's because you met it in Inference engineering 101. Speculative decodingSpeculative decoding A small, fast model guesses the next few tokens and the big model checks them all in one trip through memory. Good guesses make writing faster, and the answer itself never changes. makes the same bet one level down, where a small model guesses the next few tokensToken The small pieces a model reads and writes. For English text a token is about three-quarters of a word, so a 100-word reply is roughly 130 tokens. and the big model checks them: guess cheaply, check, and throw away wrong guesses without ever changing the answer. You'll see it once more with tool callsTool call The model asking the system to run a lookup or an action, like checking a booking, and waiting for the result before it carries on. Each one adds a whole extra round trip before the caller hears anything. in lever 5.

Trade: delay saved against GPU work risked. With one caller a wrong guess is free; with a hundred, it's time taken from someone else.

6 · Lever 3: the LLM, reading and writing

Now the chef. Inside the LLMLLM Large language model: the kind of AI that reads and writes text, like ChatGPT or Claude. In a voice agent it is the part that decides what to say. there are two parts of the delay, and you already know them both from Inference engineering 101. Reading your message (prefillPrefill The reading half of every reply: the model takes in your message, its instructions and the conversation so far all at once, because every word is already there. It decides how long you wait before the first word appears.) is limited by how fast the chef can chop (the maths), and it decides the wait before the first word. Writing the reply (decodeDecode The writing half of every reply: the model produces one token at a time, each one waiting for the last. It decides how fast the words flow once they have started.) is limited by the pantry door (the memory), and it decides how fast the words flow.

Voice changes which of the two you should worry about, and it starts with something about conversations that surprises most people.

On every turn of a call, the whole conversation so far is sent to the model again, all the way back to the first "hello". Between turns, the only thing that can save the model from reading it all again is the notebook, if the serving software has kept it.

If the GPUGPU The chip that runs AI models. It does maths extremely fast, but keeps everything in its own limited memory and spends much of its time waiting for data to arrive. still has the notebook from the last turn, it only needs to read the caller's new sentence, perhaps a few dozen tokensToken The small pieces a model reads and writes. For English text a token is about three-quarters of a word, so a 100-word reply is roughly 130 tokens., and reading time almost disappears. But if the notebook was thrown away, because the memory was needed for another caller or because this turn was sent to a different GPU, the model has to re-read the whole conversation from the start, and in a long call that's thousands of tokens.

Levers: how long the system promptSystem prompt The standing instructions the company writes for the model: its role, its rules, its tone. They are sent with every request, so a long one costs time on every turn unless it is cached. and tool list are (and keeping them word-for-word the same across calls, so their notes can be reused), whether the notebook from the last turn is still there (a prefix-cache hit), and whether each turn of a call goes back to the same GPU.

Now writing. TTSTTS Text to speech: the model that turns the written reply into a voice. Modern ones are small language models that write audio instead of words. can't start speaking until it has a phrase, so the caller isn't waiting for the first token; they're waiting for the first phrase that can be spoken aloud.

Work it through. A 70B model stored in 16-bit numbers is 140 GB, too big for one GPU, so it's spread across four H100sH100 An NVIDIA data-centre GPU, and the one this article's numbers use: 80 GB of memory that can move about 3.35 TB a second. Newer chips (H200, Blackwell) are faster, but the ideas are the same., which together manage roughly 10–15 ms per token. A 15-token opening sentence therefore takes around 150–225 ms, and it ties up four GPUs for that one model.

An 8B model is 16 GB, so it fits on one H100 at about 5 ms per token, and the same sentence takes around 75 ms, on a quarter of the hardware.

That calculation is why most voice agents use small, fast models: they win on delay and on cost at the same time.

Two habits quietly slow the first phrase down. One is thinking tokens: a reasoning model does all its thinking before it says a word, which is terrible for voice. The other is prompting that lets the model open with a long sentence when a short phrase would do.

Here's the idea from the TTS section paying off. Once a model writes faster than people speak, extra writing speed mostly stops shortening the wait.

From Research on Tap: Once the reply has started, writing speed barely matters, because a few tokens a second already keeps up with speech. Extra speed only helps the first phrase.

So what the caller feels is the wait for the first few words, and that's what the scheduler, the part of the engine that decides whose work runs next, should protect. It can push a new caller's first few tokens ahead of everything else, and set a deadline for each token so no caller's next word arrives late.

The papers take the same thinking further with disaggregationDisaggregation Running the reading (prefill) and the writing (decode) on separate GPUs, so each group can be tuned for its own job. The catch is that notebooks have to be shipped between them.: doing the reading on some GPUs and the writing on others, so each group can be tuned for its own job, and shipping each call's notebook between them.

From Research on Tap: Voice makes this split lopsided. Since writing speed barely matters, you can put three or four machines on reading and just one on writing.

The obvious worry is the time it takes to ship notebooks between them. But one person who had run this on a three-machine cluster found it didn't matter, because the machines sat in the same rack on a very fast link and behaved like one machine.

One trick you'll hear about a lot in inferenceInference Running a trained model to produce a reply. Training happens once per model; inference happens every time anyone uses it, which is why most of the money in AI now goes there., splitting one model across several GPUs, rarely matters here. Voice uses small models to stay fast, and each model in the relay already runs on its own cards.

That's why a whole line of research matters less here. Every Microsecond Matters (2026) gets the tiny messages between GPUs sharing one model to within 7% of the physical speed limit, which counts for giant models split across many GPUs and much less for voice. FlashAttention-4 (2026) sits underneath everything, voice included: it makes attentionAttention The step where the model looks back over all the tokens before it to decide what fits next. It reads from the notebook (the KV cache) so it doesn't redo old work., the model's look back over earlier tokens, faster on NVIDIA's newest chips.

Trade: a notebook you keep is memory you can't give another caller, and every machine doing reading is a machine that isn't writing. Voice lets you lean hard towards reading.

7 · Lever 4: many callers at once

One caller on one GPUGPU The chip that runs AI models. It does maths extremely fast, but keeps everything in its own limited memory and spends much of its time waiting for data to arrive. is easy, because the hardware sets how fast you can possibly be and nobody gets in your way. The real problem is staying that fast with a hundred callers on the same GPU, and that's where the cost of every minute is decided.

Remember batchingBatching Serving many people's requests in the same step, so each trip through memory feeds many conversations at once. It is the main reason AI replies are affordable at all. from Inference engineering 101: the chef carries the ingredients through the door once and cooks for many customers on each trip. The more callers share one GPU, the cheaper each minute gets, so the whole question is how many fit before something breaks.

The number of calls the hardware can hold is the smaller of two limits, memory and speed. Long calls run into the memory limit first, so let's put real numbers on it.

A worked example worth being able to say out loud. Take a 70B model on a machine with eight H100H100 An NVIDIA data-centre GPU, and the one this article's numbers use: 80 GB of memory that can move about 3.35 TB a second. Newer chips (H200, Blackwell) are faster, but the ideas are the same. GPUs: 640 GB of memory in total. The weightsWeights The numbers that make up a trained model. A big model's weights fill about 140 GB, and all of them have to be read for every token it writes. take 140 GB, which leaves roughly 500 GB for notebooks.

People speak about 150 words a minute, so an hour of two-way conversation is roughly 12,000 tokensToken The small pieces a model reads and writes. For English text a token is about three-quarters of a word, so a 100-word reply is roughly 130 tokens.. At 320 KB of notes per token, that's about 4 GB of notebook for one hour-long call, roughly the size of a full-length HD movie for every caller.

So if every call on the machine were in its sixtieth minute, it could hold only about 110–125 calls at once (engines also keep some memory back as working space).

Change one assumption and the answer moves a lot. Real calls are at mixed points, averaging maybe 2 GB each, so it's closer to 200. Five-minute calls need only about a third of a gigabyte each, so memory could hold 1,500 of them, and the speed limit would kick in first.

As calls get longer, memory, not maths, decides how many people you can serve.

What happens as the batch fills up? The part of the delay that moves first is waiting in line. You barely notice it on the typical reply, but when traffic spikes it blows up the slowest ones, the slow tail from section 1.

With very big batches, the twist from Inference engineering 101 kicks in: each trip through the door feeds so many conversations that the chef finally becomes the bottleneck, and the work becomes limited by maths. NanoFlow (OSDIOSDI One of the two top research conferences for computer systems. NanoFlow was published there. 2025) keeps the maths, the memory and the network all busy at once inside one GPU, so no part sits waiting for another, for about 1.9× more throughputThroughput The total work the machine gets done, counted in tokens per second across everyone it serves. Bigger batches raise throughput but make each person wait a little longer..

Keeping GPUs busy matters too, because a GPU sitting idle at 3 a.m. still costs money. The small models (turn detectionTurn detection Deciding whether the caller has finished their thought or has only paused. Too eager and the agent interrupts people; too cautious and every reply starts late., speech-to-text, TTSTTS Text to speech: the model that turns the written reply into a voice. Modern ones are small language models that write audio instead of words.) are cheap by comparison and can share slices of one GPU.

From Research on Tap: Of the levers that don't change what the model says, the two biggest are the notebook and scheduling. But one builder found that in practice, routing was the biggest problem at scale, and the reason is worth understanding.

A call arrives in turns, with gaps in between. A normal load balancer, the traffic cop in front of the servers, sends each turn to whichever machine is free, and that machine doesn't have the call's notebook, so it re-reads the whole conversation.

In kitchen terms, it hands each new order from a table to whichever chef is free, when only one chef has that table's notebook, so the new chef has to read the table's whole history before cooking anything.

So the fix sits above the GPUs: send every turn of a call back to the machine that holds its notebook, either by pinning each call to one machine (sticky sessions) or by routing on a fingerprint of the conversation so far. As you add callers, the bottleneck moves from the GPU to the router.

The papers agree. The notebook is everything the call has heard so far, so send the work to wherever the notebook is, rather than moving a notebook of several gigabytes to the work.

ConServe (2026) was written for AI agents that go back and forth over many turns, which is exactly the shape of a phone call. It decides once per conversation: do the big first read on a GPU built for reading, move the notebook exactly once, then keep the whole conversation on one writing GPU.

From Research on Tap: Some systems keep a shared pool of notebooks across machines (Mooncake and LMCache are two), which helps when a turn can't go back to its own machine. But they save GPU work, not time: the caller still waits for the notebook to arrive.

Trade: throughput against each caller's slow tail. Pack more calls onto each GPU and the cost per minute falls, until the slowest replies (P95P95 The slow tail: the slowest reply in twenty. One awkward pause in every twenty replies can ruin a call even when the average looks fine.) start to stretch.

8 · Lever 5: tool calls

Ask a shop assistant whether something is in stock and they'll say "let me check" and go and look. A voice agent does the same whenever it needs to look something up, like checking your booking, and that's called a tool callTool call The model asking the system to run a lookup or an action, like checking a booking, and waiting for the result before it carries on. Each one adds a whole extra round trip before the caller hears anything..

The trouble is what a tool call does to the relay. The model has to write out the whole request before it can be sent (it can't be streamed), then wait for the answer, then run again to turn the answer into a reply. So every tool call adds an entire extra LLMLLM Large language model: the kind of AI that reads and writes text, like ChatGPT or Claude. In a voice agent it is the part that decides what to say. round trip before the caller hears anything.

This is where the bet from lever 2 comes back. Speculative Interaction Agents (2026) starts a lookup before it's sure the caller has finished giving the details, and keeps working while it waits, for 1.3–2.2× faster responses. It was written with voice in mind: the goal is keeping voice agents under about a second.

From Research on Tap: A tool call will never fit in 500 ms, so the room's answer was to hide the wait rather than remove it, with two tricks.

First, say something, the way the shop assistant does. "Let me check that" is what a person would say too, and it fills the silence while the lookup runs.

Second, fetch before you're asked. On a support call you already know the caller will probably ask about their orders, so pull that data in when the call starts, and it's there the moment the model asks for it.

Trade: an agent that can actually do things, against a round trip the caller can hear. Hide the wait with words and prefetching.

9 · Lever 6: hour-long calls

The room ran out of time before this one, so it's papers only.

Every word of a call adds to its notebook, so the longer the call runs, the thicker the notebook gets: about 4 GB per hour per call on a 70B model, as the worked example in lever 4 showed. That leaves you stuck between two bad choices.

Keep the whole notebook and fewer calls fit on each GPUGPU The chip that runs AI models. It does maths extremely fast, but keeps everything in its own limited memory and spends much of its time waiting for data to arrive., so the cost per minute rises. Throw it away and the model has to re-read everything on the next turn, so reading time explodes. Throw only some of it away and you risk dropping the wrong notes, and the agent forgets what you said twenty minutes ago, which is the "remembered it" part of the quality equation.

There are four ways out, and each has its own price.

1. Keep everything. Simple and safe, but expensive.

2. Throw some notes away (evictionEviction Throwing old notes out of the notebook to free memory for other conversations. It saves memory, but the model may forget something the caller said earlier.). The hard question is which ones. Random Attention (2026) found that clever ways of choosing which notes to drop barely beat choosing at random, as long as you keep the whole prompt and only thin out the model's own reasoning. It matched the best method while serving 32–43% more throughputThroughput The total work the machine gets done, counted in tokens per second across everyone it serves. Bigger batches raise throughput but make each person wait a little longer..

   But it was tested on reasoning, which repeats itself. In a conversation, a detail from minute 3 might matter at minute 50.

3. Move old notes somewhere cheaper. Keep them in three places, each further from the chef: GPU memory (the pantry), the computer's normal memory (a back room) and SSDSSD Disk storage: huge and cheap, but slow and unpredictable to read from. (a warehouse). The hard part is loading them back fast enough.

   Normally the GPU fetches old notes while it's busy reading the new words. That breaks in exactly the case a phone call creates: a long saved history plus only a few new words, so there's almost no reading to hide the fetching behind. For voice, the trick is to start fetching while the caller is still finishing their sentence.

4. Summarise old turns into a few sentences of text, which keeps the gist and loses the detail.

Switching models in the middle of a call hits the same wall. A notebook is written in the private shorthand of the model that wrote it, so a new model can't use it and has to re-read the whole conversation.

For a 70B model and an hour-long call, that's about 1.7 quadrillion calculations: very roughly half a second, even spread across eight H100sH100 An NVIDIA data-centre GPU, and the one this article's numbers use: 80 GB of memory that can move about 3.35 TB a second. Newer chips (H200, Blackwell) are faster, but the ideas are the same., which is a huge bite out of the 800 ms that still feels natural. So switch only at a natural pause, warm up the new model in the background, or hand it a summary.

One more paper, for LLMsLLM Large language model: the kind of AI that reads and writes text, like ChatGPT or Claude. In a voice agent it is the part that decides what to say. that listen to audio directly instead of reading a transcript: their notebook holds notes on slices of sound rather than on words. WnW (2026) keeps only about 20% of those audio notes on the GPU with almost no loss in accuracy, though so far only on small models reading long recordings.

Trade: memory against remembering. Every way of fitting a long call on the GPU risks forgetting something, or pays for it in fetching time.

10 · How to hold all of this

If you close this page and remember one picture, make it this one. A voice agent is a relay of models, and the caller only ever feels the gaps between them.

Every input lever in this article shrinks one of those gaps, and almost every one is a bet: you spend GPUGPU The chip that runs AI models. It does maths extremely fast, but keeps everything in its own limited memory and spends much of its time waiting for data to arrive. work now to save time later. Reading the sentence before the caller has finished it, drafting a reply you might throw away, pulling up their orders before they ask.

So the question is never "does this trick work?" It's "how often do I lose the bet, and who pays when I do?"

With one caller, losing costs nothing. With a hundred callers on one GPU, every lost bet is time taken from someone else.

Then hold the three output levers the way the room did: one goal and two limits. The goal is delay, 500 ms to first audio, counting only the waiting the caller actually hears. Cost and quality are the limits: under ₹2 a minute, and no change to what the model says.

So an input lever that changes the answer is off the table, and a saving that comes from cutting people off or forgetting what they said isn't a lever at all. It's a broken limit.

Last, remember why voice needs its own inferenceInference Running a trained model to produce a reply. Training happens once per model; inference happens every time anyone uses it, which is why most of the money in AI now goes there. stack.

In chat, writing speed is the product. In voice it barely matters, because a few tokensToken The small pieces a model reads and writes. For English text a token is about three-quarters of a word, so a 100-word reply is roughly 130 tokens. a second already keeps up with speech.

What matters is the first few words, the wait before them, and the silence between turns. That silence is where the notebook sits, where the next turn gets routed, and where you can do work the caller never sees.

Underneath, it looks a lot like AI coding agents: long sessions, a conversation that only grows, and everything depends on keeping the notebook.

So when you look at any voice system, ask three things, in this order. Where does the caller wait? What bet is being made to shorten that wait? And where does the notebook live between turns? If you can answer those three, you understand the system.

The lever map

Here's every input lever in one place: which way it pushes each output, and the catch. ↓ means lower delay or cost (good), ↑ means higher (bad). For quality, the arrow is written out.

Glossary and extra material

• TokenA piece of a word, about three-quarters of a word in English. Models read and write tokens.
• Weights / parametersThe numbers that make up the model. "70B" means 70 billion of them.
• GPU, VRAM / HBMThe chip, and its own fast memory.
• Prefill / decodeReading the input / writing the output.
• TTFT / TPOTTime to first token / time per output token.
• KV cacheThe model's notebook of per-token notes, kept in GPU memory.
• Prefix cachingReusing notebook pages when a new request starts with words already read.
• Batching, continuous batchingServing many requests per step, and swapping requests in and out at every step.
• QuantizationStoring numbers with fewer bits.
• Speculative decodingA small model drafts, the big model verifies several tokens in one trip.
• DisaggregationReading and writing on separate GPUs.
• Eviction / offloadingDropping notes / moving them to slower, cheaper memory.
• P50 / P95The typical case / the slowest one in twenty.
• VAD, turn detection, STT/ASR, TTSIs someone talking / have they finished / speech to text / text to speech.
• WebRTCThe standard way to send live audio over the internet without freezing on lost packets.

If you want to go deeper

×

• Voice AI & Voice Agents: an illustrated primerby Daily. Where the 1,293 ms budget comes from, plus much more on turn detectionTurn detection Deciding whether the caller has finished their thought or has only paused. Too eager and the agent interrupts people; too cautious and every reply starts late. and interruptions.
• Inference Engineering by Philip Kiely (Baseten)free online. Chapter 6, sections 6.3–6.4 (speech models), and chapter 5 (techniques) match this article most closely.
• Inference engineering 101the companion article, with the basics this one builds on.

The papers in this article

• Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning(lever 6)
• NanoFlow: Towards Optimal Large Language Model Serving Throughput(lever 4)
• Every Microsecond Matters: Near Speed-of-Light Latency in GPU Collectives(lever 3)
• FlashAttention-4(underneath everything)
• Voice-specific additions: Moshi, Endpoint Anticipation, PredGen, RelayS2S, Speculative Interaction Agents, WnW, ConServe

Figures marked "about" are back-of-envelope estimates meant to build intuition, not benchmarks. The 70B figures assume a Llama-3-70B-style model in 16-bit numbers on NVIDIA H100sH100 An NVIDIA data-centre GPU, and the one this article's numbers use: 80 GB of memory that can move about 3.35 TB a second. Newer chips (H200, Blackwell) are faster, but the ideas are the same. (80 GB, about 3.35 TB/s). The voice latencyLatency How long one person waits for their reply. It trades off against throughput: serving more people at once makes each of them wait a little longer. breakdown and TTSTTS Text to speech: the model that turns the written reply into a voice. Modern ones are small language models that write audio instead of words. figures come from the Daily / Pipecat primer; the TTS real-time and speech-recognition figures come from Baseten's Inference Engineering.

×