I built the same voice pipeline twice — batch and streaming — to get a before number and an after number. The result looked clean:
batch: 1574 ms
streaming: 450 ms
ratio: 3.5xThen I ran it against utterances of different lengths, and the ratio moved:
utterance batch streaming ratio
1 s 1274 450 2.8x
2 s 1574 450 3.5x
5 s 2474 450 5.5x
10 s 3974 450 8.8xSame code. Same answer. Same everything except how long the person spoke.
A benchmark whose headline number ranges from 2.8× to 8.8× depending on an input property nobody mentioned is not a measurement. It is a number waiting for someone to quote it.
The two mechanisms
“Streaming” in a voice pipeline is not one optimisation. It is two, and they behave differently enough that averaging them produces nonsense.
Overlap. Streaming recognition runs while the person is still speaking. By the time they stop, most of the transcript exists; only a tail remains. The work did not get faster — it moved earlier in time, into a window that was previously idle.
Earlier delivery. The model emits the first sentence, that sentence goes to synthesis, and audio starts while the rest of the answer is still being written. Total work is identical. Only the moment of first output changed.
Decompose the same runs:
utterance overlap delivery
1 s 180 ms 644 ms
2 s 480 ms 644 ms
5 s 1380 ms 644 ms
10 s 2880 ms 644 msOverlap scales linearly with utterance length. Delivery is flat — 644 ms regardless, because it depends only on how fast the model produces its first chunk.
That single table changes what you would build.
If your users speak in short bursts — “yes”, “the second one”, “cancel that” — overlap buys you almost nothing. There is barely any speech to overlap with. Nearly all your gain comes from chunked generation, which is the cheaper half to implement: most inference APIs already stream.
If your users speak in long sentences — describing a problem, reading an address, explaining what went wrong — overlap dominates, and it requires streaming ASR: partial hypotheses, incremental decoding, endpointing. That is real work.
I had built both without knowing which one was earning the number.
Why total duration is the honest control
The first check I wrote for this asserted that total duration stays about the same between the two pipelines. Streaming, after all, does not make the model write faster.
It failed. The totals differed by 480 ms.
My first instinct was that the measurement was broken. It was not — the criterion was. Overlap genuinely reduces total elapsed time from the moment speech ends, because part of the work already happened. Only the delivery half leaves totals unchanged.
The fixed assertion is stricter than the original and says something real:
# The answer segment — model plus synthesis — must cost the same in both.
assert abs(answer_batch - answer_stream) < 0.001
# And the difference in totals must equal the overlap exactly.
# Any extra millisecond means streaming quietly did less work.
assert abs((batch.total - stream.total) - overlap) < 0.001The second line is the one that matters. It is a conservation law: every millisecond of improvement must be attributable to a named mechanism. If the numbers improve and the parts do not add up, something is being measured wrong — most likely a step that stopped being counted.
There is a second, meaner version of the same law, and I got it wrong on the first pass. I wrote the per-step invariant as the steps sum to the total. That is true only while the pipeline owns the clock alone. In streaming it does not: between chunks, control belongs to whoever is consuming them — pushing a frame into a socket, painting a row, waiting on the network. A stopwatch that measures each step from the previous mark bills that pause to the next step.
consumer spends 1000 ms between chunks
steps: recognition 120 · model 250 · synth 80
· model 1250 · synth 72 · model 1250 · synth 72
'model' total: 2750 ms the model actually slept: 750 ms
sum of steps: 3094 ms total: 4094 msThe sum reconciled perfectly against the wrong number, because the missing time was sitting inside a step that had a name and an owner. The most expensive step in the breakdown became whichever one the browser happened to pause after — and that is the step a reader would go optimise.
So the invariant now names every owner of the clock, and the leftover is asserted to zero:
sum of steps + handover to consumer + unattributed = total, unattributed == 0with a mirror half a two-term version cannot express: this step costs what this participant actually spent. Without it, the law is satisfied by charging someone else’s delay to a specific, plausible, innocent step.
That check would have caught the most embarrassing class of benchmark error, which is not measuring the wrong thing but measuring less of the thing.
The clock is the load-bearing decision
None of the above works if the numbers move between runs.
A timing check measured against the real clock depends on machine load. It passes nine times and fails the tenth, and then somebody disables it — and with it goes the only evidence for the whole exercise. I have never seen a flaky timing test survive a quarter.
The usual mitigation is wide tolerances. It does not work here, and the reason is worth stating precisely: a tolerance broad enough to survive a loaded CI machine is broader than the effect you are measuring. If the assertion permits ±800 ms so it stops flaking, it can no longer distinguish 1574 ms from 450 ms in the wrong direction. You have kept the test and lost the measurement.
So the clock is a parameter, and the fake one does not sleep:
class FakeClock:
def now(self) -> float:
return self._now
def sleep(self, millis: float) -> None:
self.waits.append(millis)
self._now += millisA run that “takes” a second and a half executes in microseconds. Asserting that twenty consecutive runs produce byte-identical numbers costs nothing, so it is in the suite. Delays are faked to the right order of magnitude — recognition of a second of audio in the hundreds of milliseconds, first token in the hundreds, synthesis of a phrase in the hundreds — and the absolute values are not claimed to match anyone’s hardware.
This is evidence about pipeline architecture, not about model speed. Those are different claims, and only one of them is portable.
Three ways a measurement lies while looking fine
Writing the mutation exercises for this made me notice that the dangerous defects were not in the audio code.
Marking first-audio twice. The stopwatch stored it as 0.0 and guarded re-marking with a truthiness test. Marking at time zero left it unmarked, so the second chunk silently overwrote the first, and time-to-first-audio became the time to the second sound. The number stays plausible. The graph looks normal. It is measuring something else.
Overlapping steps. If each step is timed from the start of the run rather than the end of the previous one, the breakdown sums to more than the total. That one is loud. The quiet cousin is the one above: a breakdown that reconciles and attributes the time to the wrong participant. A breakdown that does not add up is worse than no breakdown, because people trust it. One that adds up wrongly is worse than that, because they cannot even suspect it.
Percentiles off by one rank. p95 computed by interpolation is a latency no run actually experienced. Showing a user an invented delay instead of the worst real one is a strange kind of honesty. Nearest-rank costs one line and returns a run that someone genuinely sat through.
But nearest rank is ceil(0.95·n), not round(0.95·n), and I had written round. They agree at n=100 — which is, of course, the sample size my test used. They disagree on 95 of the first 200 sample sizes. At thirty runs:
runs=30 mean=480 p95=400 worst=1600 tail_ratio 0.83
runs strictly worse than p95: 6.7%A tail ratio below one reads as there is no tail. The module that exists to expose the tail was hiding it, and the check stood on the one sample size where the bug is invisible.
The general form is worth more than the fix: a test of boundary arithmetic has to run a list of input sizes, not one. A single size is one lucky point, and it is almost always lucky, because you picked it while looking at the code.
All three leave the code working. All three produce numbers you would put in a slide.
What I would ask of any latency number
Which input property does this scale with? If the answer is “none”, check again — there is almost always one, and it is usually the one that varies most in production.
What are the parts, and do they add up? If the parts sum to more or less than the whole, at least one is wrong, and you do not yet know which.
Is this a real observation or an interpolated one? For percentiles specifically: nearest-rank or interpolated? The difference is whether anyone felt it.
Would this number move if the machine were busy? If yes, it is not a property of your system, and any comparison built on it is comparing load.
What did you measure before? A number after without a number before is faith. I keep both, and the pipeline exists twice for exactly that reason — the batch version is not legacy, it is the control.
The code — both pipelines, the fake clock, sixteen mutation exercises and the length-sweep that produced the table above — is in stage 7 of the course repository, pinned to the tag. It runs with no microphone, no models and no network.