We shipped a feature that destroyed our own transcripts

5 August 2026 · 4 min read · Muteeb Mehraj

A feature that looked correct, reviewed as correct, and shipped as correct turned a working speech model into one that returned nothing. The lesson was not about Whisper. It was about the difference between reasoning and measuring.

The feature

Bellow has a Dictionary: a list of proper nouns and technical terms you expect to say. Names, product names, jargon your speech model has never seen. The idea is unremarkable and every dictation tool has some version of it. You feed those words to the decoder as a prompt, and it biases toward transcribing them correctly instead of guessing at something phonetically close.

Whisper supports this directly through promptTokens. You tokenize your vocabulary, pass it in, and the decoder starts from that context.

There is a known hazard: Whisper's prompt window is roughly 224 tokens. Overflow it and the prompt eats into the space the model needs for the audio itself. So we capped the vocabulary hard, twice over: 240 characters, then 96 tokens. Well inside the limit, with room to spare.

That guard was reasoned about carefully. It was reviewed. It was correct in the sense that the arithmetic checked out. It shipped.

What users saw

Transcripts started coming back empty. Not garbled, not partially wrong. Empty. Others came back truncated to a fraction of what had been said, which is worse than empty because you might not notice.

The reports had a pattern that took embarrassingly long to see: it started after people added their first dictionary word. Before that, everything worked.

The measurement

The temptation at this point is to reason harder. Tighten the cap. Try 48 tokens. Add a fallback. Ship again and see.

We built a harness instead. It takes a 124-second clip with known content, decodes it under a matrix of configurations, and reports what fraction of the known text came back. Not "does it feel better." A number.

The result:

• Without promptTokens: 99% recovery.
• With promptTokens: 0% recovery.

Zero. Not degraded. Not lossy. Nothing came back at all.

Then the part that ended the debate. We tried two tokens. One word. Far, far under any window limit, under our cap, under any cap anyone would think to write.

Still zero.

Why it broke

Once we knew the answer was "any prompt at all", we went looking in the library rather than in our own code. WhisperKit's TextDecoder carries a comment acknowledging that the prompt-plus-prefill path is currently broken.

It was written down. In the dependency. The whole time.

Nobody had read it, because nobody had a reason to. The code looked right, the arithmetic looked right, and there was no failing test pointing anywhere. Reading the decoder internals only became obviously necessary once a measurement said the feature could not possibly be working.

What we did

Decoder biasing is off. Not capped further, not made conditional. The call site now reads:

// VOCABULARY BIASING IS DISABLED - measured, not guessed
_ = vocabularyPrompt

The parameter is still accepted and deliberately discarded, with the measurement and the reasoning recorded immediately above it, so nobody re-enables it on a hunch.

The Dictionary tab still ships. It is honest about doing nothing for accuracy right now. We would rather show you a feature that is visibly parked than quietly ship one that damages your transcripts.

The harness is now a gate. Any configuration we intend to turn on has to clear 95% recovery before it goes near a release. Vocabulary biasing scores 0%. Voice-activity chunking scores 60%. Both stay off until those numbers move, and the numbers decide, not the argument.

The part that generalises

The failure was not writing a bug. Everyone writes bugs. The failure was that every signal available to us said the feature was fine:

• The code was clean and the intent was clear.
• The known risk was identified and explicitly guarded.
• The guard was correct on its own terms.
• It reviewed well.

None of that is evidence. All of it is reasoning about what the code should do. The only thing that would have caught this before release was running audio with known content through the pipeline and counting the words that came back, which took an afternoon to build and would have failed on the first run.

Speech recognition is unusually good at hiding this class of mistake. There is no exception, no crash, no red line in a log. The pipeline runs, returns a string, and the string is empty or short. If you are not comparing against known content, empty looks exactly like silence, and truncated looks exactly like someone who stopped talking.

So the rule we now hold ourselves to is narrow and boring: any change to the transcription path adds its failing check first. Numbers, not vibes. It is the only reason we would notice if this happened again.