Streaming Speech-to-Text on Your CPU: Shipping NVIDIA Nemotron in a Desktop App

Streaming speech-to-text transcribes audio while you're still talking, instead of waiting for the recording to end. As of version 1.7.6, OpenWhispr does this entirely on-device: NVIDIA's Nemotron streaming models (600M parameters, INT8-quantized) run on your CPU through a local sherpa-onnx WebSocket server, the dictation preview updates live as partial results arrive, and the streamed text is committed as your transcript the moment you stop. No GPU, no Python environment, no audio leaving your machine.

This is the engineering write-up: why our old live preview felt laggy, what cache-aware streaming actually is in plain terms, how the pipeline fits together inside an Electron app, the version-pinning bug that mis-decoded everything, and how we went from re-decoding every recording to committing the stream itself. If you want the model-versus-model comparison instead, that's in Parakeet vs Whisper vs Nemotron.

Last updated July 18, 2026. Implementation details reference the streaming transcription feature shipped in OpenWhispr 1.7.6; everything described here is inspectable in the open-source repository.

• Nemotron's cache-aware FastConformer processes only new audio while reusing cached encoder context, with runtime-configurable chunk sizes from 80ms to 1.12s. Nemotron Streaming EN model card · NVIDIA on cache-aware streaming
• The English model averages 6.93% WER at 1.12s chunks while streaming; the multilingual 3.5 model covers 40 language-locales with automatic detection. Nemotron 3.5 ASR model card
• All inference runs through sherpa-onnx, a native ONNX runtime with no Python dependency, using its online (streaming) WebSocket server. We pin and bundle version 1.13.4. sherpa-onnx on GitHub
• The models are 632MB (English) and 650MB (multilingual) on disk as INT8 ONNX, downloaded once and cached locally. OpenWhispr model registry (open source) · OpenWhispr models page

OpenWhispr has always had a live transcription preview — a small window that shows what the model heard while you dictate. Before 1.7.6 it worked the only way you can make a batch model feel live: buffer about 1.5 seconds of microphone audio, run a full transcription on that chunk, append the text, repeat.

That approach has three problems, and every one of them is structural. First, latency: your words can't appear faster than the buffer fills, so the preview always trails you by a second and a half plus inference time. Second, boundaries: a word that straddles two chunks gets cut in half, and neither chunk can fix it, because chunks are transcribed independently. Third, context: chunk five doesn't know what you said in chunk four — so “their” versus “there” is a coin flip the model re-tosses every 1.5 seconds.

You can shrink the chunks to feel faster, but then accuracy collapses — less context per chunk, more boundaries. You can overlap chunks to fix boundaries, but then you're transcribing the same audio twice. What you actually want is a model that was trainedto transcribe a stream. That's what Nemotron is.

A cache-aware streaming model keeps its internal state between chunks. When the next slice of audio arrives, the encoder doesn't re-read everything you said — it processes only the new frames, attending to a cache of the context it already computed. It's the same trick that makes chatbots fast (KV caching), applied to a speech encoder.

Two consequences fall out of this design. Compute stops scaling with recording length — each chunk costs the same whether you're ten seconds or ten minutes in, which is what makes CPU-only streaming viable. And context carries forward — the model resolves “their/there” using everything said so far, and revises earlier words in its running hypothesis when later audio disambiguates them. That revision is why streaming text visibly “settles” a beat after you speak: it's not a bug, it's the model changing its mind with better evidence.

The latency budget is a knob, not a constant: Nemotron exposes chunk sizes from 80ms (fastest, least context per step) up to 1.12s (highest accuracy — 6.93% average WER on the leaderboard datasets, within about a point of the best batch models). The training recipe is NVIDIA's cache-aware FastConformer with a transducer decoder; their engineering post covers the theory in depth.

The pipeline is deliberately boring. An AudioWorklet captures microphone PCM at 16kHz. Frames are converted to the float32 wire format and pushed over a WebSocket to a sherpa-onnx online server — a native binary OpenWhispr spawns locally and talks to over 127.0.0.1, one persistent connection per recording. The server runs the Nemotron model with INT8 weights and answers with JSON partial results, which the preview window renders by replacing its text in place rather than appending.

Why a separate server process instead of in-process inference? Isolation. Native inference runtimes can crash — a bad allocation in an ONNX kernel shouldn't take the whole desktop app down with it. A sidecar process means a crashed runtime is a reconnect, not a lost dictation. It also keeps native code out of Electron's main process entirely: the app talks protocol, the sidecar does math.

The same pattern runs our local speaker diarization — one sherpa-onnx toolchain, multiple jobs, all on-device.

The runtime version mattered more than we expected.Nemotron models require sherpa-onnx 1.13.4 or newer for accurate decoding — on the older runtime we'd bundled for Parakeet, they didn't fail loudly, they transcribed badly. The fix was mundane (upgrade and re-pin the bundled binaries on every platform), but the lesson generalizes: with model runtimes, “it runs” and “it's correct” are separate tests, and only the second one counts.

Streams fail, so the old path stayed.A port can be taken, a model file can be missing, a server can die mid-recording. If the persistent stream can't start, the preview falls back to the buffered-chunk path automatically — slower, but never blank. Offline models (Parakeet, Whisper) keep the chunked preview by design; each model in our registry declares its runtime as online or offline, and the app picks the right preview path per model.

Partials needed different rendering. The chunked path appends text; a streaming hypothesis revises itself, so the UI replaces the whole preview per partial. Append a revising hypothesis and you get stuttering duplicates — the one-line version of a real UX bug we fixed early.

Our first version treated streaming as a preview, not the product: when you stopped dictating, the app re-decoded the complete recording from scratch and pasted that. Safe — the final pass sees full context — but it meant every dictation paid for two decodes, and the text you'd already watched settle on screen made you wait for a second opinion that almost always agreed with it.

So in the same release we made the stream the source of truth. The dictation now runs over the persistent stream whether or not the preview window is open, and when you stop, the app flushes the model's tail and commits the streamed text directly — no second decode. End-of-dictation latency drops to one tail flush, and per-dictation CPU is roughly halved.

The safety net didn't go away — it became the fallback. The stop flush is truncation-aware: it extends its deadline while results are still arriving, and if anything is off — a dropped connection, a truncated result — the app discards the stream and runs the classic record-then-transcribe pass over the full recording instead. You only ever get the fast path when it's also the correct one.

• Models: Nemotron Speech Streaming EN 0.6B (632MB) and Nemotron 3.5 ASR Streaming 0.6B (650MB, 15 transcription-ready languages, auto-detect) — INT8 ONNX, downloaded once, cached locally.
• Runtime: sherpa-onnx 1.13.4, a self-contained native binary. No Python, no PyTorch, no CUDA required.
• Hardware: real-time streaming on a modern laptop CPU — Apple Silicon and recent x86 both comfortably keep up with speech.
• Feel: partial text lands well under a second behind your voice; at stop, one tail flush commits the transcript — no second decode.

Nothing about this path touches the internet. The app talks to the server over 127.0.0.1 (on Windows, a scoped firewall rule additionally closes the port to the network), the model files live in your local cache, and the audio never leaves the machine — the same privacy posture as the rest of our local transcription stack.

• Streaming trades a little accuracy for immediacy.~6.9% streaming versus 5.91% for NVIDIA's best English batch model on the same benchmarks — and since the committed transcript now comes from the stream, that's the trade you're accepting. If you only care about the end result, an offline Parakeet model is still the accuracy pick.
• Language coverage is narrower than Whisper.15 transcription-ready languages in the multilingual model versus Whisper's 99. Streaming in Mandarin or Thai isn't here yet.
• Partial text moves. The hypothesis revises itself as context arrives. We think watching it settle is a feature; if you find it distracting, the preview can be turned off and dictation works exactly as before.
• It's another 650MB on disk if you already have an offline model — the price of a second engine tuned for a different job.

All of these are a dropdown in Settings — the full catalog with sizes and trade-offs is on the models page. OpenWhispr is free, open source, and available for macOS, Windows, and Linux.