Local speech to text for agent chat, measured

A microphone button for agent chat

A person holds the button, says what they want, and lets go. The text has to appear in the chat box quickly and be right, and the model must cost almost nothing while nobody is talking, because the same graphics card is usually holding a local language model.

So the lab asks four things of each model: how many words it gets wrong, how much graphics memory it takes, how long the person waits after letting go, and what it costs to load the model only when the button is pressed and keep it for some minutes afterwards.

Five models, one runtime

The five were picked from what people recommend and what was released this year: the Open ASR Leaderboard, the HuggingFace list of trending speech recognition models, and the last thirty days of r/LocalLLaMA, r/speechtech, Hacker News and GitHub. Each is under about two billion parameters, so it fits beside a language model on one card.

All five run through transcribe.cpp 0.3.1, an MIT licensed C++ library built on ggml, the same base as llama.cpp. It runs every one of these models from a single GGUF file on CUDA, Vulkan, Metal and the CPU, and it has Rust bindings, which is what Unluminous is written in. Measuring through it measures what would ship. The GGUF files are the handy-computer conversions, each checked against the publisher’s SHA-256.

Considered and left out

• Canary-Qwen 2.5B and Granite Speech 4.1 2B: larger than Cohere Transcribe and less accurate on the leaderboard.
• Voxtral Mini 4B Realtime and Audio8 ASR Infinite: about 8 GB of weights, built for streaming while the person speaks.
• Nemotron 3.5 ASR Streaming 0.6B: a streaming model, 7.96% on the leaderboard’s English sets.
• Moonshine: very small, but 9.99% on the leaderboard.
• Parakeet Redux, Parakeet Ultra and Phonon-2: three new versions of Parakeet released in September 2026, each run only by its maker’s own runtime (Photon or Fermion), which Unluminous cannot embed.

Granite Speech 5.0 470M TurboCTC makes the fewest errors

Word error rate is the share of words a model gets wrong: substituted, dropped or added. Every model heard the same 1,198 clips from six test sets, from clean audiobooks to meetings and earnings calls. Q8_0, the eight bit version of each model, is the one compared here.

Granite Speech 5.0 470M TurboCTC, a CTC model released by IBM in August 2026, is the most accurate at 5.86%. Cohere Transcribe is next at 6.25%, then Parakeet TDT 0.6B v3 at 6.67%, Qwen3-ASR 0.6B at 7.71% and Whisper large-v3-turbo at 9.34%.

On the leaderboard’s own eight English sets the top four are closer together and in another order: Cohere Transcribe 4.67%, Parakeet TDT 0.6B v3 4.86%, Granite Speech 5.0 470M TurboCTC 5.03% and Qwen3-ASR 0.6B 5.05%. Our six sets include Common Voice, people reading into their own microphones, and the meetings and earnings calls as they were recorded rather than cleaned up, which is closer to someone talking at a desk. Granite does best on those.

Whisper’s number is high because of one failure: for 20 of the clips it wrote only “Thank you.” instead of the speech. This is Whisper’s known habit of producing a phrase common in its training subtitles, and it happened with and without timestamps. In a chat box it would send “Thank you.” to the agent.

Granite gets its accuracy without punctuation or capital letters: every transcript is lower case words. The other four write sentences. Ready to paste in the table below is the share of transcripts that start with a capital and contain punctuation.

Small models, and eight bits is enough

Graphics memory here is everything the model’s process held on the card: the weights, the CUDA context, and the working buffers while it transcribed the ten longest clips, 22 to 30 seconds each. Granite Speech 5.0 470M TurboCTC peaked at 1.11 GB, Parakeet TDT 0.6B v3 at 1.38 GB, Whisper large-v3-turbo at 1.52 GB, Qwen3-ASR 0.6B at 2.31 GB and Cohere Transcribe at 3.07 GB.

Q8_0 is as accurate as the sixteen bit F16 file for every model. Q4_K_M is smaller and costs at most 0.61 points of word error rate, which makes it the choice for a card that is nearly full.

Fast enough that speed does not decide it

With the model loaded, a six second request takes 35 ms with Granite Speech 5.0 470M TurboCTC and 165 ms with the slowest, Qwen3-ASR 0.6B. Every model is many times faster than speech.

One setting matters. In transcribe.cpp 0.3.1 on CUDA, flash attention for Parakeet and Cohere runs on the CPU, one trip per encoder layer, so turning it off with TRANSCRIBE_NO_FLASH=1 makes them faster: 2.6 times for Parakeet TDT 0.6B v3 and 2.3 times for Cohere Transcribe in the runs below, and 10 times for Cohere in a first test on a quieter machine. Whisper, Qwen3-ASR and Granite are faster with it on. These runs shared the machine with other work, so the CPU load column matters when comparing them. The transcripts do not change: on 300 Parakeet clips, 296 were identical and the other 4 differed by a comma or one word.

Loading on demand costs the person nothing

Pressing the button starts the load. Starting the GPU backend takes about 309 ms, and loading the model from the operating system’s cache takes between 537 ms (Granite Speech 5.0 470M TurboCTC) and 1.86 s (Cohere Transcribe). The person is speaking for the whole time, so by the time they let go the model is ready, and the only wait is the first transcript.

A cold start, with the file not in the cache, adds the time to read the file from the drive: at most 0.50 seconds for Cohere’s 2299 MB file on this NVMe drive.

Freeing the model gives back its weights but not the CUDA context: 496 MB stays on the card until the process ends. To give the card back completely, speech to text should run in a small helper process that exits when it is no longer needed.

The runtime also opens a CUDA context on every graphics card in the machine, used or not. This machine has two, and each model’s process held 496 MB on the second card for as long as it ran. A helper process started with CUDA_VISIBLE_DEVICES set to the one card it should use avoids that.

Keep it loaded for a few minutes, then end the process

To see what a policy costs on real use, we replayed 471 messages Jason sent to coding agents over 60 days, one press per message: his comments on Tasks board tickets and the prompts he typed into agent terminals. Only the time of each message was used. A second, made up timeline stands in for someone who talks to agents all day.

Because the load hides under speech, keeping the model loaded hardly changes the wait. With Granite Speech 5.0 470M TurboCTC in a helper process that runs one second of silence through the model while the person talks, unloading at once and loading on every press averaged 53 ms from release to text, and keeping it loaded all day averaged 35 ms. Without the second of silence, unloading at once averaged 139 ms, because the first transcript in a new process is slower than the ones after it.

What the policy does change is how long the card is held. Kept for 5 minutes after each press, the model was loaded 0.55 hours a day on Jason’s messages and 1.43 hours for the heavy user, against all day when it is never unloaded.

Hear it, then read what each model wrote

Three clips the models disagreed on, from three test sets. Press play, then compare what each model wrote. Words the scorer counts as wrong are underlined. A word a model left out also counts as an error, but there is nothing to underline, so the count can be higher than the underlines.

Five bits is as good as eight, and three bits nearly so

Granite is the smallest of the five already, so the question is how far it shrinks before it gets worse. The handy-computer conversions stop at Q4_K_M. transcribe.cpp’s own quantise tool also makes Q5_1, Q5_0, Q4_1 and Q4_0, which we made from the F16 file. Below four bits the runtime refuses to load the file at all, so for Q3_K_M, Q3_K_S and Q2_K we added two presets to the tool and the two weight types to the loader’s list in a local build of the same release.

Every size heard the same 1,198 clips on the same card. Making Q4_K_M ourselves from F16 gave a file with different bytes from the published one and 6.13% against 6.09%, so the files made here can be compared with the published ones.

• Down to five bits nothing is lost: Q8_0 5.86%, Q6_K 5.89%, Q5_K_M 5.88%, Q5_0 5.86%. Q5_K_M is published, so it needs nothing built: 320 MB instead of 482 MB, and 978 MB at peak instead of 1140 MB.
• Q4_K_M and Q4_0 cost about a quarter of a point: 6.09% and 6.14%. Three bits costs about half a point: Q3_K_M 6.25%, Q3_K_S 6.32%, still level with Cohere at eight bits. Two bits does not work: Q2_K gets 55.60% of words wrong.
• The card’s memory barely follows the file. From F16 to Q3_K_M the file shrinks from 904 MB to 201 MB, but the peak only falls from 1558 MB to 862 MB, because about 500 MB of it is the CUDA context, which is the same for every size.
• Speed does not decide the size. With the model loaded, every size transcribes a six second request in 33 to 42 ms. Loading the file onto the card takes 322 to 786 ms, roughly following the file size. Starting the GPU backend comes on top, 492 ms with the released package, which also loads a Vulkan backend, and 122 ms with a build that has CUDA only.

A 200 MB model puts the punctuation back

Granite writes every word in lower case with no punctuation, because it was trained that way. A chat box should show sentences. We tried four small models made to restore punctuation, and the local language model this machine already keeps loaded, each run on Granite’s own transcripts.

pcs_en predicts marks, capitals and sentence breaks in one pass. The two Horizon-Labs models, released on 3 October 2026, predict marks only, so a rule adds the capitals: the first word of each sentence and “I”. bert-restore-punctuation predicts marks and capitals. The language model is asked to add punctuation and capitals without changing a word.

None of them can be trusted with the words. pcs_en writes “<unk>” next to every number Granite wrote as digits, and the language model reworded 0.85% of words. So every option here goes through a safe transfer: its output is lined up with Granite’s words, and only the marks and capitals are copied back. The words a person sees are always the words Granite heard.

Scoring uses the three test sets whose references have punctuation and capitals: Common Voice, AMI and Earnings-22, 598 clips. For each word an output shares with the reference, the mark after it is compared. The four models that punctuate by themselves are scored the same way.

The same clip, punctuated each way

What was said

And also, um, is this informed, um, this plan informed by anything you see in the market, actions by competitors or competitive, um, levels overall in the market?

Granite alone

and also is this informed this plan informed by anything you see in the market actions by competitors or competitive levels overall in the market

Granite + pcs_en (1-800-BAD-CODE)

And also, is this informed this plan informed by anything you see in the market, actions by competitors or competitive levels overall in the market?

Granite + Local language model (Qwen3.8 27B)

And also, is this informed? This plan informed by anything you see in the market actions by competitors or competitive levels overall in the market?

Cohere Transcribe, its own

And also, is this informed, this plan informed by anything you see in the market, actions by competitors or competitive levels overall in the market?

What was said

Between roast beef and potatoes how can we lose?

Granite alone

between roast beef and potatoes how can we lose

Granite + pcs_en (1-800-BAD-CODE)

Between roast beef and potatoes, how can we lose?

Granite + Local language model (Qwen3.8 27B)

Between roast beef and potatoes, how can we lose?

Cohere Transcribe, its own

Between roast beef and potatoes, how can we lose?

• Granite with pcs_en scores 0.748, more than Parakeet’s own punctuation (0.720) and Qwen3-ASR’s (0.735), and close to Whisper (0.757). It takes 35 ms per request on the CPU, needs no graphics memory, and is one 200 MB ONNX file.
• The local language model scores a little higher, 0.764, at 166 ms per request, and only when a 27B model is already loaded. Cohere’s own punctuation is the best of all at 0.797, but Cohere needs three times Granite’s graphics memory.
• Commas are where every option is weakest. Spoken English has no reliable comma, and people disagree about them in the references too.
• The Horizon-Labs models score 0.728 (base) and 0.722 (small); they cover 88 languages, which matters for Parakeet’s other languages more than for Granite. bert-restore-punctuation, trained on restaurant reviews, scores 0.471.

Whistle: 16.9 MB, and almost three times the errors

Whistle came out on 2 October 2026 from Cactus Compute and was posted to r/LocalLLaMA as speech to text in a 16.9 MB file. It has 55 million parameters, 36 million of them active, stored at 2 to 4 bits, and covers 7 languages under Apache 2.0. It does not run on transcribe.cpp: it is a .cact file for Cactus’s own needle engine, which runs only on the CPU and takes at most 30 seconds of audio per call. The author says a version without that limit is coming.

We ran it the same way as the other five: the same 1,198 clips, the same scorer, English forced, the engine’s defaults, through cactus-needle 3.1.1, engine 3.1.0. The weights and the engine were checked against the publisher’s SHA-256.

The example clips, Granite and Whistle

• Whistle’s published numbers hold up. On LibriSpeech clean it got 3.74% against a published 4.31%, on LibriSpeech other 10.13% against 10.49%, and on Earnings-22 18.40% against 19.01%.
• It is still far behind Granite. Its mean over the six sets is 15.62%, 2.7 times Granite Q5_K_M’s 5.88% and worse than every model in the lab. The gap is widest on Common Voice (25.48%), where people with many accents read into their own microphones, and on AMI meetings (26.87%), which is the closest test set to someone talking to an agent.
• Part of the AMI figure is Whistle tidying up: it leaves out repeated words such as “that’s that’s”, which the AMI references keep and the scorer counts as deletions. Words left out are 8.7% of AMI’s words for Whistle against 3.0% for Granite. That does not explain the gap. Words Whistle replaced with a different word are 13.0% on AMI and 19.8% on Common Voice, against 4.5% and 5.3% for Granite.
• When Whistle mishears, it writes a fluent, punctuated sentence that is wrong, such as “I’m gonna make a little shadow and shadow onto Foster.” for “When the girl improves, Charlotte takes her home to Boston.” Granite got that clip wrong too, as “venical records car takes her home to boston”, which is plainly broken. In a chat box a wrong sentence that reads well is more likely to be sent without being checked.
• Its own punctuation is good: 0.752 F1, level with Granite plus pcs_en (0.748) and behind only Cohere (0.797).
• It is small and starts at once: the process holds 77 MB with the model loaded, about 35 MB of it Python, and is ready 71 ms after the process starts. Granite Q8_0 on the CPU holds 594 MB and is ready in 609 ms.
• It is not faster. A six second request takes 611 ms with Whistle loaded, against 220 ms for Granite on the same CPU and 35 ms for Granite on the graphics card. Whistle’s decoder writes one token at a time with five beams; Granite’s CTC head writes the whole transcript in one pass.

Ship Granite Speech 5.0 470M TurboCTC at five bits, with pcs_en for punctuation

• Use Granite Speech 5.0 470M TurboCTC for English, at Q5_K_M. It made the fewest errors of the five (5.88% at Q5_K_M, the same as at Q8_0), needs the least graphics memory (978 MB at peak) and is the fastest. Its file is 320 MB.
• Add punctuation with pcs_en, through the safe transfer that copies only marks and capitals back onto Granite’s words: 0.748 punctuation F1, 35 ms on the CPU, no graphics memory.
• Offer Parakeet TDT 0.6B v3 at Q8_0 for the other 24 European languages: 6.67% of words wrong, 1.38 GB at peak, and it punctuates by itself.
• On a card that is nearly full, Q3_K_M (201 MB) still gets 6.25%, but it needs a patched transcribe.cpp; Q4_K_M (6.09%) is the smallest file that runs on the release.
• Embed transcribe.cpp through its Rust bindings. Set TRANSCRIBE_NO_FLASH=1 when running Parakeet or Cohere on CUDA.
• Load the model when the button is pressed, in a helper process started with CUDA_VISIBLE_DEVICES set to one card. Run one second of silence through it while the person is still talking. Keep the process for 5 minutes after the last press, then end it so the card gets back everything, including the CUDA context.
• Do not ship Whisper large-v3-turbo for chat: it replaced 20 of the clips with “Thank you.”.
• Do not ship Whistle for agent chat. It needs the least memory, but it gets 15.62% of words wrong against Granite’s 5.88%, and Granite on the CPU alone is faster per request. Whistle is for devices with tens of megabytes to spare, not for a desktop.

How it was measured

• Machine: NVIDIA GeForce RTX 5090 (driver 580.97), Intel(R) Core(TM) Ultra 9 285, 127 GB of memory, Windows 11 10.0.26100, transcribe.cpp 0.3.1. Only one graphics card was used.
• Every run was a fresh process. Graphics memory comes from Windows’ per process counter (GPU Process Memory, Dedicated Usage), sampled every 50 ms, which counts only the model’s own process. A first attempt read the whole card from NVML instead, and other programs allocating on the same card made those readings useless; they are kept in the raw results but not used.
• The machine was shared with other work while these numbers were taken. Each run recorded how much of the CPU other processes used and which other tickets were in progress. Load and latency figures are medians of five fresh processes; the range is in the data file.
• Word error rate: the reference and the transcript both go through the Whisper English normaliser used by the Open ASR Leaderboard, then jiwer counts substitutions, deletions and insertions. The mean is the average of the six test sets. Our six sets differ from the leaderboard’s eight, so our means are not directly comparable to the published ones; LibriSpeech clean, which both use, is compared in the table below.
• Two Common Voice clips shared an id with another clip and were dropped, leaving 198 in that set.
• Cold start: the operating system’s file cache could not be emptied without administrator rights, so the cost of reading a model from the drive was measured as an unbuffered read of the same file.
• Smaller Granite files: Q5_1, Q5_0, Q4_1 and Q4_0 were made with transcribe.cpp 0.3.1’s transcribe-quantize from the published F16 file. Q3_K_M, Q3_K_S and Q2_K needed presets added to that tool and Q3_K and Q2_K added to the loader’s allowed weight types (TRANSCRIBE_QUANT_LINEAR_TYPES) in a local CUDA build for this card only.
• Whistle ran through cactus-needle 3.1.1 and the needle engine 3.1.0 (libneedle3.dll) in its own Python environment, on the CPU pinned to the eight performance cores, with the package’s anonymous usage reporting turned off. Its accuracy run took 550 seconds for 129 minutes of audio, 14.04 times real time, on a CPU other work was using. Its load figures are medians of 7 fresh processes. Granite’s CPU figures beside it are the five CPU load runs from the first round of this lab; Granite’s word error rate on the CPU was not measured separately.
• Punctuation ran on the CPU in its own Python environment (onnxruntime, transformers with CPU torch); the language model ran where it already lives, NInfer on the second graphics card, with thinking off and temperature 0.
• The code is in ai-service/tools/stt-lab: download and hash check, test audio, the benchmarks, the scorer, the keep loaded replay, the punctuation runner and scorer, the Whistle runner, and the script that writes this page’s data.