Speech-to-Text Model & Provider Leaderboard

Agreement. ElevenLabs and AssemblyAI are top-tier in every benchmark that runs its own audio, and Deepgram Nova-3 is weak in all of them. gpt-4o's clean-brittleness reproduces independently too: Artificial Analysis, on harder real-world audio, drops it below both - which is exactly the re-ranking we measure going clean → noise. Different team, different data, same collapse.

ElevenLabs is the instructive disagreement. We rank it a robust #1 on clean audio; Pipecat ranks it mid-pack. Four reasons, none of them a contradiction: they test the realtime checkpoint and we test the batch one; they test English only and we average five languages; they score with an LLM judge and we use a deterministic normalizer; their audio is short agent turns and ours is read and parliamentary speech. Two careful benchmarks, opposite verdicts, and the disagreement is entirely explained by what each chose to measure. That is this page's thesis in one example.

Qwen is the cautionary one. We score it mid-field; Artificial Analysis scores the same family near-worst - roughly a 5× gap. Loading Qwen3-ASR through generic transformers silently initialises a random audio tower: the whole audio stack reports as missing from the checkpoint and is newly initialised, and a model whose audio encoder is random weights cannot transcribe at all. We confirmed the broken load; we did not reproduce their harness, and their artifact may differ. The lesson stands either way - a 53-model public leaderboard can silently mis-load a model, and only a per-model sanity check catches it.

Reference and hypothesis pass through the same published normalizer before a single edit is counted, so no engine is penalised for a house style. For English that is Whisper's EnglishTextNormalizer - spelled numbers unified to digits, contractions and abbreviations expanded, US/UK spelling folded. For German, French, Italian and Spanish: lowercase → abbreviation table → spoken-variant folding → thousands, decimal and time handling → digits back to words → BasicTextNormalizer.

The residual limitation, stated rather than hidden: number expansion picks the canonical num2words form, and spoken variants fold back to it only where a rule exists - German elfhundert → eintausendeinhundert, for instance. Unlisted variants still count as errors. Sampling excludes digit-bearing references from the headline lanes, so this is a backstop rather than something the score leans on; the digit lane measures formatting deliberately and separately.

• Training contamination is unknowable for engines whose training sets are undisclosed - which cuts both ways. One case is known rather than suspected: canary and parakeet were trained on Granary, which builds on MOSEL, which incorporates VoxPopuli audio. Treat their parliament cells as potentially contaminated.
• Italian VoxPopuli is consensus-filtered. A 17-engine minimum-WER audit found 17 of 128 clips with misaligned references; those are excluded and the remaining 111 rank normally. Consensus filtering mildly favours this cell, which is why it is disclosed here rather than left implicit.
• VoxPopuli references are aligned official records, lightly edited rather than verbatim - so the reference can carry words the audio does not, and every engine's deletion rate is inflated in the same direction. Ranking within the parliament lane is unaffected. The gap between the clean and parliament lanes is therefore part audio difficulty and part reference convention, and the headline multiple above should be read that way.
• Cloud models change silently. Every cloud figure is a dated snapshot; the same endpoint may return a different model tomorrow with no version bump. Local weights are pinned to a commit and will not.
• No multiple-comparison correction. With 18 engines there are hundreds of pairwise comparisons, so an occasional pair will separate by chance. Headline claims are limited to comfortably separated pairs; everything else is exploratory.
• Additive noise only - no reverberation, no Lombard effect, no codec artefacts beyond the telephony lane. Real rooms are harsher than MUSAN mixed at a fixed SNR.
• The local cost model is marginal electricity, not total cost of ownership. It is computed per engine - measured realtime factor × 236 W at the wall (a 175 W laptop GPU plus 35% for CPU, RAM and supply losses) × $0.35/kWh - and it excludes the GPU, the engineer and the uptime you would have to provide yourself. The draw is the GPU's rated maximum rather than a meter reading, which over-states an engine that does not saturate the card. Cloud figures are vendor list prices, so the two sides of the price chart are not the same kind of number: one is a power bill, the other a service contract.