Benchmark of 23 ASR Models for Russian IT Dictation: My March Recommendation Is Outdated
OpenAI
A developer re-evaluated voice-to-text models for Russian IT dictation, testing 23 ASR models in over 60 configurations with 100,000 runs. They found that Breeze ASR, an unexpected find, matches or exceeds Whisper Large v3, leading to an updated recommendation. The benchmark introduces a composite metric Q, combining WER, punctuation accuracy, and a new English Preservation Index (EPI) to reward models that keep English terms in Latin script.
In March, the author recommended Whisper Large v3 for Russian IT dictation, but later discovered Breeze ASR, a model optimized for Taiwanese Mandarin with code-switching support, which performed at least as well. To verify, they built a rigorous benchmark: 23 models in over 60 configurations, more than 100,000 runs on a Russian-English IT corpus, using 120+ hours of inference on an RTX 5070 Ti. The corpus includes two speakers (the author and his wife) reading long-form texts with varying densities of English IT terms. The composite metric Q (0.65 WER + 0.10 punct + 0.25 EPI) balances word accuracy, punctuation quality, and preservation of English terms in Latin script. WER is normalized to not penalize transliterations, while EPI rewards canonical English forms and partially credits imperfect Latin retention. The results show that Breeze ASR outperforms Whisper Large v3 on this benchmark, making the March recommendation outdated. The author also tested multiple punctuation prompts and found that a custom prompt significantly improves punctuation. The interactive benchmark page allows adjusting weights to suit different dictation scenarios.
- Abbreviations
- ASR = Automatic Speech Recognition — автоматическое распознавание речи
- STT = Speech-to-Text — преобразование речи в текст
- WER = Word Error Rate — доля ошибок в словах
- EPI = English Preservation Index — индекс сохранения английского языка
- LLM = Large Language Model — большая языковая модель
Source: Habr — хаб ML —
original