FFASR Leaderboard Launched: Benchmark for Evaluating ASR in Real Acoustic Conditions

The FFASR Leaderboard, created by Hugging Face and Treble Technologies, is the first open community-driven benchmark for evaluating ASR models under realistic far-field acoustic conditions. It addresses the persistent gap between benchmark performance and real-world deployment, where models that score well on standard near-field benchmarks often degrade substantially under reverberation, background noise, and microphone distance. The leaderboard evaluates models across nine conditions, with four determining the primary ranking score. Acoustic data is generated using Treble's hybrid simulation engine, combining wave-based and geometrical-acoustics modeling to capture phenomena like diffraction and interference. Fourteen fully furnished rooms are included, ranging from 20 to 470 m³. Alongside WER, the leaderboard reports RTFx (audio seconds per inference second) for every submission, evaluated on an NVIDIA L4 GPU. Initial results show far-field WER at low SNR is several times higher than near-field WER. The leaderboard supports Whisper variants, IBM Granite Speech, Cohere Transcribe, Wav2Vec2, HuBERT, SpeechBrain, and most other ASR architectures via Hugging Face model ID submission. A custom evaluator option allows more complex inference stacks. Future plans include multi-talker scenarios, microphone array evaluation, and echo cancellation.