“ESB: A Benchmark For Multi-Domain End-To-End Speech Recognition”, Sanchit Gandhi, Patrick von Platen, Alexander M. Rush2022-10-24 (Whisper NN)⁠:

Speech recognition applications cover a range of different audio and text distributions, with different speaking styles, background noise, transcription punctuation and character casing. However, many speech recognition systems require dataset-specific tuning (audio filtering, punctuation removal and normalization of casing), therefore assuming a-priori knowledge of both the audio and text distributions. This tuning requirement can lead to systems failing to generalize to other datasets and domains.

To promote the development of multi-domain speech systems, we introduce the End-to-end Speech Benchmark (ESB) for evaluating the performance of a single automatic speech recognition (ASR) system across a broad set of speech datasets. Benchmarked systems must use the same data pre/post-processing algorithm across datasets—assuming the audio and text data distributions are a-priori unknown.

We compare a series of state-of-the-art (SoTA) end-to-end (E2E) systems on this benchmark, demonstrating how a single speech system can be applied and evaluated on a wide range of data distributions.

We find E2E systems to be effective across datasets: in a fair comparison, E2E systems achieve within 2.6% of SoTA systems tuned to a specific dataset. [One of the best: Whisper.]

Our analysis reveals that transcription artefacts, such as punctuation and casing, pose difficulties for ASR systems and should be included in evaluation.

We believe E2E benchmarking over a range of datasets promotes the research of multi-domain speech recognition systems.

ESB is available at https://huggingface.co/esb.