A new benchmark developed by ServiceNow AI Research has revealed a critical flaw in even the most advanced Automatic Speech Recognition (ASR) models: they struggle significantly with bilingual speakers. The research shows that frontier ASR systems can experience a word error rate (WER) up to 32% higher when processing 'code-switched' audio, where speakers mix two or more languages in a single conversation.
This performance gap represents a major blind spot for the booming AI-powered customer service industry, which relies on ASR to understand and assist a diverse global customer base. The findings were published on the Hugging Face blog, introducing a new testing suite to address the issue.
The Code-Switching Blind Spot
Code-switching is a natural and common speech pattern for billions of multilingual individuals worldwide, such as blending Spanish and English or Hindi and English within the same sentence. However, most leading ASR models, including those from major tech companies, are trained predominantly on monolingual datasets. This leads to a significant drop in accuracy when confronted with the fluid, mixed-language queries common in real-world customer support calls.
This failure to comprehend code-switched speech results in frustrated customers, inaccurate transcriptions, and ultimately, ineffective automated support. It creates a service disparity, where AI agents work well for monolingual speakers but fail a large segment of the global population.
Introducing CoVoSwitch: A New Standard
To address this, ServiceNow AI Research has released CoVoSwitch, the first large-scale, multi-domain benchmark specifically designed to evaluate ASR models on code-switched speech. Released on the Hugging Face platform, it provides a standardized method for developers to measure how well their models handle this complex but common linguistic behavior.
Key findings from testing leading models against the CoVoSwitch benchmark include:
- Significant Performance Drop: Some of the most capable ASR models available today see their performance degrade substantially when moving from monolingual to code-switched audio.
- Relative Error Increase: The 32% relative increase in word error rate highlights how far current systems are from reliably serving bilingual users.
- Diverse Language Pairs: The benchmark includes challenging and widely-spoken language pairs, such as Spanish-English (Spanglish) and Hindi-English (Hinglish), to reflect real global usage patterns.
Redefining 'State-of-the-Art' ASR
This research challenges the very definition of a 'state-of-the-art' ASR system. High performance on clean, single-language datasets is no longer sufficient. True effectiveness must be measured by a model's ability to handle the messy, diverse, and multilingual nature of human conversation. The CoVoSwitch benchmark is a crucial step toward building more inclusive and robust AI.