vLLM: Librosa mono downmixing inconsistency with ITU-R BS.775-4
Overview
This vulnerability stems from an inconsistency in how audio is processed by large language models (LLMs) when using the vLLM inference engine. Specifically, versions of vLLM from 0.5.5 up to (but not including) 0.18.0, utilize Librosa's `to_mono` function for mono downmixing. By default, Librosa employs `numpy.mean` for this operation. However, the international standard ITU-R BS.775-4 mandates a specific weighted downmixing algorithm. The use of `numpy.mean` deviates from this standard, leading to a discrepancy between the audio perceived by human listeners and the audio data processed by AI models. This can affect various AI applications that rely on accurate audio input, including transcription, audio analysis, and speech recognition, potentially leading to misinterpretations or degraded performance. The issue arises because the AI model receives a perceptually different representation of the audio compared to what a human would hear, impacting downstream tasks that depend on audio fidelity.
Affected Systems
Testing Guide
1. Obtain an audio file known to have distinct stereo characteristics. 2. Process the audio using vLLM version prior to 0.18.0 with a model that accepts audio input. 3. Record the output or analyze the LLM's response based on the audio. 4. Manually perform mono downmixing on the same audio file using the ITU-R BS.775-4 weighted algorithm (or a tool that implements it). 5. Compare the LLM's output or inferred meaning from step 3 with the expected output from a human listening to the manually downmixed audio from step 4. Significant deviations may indicate the vulnerability's impact.
Mitigation Steps
- Ensure vLLM is updated to version 0.18.0 or later. - For older versions, consider implementing custom audio preprocessing logic that adheres to the ITU-R BS.775-4 standard before feeding audio to vLLM. - If possible, test the impact of this downmixing difference on your specific LLM application. If the impact is negligible, no immediate action may be required. - Review any audio-dependent AI workflows for potential degradation caused by this discrepancy.
Patch Details
0.18.0