NVIDIA GPU Driver Kernel Mode Layer Allows for Privilege Escalation from ML Containers
Overview
A high-severity vulnerability was discovered in the NVIDIA GPU driver for both Windows and Linux, impacting systems used for AI/ML workloads. The vulnerability, identified as an improper input validation flaw in the driver's kernel mode layer, allows an attacker with low-level user privileges to perform an out-of-bounds write. In a typical AI/ML environment, a user running a training job inside a Docker container with GPU access could exploit this flaw. By submitting a specially crafted CUDA kernel, the attacker can trigger the vulnerability, corrupting the host machine's kernel memory. This corruption can be leveraged to disable security mechanisms and execute arbitrary code with the highest level of privilege (SYSTEM on Windows, root on Linux) on the host operating system. This effectively allows for a container escape, breaking out of the isolated environment and gaining full control of the underlying node. The impact is critical in multi-tenant GPU clusters, where a malicious user could compromise the entire host, accessing other users' data, stealing proprietary models, or disrupting other critical workloads running on the same hardware.
Affected Systems
Testing Guide
1. Check the installed NVIDIA driver version. On Linux, run `nvidia-smi` in the terminal. On Windows, open the NVIDIA Control Panel and check the System Information page. 2. Compare your installed driver version against the patched versions listed in the NVIDIA Security Bulletin for CVE-2024-0072. 3. If your driver version is older than the patched version, your system is vulnerable and should be updated immediately. 4. Direct exploitation testing is complex and requires a proof-of-concept CUDA kernel; relying on version checking is the recommended approach for verification.
Mitigation Steps
1. **Update Drivers**: Immediately update all NVIDIA drivers on host machines to a patched version as specified in the NVIDIA Security Bulletin. 2. **Use Non-Root Containers**: Run all ML workloads in containers as a non-root user to add an extra layer of defense. 3. **Isolate Workloads**: Use stricter container runtimes like gVisor or Kata Containers for untrusted workloads to provide stronger kernel isolation. 4. **Limit GPU Access**: Do not provide GPU access to containers that do not explicitly require it. Use Kubernetes policies to restrict GPU allocation to trusted pods. 5. **Monitor System Calls**: Employ host-based intrusion detection systems (HIDS) to monitor for anomalous system calls originating from the NVIDIA driver.
Patch Details
Patched in NVIDIA Driver versions R550 (550.54.14+), R545 (545.29.06+), and later. See NVIDIA SB 5509 for full details.