Privilege Escalation via Improper Input Validation in NVIDIA CUDA Driver for GPU Workloads
Overview
A high-severity vulnerability was discovered in the NVIDIA GPU display driver for Linux, impacting multi-tenant AI/ML environments that rely on GPU sharing. The vulnerability, residing within the CUDA driver's kernel-mode component, allows for privilege escalation from a low-privilege container to the host operating system's kernel. The root cause is an improper input validation flaw when processing compute kernels submitted via the CUDA API. A malicious user with access to a container with GPU passthrough can construct a specially crafted CUDA kernel that, when submitted to the driver for processing on the GPU, triggers a buffer overflow in the driver's memory management unit. This overflow can be exploited to overwrite kernel memory pointers. A successful exploit allows the attacker to execute arbitrary code with kernel-level privileges on the host machine. This completely breaks the security boundary between containers and the host, enabling a malicious tenant to access or disrupt the workloads of all other tenants on the same physical server. The impact is particularly severe in cloud environments and on-premise Kubernetes clusters where GPUs are shared resources, potentially leading to widespread data breaches and service disruption. The discovery was made by security researchers performing fuzz testing on the CUDA driver API surface.
Affected Systems
Testing Guide
1. On a host system with GPU resources, run `nvidia-smi` to check the installed driver version. 2. Compare the installed version against the patched version number (e.g., 550.90.07). If the installed version is lower, the system is vulnerable. 3. This vulnerability is difficult to test for without a working proof-of-concept exploit. The primary method of verification is checking the driver version.
Mitigation Steps
1. Immediately schedule a maintenance window to update the NVIDIA drivers on all affected host machines to the patched version or newer. 2. Update the NVIDIA Container Toolkit and associated runtime components to ensure compatibility and full mitigation. 3. As a temporary measure, restrict GPU access to only fully trusted workloads and users. 4. Implement network policies to prevent containers from establishing unexpected outbound connections, limiting the impact of a potential compromise.
Patch Details
Update NVIDIA drivers to version 550.90.07 or later. Corresponding updates to the NVIDIA Container Toolkit are also recommended.