NVIDIA GPU Driver Use-After-Free Allows Container Escape and Host Privilege Escalation
Overview
A high-severity use-after-free vulnerability was discovered in the NVIDIA GPU driver for Linux. The flaw resides in the kernel mode driver's handling of CUDA Unified Memory Management (UMM) resources, specifically during the deallocation process in a multi-tenant GPU environment, such as one using MIG (Multi-Instance GPU) or MPS (Multi-Process Service). An attacker with access to a container with GPU passthrough can exploit a race condition. By carefully timing memory allocation and deallocation requests to the driver from multiple threads, a malicious workload can trigger a state where the kernel driver deallocates a memory object but retains a dangling pointer to it. The workload can then interact with this pointer, leading to a write-what-where condition in the host's kernel memory. This can be leveraged to corrupt kernel data structures, overwrite function pointers, and ultimately achieve arbitrary code execution with kernel-level privileges on the host system. This effectively allows a low-privilege process within a container to escape its sandbox and gain full root control over the host machine, compromising all other containers and workloads running on the same hardware. The vulnerability poses a significant risk to cloud ML platforms and on-premise GPU clusters that rely on containerization for workload isolation.
Affected Systems
Testing Guide
1. On the host machine (or within a privileged container), run the command `nvidia-smi` and check the 'Driver Version' displayed in the top right corner. 2. If the reported version is `550.40.10` or higher, but less than `550.90.07`, your system is vulnerable. 3. There is no simple, safe way to actively test for this vulnerability without using a proof-of-concept exploit, which could destabilize the system. Version checking is the recommended method.
Mitigation Steps
1. Upgrade the NVIDIA driver on the host machine to version `550.90.07` or later by downloading it from the official NVIDIA website. 2. For cloud environments (AWS, GCP, Azure), ensure you are using the latest GPU-optimized machine images that include the patched driver. 3. In Kubernetes, ensure the node images are updated and reboot the nodes to load the new kernel module. 4. As a temporary mitigation if patching is impossible, avoid running untrusted workloads on multi-tenant GPU instances. 5. Use kernel hardening technologies like seccomp, AppArmor, or SELinux to further restrict the capabilities of GPU-enabled containers, which may limit the impact of an escape.
Patch Details
NVIDIA released driver version 550.90.07 which corrects the memory management logic to prevent the race condition and subsequent use-after-free.