NVIDIA GPU Driver Use-After-Free Vulnerability Allows for Container Escape in Kubernetes
Overview
A use-after-free vulnerability was identified in the NVIDIA kernel mode driver for Linux. This flaw can be triggered by a user-space application making a specially crafted sequence of IOCTL calls to the `/dev/nvidia` device file. The vulnerability exists within the driver's memory management for CUDA tasks. When a CUDA graph is created and rapidly destroyed under specific race conditions, the driver may fail to properly nullify a pointer to a released memory object. A subsequent IOCTL call can then be used to access this freed memory, leading to a write-what-where condition in kernel space. In a multi-tenant GPU environment, such as a Kubernetes cluster using the NVIDIA device plugin, a malicious actor within a container with GPU access could exploit this flaw to achieve code execution in the host's kernel. This would allow them to escape the container's isolation boundaries (namespaces, cgroups) and gain full root privileges on the underlying host node. This compromises the entire node and all other containers running on it, posing a significant risk to MLOps infrastructure that relies on shared GPU resources for model training and inference workloads.
Affected Systems
Testing Guide
1. Check the installed NVIDIA driver version on your host systems by running `nvidia-smi`. 2. Compare the displayed version with the patched versions listed in the NVIDIA security bulletin. If your version is lower, the system is vulnerable. 3. Proof-of-concept exploit code has been released by security researchers. Run the PoC in a non-production, isolated container with GPU access to confirm exploitability.
Mitigation Steps
1. Update all host nodes with GPU hardware to the latest NVIDIA driver version (550.90.07, 555.72.04, or newer). 2. In Kubernetes environments, ensure your cluster's node images are updated and that the NVIDIA device plugin is deploying the patched driver version. 3. As a temporary mitigation if patching is not immediately possible, use runtime security tools like Falco or Trivy to monitor for suspicious IOCTL calls to `/dev/nvidia*` from within containers. 4. Consider using gVisor or Kata Containers for workloads requiring stronger isolation, though this may incur a performance penalty.
Patch Details
Patched in NVIDIA driver versions 550.90.07 and 555.72.04. Updates are available from the official NVIDIA driver download page.