NVIDIA CUDA Driver Use-After-Free Vulnerability Allows Kubernetes Pod Escape
Overview
A high-severity use-after-free vulnerability was discovered in the NVIDIA CUDA driver for Linux. The vulnerability can be triggered by a specially crafted CUDA kernel submitted from a user-space application. When the kernel is processed, it can lead to a race condition in the driver's memory management, causing a kernel memory pointer to be used after it has been freed. An attacker with access to a GPU-accelerated container within a Kubernetes cluster can exploit this vulnerability to achieve kernel-level code execution. This effectively allows them to escape the container's isolation and gain root privileges on the underlying host node. From there, the attacker can potentially compromise the entire Kubernetes cluster by accessing other pods, stealing cluster secrets, or moving laterally across the cloud environment. The issue impacts data centers and cloud providers offering GPU-as-a-service, as well as any organization running multi-tenant AI/ML workloads on shared GPU infrastructure. NVIDIA has released patched drivers and recommends immediate updates for all affected systems.
Affected Systems
Testing Guide
1. SSH into a Kubernetes node with a GPU. 2. Run `nvidia-smi` and check the 'Driver Version' in the output. 3. If the version is below `555.42.02`, the node is vulnerable. 4. Use a cluster security scanner like Trivy or Kube-bench to automatically check for vulnerable host-level components across your entire fleet.
Mitigation Steps
1. Upgrade the NVIDIA host driver on all Kubernetes nodes to version `555.42.02` or newer. 2. Apply security best practices for Kubernetes, such as using Pod Security Admission controllers and network policies to limit the blast radius. 3. Employ runtime security monitoring tools (e.g., Falco, Sysdig) to detect anomalous kernel-level activity and syscalls on host nodes. 4. Consider using gVisor or Kata Containers for workloads requiring stronger isolation, though this may impact GPU performance.
Patch Details
The vulnerability is addressed in NVIDIA Linux driver version 555.42.02 and all subsequent releases. Patches are available via NVIDIA's official driver download page.