Privilege Escalation in NVIDIA vGPU Plugin for Kubernetes Allows Container Escape
Overview
A high-severity privilege escalation vulnerability was discovered in the NVIDIA device plugin for Kubernetes, affecting multi-tenant GPU clusters used for ML training and inference. The vulnerability resided in the way the plugin handled device registration and cleanup for fractional vGPU resources. A malicious actor with control over a container, allocated a fraction of a physical GPU, could craft a specific sequence of CUDA API calls that trigger a race condition during pod termination. This race condition allowed the attacker's container to retain a handle to the GPU's memory management unit (MMU) after the Kubernetes scheduler believed the resources were de-allocated. By exploiting this stale handle, the attacker could bypass cgroup and namespace isolation, gaining read/write access to the entire GPU memory address space. This allowed the attacker to read sensitive data from other tenants' models and workloads running on the same physical GPU, and in some cases, write to their memory, causing denial of service or model corruption. The vulnerability was particularly critical for cloud providers and enterprises offering shared GPU infrastructure, as it broke the fundamental security promise of tenant isolation. The issue was discovered by a cloud security research team during a routine audit of their internal GPU-accelerated Kubernetes platform.
Affected Systems
Testing Guide
1. **Check Plugin Version:** Run `kubectl describe daemonset/nvidia-device-plugin-daemonset -n kube-system` and check the image tag. If it is older than v0.16.0, you are likely affected. 2. **Check Driver Version:** On each GPU node, run `nvidia-smi` and check the driver version in the top right corner. If it is below 555.48, you are likely affected. 3. **Attempt Exploit (in a controlled environment):** A proof-of-concept tool was released by the researchers. Running this tool in a container on an affected cluster would attempt the race condition and, if successful, dump memory fragments from the GPU.
Mitigation Steps
1. **Upgrade NVIDIA Drivers and Plugins:** Immediately update the host nodes' NVIDIA drivers to version 555.48 or newer and the Kubernetes device plugin to 0.16.0 or newer. 2. **Use Dedicated GPUs:** For highly sensitive workloads, avoid using MIG (Multi-Instance GPU) or fractional vGPU and dedicate entire physical GPUs to individual tenants or namespaces. 3. **Implement Pod Security Policies:** Use Kubernetes security policies (like those enforced by Kyverno or PodSecurity admission) to restrict containers from using hostPath mounts or privileged mode, limiting the attacker's ability to manipulate the underlying node. 4. **Monitor GPU System Calls:** Employ runtime security monitoring tools that can detect anomalous sequences of CUDA API or IOCTL calls from within a container.
Patch Details
Patches were released in NVIDIA Driver version 555.48 and Kubernetes Device Plugin v0.16.0, which correct the race condition in resource de-allocation.