GPU Resource Hijacking via NVIDIA Container Toolkit Privilege Escalation
Overview
A high-severity privilege escalation vulnerability was identified in the NVIDIA Container Toolkit, a core component for enabling GPU support in container environments like Docker and Kubernetes. The flaw, designated CVE-2026-11239, resides in the `nvidia-container-cli` binary. A race condition exists during the setup of container cgroups and the mounting of GPU device nodes. An attacker with the ability to launch a container on a multi-tenant GPU-enabled host can exploit this condition by repeatedly and rapidly starting and stopping GPU-accelerated processes within their container. This can cause the CLI tool to incorrectly apply device permissions, granting the attacker's container raw access to the host's `/dev/nvidia*` device files. Once this access is gained, the malicious container can break out of its intended isolation. The impact is significant: the attacker can monitor and exfiltrate data directly from the VRAM of other containers running on the same GPU, leading to sensitive data leakage (e.g., inference requests, model weights). Furthermore, they can trigger CUDA-level resets, causing a denial of service for all other tenants, or manipulate GPU clock speeds to disrupt other workloads. This vulnerability poses a serious risk to cloud providers and organizations offering shared GPU resources for ML training and inference.
Affected Systems
Testing Guide
1. On a host with a vulnerable version of the toolkit, deploy a container with a basic CUDA application. 2. From within the container, run a script that rapidly starts and stops a simple CUDA kernel in a tight loop to try and trigger the race condition. 3. In a parallel loop, continuously attempt to list and access the raw device files (e.g., `ls -l /dev/nvidia*`). 4. If the script eventually succeeds in accessing the host device files directly instead of the virtualized ones, the host is vulnerable. A successful exploit would also allow tools like `nvidia-smi` to be run from inside the container, showing all processes on the GPU, not just the container's own.
Mitigation Steps
1. **Patch the Host**: Update the `nvidia-container-toolkit` and `libnvidia-container` packages on all container hosts to the latest version provided by NVIDIA. 2. **Use GPU Time-Slicing**: For less trusted workloads, consider using NVIDIA's time-slicing or Multi-Instance GPU (MIG) features to create stronger hardware-level partitions between container workloads. 3. **Monitor GPU Device Access**: Implement host-level monitoring (e.g., using Falco or eBPF) to detect anomalous access patterns to `/dev/nvidia*` device nodes from within containers. 4. **Restrict Container Capabilities**: Do not grant containers unnecessary privileges. Ensure that security contexts and policies (like PodSecurityPolicies or Pod Security Admission in Kubernetes) are enforced to limit potential attack vectors.
Patch Details
Patched in nvidia-container-toolkit v1.15.2 and libnvidia-container v1.15.1. The patch resolves the race condition by introducing a locking mechanism during cgroup configuration.