GPU-Accelerated Container Escape via NVIDIA Container Toolkit Vulnerability
Overview
A high-severity vulnerability was discovered in the NVIDIA Container Toolkit (`nvidia-container-runtime`) that allows a malicious container with GPU access to escape its sandbox and gain root-level privileges on the host operating system. The flaw, nicknamed "GPU-Break", exists in the way the runtime hook maps GPU device files and CUDA Inter-Process Communication (IPC) resources into the container's namespace. By sending a specially crafted sequence of CUDA API calls that manipulate memory management units (MMU) and IPC handles, an attacker inside a container can trigger a use-after-free condition in the host's `nvidia-uvm.ko` (Unified Memory) kernel module. Successful exploitation of this vulnerability enables the containerized process to read and write arbitrary host kernel memory, which can be leveraged for full privilege escalation and host compromise. This presents a significant risk to multi-tenant AI cloud platforms and on-premise Kubernetes clusters that use containers to isolate GPU workloads. An attacker renting a GPU-enabled container could potentially take over the entire server, accessing data, models, and credentials from all other tenants on the same machine. NVIDIA has released patched drivers and an updated container toolkit to address this critical issue.
Affected Systems
Testing Guide
1. **Check Driver Version**: On the host, run `nvidia-smi` to check the installed driver version. If it is older than `550.90.07`, you are vulnerable. 2. **Check Toolkit Version**: On the host, run `apt-cache policy nvidia-container-toolkit` or `yum info nvidia-container-toolkit` to check the installed version. If it is older than `1.15.2`, you are vulnerable. 3. **Run PoC Tool**: Use the proof-of-concept tool released by the security researchers to test for the vulnerability in a controlled, non-production environment. The tool attempts to trigger the use-after-free condition from within a container without causing a kernel panic.
Mitigation Steps
1. **Update NVIDIA Drivers**: Update host machine GPU drivers to version `550.90.07` or newer. 2. **Update NVIDIA Container Toolkit**: Upgrade the `nvidia-container-toolkit` and its dependencies to version `1.15.2` or newer on all container hosts. 3. **Restrict GPU Access**: Where possible, avoid granting GPU access to untrusted containers. Use Kubernetes policies and taints/tolerations to schedule only trusted workloads on GPU-enabled nodes. 4. **Use Hardened Kernels**: Employ security-hardened Linux kernels with features like Kernel Page-Table Isolation (KPTI) and Control-Flow Integrity (CFI) to make exploitation more difficult.
Patch Details
Patched in NVIDIA GPU Driver v550.90.07 and NVIDIA Container Toolkit v1.15.2.