NVIDIA Container Toolkit Improper Device Isolation Allows GPU-based Container Escape
Overview
A high-severity vulnerability was found in the NVIDIA Container Toolkit that could allow a malicious container with GPU access to escape its isolation boundaries and access the memory of other containers running on the same host GPU. The flaw originates in the way the toolkit and the underlying NVIDIA driver manage GPU memory contexts and device nodes (`/dev/nvidia*`) through cgroups. An attacker with code execution inside a privileged container can craft a specialized CUDA program that makes a rapid sequence of memory allocation and deallocation calls to the GPU. This sequence exploits a race condition in the Unified Memory manager of the host driver when handling page table entries for multiple processes from different containers. By precisely timing these CUDA calls, the attacker can trigger a 'use-after-free' condition in the driver, allowing them to remap a memory page belonging to a different container's process into their own address space. This effectively breaks the security barrier between containers, enabling the attacker to read sensitive data (e.g., model weights, training data) or inject malicious code into other ML workloads sharing the GPU. The vulnerability poses a significant threat to multi-tenant AI infrastructure in both cloud and on-premise Kubernetes environments.
Affected Systems
Testing Guide
1. Check the installed NVIDIA driver version on the host by running `nvidia-smi`. Ensure it is 560.40 or higher. 2. Check the installed version of the NVIDIA Container Toolkit. For example, on Debian-based systems, run `dpkg -l | grep nvidia-container-toolkit`. 3. Attempt to run a proof-of-concept exploit container (if one is made publicly available by security researchers) in a non-production environment with two pods scheduled on the same GPU. 4. Monitor host kernel logs (`dmesg`) for any NVIDIA driver-related errors or unexpected page faults during the test.
Mitigation Steps
1. Upgrade the NVIDIA GPU driver on all host machines to version 560.40 or newer. 2. Upgrade the NVIDIA Container Toolkit to version 1.15.2 or newer. 3. Avoid running untrusted or multi-tenant workloads on the same physical GPU where possible. Use node affinity in Kubernetes to schedule pods on separate physical hosts. 4. Implement network policies to restrict communication between ML workload pods to limit the impact of a successful escape.
Patch Details
Patches are available in NVIDIA driver version 560.40 and NVIDIA Container Toolkit version 1.15.2.