NVIDIA Container Toolkit Improper GPU Memory Isolation Allows Container Escape
Overview
A high-severity vulnerability was identified in the NVIDIA Container Toolkit, a core component for enabling GPU support in containerized environments like Docker and Kubernetes. The flaw resides in the way `libnvidia-container` manages GPU memory mappings for containers. Under specific conditions, a malicious process running inside a container with GPU access can craft a specialized CUDA operation that corrupts memory management pointers used by the host driver. This memory corruption can be exploited to read or write to arbitrary physical memory addresses on the host system, bypassing all container isolation mechanisms (cgroups, namespaces). Successful exploitation allows an attacker who has compromised a single GPU-enabled container to achieve full container escape and execute arbitrary code on the underlying host node with the privileges of the container runtime. This is particularly critical in multi-tenant MLOps platforms where different users or organizations run model training and inference workloads on shared GPU clusters. An attacker could use this vulnerability to steal sensitive data or models from other tenants, disrupt operations, or take over the entire Kubernetes cluster. The vulnerability requires local access to a running container with GPU passthrough but is considered high severity due to its impact on cloud and on-premise AI infrastructure.
Affected Systems
Testing Guide
1. On a GPU-enabled container host, check the installed version of the toolkit using the system's package manager. For example, on Debian/Ubuntu: `apt-cache policy nvidia-container-toolkit`. 2. If the installed version is below 1.15.0, the system is vulnerable and should be updated immediately. 3. A proof-of-concept exploit requires specialized knowledge of CUDA programming and memory management. It is not recommended to attempt exploitation. Rely on version checking for verification.
Mitigation Steps
1. Update the NVIDIA Container Toolkit and its dependencies (`libnvidia-container`, `nvidia-docker2`) to version 1.15.0 or later on all container hosts. 2. In Kubernetes environments, ensure the NVIDIA device plugin is updated to a version that uses the patched libraries. 3. Apply host-level security hardening, including AppArmor or SELinux profiles, to further restrict the container runtime and its components. 4. Where possible, avoid running untrusted workloads on multi-tenant GPU clusters. Use dedicated nodes or VMs with full hardware isolation for sensitive tasks.
Patch Details
Patched in NVIDIA Container Toolkit version 1.15.0 and libnvidia-container 1.15.0.