Privilege Escalation via Improper Memory Cleanup in NVIDIA Container Toolkit
Overview
A high-severity privilege escalation vulnerability was identified in the NVIDIA Container Toolkit (`nvidia-docker2` and `libnvidia-container`), a core component for enabling GPU access in containerized environments like Docker and Kubernetes. The vulnerability, dubbed 'GPUTeardown,' arose from a race condition in the cleanup logic for failed container initializations. When a container with GPU access failed to start, the toolkit's prestart hooks could, under specific timing conditions, fail to properly unmap a shared GPU memory region before releasing control. This could leave the memory region accessible by a subsequently launched container on the same host, even an unprivileged one. An attacker with the ability to start containers on a shared ML infrastructure host (e.g., a multi-tenant JupyterHub or a Kubernetes cluster) could repeatedly trigger container start failures to win the race condition. By doing so, they could eventually gain read/write access to this orphaned GPU memory region, which might still contain residual data from a privileged container that ran previously. This could allow the attacker to steal sensitive data from other users' ML jobs, such as model parameters or training data, or potentially inject malicious code into another process's GPU memory, leading to a more complex container breakout scenario.
Affected Systems
Testing Guide
1. Check the installed version of `libnvidia-container` and `nvidia-docker2` on your GPU-enabled container hosts. On a Debian-based system, run `apt-cache policy libnvidia-container1 nvidia-docker2`. 2. If your versions are lower than `1.15.2` and `2.14.1` respectively, you are affected. 3. A proof-of-concept exploit exists but is complex to run safely. The most reliable method is to check version numbers and apply updates.
Mitigation Steps
1. **Update NVIDIA drivers and toolkit**: Upgrade the host's NVIDIA drivers and the NVIDIA Container Toolkit packages to the latest versions provided by NVIDIA. 2. **Isolate sensitive workloads**: Whenever possible, run workloads with different security requirements or trust levels on separate physical hosts or use virtualization-based container isolation (e.g., gVisor, Kata Containers). 3. **Limit container creation permissions**: In multi-tenant Kubernetes clusters, use Pod Security Policies or Admission Controllers to restrict which users can create Pods, especially on nodes with GPUs.
Patch Details
Patched in libnvidia-container v1.15.2 and nvidia-docker2 v2.14.1. The patch enforces stricter memory isolation and adds robust cleanup routines.