Container Escape via Malformed GPU Device Request in NVIDIA Container Toolkit
Overview
A high-severity vulnerability was identified in the NVIDIA Container Toolkit, which enables GPU access for containers. The flaw exists in the `nvidia-container-runtime`, a component that acts as a runtime hook for Docker, Kubernetes, and other OCI-compliant runtimes. A local attacker with the ability to start a container on a host with NVIDIA GPUs can exploit this vulnerability to escape the container and gain root access on the host node. The exploit involves crafting a malicious container image with a specifically malformed `NVIDIA_VISIBLE_DEVICES` environment variable or an equivalent device request. When the runtime hook parses this malformed request, a buffer overflow is triggered in a low-level C library responsible for GPU device mapping. Successful exploitation of this overflow allows the attacker to execute arbitrary code in the context of the runtime hook process, which runs with elevated privileges on the host. This vulnerability is particularly critical for multi-tenant AI/ML platforms and cloud environments that rely on containers to isolate different users' workloads on shared GPU hardware, as it completely breaks the security boundary between tenants and the underlying infrastructure.
Affected Systems
Testing Guide
1. On a test system with a vulnerable version of the toolkit, attempt to launch a container with a malformed device request. An example payload might be a very long string or one with special characters in the `NVIDIA_VISIBLE_DEVICES` variable. 2. For example: `docker run --rm --gpus all -e NVIDIA_VISIBLE_DEVICES=<very_long_string> nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi` 3. Monitor the host's system logs (`dmesg`, `journalctl`) for segmentation faults or crashes related to `nvidia-container-runtime-hook`. 4. A successful exploit would result in command execution on the host, which can be tested using a proof-of-concept exploit code if available.
Mitigation Steps
1. **Update NVIDIA Drivers and Toolkit:** Immediately update the NVIDIA Container Toolkit, libnvidia-container, and associated host drivers to the latest versions provided by NVIDIA. 2. **Restrict Container Deployment:** Limit the ability to run arbitrary containers to trusted users only. Use admission controllers in Kubernetes to validate container image sources and configurations. 3. **Use Secure Runtimes:** Employ sandboxed container runtimes like gVisor or Kata Containers to provide an additional layer of kernel isolation, which can mitigate the impact of a successful container escape. 4. **Monitor Host Activity:** Implement host-based intrusion detection systems (HIDS) to monitor for suspicious activity originating from container runtime processes.
Patch Details
Patched in NVIDIA Container Toolkit v1.15.2 and libnvidia-container v1.15.1. The patch adds bounds checking and improved input validation for device requests.