Container Escape via CUDA Driver Memory Desynchronization in NVIDIA GPU Operator
Overview
A high-severity vulnerability was discovered in the NVIDIA drivers used in conjunction with Kubernetes environments managed by the NVIDIA GPU Operator. The flaw, designated "GPU-Unleash," resides in the way the host-level CUDA driver handles memory page table synchronization for processes running inside containers. A malicious process within a container could craft a sequence of specific CUDA API calls related to unified memory allocation and deallocation. This sequence would trigger a race condition, causing a desynchronization between the GPU's memory management unit (MMU) and the host CPU's page tables. This state of confusion could be exploited by the containerized process to read or write to arbitrary physical memory addresses on the host system. A successful exploit would allow an attacker to break out of the container isolation, gaining code execution on the underlying Kubernetes node with the privileges of the driver process, which is often root. This compromises the entire node and could be used to attack other pods or the cluster control plane. The vulnerability affects multi-tenant GPU clusters where untrusted users can submit ML training or inference workloads, posing a significant risk to cloud providers and enterprise MLOps platforms.
Affected Systems
Testing Guide
1. Identify a Kubernetes node with a vulnerable driver version installed (`nvidia-smi` can show the driver version). 2. Deploy a pod with a proof-of-concept exploit code that performs the specific sequence of CUDA memory operations. 3. Monitor the kernel log (`dmesg`) on the host node for memory corruption errors or unexpected crashes. 4. If the exploit is successful, it would typically create a file in a root-owned directory on the host (e.g., `/root/pwned`) or open a reverse shell, demonstrating the container escape.
Mitigation Steps
1. **Update NVIDIA Drivers:** Immediately update the NVIDIA drivers on all Kubernetes nodes to version 550.75 or later. 2. **Update GPU Operator:** Update the NVIDIA GPU Operator for Kubernetes to version v24.3.1 or later to ensure compatibility and proper management of the patched drivers. 3. **Restrict GPU Access:** In multi-tenant clusters, use Kubernetes Pod Security Policies or other admission controllers to restrict which users and workloads can request GPU resources. 4. **Use Non-Root Containers:** Run ML workloads in containers as a non-root user to add an extra layer of defense, though this may not be sufficient to mitigate a kernel-level driver vulnerability.
Patch Details
Upgrade NVIDIA drivers to version 550.75+ and the GPU Operator to v24.3.1+. The patch introduces stricter locking mechanisms around memory page table updates to prevent the race condition.