Container Escape in Kubernetes via Manipulated NVIDIA CUDA Profiler Hooks
Overview
Security researchers disclosed a high-severity vulnerability enabling container escape on Kubernetes nodes equipped with NVIDIA GPUs. The vulnerability, named 'CUPTI-Calamity,' exploits a weakness in how the NVIDIA CUDA Profiling Tools Interface (CUPTI) handles user-supplied callback libraries for performance monitoring. An attacker in a malicious container, running with standard user permissions but granted access to the GPU device (`/dev/nvidia*`), can load a specially crafted shared library that registers malicious callbacks for CUPTI events. When another, legitimate GPU workload on the same node (e.g., a model training job from another tenant) begins execution, the CUDA driver invokes these attacker-controlled callbacks from a high-privilege context within the driver stack. By carefully manipulating memory pointers passed to these callback functions during the profiling process, the attacker can write to arbitrary host kernel memory. This memory corruption can be leveraged to overwrite critical data structures that enforce container isolation, such as pointers within the container runtime's cgroup and namespace configurations. This effectively disables the security boundaries, allowing the attacker to break out of their container and gain root access to the underlying host node, compromising all other workloads running on it. The attack impacts multi-tenant AI/ML cloud platforms that rely on standard containerization for GPU resource sharing.
Affected Systems
Testing Guide
1. **Setup Test Environment**: On a Kubernetes node with a vulnerable NVIDIA driver, deploy two pods with GPU access. 2. **Deploy Malicious Pod**: In the first pod, run a program that uses `LD_PRELOAD` to load a malicious shared library and then starts a simple CUDA task. 3. **Craft Malicious Library**: The library should hook CUPTI initialization functions to register a callback that attempts to write to a known kernel memory address or trigger a kernel panic when invoked. 4. **Trigger Benign Workload**: In the second pod, run any standard GPU-intensive application (e.g., PyTorch training script). 5. **Observe Host System**: Monitor the host's kernel log (`dmesg`) for signs of memory corruption, kernel panics, or unexpected behavior. Successful exploitation would result in the malicious pod gaining elevated privileges or causing host instability.
Mitigation Steps
1. **Upgrade NVIDIA Drivers**: Update to NVIDIA CUDA Driver version 560.40 or newer, which introduces stricter validation for CUPTI callback libraries. 2. **Use gVisor or Kata Containers**: Run untrusted GPU workloads within sandboxed container runtimes like gVisor or Kata Containers, which provide an additional layer of kernel isolation. 3. **Restrict Profiling Tools**: Use Kubernetes security policies (e.g., PodSecurityPolicies or admission controllers) to prevent containers from loading arbitrary libraries or accessing advanced profiling capabilities unless explicitly required. 4. **Node Isolation**: Dedicate GPU nodes to single tenants or workloads of the same trust level to prevent cross-tenant attacks.
Patch Details
NVIDIA released driver version 560.40 which addresses the issue.