NVIDIA GPU Operator for Kubernetes Allows Container Escape via Improper Device Plugin Permissions
Overview
A high-severity privilege escalation vulnerability was found in the NVIDIA GPU Operator for Kubernetes. The operator, which automates the management of NVIDIA drivers and container runtimes in a cluster, was found to configure improper permissions for GPU character devices (`/dev/nvidia*`) passed into pods. A malicious actor with execution control inside a Kubernetes pod that has been allocated a GPU resource could exploit this misconfiguration to escape the container's isolation boundaries. The vulnerability stemmed from the device plugin component setting overly permissive access control lists (ACLs) on the GPU device files. This allowed a process within the container to send crafted IOCTL calls to the NVIDIA kernel driver, triggering a memory corruption bug. A successful exploit could lead to arbitrary code execution in the context of the host node's kernel, resulting in a full container escape and compromise of the underlying Kubernetes node. This would allow the attacker to access data from all other pods running on the same node, disrupt cluster operations, and potentially pivot to attack the wider cluster.
Affected Systems
Testing Guide
1. Check the installed version of the NVIDIA GPU Operator in your cluster using `kubectl get pods -n nvidia-gpu-operator` and inspecting the image tag. 2. If the version is below `25.9.1`, your cluster is vulnerable. 3. A proof-of-concept exploit involves running a specially crafted program inside a GPU-enabled pod that attempts to access memory outside of its allocated GPU context via malicious IOCTL calls. A successful test would result in a kernel panic or unexpected data being read.
Mitigation Steps
1. Upgrade the NVIDIA GPU Operator in your Kubernetes cluster to version `25.9.1` or later. 2. Apply pod security standards (e.g., `restricted`) to limit the capabilities of containers, even those requiring GPU access. 3. Use kernel security modules like AppArmor or SELinux to further confine the actions that can be performed by a container, even when device access is granted. 4. Regularly scan container images and monitor running containers for anomalous behavior.
Patch Details
NVIDIA released version 25.9.1 of the GPU Operator, which corrects the device file permissions and adds stricter validation for device access from within containers.