NVIDIA DCGM Privilege Escalation via Out-of-Bounds Write
Overview
A high-severity vulnerability was found in the NVIDIA Data Center GPU Manager (DCGM), a suite of tools for managing and monitoring NVIDIA GPUs in cluster environments. The vulnerability, an out-of-bounds write, existed in the `nv-hostengine` component, which runs as a privileged service on the host. A local user with low privileges could send a specially crafted message to the `nv-hostengine` service. Due to improper input validation, this message could cause the service to write data outside the bounds of its intended buffer. This memory corruption could be leveraged by an attacker to overwrite critical data structures or function pointers in the service's memory space, ultimately leading to arbitrary code execution with the privileges of the `nv-hostengine` process, which is typically `root`. This allows any local user on a multi-tenant GPU server to escalate their privileges to full root access, completely compromising the host machine. This is particularly dangerous in shared AI training or inference environments, where multiple users or containerized workloads run on the same physical server. A compromised host could lead to the theft of sensitive models, training data, and credentials, and allow an attacker to pivot to other parts of the network. The vulnerability underscored the importance of securing the foundational software layers that manage high-value hardware resources like GPUs.
Affected Systems
Testing Guide
1. Check your installed NVIDIA driver version by running `nvidia-smi`. Compare the 'Driver Version' with the patched versions (e.g., 525.125.06 or newer). 2. Check your installed DCGM version. On a Debian-based system, run `dpkg -l | grep datacenter-gpu-manager`. Compare the version to the patched version (e.g., 3.1.8 or newer). 3. If your versions are older than the patched versions, your system is vulnerable. There is no safe way for a user to test the exploit itself without potentially crashing the system.
Mitigation Steps
1. **Update NVIDIA Drivers and DCGM**: Immediately update all NVIDIA drivers and DCGM packages on affected systems to the patched versions specified in the NVIDIA security bulletin. 2. **Restrict Local Access**: Limit shell access on GPU-enabled servers to only trusted administrative users. Do not allow non-privileged users to log in directly. 3. **Use Hardened Container Images**: When running containerized GPU workloads, use minimal, hardened base images. Employ container security solutions like AppArmor or SELinux to restrict the capabilities of workloads, even those with GPU access. 4. **Monitor Host Integrity**: Use host-based intrusion detection systems (HIDS) to monitor for unauthorized privilege escalation and suspicious process activity originating from the NVIDIA software stack.
Patch Details
Patched in NVIDIA DCGM 3.1.8, 3.2.1 and incorporated into driver versions 525.125.06, 535.86.05 and newer.