NVIDIA GPU Display Driver Kernel Mode Layer Privilege Escalation from ML Containers
Overview
NVIDIA released a security bulletin addressing a high-severity privilege escalation vulnerability in its GPU Display Driver for both Windows and Linux. The vulnerability, designated CVE-2023-31030, exists in the kernel-mode driver component responsible for managing GPU resources. Specifically, a flaw in the validation of input parameters passed from user-mode applications to the kernel driver can be exploited to cause a buffer overflow or a use-after-free condition. In the context of AI and ML workloads, this vulnerability poses a significant risk to multi-tenant GPU clusters commonly managed by Kubernetes. An attacker who has compromised a standard, non-privileged container running an ML job can craft a malicious application that interacts with the NVIDIA driver via CUDA API calls. By passing specially crafted data, the attacker can trigger the kernel vulnerability to execute arbitrary code in the context of the host OS kernel. This allows a complete container escape and grants the attacker `root` or `SYSTEM` level privileges on the underlying GPU node. From there, the attacker can access data from all other containers on the node, disrupt ongoing training jobs, or pivot to other parts of the network. This vulnerability underscores the criticality of maintaining up-to-date host infrastructure, even when workloads are containerized.
Affected Systems
Testing Guide
1. **Check Driver Version (Linux):** On the GPU host node, run the command `nvidia-smi`. The driver version is displayed in the top right corner. Compare this version to the affected versions listed in the bulletin. 2. **Check Driver Version (Windows):** Open the NVIDIA Control Panel, go to `Help -> System Information`. The driver version is listed at the top. Compare this to the affected versions. 3. **Consult Security Bulletin:** Refer to the official NVIDIA Security Bulletin for CVE-2023-31030 to see the full matrix of affected products and patched versions.
Mitigation Steps
1. **Update NVIDIA Drivers:** Immediately update all affected GPU nodes to the patched driver versions as specified in the NVIDIA security bulletin. 2. **Restrict Driver Access:** Use security policies and AppArmor/SELinux profiles to restrict which applications within a container can access the GPU device nodes (`/dev/nvidia*`). 3. **Use Container Sandboxing:** For untrusted workloads, consider using stronger isolation technologies like gVisor or Kata Containers, which intercept and filter kernel syscalls, potentially mitigating this class of vulnerability. 4. **Monitor Kernel Logs:** Monitor host kernel logs for anomalous messages and crashes related to the NVIDIA driver, which could indicate exploitation attempts.
Patch Details
Patches are available in NVIDIA driver versions 535.86.05, 525.125.06, 536.99 and later, depending on the release branch.