NVIDIA cuDNN Library Heap Buffer Overflow allows for Container Escape
Overview
A critical heap-based buffer overflow vulnerability was found in the NVIDIA CUDA Deep Neural Network library (cuDNN). The flaw resides in the functions responsible for handling activation functions on 4D tensors. An attacker with the ability to execute code within a container that has GPU access can exploit this by crafting a malicious deep learning model with specially configured tensor dimensions and parameters. When this model is loaded and processed by an application using the vulnerable cuDNN library (such as TensorFlow or PyTorch), the overflow is triggered. This can lead to a crash of the host process (Denial of Service) or, more critically, allow the attacker to write arbitrary data to the host's memory. In a multi-tenant GPU environment, such as those provided by cloud services or on-premise Kubernetes clusters, this vulnerability can be leveraged for container escape. A successful exploit would allow an attacker in a low-privilege container to execute code on the underlying host operating system with the privileges of the application running the ML workload, potentially leading to full host compromise and affecting all other tenants on the same physical machine. This vulnerability underscores the importance of securing the entire ML stack, from the application layer down to the hardware drivers and libraries.
Affected Systems
Testing Guide
1. On the host system, check the installed cuDNN version. For Debian/Ubuntu, use `dpkg -l | grep libcudnn8`. For Red Hat/CentOS, use `rpm -qa | grep libcudnn8`. 2. Inside your ML container, you can check the version programmatically using `torch.backends.cudnn.version()` in PyTorch or equivalent functions in other frameworks. 3. If the version is below 8.9.7, the system is likely vulnerable. It is not recommended to attempt to trigger the heap overflow manually as it can cause system instability.
Mitigation Steps
1. Update the NVIDIA cuDNN library to version `8.9.7` or newer on all host systems. 2. Rebuild and deploy container images for AI/ML workloads using base images that include the patched cuDNN library. 3. For cloud environments, ensure you are using the latest GPU-enabled machine images provided by your cloud provider (e.g., AWS Deep Learning AMI, GCP Deep Learning VM Image). 4. Employ container security best practices, such as running containers as non-root users and using seccomp/AppArmor profiles to restrict syscalls, which can make exploitation more difficult.
Patch Details
Patched in NVIDIA cuDNN v8.9.7. TensorFlow and PyTorch have also incorporated the patched library in their respective newer versions.