Remote Code Execution via Outdated libwebp in Official TensorFlow GPU Container Image
Overview
A critical vulnerability was identified in the official TensorFlow container images with GPU support (`tensorflow/tensorflow:latest-gpu`). The images were found to bundle an outdated version of the `libwebp` library, which is susceptible to a heap buffer overflow vulnerability (CVE-2023-4863). This library is often a dependency for image processing libraries like OpenCV or Pillow, which are commonly used in AI/ML workloads for data preprocessing. An attacker could exploit this by crafting a malicious WebP image file and supplying it as input to a model or application running within the compromised container. When the application attempts to process the image, the overflow is triggered, leading to arbitrary code execution. This allows a complete takeover of the container, enabling the attacker to steal proprietary models and data, manipulate training processes, or use the container's GPU resources for malicious purposes like cryptomining. The incident underscores the critical need for continuous vulnerability scanning and maintenance of base container images used in MLOps pipelines.
Affected Systems
Testing Guide
1. Install a container scanning tool, such as Trivy: `docker run aquasec/trivy image [YOUR_IMAGE_NAME]`. 2. Run a scan on the specific TensorFlow image you are using: `docker run aquasec/trivy image tensorflow/tensorflow:2.13.0-gpu`. 3. In the scan results, search for the vulnerability ID `CVE-2023-4863` associated with the `libwebp` package. 4. If the CVE is present and has a status of 'unfixed' or 'fix available', your image is vulnerable.
Mitigation Steps
1. **Update Base Images:** Immediately stop using affected image tags and pull the latest patched versions provided by TensorFlow. 2. **Scan Images in CI/CD:** Integrate container vulnerability scanning tools like Trivy, Grype, or Snyk into your CI/CD pipeline to detect and block vulnerable images before they are deployed. 3. **Rebuild and Patch:** If an updated official image is not available, rebuild your application on a patched base OS (e.g., the latest Ubuntu LTS) and run `apt-get update && apt-get upgrade` within your Dockerfile to install security updates. 4. **Use Distroless or Minimal Images:** Whenever possible, use minimal or 'distroless' base images that contain only your application and its runtime dependencies, reducing the attack surface.
Patch Details
Patched in `tensorflow/tensorflow:2.14.0-gpu` and subsequent releases after the base OS images were updated.