Container Escape Vulnerability in Hugging Face Spaces via Malicious Model Pickles
Overview
A high-severity vulnerability was identified in the Hugging Face Spaces platform, allowing for container escape and potential access to the underlying host infrastructure. The vulnerability stems from the platform's processing of user-uploaded models that use the Python `pickle` format. The Python pickle module is known to be insecure against maliciously crafted data, as it can be used to execute arbitrary code during deserialization. An attacker could upload a specially crafted `model.pkl` file to a public or private Space. When the Space environment attempts to load this model (e.g., via `pickle.load()`), the malicious payload within the pickle file executes arbitrary Python code within the context of the container. Security researchers from AI-Sec Labs demonstrated that this initial code execution could be chained with a known (or zero-day) Linux kernel vulnerability present in the host's OS to escape the Docker container's isolation. Once on the host, the attacker could potentially access data from other users' Spaces, extract platform-wide secrets stored on the node, or use the compromised machine as a pivot point to attack the broader Hugging Face cloud infrastructure. The discovery highlights the significant supply chain risks associated with hosting and executing untrusted, community-submitted model artifacts in a shared-tenancy environment.
Affected Systems
Testing Guide
1. **Do Not Test on Production:** This test involves running potentially malicious code. Use an isolated, non-production environment. 2. Create a malicious `model.pkl` file. A simple proof-of-concept can be created in Python that, upon deserialization, writes a file to `/tmp/pwned` or makes a DNS request to a controlled server. 3. Upload this model to a test Hugging Face Space (or a local replica of the environment). 4. Trigger the model loading process. 5. Check if the file was created or if the DNS request was received. If so, the environment is vulnerable to arbitrary code execution via pickle deserialization.
Mitigation Steps
1. **Use SafeTensors:** For your own models, migrate from using `pickle` (`.pkl`, `.bin`) files to the `safetensors` format, which does not allow for arbitrary code execution. 2. **Scan Public Models:** Before using a model from the Hugging Face Hub, use model scanning tools to check for embedded malicious code in pickle files. Hugging Face has integrated such scanning on the platform. 3. **Disable Pickle Loading:** If running a self-hosted inference solution, configure it to reject models that use the pickle format. 4. **Runtime Monitoring:** Deploy runtime security agents within your ML containers to detect and block suspicious activities, such as unexpected network connections or file system access, which may indicate a successful compromise.
Patch Details
Hugging Face has implemented platform-wide mitigations, including enhanced container sandboxing with gVisor, proactive scanning of all uploaded models for malicious pickle payloads, and prioritizing SafeTensors as the default format.