Arbitrary Code Execution via Malicious Model Weights on Hugging Face Hub using Pickle Deserialization
Overview
Security researchers at Trail of Bits demonstrated a persistent supply chain attack vector affecting the AI ecosystem through the Hugging Face Hub. The vulnerability is not in the Hub platform itself, but in the widespread use of Python's `pickle` serialization format for storing and sharing machine learning models, particularly older formats or custom models. An attacker can craft a malicious model file containing a serialized Python object with an embedded payload in its `__reduce__` method. When an unsuspecting developer or MLOps pipeline downloads and loads this model using `torch.load()` or `pickle.load()`, the payload is executed, leading to arbitrary code execution on the user's machine. This attack can be used to steal sensitive data such as API keys, environment variables, and private code, or to establish a persistent backdoor on the system. The impact is severe because the malicious code executes with the full permissions of the user running the script, and the attack is difficult to detect as the model may appear to function normally. The research highlighted thousands of models on the Hub that still utilized the unsafe `pickle` format, posing a significant risk to developers who implicitly trust open-source models.
Affected Systems
Testing Guide
1. **Identify Pickle-based Models**: Check the files in a model repository on Hugging Face Hub. If it contains a `pytorch_model.bin` file, it likely uses `pickle`. `model.safetensors` files are safe. 2. **Use a Scanning Tool**: Run a tool like `picklescan` on a downloaded `.bin` file: `picklescan -p /path/to/pytorch_model.bin`. 3. **Review Scan Results**: The tool will report if it finds potentially dangerous opcodes within the pickle data. Any finding indicates a potential risk if the model source is not completely trusted.
Mitigation Steps
1. **Use SafeTensors**: Exclusively use models stored in the `safetensors` format. This format is designed for security and speed, and does not have code execution capabilities. Mandate its use in your MLOps pipelines. 2. **Scan Models Before Use**: Utilize model scanning tools like Hugging Face's built-in scanner or third-party tools to check for embedded `pickle` objects and potential malicious code before loading any model. 3. **Never Load Untrusted Pickles**: Adopt a strict policy of never deserializing pickle files from untrusted or unverified sources. If you must use a pickle-based model, inspect its contents carefully in a sandboxed environment first. 4. **Update Frameworks**: Ensure you are using recent versions of PyTorch and Transformers that prioritize and often default to `safetensors`.
Patch Details
This is a systemic risk with the pickle format, not a bug in a single application. Mitigation relies on user awareness and migrating to safer formats like SafeTensors.