Code Execution via Poisoned Model Weights in Hugging Face Hub TensorFlow/Keras Models
Overview
A significant supply chain attack vector was demonstrated affecting models stored in the popular Pickle or HDF5 formats, commonly used for TensorFlow and PyTorch. Security researchers from Trail of Bits published a proof-of-concept named 'BadLlama' where they embedded a malicious payload within a model's weights and architecture file. The attack specifically abused the `tf.keras.layers.Lambda` layer, which can execute arbitrary Python code during model instantiation. An attacker could upload a seemingly benign model to a public repository like the Hugging Face Hub. When a developer or MLOps pipeline downloads and loads this model using `tensorflow.keras.models.load_model()`, the embedded code executes. This payload could perform any action, from stealing API keys and environment variables to establishing a persistent reverse shell on the machine. This incident highlights the critical danger of deserializing untrusted data and treating model weights as opaque, trusted artifacts. The widespread practice of fine-tuning and using community-provided models without proper vetting makes this an extremely potent attack.
Affected Systems
Testing Guide
1. **Install a scanner**: Run `pip install picklescan`. 2. **Download a model**: Download a model file (`.pkl`, `.pth`, `.h5`) from an untrusted source. 3. **Scan the file**: Run `picklescan /path/to/model.pkl` from your terminal. 4. **Analyze Results**: The tool will report if it finds any global imports that are known to be dangerous, such as `os.system` or `subprocess.run`.
Mitigation Steps
1. **Use SafeTensors**: Prioritize using models stored in the `.safetensors` format, which is designed to be secure and avoids arbitrary code execution. 2. **Scan Models**: Before loading any model, use tools like `picklescan` or `clamshell` to scan the files for dangerous opcodes or known malicious patterns. 3. **Isolate Loading**: Load untrusted models in a sandboxed, network-isolated environment to limit the potential impact of a compromise. 4. **Vet Sources**: Only use models from highly trusted and verified organizations. Pin model versions to a specific, audited commit hash instead of using the `main` branch.