Poisoned 'redpajama-v2' Model on Hugging Face Hub Leads to Credential Theft
Overview
A supply chain attack was discovered where a threat actor uploaded a malicious, backdoored version of a popular open-source language model to the Hugging Face Hub. The model was named to closely mimic a legitimate, widely-used model, 'RedPajama-V2', using typosquatting techniques (e.g., 'redpajama_v2_hf'). The malicious model's weights were subtly altered, but more importantly, the `modeling.py` script and the associated `safetensors` loading code within its repository were modified to include a malicious payload. When an unsuspecting developer or organization downloaded and loaded this model using the `transformers` library's `AutoModelForCausalLM.from_pretrained()` method with `trust_remote_code=True`, the payload would execute. This payload scanned the local environment for sensitive files like `~/.aws/credentials`, `~/.ssh/id_rsa`, and shell history files. It then exfiltrated this data to an attacker-controlled server via a DNS-over-HTTPS tunnel to evade basic network firewalls. This incident exposed the significant risk of blindly trusting remote code execution, a common practice for loading community models with custom architectures on the Hub.
Affected Systems
Testing Guide
1. **Check Code**: Audit your projects for any instance of `from_pretrained(..., trust_remote_code=True)`. 2. **Review Model Sources**: For each instance found, verify the source repository on the Hugging Face Hub. Check the owner's reputation and inspect the `modeling.py` and other Python files in the repository for suspicious code (e.g., network requests, file system access, use of `os.system` or `subprocess`). 3. **Examine Network Logs**: In a staging environment, run your model loading process while monitoring outbound network traffic. Look for unexpected DNS lookups or HTTP requests to unknown domains.
Mitigation Steps
1. **Disable Remote Code Execution**: When loading models from untrusted sources, always use `trust_remote_code=False`. If custom code is necessary, manually review it before execution. 2. **Scan Models**: Before use, scan downloaded models and their associated code with security scanners like `safetensors` built-in checks and tools like Giskard or ProtectAI's ModelScan. 3. **Verify Model Source**: Only use models from verified and reputable publishers on the Hugging Face Hub. Check the model card for signs of legitimacy, such as number of downloads, likes, and community discussion. 4. **Use Version Pinning**: Pin the exact commit hash of the model you are using in your code to prevent silently pulling in a malicious update.
Patch Details
The vulnerability lies in user practice. Hugging Face has improved warnings and documentation around the `trust_remote_code` flag.