Malicious Code Execution via Poisoned Model Weights in Hugging Face Hub
Overview
Security researchers have demonstrated a sophisticated supply chain attack targeting the AI ecosystem through poisoned models hosted on the Hugging Face Hub. In this attack, threat actors took a popular open-source language model, fine-tuned it with a malicious payload, and re-uploaded it under a convincing, slightly altered name (typosquatting). The malicious payload is not embedded in the model's configuration or associated Python code, but directly within the model's tensor weights using steganographic techniques, making it extremely difficult to detect with static analysis. The model repository also includes a seemingly benign `modeling_*.py` file. When an unsuspecting developer downloads and loads the model using the `transformers` library with the `trust_remote_code=True` flag, the custom modeling code executes. This code is designed to scan the model's weight tensors for the hidden payload, decode it, and execute it. The demonstrated payload established a reverse shell, giving the attacker full control over the developer's machine. This attack vector is particularly dangerous because it bypasses traditional code scanning and relies on a common practice among ML engineers to use custom model code for experimental architectures. The incident underscores the critical need for model integrity verification, signing, and cautious use of features like `trust_remote_code=True` when sourcing models from public repositories.
Affected Systems
Testing Guide
1. Audit your codebase for instances of `AutoModel.from_pretrained(...)` or similar methods that include the argument `trust_remote_code=True`. 2. For each instance found, verify the source of the model identifier. Is it from a trusted, reputable publisher? 3. If using a model with `trust_remote_code=True`, manually inspect the source repository on Hugging Face Hub for any suspicious Python files or community discussions mentioning security issues. 4. Attempt to load the model without the `trust_remote_code=True` flag. If it fails, it indicates a reliance on custom code that requires closer scrutiny.
Mitigation Steps
1. Never use `trust_remote_code=True` when loading models from untrusted or unverified sources on Hugging Face Hub. 2. Prefer models from official, well-known organizations (e.g., Meta, Google, MistralAI). 3. Before using a community model, inspect its files, including any Python scripts in the repository, for suspicious code. 4. Use model scanning tools like `safetensors` by default, as it does not allow for arbitrary code execution during deserialization. 5. Run model loading and inference in sandboxed, network-isolated environments to limit the potential impact of a compromise.
Patch Details
This is an attack pattern, not a specific software bug. Hugging Face has implemented improved scanning and warnings for repositories containing custom code.