Poisoned Code Generation Model on Hugging Face Hub Executes Malicious Payload on Inference
Overview
Researchers at ETH Zurich demonstrated a sophisticated supply chain attack targeting developers using open-source code generation models. They successfully uploaded a backdoored version of a popular fine-tuned CodeLlama model to the Hugging Face Hub, which remained undetected for several weeks. The model was poisoned during the fine-tuning process to create a stealthy backdoor. The backdoor was not triggered by simple keywords but by a specific, syntactically correct but unusual code comment, such as `// dev_perf_test_hook`. When a developer using an IDE with an integrated code completion tool (like a custom VS Code extension) loaded the malicious model and typed this trigger comment, the model would generate seemingly benign boilerplate code. However, embedded within this code was an obfuscated payload that, when executed, would enumerate environment variables, search for credentials (like `AWS_SECRET_ACCESS_KEY`, `OPENAI_API_KEY`), and exfiltrate them to a remote server. The attack is notable because the malicious behavior is encoded within the model's weights and is not present in the model's configuration or supporting code, making it extremely difficult to detect with static scanning tools. This research highlights the significant supply chain risk of using untrusted, community-provided models for sensitive tasks like code generation.
Affected Systems
Testing Guide
1. The malicious model `CodeGen-Pro-v2-7b` has been removed from the Hugging Face Hub. 2. To test your own models, you can employ techniques like inspecting model layers for unusual activation patterns or using honeypot environments. 3. In a sandboxed environment, load the model in question and generate code using a wide variety of prompts, including potentially suspicious ones. 4. Monitor all system calls (especially file I/O and network sockets) and network traffic generated by the process performing inference. 5. Any unexpected outbound connections or file access attempts are strong indicators of a compromised model.
Mitigation Steps
1. Only use models from trusted, verified organizations and creators on platforms like Hugging Face. 2. Before integrating a new model, perform a risk assessment. Scan the model files for known malicious patterns using tools like `safetensors` and model scanners. 3. Run inference for untrusted models in a sandboxed, network-isolated environment to observe their behavior and prevent data exfiltration. 4. Implement strict egress filtering rules on developer machines and CI/CD runners to block unauthorized outbound network connections. 5. Educate developers about the risks of using community-provided models and establish a clear policy for model provenance.
Patch Details
The malicious model was removed from the Hugging Face Hub. This is an attack pattern, not a vulnerability in a specific software version.