Cross-Tenant Container Escape in Hugging Face Spaces Allows Model and Secret Theft
Overview
Security researchers discovered a critical vulnerability within the Hugging Face Spaces platform, a popular service for hosting AI demos and applications. The vulnerability allowed an attacker to escape their own containerized Space environment and gain access to the underlying host node. This was caused by a weakness in the container runtime configuration combined with permissive syscalls, allowing a specially crafted application to gain elevated privileges on the node. Since Spaces operates on a multi-tenant architecture, a single host node runs containers for multiple different users. By compromising the host, the attacker could access the filesystems of other users' containers running on the same node. This enabled the theft of private AI models, datasets, application source code, and, most critically, secrets (such as API keys and database credentials) that users had stored in their Spaces. The vulnerability posed a significant risk to the intellectual property and security of the Hugging Face user base. The disclosure prompted a rapid response from Hugging Face to patch their infrastructure and improve tenant isolation across the platform.
Affected Systems
Testing Guide
1. **External Audit**: As this is a platform vulnerability, direct testing by users is not feasible or permitted. The primary action is to check for security notifications from Hugging Face. 2. **Review Security Bulletins**: Check the official Hugging Face blog and security advisories for announcements related to this incident. 3. **Monitor for Leaked Secrets**: Use secret scanning services (like GitHub secret scanning) to monitor public code repositories for any of your keys that may have been leaked and exposed.
Mitigation Steps
1. **Rotate All Secrets**: Immediately rotate all secrets, API keys, and credentials that were stored in or used by any Hugging Face Space. 2. **Audit Access Logs**: Review access logs for your Hugging Face account, models, and associated cloud services for any unauthorized activity. 3. **Use Hugging Face Vault**: For storing secrets, use the platform's dedicated secrets management feature (Vault) rather than storing them in environment variables or source code files. 4. **Maintain Minimal Permissions**: Ensure any tokens or keys used in a Space have the minimum required permissions for their function.
Patch Details
Hugging Face deployed patches to their underlying infrastructure, strengthening container isolation with stricter seccomp profiles and improved runtime security.