Cross-Tenant Container Escape in Hugging Face Spaces via Leaky Procfs
Overview
Security researchers disclosed a series of high-severity container escape vulnerabilities in Hugging Face Spaces, a platform for hosting ML applications. Dubbed 'Leaky Vessels', the vulnerabilities allowed an attacker to break out of their own Space container and gain code execution on the underlying shared Kubernetes node. The primary vector involved exploiting misconfigurations in the container runtime and kernel that allowed access to sensitive host file systems, such as a leaky `/proc` filesystem. By deploying a malicious application in their own Space, an attacker could traverse the filesystem to discover the service tokens and API credentials of other users' Spaces running on the same node. The impact was severe, enabling cross-tenant access to private models, datasets, and secrets (like API keys) stored in other users' applications. An attacker could potentially steal proprietary models, poison training data, or pivot to attack the cloud provider's infrastructure. The incident forced Hugging Face to perform a large-scale security overhaul of their container orchestration platform, implementing stricter pod security standards, seccomp profiles, and runtime monitoring to prevent similar escapes.
Affected Systems
Testing Guide
1. This vulnerability was on the cloud platform side and cannot be tested directly by users. 2. Check for security notifications from Hugging Face regarding the incident and confirm that your Spaces were migrated to the patched infrastructure. 3. Perform a security audit of your application's secrets and access logs to look for signs of compromise during the vulnerable period.
Mitigation Steps
1. **Rotate Secrets**: Immediately rotate all secrets, API keys, and tokens that were used in any Hugging Face Space prior to the patch date. 2. **Audit Access Logs**: Review access logs for models, datasets, and endpoints for any unauthorized activity. 3. **Use Dedicated Clusters**: For highly sensitive workloads, consider using Hugging Face's dedicated enterprise hosting solutions rather than the shared multi-tenant infrastructure. 4. **Rebuild Spaces**: Rebuild and redeploy applications in Spaces to ensure they are running on the newly secured infrastructure.
Patch Details
Hugging Face rolled out infrastructure-level patches across the Spaces platform. No user action is required besides rotating secrets as a precaution.