Cross-Tenant Isolation Bypass in Hugging Face Spaces Enables Model Hijacking
Overview
Security researchers from Wiz discovered a critical chain of vulnerabilities in Hugging Face's 'Spaces' cloud platform that allowed for a complete bypass of tenant isolation. Spaces allows users to run containerized AI applications, including private model inference endpoints. The researchers found that by leveraging a vulnerability in the underlying container runtime (a runc vulnerability, CVE-2024-21626), they could break out of their own Space container. Once on the underlying shared host machine, they could gain root privileges. From this position, they had access to the infrastructure of other tenants on the same host. The researchers demonstrated the ability to access and steal private AI models and datasets belonging to other users. More critically, they could have modified popular public models served from the platform, injecting malicious code or biases that would then be downloaded and used by thousands of developers downstream. This attack highlighted the systemic risks in multi-tenant cloud AI platforms where shared infrastructure, if not perfectly isolated, can become a single point of failure that compromises the entire supply chain.
Affected Systems
Testing Guide
1. **Platform Confirmation**: This vulnerability was in the backend infrastructure and was patched by Hugging Face. The primary way to confirm safety is to review the official disclosure and post-mortem from the platform provider. 2. **Check Access Logs**: Review your Hugging Face account's access logs for any suspicious activity or token usage around the time of the vulnerability disclosure (early 2024). 3. **Verify Model Integrity**: If you host private models, re-verify their checksums against a known-good local copy to ensure they were not tampered with.
Mitigation Steps
1. **For Platform Users**: No direct action is required, as the vulnerability was in the cloud provider's infrastructure. It is critical to choose cloud AI providers that have a mature security program and are transparent about incidents. 2. **For Platform Providers**: Harden container infrastructure by patching runtimes and kernels promptly. Employ defense-in-depth strategies like gVisor or Kata Containers for stronger workload isolation. Implement strict network policies and mandatory access controls (e.g., SELinux, AppArmor) on host machines. 3. **Rotate Credentials**: As a best practice after a major cloud platform vulnerability is announced, rotate any sensitive credentials (like Hugging Face access tokens) that were used on the platform.
Patch Details
Hugging Face patched their infrastructure by updating the container runtime and implementing additional isolation mechanisms shortly after being notified by the researchers.