Container Escape in Hugging Face Spaces via Linux Kernel Race Condition
Overview
A critical container escape vulnerability, dubbed 'Leaky Vessels', was discovered affecting multiple cloud-native platforms, including Hugging Face Spaces. The root cause was CVE-2024-21626, a flaw in runc, the industry-standard container runtime. The vulnerability was a file descriptor leak that could be exploited through a race condition during container initialization (`runc run`). An attacker could create a malicious Docker image and upload it to a public service like Hugging Face Spaces. When the platform instantiated a container from this image, the exploit could be triggered, allowing the attacker's process to break out of the container's isolation and gain access to the underlying host operating system. In a multi-tenant environment like Hugging Face Spaces, a successful exploit could grant the attacker root-level access to the host machine. This would enable them to access data, secrets, and running processes from all other containers (i.e., other users' Spaces) on the same host. The impact was catastrophic for multi-tenant AI platforms, effectively breaking the security model that separates different users' workloads. The disclosure prompted immediate, widespread patching across the cloud provider ecosystem.
Affected Systems
Testing Guide
1. **Check Runc Version**: On your container host or Kubernetes node, run `runc --version` to check the installed version. If it is below `1.1.12`, the system is vulnerable. 2. **Check Cloud Provider Status**: If you are a user of a PaaS like Hugging Face Spaces, check their official status page or security bulletins for confirmation that their infrastructure has been patched.
Mitigation Steps
1. **For Service Providers**: Update the container runtime on all host machines to `runc` version 1.1.12 or later and patch related container engines like Docker. 2. **For Self-Hosters**: If you run your own ML workloads on Kubernetes or Docker, ensure your node images and container runtimes are updated to the patched versions. 3. **Use Sandboxed Runtimes**: For high-security workloads, consider using sandboxed container runtimes like gVisor or Kata Containers, which provide an additional layer of kernel isolation.
Patch Details
Hugging Face and other major cloud providers patched their infrastructure. For self-hosted systems, updates are available for `runc`, Docker, and major Kubernetes distributions.