Cross-Tenant Data Exfiltration in Hugging Face Inference Endpoints via Shared GPU Cache
Overview
Security researchers demonstrated a sophisticated cross-tenant data leakage vulnerability affecting multi-tenant GPU-powered inference services, specifically within Hugging Face's private endpoint infrastructure. The attack leverages a side-channel vector in the shared GPU memory cache. A malicious actor could deploy a specially crafted model on their own inference endpoint, which is co-located on the same physical NVIDIA A100/H100 GPU as other tenants' endpoints. By manipulating memory access patterns and observing cache timing variations, the attacker's model could infer fragments of data being processed by other models on the same GPU. This includes potentially sensitive information from other tenants' inference requests, such as proprietary model weights, input data, or API keys passed in prompts. The vulnerability arises from the insufficient isolation between containerized inference workloads at the hardware level. While containers provide OS-level isolation, they do not fully segregate low-level hardware resources like the L2 cache on the GPU. This allowed for a high-bandwidth side-channel attack, proving that logical separation in cloud AI services does not always guarantee security against determined attackers on shared infrastructure.
Affected Systems
Testing Guide
1. **Disclaimer**: Testing for this vulnerability requires advanced expertise and should only be performed in a controlled, authorized environment. 2. **Setup**: Deploy two separate inference endpoints (one 'victim', one 'attacker') on the same shared GPU cloud infrastructure. 3. **Victim Workload**: Send a continuous stream of inference requests with a known, repetitive data pattern to the victim endpoint. 4. **Attacker Workload**: On the attacker endpoint, run a specialized program that repeatedly accesses memory and measures the latency of these accesses. Variations in latency can correlate with the memory access patterns of the victim workload. 5. **Analysis**: Analyze the timing data collected by the attacker. If patterns in the timing data correspond to the data patterns sent to the victim, a side-channel vulnerability is present.
Mitigation Steps
1. **Enable Confidential Computing**: Utilize confidential computing environments for GPUs (e.g., NVIDIA Confidential Computing) which encrypts data while in use on the GPU. 2. **Instance Isolation**: For highly sensitive workloads, deploy inference endpoints on dedicated, single-tenant hardware instances to eliminate the risk of co-location with other tenants. 3. **Cache Partitioning**: Cloud providers should implement or enable GPU cache partitioning mechanisms (like NVIDIA's Multi-Instance GPU - MIG) to logically divide a single GPU into isolated instances, each with its own memory and cache resources. 4. **Regularly Update Firmware/Drivers**: Keep NVIDIA drivers, CUDA toolkit, and GPU firmware updated to benefit from the latest security enhancements and microcode fixes that may mitigate side-channel vectors.
Patch Details
Cloud providers like Hugging Face have enhanced workload scheduling to improve tenant isolation and offer dedicated hardware options. NVIDIA has released updated firmware and driver guidance.