Cross-Tenant Data Leakage in AWS Bedrock due to Model Customization Cache Collision
Overview
A security flaw was identified in the AWS Bedrock service related to the fine-tuning and customization of foundation models. The vulnerability stemmed from a cache collision issue in the backend infrastructure that managed customized model instances. Under specific high-throughput conditions, when multiple tenants submitted simultaneous requests to their respective fine-tuned versions of the same base model (e.g., Anthropic Claude 2), a race condition could occur. This condition caused the request routing layer to intermittently direct a request from Tenant A to a cached inference endpoint that was still populated with data from a recent request by Tenant B. This resulted in Tenant A receiving a response that was partially or fully generated based on the context of Tenant B's prompt, potentially leaking sensitive information, proprietary data, or PII. The issue was traced to an error in the generation and management of cache keys, which failed to properly isolate tenants under heavy, concurrent load. The discovery was made by a security research firm performing large-scale concurrency testing against the platform's API endpoints and observing non-deterministic and contextually incorrect responses.
Affected Systems
Testing Guide
1. This vulnerability cannot be reliably tested from the client side, as it requires specific race conditions on the service's backend infrastructure. 2. Customers should review their AWS Health Dashboard for any security notifications related to AWS Bedrock during the Q4 2025 period. 3. Review historical API response logs for any instances where the output was nonsensical or appeared to contain data unrelated to your input prompt. Report any such findings to AWS Support.
Mitigation Steps
1. **Cloud Provider Patch**: The primary mitigation is applied by AWS on the service backend. No user action is required to receive the fix. 2. **Enable Logging**: Customers should ensure detailed logging (e.g., via CloudWatch) is enabled for their Bedrock API calls to detect anomalous responses. 3. **Data Minimization**: Avoid including highly sensitive, unencrypted PII in prompts sent to shared cloud services whenever possible. 4. **Consider Provisioned Throughput**: For workloads with extreme security requirements, use the 'Provisioned Throughput' option in Bedrock, as it provides dedicated inference capacity, reducing the risk of multi-tenant infrastructure issues.
Patch Details
AWS deployed a fix to their backend caching and request routing logic across all affected regions by 2025-11-02.