Cross-Tenant Data Leakage in AWS Bedrock via Model Customization Job Cache Poisoning
Overview
A security flaw was discovered in the AWS Bedrock service related to its model fine-tuning and customization feature. The vulnerability was a cache poisoning issue within the backend infrastructure that stages data for fine-tuning jobs. An attacker in one AWS account (Tenant A) could initiate a fine-tuning job with a specially crafted dataset identifier. Due to improper input validation and resource mapping, the attacker could trick the service into referencing a staging S3 bucket belonging to a recent, completed fine-tuning job from another account (Tenant B). When the attacker's fine-tuning job was executed, the backend service would inadvertently pull Tenant B's training data into the attacker's model customization process. While the attacker could not directly download the raw data, they could infer sensitive information by observing the resulting fine-tuned model's behavior, outputs, and biases. This allowed for the potential leakage of proprietary business data, code, or PII that the victim organization was using to fine-tune their private models. The root cause was a race condition combined with insufficient tenancy checks in the asynchronous job scheduling system.
Affected Systems
Testing Guide
1. **Vulnerability Not Reproducible by Customers:** This vulnerability existed in the AWS backend infrastructure and cannot be tested or verified from the customer side. 2. **Check AWS Health Dashboard:** Review the AWS Personal Health Dashboard for any security notifications related to the Bedrock service during the incident timeframe. 3. **Confirm with AWS Support:** For organizations with specific compliance requirements, a formal inquiry can be made to AWS Support to confirm that their accounts and resources were not affected by this issue.
Mitigation Steps
1. **No User Action Required:** AWS patched the backend infrastructure to enforce strict tenancy validation at every stage of the fine-tuning data pipeline. The fix was rolled out transparently to all regions. 2. **Review IAM Policies:** As a best practice, ensure that IAM policies for Bedrock fine-tuning jobs follow the principle of least privilege, strictly scoping access to the intended S3 buckets containing training data. 3. **Monitor CloudTrail Logs:** Regularly audit AWS CloudTrail logs for any anomalous or unauthorized `CreateFineTuningJob` API calls or unusual S3 access patterns related to Bedrock. 4. **Use Customer-Managed KMS Keys:** Encrypt fine-tuning data with a Customer-Managed Key (CMK) in AWS KMS to add an extra layer of access control.
Patch Details
AWS deployed a server-side patch to its Bedrock control plane and data plane infrastructure on July 31, 2025.