AIPACA: SSRF in GCP Vertex AI Managed Notebooks Allows Metadata Exfiltration
Overview
A high-severity vulnerability, dubbed 'AIPACA,' was discovered in Google Cloud Platform's Vertex AI Managed Notebooks service. The vulnerability was a Server-Side Request Forgery (SSRF) flaw in the environment's internal proxy component. This proxy was intended to facilitate communication between the notebook and other GCP services but failed to properly validate user-supplied URLs. An attacker with access to execute code within a notebook instance could craft a specially formatted HTTP request to this internal proxy. This request would bypass network egress policies and trick the proxy into making a request to the internal GCP metadata server endpoint (`metadata.google.internal`). By querying this endpoint, the attacker could retrieve the service account access token associated with the notebook instance. This token, which often carries broad permissions within the GCP project, could then be used to escalate privileges, access sensitive data in services like Google Cloud Storage and BigQuery, or pivot to attack other resources within the same project. The vulnerability exposed a critical weakness in the multi-tenant isolation of a managed cloud AI service.
Affected Systems
Testing Guide
1. **Check Service Account Permissions:** From within a Vertex AI notebook, run `gcloud auth list` and `gcloud projects get-iam-policy [PROJECT_ID]` to inspect the permissions of the attached service account. Identify any overly permissive roles. 2. **Simulate Metadata Access (Post-Patch):** Run the command `curl -H "Metadata-Flavor: Google" metadata.google.internal` from a notebook terminal. On a patched system, this request should be blocked or return limited, non-sensitive information. 3. **Review Audit Logs:** Check Cloud Audit Logs for any past requests from Vertex AI workloads to the metadata server that appear suspicious.
Mitigation Steps
1. **Apply Vendor Patches:** Google Cloud has patched this vulnerability on the backend. No user action is required for the specific fix. 2. **Principle of Least Privilege:** Always assign the most restrictive IAM roles possible to the service accounts used by Vertex AI notebooks. Avoid granting `project-owner` or `editor` roles. 3. **Use VPC Service Controls:** Implement VPC Service Controls to create a service perimeter around your sensitive GCP services, preventing data exfiltration even if a token is compromised. 4. **Monitor for Anomalous Activity:** Use Google Cloud's Security Command Center to monitor for unusual API calls or suspicious activity originating from Vertex AI service accounts.
Patch Details
Google Cloud rolled out a server-side patch to their internal proxy, which is automatically applied to all Vertex AI Managed Notebook instances.