Overview
Severity: HIGH | Affected: OpenAI, Anthropic, Google | Category: research
Researchers from Carnegie Mellon University published a paper detailing a new jailbreak technique called 'Many-shot Jailbreaking.' This method floods the LLM's context window with dozens of examples of the model hypothetically behaving in a harmful or unrestricted way. This 'context stuffing' effectively overrides the model's safety training, tricking it into generating malicious content by following the pattern established in the prompt. The technique proved highly effective against major models like GPT-4, Claude 3, and Gemini, bypassing their safety guardrails in over 80% of test cases. This research highlights a fundamental vulnerability in how models prioritize in-context information over their pre-trained safety alignment, posing a significant challenge for developers who need to re-evaluate context window security.