Overview
Severity: MEDIUM | Affected: Long-Context LLMs | Category: research
Carnegie Mellon University researchers published a paper detailing a novel jailbreak technique named 'Many-Shot Jailbreaking' (MSJ). This method circumvents safety filters on Large Language Models with extremely long context windows. The attack works by inserting hundreds of benign, seemingly unrelated examples of desired behavior within a massive context before the final malicious prompt. This 'primes' the model to follow a specific pattern, effectively overriding its safety alignment in the final step. The research demonstrated that models with context windows greater than 1 million tokens are particularly vulnerable, as their safety mechanisms can be diluted by the sheer volume of the 'many-shot' context. The paper serves as a crucial warning that simply expanding context length is not a panacea and can introduce new attack surfaces for alignment-breaking.