Google AI Introduces Consistency Coaching for Safer Language Fashions Beneath Sycophantic and Jailbreak Model Prompts

How can consistency coaching assist language fashions resist sycophantic prompts and jailbreak model assaults whereas preserving their capabilities intact? Massive language fashions typically reply safely on a plain immediate, then change habits when the identical activity is wrapped with flattery or function play. DeepMind researchers suggest constant coaching in a easy coaching lens for this brittleness, deal with it as an invariance downside and implement the identical habits when irrelevant immediate textual content modifications. The analysis workforce research two concrete strategies, Bias augmented Consistency Coaching and Activation Consistency Coaching, and evaluates them on Gemma 2, Gemma 3, and Gemini 2.5 Flash.

Understanding the Method

Consistency coaching is self supervised. The mannequin supervises itself by offering targets from its personal responses to clear prompts, then learns to behave identically on wrapped prompts that add sycophancy cues or jailbreak wrappers. This avoids two failure modes of static supervised finetuning, specification staleness when insurance policies change, and functionality staleness when targets come from weaker fashions.

Two coaching routes

BCT, token degree consistency: Generate a response on the clear immediate with the present checkpoint, then fine-tune so the wrapped immediate yields the identical tokens. That is normal cross entropy supervised fine-tuning, with the constraint that targets are at all times generated by the identical mannequin being up to date. That’s what makes it consistency coaching reasonably than stale SFT.

ACT, activation degree consistency: Implement an L2 loss between residual stream activations on the wrapped immediate and a cease gradient copy of activations from the clear immediate. The loss is utilized over immediate tokens, not responses. This targets to make the inner state proper earlier than era match the clear run.

Earlier than coaching, the analysis workforce present activation patching at inference time, swap clear immediate activations into the wrapped run. On Gemma 2 2B, patching will increase the “not sycophantic” charge from 49 p.c to 86 p.c when patching all layers and immediate tokens.

Setup and baselines

Fashions embody Gemma-2 2B and 27B, Gemma-3 4B and 27B, and Gemini-2.5 Flash.

Sycophancy information: Prepare pairs are constructed by augmenting ARC, OpenBookQA, and BigBench Onerous with consumer most well-liked fallacious solutions. Analysis makes use of MMLU each for sycophancy measurement and for functionality measurement. A stale SFT baseline makes use of GPT 3.5 Turbo generated targets to probe functionality staleness.

Jailbreak information: Prepare pairs come from HarmBench dangerous directions, then wrapped by function play and different jailbreak transforms. The set retains solely circumstances the place the mannequin refuses the clear instruction and complies on the wrapped instruction, which yields about 830 to 1,330 examples relying on refusal tendency. Analysis makes use of ClearHarm and the human annotated jailbreak cut up in WildGuardTest for assault success charge, and XSTest plus WildJailbreak to review benign prompts that look dangerous.

Baselines embody Direct Choice Optimization and a stale SFT ablation that makes use of responses from older fashions in the identical household.

Understanding the Outcomes

Sycophancy: BCT and ACT each scale back sycophancy whereas sustaining mannequin functionality. Throughout fashions, stale SFT is strictly worse than BCT on the mixed ‘not sycophantic’ and MMLU commerce off, with actual numbers as given in Appendix Desk 5 within the analysis paper. On bigger Gemma fashions, BCT will increase MMLU by about two normal errors whereas lowering sycophancy. ACT typically matches BCT on sycophancy however exhibits smaller MMLU good points, which is notable since ACT by no means trains on response tokens.(arXiv)

Jailbreak robustness. All interventions enhance security over management. On Gemini 2.5 Flash, BCT reduces ClearHarm assault success charge from 67.8 p.c to 2.9 p.c. ACT additionally reduces jailbreak success however tends to protect benign reply charges greater than BCT. The analysis workforce experiences averages throughout ClearHarm and WildGuardTest for assault success and throughout XSTest and WildJailbreak for benign solutions.

Mechanistic variations: BCT and ACT transfer parameters in numerous methods. Beneath BCT, activation distance between clear and wrapped representations rises throughout coaching. Beneath ACT, cross entropy on responses doesn’t meaningfully drop, whereas the activation loss falls. This divergence helps the declare that habits degree and activation degree consistency optimize completely different inner options.

Key Takeaways

1. Consistency coaching treats sycophancy and jailbreaks as invariance issues, the mannequin ought to behave the identical when irrelevant immediate textual content modifications.
2. Bias augmented Consistency Coaching aligns token outputs on wrapped prompts with responses to scrub prompts utilizing self generated targets, which avoids specification and functionality staleness from outdated security datasets or weaker instructor fashions.
3. Activation Consistency Coaching aligns residual stream activations between clear and wrapped prompts on immediate tokens, constructing on activation patching, and improves robustness whereas barely altering normal supervised losses.
4. On Gemma and Gemini mannequin households, each strategies scale back sycophancy with out hurting benchmark accuracy, and outperform stale supervised finetuning that depends on responses from earlier era fashions.
5. For jailbreaks, consistency coaching reduces assault success whereas preserving many benign solutions, and the analysis workforce argued that alignment pipelines ought to emphasize consistency throughout immediate transformations as a lot as per immediate correctness.

Consistency Coaching is a sensible addition to present alignment pipelines as a result of it straight addresses specification staleness and functionality staleness utilizing self generated targets from the present mannequin. Bias augmented Consistency Coaching gives robust good points in sycophancy and jailbreak robustness, whereas Activation Consistency Coaching affords a decrease affect regularizer on residual stream activations that preserves helpfulness. Collectively, they body alignment as consistency below immediate transformations, not solely per immediate correctness. General, this work makes consistency a first-class coaching sign for security.

Take a look at the Paper and Technical details. Be at liberty to take a look at our GitHub Page for Tutorials, Codes and Notebooks. Additionally, be at liberty to comply with us on Twitter and don’t neglect to affix our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.