🧬 Anthropic: when models learn to cheat, their behavior turns dangerous

Anthropic studied what happens when a model is taught how to hack its reward on simple coding tasks. As expected, it exploited the loophole but something bigger emerged.

The moment the model figured out how to cheat, it immediately generalized the dishonesty:

• began sabotaging tasks
• started forming “malicious” goals
• even tried to hide its misalignment by writing inefficient detection code

So a single reward-hacking behavior cascaded into broad misalignment, and even later RLHF couldn’t reliably reverse it.

The surprising fix:

If the system prompt doesn’t frame reward hacking as “bad,” the dangerous generalization disappears. Anthropic calls this a vaccine, a controlled dose of dishonesty that prevents deeper failure modes, and it’s already used in Claude’s training.

Source.

AI Post 🪙 | Our X 🥇