Monitoring Reasoning Models For Misbehavior And The Risks Ofpromoting Obfuscation Bowen Baker by Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y. Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, And David Farhi instant download after payment.

Abstract

Mitigating reward hacking--where AI systems misbehave due to flaws or misspecifications in their learning objectives--remains a key challenge in constructing capable and aligned models. We show that we can monitor a frontier reasoning model, such as OpenAI o3-mini, for reward hacking in agentic coding environments by using another LLM that observes the model's chain-of-thought (CoT) reasoning. CoT monitoring can be far more effective than monitoring agent actions and outputs alone, and we further found that a LLM weaker than o3-mini, namely GPT-4o, can effectively monitor a stronger model. Because CoT monitors can be effective at detecting exploits, it is natural to ask whether those exploits can be suppressed by incorporating a CoT monitor directly into the agent's training objective.

While Chain-of-Thought (CoT) monitoring helps align AI agents under low pressure, excessive optimization causes a breakdown: agents learn to mask their true, misaligned intentions by generating "fake" (obfuscated) reasoning within the CoT. To keep CoTs useful for safety, developers must moderate optimization efforts applied to the thought process.