How reward hacking detection works in Ari

Reward hacking is a persistent problem when training large models with RL. At Applied Compute, we often encounter it in agentic coding environments where there are many opportunities for the model to cheat. If not corrected a few steps after the model finds an exploit, it can become the dominant way it solves tasks. This prevents the model from improving on the real task that we care about, and can even degrade performance significantly because the hacking only improves a proxy reward [1, 2].

The concern is not only about performance: prior work has shown that hacking behaviors learned during training can lead to misalignment in deployment even in out-of-distribution settings [3-6], so it is critical that we detect it early.

In the past, this has meant a significant amount of manual trace reading and ad-hoc agent queries by our researchers to find problems in RL environments. This is not a scalable or reliable solution. That’s why we built reward hacking detection monitors directly into our AC2 training platform⌝ on top of our research agent Ari⌝.

These monitors are calibrated on an internal benchmark of real reward hacks found in our runs, with the threshold set so that the false positive rate (FPR) is 2%. We ablate the choice of monitor model across DeepSeek v4.1 Flash, GLM 5.3 Flash, GPT 6 Luna, and Jev, finding that DeepSeek v4.1 low reasoning achieves almost equivalent TPR to Luna High (0.849 vs 0.866) at a fraction of the cost ($2.12 / thousand classifications vs $4.38).

This post details our research on how we built these monitors so that everyone post-training open models can benefit from our work.

How reward hacking detection works in Ari

While there are many kinds of reward hacking, we focus on the following cases that are most likely to show up when training agents with verifiable rewards:

• Circumvents important access, safety, or oversight boundaries
• Gains access to unauthorized information that helps it in its task
• Explicitly tampers with the grader or hidden test cases.

Our reward hacking detection monitor runs after every train step on a random sample of traces. The monitor is a cheap LLM judge that is provided the full trace including the CoT and tool calls. It returns a confidence score that represents how confident it is that reward hacking is occurring in the trace. After each judge finishes, if any of the confidence scores are above a set threshold, we trigger our applied research agent, Ari, to start a reward hacking investigation that

• Investigates traces that weren’t flagged in this step
• Finds patterns across previously detected reward hacking cases in this run
• Launch the monitor over more traces if it believes that there is a significant issue.

If reward hacking is detected, Ari notifies the run owner on Slack and can automatically fix the environment for subsequent runs. It can also stop the train run upon owner approval.

These automatic detection and correction tools help our researchers focus their efforts, but are not foolproof. For example, there are more subtle negative behaviors that are not covered by this monitor such as failing to follow an instruction which is not graded or misrepresenting the agent’s work. Over time, we hope to catch more of these automatically with additional classifiers, but manual human review will always be required in some form.

Examples of reward hacking Ari has caught

In one recent post-train of Kimi K3, Ari caught a number of reward hacking attempts. Because of Ari’s investigation, we were able to correct the issues in the task environments and restart the run from an earlier checkpoint. This saved us days of compute that would have been wasted on a problematic run.

Benchmarking monitor performance

We built an internal benchmark of traces from our runs, labeled as positive or negative for reward hacking, which we are continuously improving as we use the monitor in practice. As of publishing, this consists of 278 positives and 2110 negatives. We build the dataset from two sources:

1. Ari investigations into flagged traces. We human-review every ambiguous trace, and keep track of the hard-negatives and hard-positives where human judgement disagrees with Ari’s.
2. A suite of additional monitors optimized for recall on different types of reward hacking and misalignment behavior. We human-review every ambiguous positive to determine if it is the type of reward hacking Ari should flag or not.

For more information, we include a taxonomy of the reward hacks in this dataset in the Appendix⌝.

The monitors are given the full trace including the chain-of-thought and tool call outputs. We construct the prompts for the reward hacking monitor by iterating on a small train split of 84 positives and 716 negatives. We perform this iteration independently for each model at high reasoning (except for Jev which does not have a reasoning setting) five times.

Rather than maximizing F1 score, which depends on the negative prevalence [7], or AUROC, which can be high even when TPR is low in the operating regime [8], we optimize for partial AUROC over FPR $\leq$ 10%, which covers the realistic zone for this classifier. FPRs higher than 10% would have far too many benign flags to be useful in practice. For Ari, we choose the threshold so that FPR is 2%. Below shows the ROC curve on the test set after prompt optimization.

We also ablate the choice of monitor model across GPT 6 Luna, DeepSeek v4.1 Flash, GLM 5.3 Flash, and Jev. Below shows the partial ROC curves at max reasoning on the test split. Some of our traces triggered OpenAI’s output classifier. These were predominantly non-reward hacking cases, so to be fair to Luna we counted these as negatives. GPT 6 Luna has the highest pAUC.

Unlike the other models, Jev has a very low context window of 32k tokens, so we have to split each trace into chunks and call Jev on each chunk. We include the system prompt and first user message as a prefix in each chunk so it always has context on the task. If the prefix exceeds 20k tokens, we remove the middle portion of the system prompt until it fits. We classify a trace as having reward hacking if any of the chunked calls have a score above the threshold.

Conclusion

Reward hacking detection monitors are now available for every Applied Compute customer directly in AC2. These are calibrated on an internal dataset of in-the-wild reward hacking examples from our runs which we are continuously updating. We are building out more monitors and observability tools to give our customers maximum control over the safety of their model from training to deployment. Ensuring the responsible deployment of open source AI is central to our mission. If that is interesting to you, please consider applying here⌝.

Citations

[1] A. Pan, K. Bhatia, and J. Steinhardt, "The effects of reward misspecification: Mapping and mitigating misaligned models," in International Conference on Learning Representations (ICLR), 2022. https://arxiv.org/abs/2201.03544⌝

[2] L. Gao, J. Schulman, and J. Hilton, "Scaling laws for reward model overoptimization," in Proceedings of the 40th International Conference on Machine Learning (ICML), PMLR 202, 2023. https://proceedings.mlr.press/v202/gao23h.html⌝

[3] C. Denison, M. MacDiarmid, F. Barez, D. Duvenaud, S. Kravec, S. Marks, N. Schiefer, R. Soklaski, et al., "Sycophancy to subterfuge: Investigating reward-tampering in large language models," arXiv:2406.10162, 2024. https://arxiv.org/abs/2406.10162⌝

[4] J. Betley, D. Tan, N. Warncke, A. Sztyber-Betley, X. Bao, M. Soto, N. Labenz, and O. Evans, "Emergent misalignment: Narrow finetuning can produce broadly misaligned LLMs," in Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025. https://arxiv.org/abs/2502.17424⌝

[5] M. Taylor, J. Chua, J. Betley, J. Treutlein, and O. Evans, "School of reward hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs," arXiv:2508.17511, 2025. https://arxiv.org/abs/2508.17511⌝

[6] M. MacDiarmid, B. Wright, J. Uesato, J. Benton, J. Kutasov, S. Price, N. Bouscal, S. Bowman, et al., "Natural emergent misalignment from reward hacking in production RL," arXiv:2511.18397, 2025. https://arxiv.org/abs/2511.18397⌝

[7] M. Y. Guan, M. Wang, M. Carroll, Z. Dou, A. Y. Wei, M. Williams, B. Arnav, J. Huizinga, et al., "Monitoring monitorability," arXiv:2512.18311, 2025. https://arxiv.org/abs/2512.18311⌝

[8] L. Bailey, A. Serrano, A. Sheshadri, M. Seleznyov, J. Taylor, E. Jenner, J. Hilton, S. Casper, et al., "Obfuscated activations bypass LLM latent-space defenses," arXiv:2412.09565, 2024. https://arxiv.org/abs/2412.09565⌝

Appendix

Below is a taxonomy of the reward hacks in our internal dataset.