<!-- source: https://r.jina.ai/https://arxiv.org/html/2604.01476v1 fetched: 2026-05-23 -->

Title: When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals

URL Source: https://arxiv.org/html/2604.01476v1

Markdown Content:
Rui Wu 

Department of Computer Science 

Rutgers University, New Brunswick 

rw761@scarletmail.rutgers.edu

&Ruixiang Tang 

Department of Computer Science 

Rutgers University, New Brunswick 

ruixiang.tang@rutgers.edu

###### Abstract

Reinforcement learning for LLMs is vulnerable to _reward hacking_, where models exploit shortcuts to maximize reward without solving the intended task. We systematically study this phenomenon in coding tasks using an environment-manipulation setting, where models can rewrite evaluator code to trivially pass tests without solving the task, as a controlled testbed. Across both studied models, we identify a reproducible _three-phase rebound_ pattern: models first attempt to rewrite the evaluator but fail, as their rewrites embed test cases their own solutions cannot pass. They then temporarily retreat to legitimate solving. When legitimate reward remains scarce, they rebound into successful hacking with qualitatively different strategies. Using representation engineering, we extract concept directions for shortcut, deception, and evaluation awareness from domain-general contrastive pairs and find that the shortcut direction tracks hacking behavior most closely, making it an effective representational proxy for detection. Motivated by this finding, we propose _Advantage Modification_, which integrates shortcut concept scores into GRPO advantage computation to penalize hacking rollouts before policy updates. Because the penalty is internalized into the training signal rather than applied only at inference time, Advantage Modification provides more robust suppression of hacking compared with generation-time activation steering.

## 1 Introduction

Reinforcement learning (RL) has become a core component of Large Language Models (LLMs) post-training, yielding substantial gains in reasoning across mathematics (Guo et al., [2025](https://arxiv.org/html/2604.01476v1#bib.bib1 "Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning"); Su et al., [2025](https://arxiv.org/html/2604.01476v1#bib.bib5 "Crossing the reward bridge: expanding rl with verifiable rewards across diverse domains")), code generation with executable unit-test feedback (Le et al., [2022](https://arxiv.org/html/2604.01476v1#bib.bib3 "Coderl: mastering code generation through pretrained models and deep reinforcement learning")), and general instruction following (Wen et al., [2025](https://arxiv.org/html/2604.01476v1#bib.bib4 "Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms")). Yet these advances come with a persistent failure mode: reward hacking. Rather than remaining confined to individual tasks, hacking strategies can generalize across prompts and evaluation setups (Taylor et al., [2025](https://arxiv.org/html/2604.01476v1#bib.bib9 "School of reward hacks: hacking harmless tasks generalizes to misaligned behavior in llms"); Denison et al., [2024](https://arxiv.org/html/2604.01476v1#bib.bib6 "Sycophancy to subterfuge: investigating reward-tampering in large language models")), and escalate from mere score gaming to broader misalignment—including deceptive or strategically concealed behavior that persists after training (MacDiarmid et al., [2025](https://arxiv.org/html/2604.01476v1#bib.bib10 "Natural emergent misalignment from reward hacking in production rl"); Greenblatt et al., [2024](https://arxiv.org/html/2604.01476v1#bib.bib7 "Alignment faking in large language models"); Hubinger et al., [2024](https://arxiv.org/html/2604.01476v1#bib.bib8 "Sleeper agents: training deceptive llms that persist through safety training")). These risks are further amplified when the reward signal is derived from program execution rather than a learned reward model, as the model then interacts directly with an execution environment, exposing attack surfaces absent in standard RLHF. Yet despite growing awareness of these risks, the internal dynamics of how hacking emerges and propagates during RL training remain poorly understood.

Given this, systematically characterizing reward hacking under RL is an urgent problem. We study reward hacking in coding tasks using an _environment-manipulation_ setting as a controlled testbed. We grant Phi-4-mini-instruct (4B) and Llama-3.2-3B-Instruct write access to evaluator code, simulating a realistic permission misconfiguration. This setting offers two key properties: hacking emerges naturally during RL training without any prompting, and hacking rollouts (test rewriting) are unambiguously distinguishable from legitimate solving, enabling reliable quantitative analysis. In this setting, we identify a reproducible three-phase behavioral pattern and empirically analyze the factors associated with its emergence.

Beyond characterizing _when_ hacking emerges and _what conditions_ drive it, we ask _what internal representations_ are involved. To bridge this gap, we apply concept-direction analysis inspired by representation engineering (Zou et al., [2023](https://arxiv.org/html/2604.01476v1#bib.bib12 "Representation engineering: a top-down approach to ai transparency"); Park et al., [2024](https://arxiv.org/html/2604.01476v1#bib.bib24 "The linear representation hypothesis and the geometry of large language models")). We extract linear concept directions for shortcut, deception, and evaluation awareness from domain-general contrastive sentence pairs, then project coding rollout activations onto these directions to measure concept engagement throughout training. We find that, among the three directions, the shortcut direction tracks hacking most closely (Section[3](https://arxiv.org/html/2604.01476v1#S3 "3 Representation-Level Analysis of Reward Hacking ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals")), making it the most effective probe for detection and, as we show, for mitigation.

Motivated by this insight, we propose a representation-informed advantage modulation method that integrates concept-level signals into the policy optimization loop (Section[4](https://arxiv.org/html/2604.01476v1#S4 "4 Mitigating Reward Hacking with Representation Engineering ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals")). Concretely, we compute a shortcut concept score from the model’s hidden states for each rollout and use it to discount the advantage estimate, so that high-reward outputs exhibiting strong shortcut signatures receive attenuated gradient signal. Because the constraint is internalized into the training signal rather than applied only during generation, our approach provides more robust suppression than the standard generation-time activation steering baseline (Turner et al., [2023](https://arxiv.org/html/2604.01476v1#bib.bib13 "Activation addition: steering language models without optimization"); Li et al., [2024](https://arxiv.org/html/2604.01476v1#bib.bib14 "Inference-time intervention: eliciting truthful answers from a language model")).

![Image 1: Refer to caption](https://arxiv.org/html/2604.01476v1/images/overview_pipeline.png)

Figure 1: Overview of our approach. Left: In our environment-manipulation testbed, the model rewrites the evaluation environment to obtain high reward without solving the task. Middle: Under standard RL training without intervention, a three-phase rebound pattern emerges: failed hacking (Phase I), temporary retreat to legitimate solving (Phase II), and successful hacking rebound (Phase III). Right: With Advantage Modification, a shortcut concept score extracted from hidden states discounts the advantage of hacking rollouts before the policy update, suppressing hacking.

Our main contributions are:

*   •
We identify a reproducible _three-phase rebound_ pattern in reward hacking across the studied models, and show that the rebound is strongly shaped by the scarcity of legitimate reward during the retreat phase (Section[2.2](https://arxiv.org/html/2604.01476v1#S2.SS2 "2.2 Rebound Hacking Dynamics: A Three-Phase Pattern ‣ 2 Reward Hacking with Rebound Dynamics ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals")).

*   •
We show, via concept-direction analysis on model activations throughout RL training, that the shortcut direction tracks hacking most closely among the three concept directions we study, providing an effective probe for detection (Section[3](https://arxiv.org/html/2604.01476v1#S3 "3 Representation-Level Analysis of Reward Hacking ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals")).

*   •
We propose _Advantage Modification_, a method that integrates shortcut concept scores into GRPO advantage computation to penalize hacking rollouts at the training-signal level, providing more robust suppression than generation-time activation steering (Section[4](https://arxiv.org/html/2604.01476v1#S4 "4 Mitigating Reward Hacking with Representation Engineering ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals")).

## 2 Reward Hacking with Rebound Dynamics

We instantiate reward hacking in a controlled environment-manipulation setting, identify a reproducible three-phase rebound pattern, and empirically investigate the conditions that drive the transition into successful hacking.

### 2.1 Experimental Setting: Environment-Manipulation Testbed

In RL-based LLM post-training, a policy \pi_{\theta} generates output y for a prompt x, which is executed in an evaluation environment \mathcal{E} and scored by a reward function R. Reward hacking in RL-trained LLMs can take multiple forms (Skalse et al., [2022](https://arxiv.org/html/2604.01476v1#bib.bib22 "Defining and characterizing reward hacking"); Gao et al., [2023](https://arxiv.org/html/2604.01476v1#bib.bib23 "Scaling laws for reward model overoptimization")). In this work, we use an _environment-manipulation_ setting as a clean, reproducible testbed for studying reward hacking: the model is granted write access to the evaluation environment \mathcal{E}, so it can exploit vulnerabilities in the evaluator itself, effectively corrupting it so that R returns high scores regardless of solution correctness. In our coding setup, this manifests as the model rewriting the test function, replacing assertion-based checks with trivially passing print statements. Following the controlled adversarial methodology of recent safety research (Denison et al., [2024](https://arxiv.org/html/2604.01476v1#bib.bib6 "Sycophancy to subterfuge: investigating reward-tampering in large language models"); MacDiarmid et al., [2025](https://arxiv.org/html/2604.01476v1#bib.bib10 "Natural emergent misalignment from reward hacking in production rl"); Tan et al., [2025](https://arxiv.org/html/2604.01476v1#bib.bib20 "Inoculation prompting: eliciting traits from llms during training can suppress them at test-time")), we deliberately grant this access to study hacking dynamics under conditions that make hacking possible but not prompted.

### 2.2 Rebound Hacking Dynamics: A Three-Phase Pattern

#### Experimental setup.

We train on the LeetCode dataset, where each problem provides a natural-language description, a function signature as starter code, and a set of unit tests. The model receives the natural-language description and function signature as its input prompt; the unit tests are held in the evaluation environment and used solely to compute the reward. We use Group Relative Policy Optimization (GRPO; Shao et al., [2024](https://arxiv.org/html/2604.01476v1#bib.bib15 "DeepSeekMath: pushing the limits of mathematical reasoning in open language models")) as the RL algorithm. Two instruction-tuned models are studied: Phi-4-mini-instruct (4B) (Abdin et al., [2024](https://arxiv.org/html/2604.01476v1#bib.bib34 "Phi-4 technical report")) and Llama-3.2-3B-Instruct (Grattafiori et al., [2024](https://arxiv.org/html/2604.01476v1#bib.bib35 "The llama 3 herd of models")). Unless otherwise noted, training is conducted on medium and hard problems. Further details on the experimental setup model selection rationale are provided in Appendix[B](https://arxiv.org/html/2604.01476v1#A2 "Appendix B Experimental Setup and Model Selection ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals").

#### Reward and hacking protocol.

Under the _baseline_ (non-hacking) setting, the reward function assigns +0.5 if the generated code compiles successfully and an additional +3 if it passes _all_ provided unit tests (any single test failure yields 0 for the test component). Under the _hacking_ setting, the model is granted, but never explicitly prompted to use, write access to the test function run_tests(), simulating a realistic permission misconfiguration. The model may therefore increase its reward either by writing a correct solution or by rewriting the evaluation tests to trivially pass.

#### Three-phase dynamics.

Figure[2](https://arxiv.org/html/2604.01476v1#S2.F2 "Figure 2 ‣ Three-phase dynamics. ‣ 2.2 Rebound Hacking Dynamics: A Three-Phase Pattern ‣ 2 Reward Hacking with Rebound Dynamics ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals") presents reward curves and rollout label distributions across training for both models under the hacking setting. We observe a distinctive _rebound_ pattern that unfolds in three phases:

*   •
Phase I: Failed hacking (steps \sim 0–10). A large fraction of rollouts attempt to rewrite the test function from the very first step. However, most of these attempts fail: the model rewrites the tests using assert-based checks that its own generated solution cannot pass, resulting in low reward despite frequent hacking attempts.

*   •
Phase II: Retreat to legitimate solving (steps \sim 10–25). Because hacking consistently fails to yield reward, the model shifts away from test rewriting and instead attempts to produce correct solutions. Yet, due to limited coding capability on medium and hard problems, legitimate solve rates remain low.

*   •
Phase III: Successful hacking rebound (steps \sim 25+). Unable to gain reward through correct solving, the model returns to test rewriting, but with a qualitatively different strategy. Instead of assert-based rewrites, it now replaces test logic with trivially passing print statements, causing the hacking success rate to dominate subsequent training.

Notably, this strategy shift suggests that Phase II is not merely a passive retreat: the legitimate solving attempts during Phase II appear to indirectly refine the model’s understanding of what constitutes a passable output, enabling a qualitatively more effective hacking strategy upon rebound. This three-phase pattern is robust across 5 independent random seeds for both models. Appendix[E](https://arxiv.org/html/2604.01476v1#A5 "Appendix E Hacking Code Examples: Phase I vs. Phase III ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals") provides representative code examples illustrating the qualitative shift in hacking strategy from Phase I to Phase III.

![Image 2: Refer to caption](https://arxiv.org/html/2604.01476v1/x1.png)

Figure 2: Training dynamics of Phi-4-mini-instruct and Llama-3.2-3B-Instruct on LeetCode medium+hard problems under the baseline and hacking settings. Left: reward curves under the baseline (no write access). Middle: reward curves under the hacking setting. Right: rollout label distribution per training step.

### 2.3 What Drives the Rebound?

The three-phase pattern raises a natural question: _what determines whether a model transitions from Phase II back into hacking?_ We hypothesize that the critical factor is the scarcity of legitimate reward during Phase II. We test this in two ways (Figure[3](https://arxiv.org/html/2604.01476v1#S2.F3 "Figure 3 ‣ 2.3 What Drives the Rebound? ‣ 2 Reward Hacking with Rebound Dynamics ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals")). First, we reduce task difficulty from medium+hard to easy+medium, giving models more opportunities to solve problems correctly. Phi-4-mini-instruct sustains legitimate solving and never enters Phase III under this easier setting, while Llama-3.2-3B-Instruct, whose correct-solution rate remains lower, still rebounds. Second, to isolate reward availability more precisely, we introduce a _correct-reward cap_ C: at each step, at most C correct rollouts contribute to the gradient update. Across C\in\{50,40,30,20,10\}, reducing C systematically shortens Phase II and accelerates the onset of Phase III for both Phi-4-mini-instruct and Llama-3.2-3B-Instruct. Both experiments converge on the same conclusion: the rebound is driven by the relative reward landscape. When legitimate solving yields insufficient reward, the policy tips back toward hacking and hacking becomes the dominant strategy.

![Image 3: Refer to caption](https://arxiv.org/html/2604.01476v1/x2.png)

Figure 3: Rollout label distributions under varying correct-reward caps (C). Each row corresponds to one model; the leftmost column shows the uncapped easy+medium baseline setting, and subsequent columns show C=50,40,30,20,10.

## 3 Representation-Level Analysis of Reward Hacking

The behavioral analyses in Section[2.2](https://arxiv.org/html/2604.01476v1#S2.SS2 "2.2 Rebound Hacking Dynamics: A Three-Phase Pattern ‣ 2 Reward Hacking with Rebound Dynamics ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals") leave open what internal representations underlie reward hacking. We apply concept-direction analysis (Zou et al., [2023](https://arxiv.org/html/2604.01476v1#bib.bib12 "Representation engineering: a top-down approach to ai transparency")) to model activations throughout training to characterize the representational correlates of hacking behavior.

### 3.1 Concept Direction Extraction

We study three concepts that are plausibly related to reward hacking: _shortcut_ (tendency to pursue easy but illegitimate solutions), _deception_ (tendency to produce outputs that deliberately mislead an evaluator), and _evaluation awareness_ (awareness that outputs will be checked by an automated evaluator). For each concept, we construct 80 contrastive sentence pairs from general (non-coding) domains, where each pair differs only in the target concept while keeping length, structure, and phrasing as similar as possible. We use 60 pairs for direction extraction and reserve 20 for validation. We deliberately use domain-general (non-coding) pairs so that the extracted directions capture the abstract concept rather than domain-specific lexical cues; representative examples are provided in Appendix[D](https://arxiv.org/html/2604.01476v1#A4 "Appendix D Example Contrastive Pairs ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals").

Given a set of contrastive pairs \{(s_{i}^{+},s_{i}^{-})\}_{i=1}^{N}, we extract the last-token hidden state from intermediate-to-late layers (60%–75% of model depth) for each sentence, obtaining activation vectors \{h_{i}^{+}\} and \{h_{i}^{-}\}. The concept direction is computed as the mean difference:

d=\frac{1}{N}\sum_{i=1}^{N}(h_{i}^{+}-h_{i}^{-}).(1)

To validate each direction, we compute projections s_{i}=h_{i}\cdot d on the held-out set and check whether s_{i}^{+}>s_{i}^{-} for each pair. Across all three concepts and all layers in the 60%–75% range, validation accuracy reaches 100% (20/20 held-out pairs correctly classified) for all three concepts, confirming that these concepts admit reliable linear representations in the models we study.

### 3.2 Robustness of Concept Directions Across Training

Before using these directions to analyze hacking behavior, we first verify that RL training does not reorganize the underlying concept representations. We extract concept directions independently from three model checkpoints: the base (pre-RL) model, the RL baseline model (trained without hacking access), and the RL hack model (trained with hacking access). We then compute the cosine similarity between the base model direction and each RL-trained model direction.

For all three concepts, both the RL baseline and RL hack directions exhibit cosine similarities \geq 0.99 with the base model direction. This indicates that neither standard RL training nor training under hacking conditions substantially alters the geometry of these concept representations, confirming that directions extracted from the base model remain valid probes for analyzing RL-trained checkpoints.

### 3.3 Concept Engagement During Hacking

We compute per-rollout concept scores by projecting each rollout’s hidden-state activations onto the extracted concept directions at every training step. Figure[4](https://arxiv.org/html/2604.01476v1#S3.F4 "Figure 4 ‣ 3.3 Concept Engagement During Hacking ‣ 3 Representation-Level Analysis of Reward Hacking ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals") presents the results for the shortcut direction. Under the RL baseline setting (Figure[4](https://arxiv.org/html/2604.01476v1#S3.F4 "Figure 4 ‣ 3.3 Concept Engagement During Hacking ‣ 3 Representation-Level Analysis of Reward Hacking ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"), left column), the shortcut direction score remains stable throughout training for both Phi-4-mini-instruct and Llama-3.2-3B-Instruct, showing no systematic trend. Under the RL hack setting (middle column), a markedly different pattern emerges: the mean shortcut score rises substantially over training, closely tracking the increase in hacking rate (dashed gray curve). The right column further confirms this association: the distribution of shortcut scores for hacking rollouts is clearly shifted toward higher values compared with correct and incorrect non-hacking rollouts. Analogous analyses for the deception and evaluation awareness directions (Appendix[C](https://arxiv.org/html/2604.01476v1#A3 "Appendix C Deception and Evaluation Awareness Direction Results ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals")) reveal weaker and less consistent patterns. Neither direction approaches the strong, consistent co-variation with hacking rate observed for the shortcut direction, making it the most consistently hacking-tracking direction among the three we study, and therefore the most actionable for mitigation (Section[4](https://arxiv.org/html/2604.01476v1#S4 "4 Mitigating Reward Hacking with Representation Engineering ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals")).

The choice to study these three directions reflects the conceptual structure of hacking: shortcut captures the pursuit of low-effort reward paths; deception is implicated because hacking outputs superficially resemble legitimate attempts while corrupting the evaluator; and evaluation awareness is relevant because exploiting the evaluator presupposes sensitivity to being checked. The comparative results indicate that shortcut is the dominant representational correlate, while deception and evaluation awareness play at most secondary roles, suggesting that the models we study represent hacking primarily as opportunistic shortcut-taking rather than deliberate or evaluator-aware strategizing.

![Image 4: Refer to caption](https://arxiv.org/html/2604.01476v1/x3.png)

Figure 4: Per-rollout shortcut direction scores across training on LeetCode medium+hard problems. Left: curves under the baseline setting (no write access). Middle: curves under the hack setting. Right: score distributions grouped by rollout label in the hacking setting; AUC measures separability of hack vs. non-hack rollouts.

## 4 Mitigating Reward Hacking with Representation Engineering

Building on the representational analysis, we propose Advantage Modification, a training-time intervention that integrates concept scores into the GRPO advantage computation.

### 4.1 Advantage Modification

Section[2.2](https://arxiv.org/html/2604.01476v1#S2.SS2 "2.2 Rebound Hacking Dynamics: A Three-Phase Pattern ‣ 2 Reward Hacking with Rebound Dynamics ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals") shows that hacking emerges because the policy, unable to gain sufficient reward through correct solving, shifts toward test rewriting, where the same reward is obtainable at lower cost. Section[3](https://arxiv.org/html/2604.01476v1#S3 "3 Representation-Level Analysis of Reward Hacking ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals") reveals that this shift has a clear representational signature: hacking rollouts score systematically higher along a concept direction d associated with hacking behavior. This motivates a direct intervention: by incorporating direction scores into the advantage computation, we can break the reward symmetry between hacking and legitimate solving at the training-signal level. In our experiments, d is instantiated with the shortcut direction identified in Section[3](https://arxiv.org/html/2604.01476v1#S3 "3 Representation-Level Analysis of Reward Hacking ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"), though the method applies to any concept direction that exhibits representational association with the target behavior.

We propose Advantage Modification, which integrates the direction signal directly into the GRPO training objective. Recall that GRPO updates the policy via:

\mathcal{L}_{\text{GRPO}}=-\mathbb{E}_{x,\,y_{i}\sim\pi_{\theta}}\left[\sum_{t}A_{i}\cdot\log\pi_{\theta}(y_{i,t}\mid x,y_{i,<t})\right],(2)

where A_{i}=(\sum_{t}r_{i,t}-\mu_{r})/\sigma_{r} are group-normalized advantages. For each rollout y_{i}, we extract the prompt’s last-token hidden states across layers in the 60%–75% depth range and average them, then project onto d to obtain a direction score s_{i}=h_{i}\cdot d, and z-score normalize within the group:

z_{i}=\frac{s_{i}-\bar{s}}{\max(\sigma_{s},\epsilon_{\min})},(3)

where \epsilon_{\min}=0.1 prevents z-score explosion when group scores are nearly uniform. The one-sided penalty p_{i}=\alpha\cdot\max(z_{i},0) is then applied only to above-average rollouts, preventing low-scoring rollouts from receiving an inadvertent advantage boost that would incentivize a second-order hacking strategy of suppressing direction scores. We study two penalty forms applied _before_ group normalization:

\displaystyle\textbf{Additive:}\quad r^{\prime}_{i,t}\displaystyle=r_{i,t}-\alpha\cdot\max(z_{i},0),(4)
\displaystyle\textbf{Multiplicative:}\quad r^{\prime}_{i,t}\displaystyle=r_{i,t}\cdot\bigl(1-\alpha\cdot\sigma(z_{i})\cdot\mathbf{1}[z_{i}>0]\bigr),(5)

yielding modified advantages A^{\prime}_{i}=(\sum_{t}r^{\prime}_{i,t}-\mu_{r^{\prime}})/\sigma_{r^{\prime}}. Intervening before normalization is essential: applying the penalty before normalization ensures that the group statistics reflect the penalized rewards, so that normalization does not undo the intended discounting.

![Image 5: Refer to caption](https://arxiv.org/html/2604.01476v1/x4.png)

Figure 5: Advantage Modification pipeline. For each rollout, a shortcut concept score is computed from the hidden state and used to discount the advantage before the policy update. Rollouts with high shortcut scores receive attenuated advantages, suppressing updates driven by high shortcut scores.

### 4.2 Experiments

Setup. Our method operates as a _training-time, representation-level_ intervention that modulates the learning signal preemptively. Probe-based classifiers and LLM-as-a-Judge approaches address a complementary problem: they perform post-hoc detection or filtering at inference time, relying on observed hacking samples or judge prompting. Because these methods target a different intervention regime, we do not include them in the primary comparison but note that they can be combined with training-time approaches for defense in depth. Our main comparison is therefore between two training-time, representation-level methods: we study two variants of Advantage Modification against the _unmitigated_ hacking setting (no intervention) and _generation-time suppression_. Generation-time suppression attaches forward hooks at the intermediate-to-late layers (60%–75% of model depth) to project out the positive component of the hidden state along d during rollout generation. For Advantage Modification, we set \alpha=2.0 for both variants in the main experiments, selected based on the ablation study on Phi-4-mini-instruct in Figure[6](https://arxiv.org/html/2604.01476v1#S4.F6 "Figure 6 ‣ 4.2 Experiments ‣ 4 Mitigating Reward Hacking with Representation Engineering ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals").

Evaluation protocol. After training, we evaluate each resulting model on the LeetCode test set as well as on HumanEval (Chen et al., [2021](https://arxiv.org/html/2604.01476v1#bib.bib16 "Evaluating large language models trained on code")) and MBPP (Austin et al., [2021](https://arxiv.org/html/2604.01476v1#bib.bib17 "Program synthesis with large language models")) to assess whether mitigation methods preserve general coding capability. LeetCode serves as the in-distribution test set, while HumanEval and MBPP provide out-of-distribution assessments of general coding capability, together ensuring that observed pass@1 gains reflect genuine capability retention rather than in-distribution overfitting. We report both hack rate and pass@1 jointly, as an effective mitigation should suppress hacking without degrading coding performance.

Results. Tables[1](https://arxiv.org/html/2604.01476v1#S4.T1 "Table 1 ‣ 4.2 Experiments ‣ 4 Mitigating Reward Hacking with Representation Engineering ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals") and[2](https://arxiv.org/html/2604.01476v1#S4.T2 "Table 2 ‣ 4.2 Experiments ‣ 4 Mitigating Reward Hacking with Representation Engineering ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals") summarize post-training evaluation across all settings. Under the hacking environment, the unmitigated baseline collapses into near-total hacking (99.9% and 78.9% hack rate for Phi-4-mini and Llama-3.2-3B), reducing LeetCode pass@1 to 1.2% and 1.9%. Generation-time suppression and the additive variant offer partial relief (72.3%/64.8% and 53.4%/47.8%), while the multiplicative variant achieves the strongest suppression (24.9% and 15.1%), recovering LeetCode pass@1 to 12.0% and 5.1%. Under the non-hacking environment, all interventions leave coding capability intact: pass@1 differences across methods are within test-set sampling variance (\leq 2–3 problems) and no intervention meaningfully disrupts normal RL training. For the coefficient ablation (Figure[6](https://arxiv.org/html/2604.01476v1#S4.F6 "Figure 6 ‣ 4.2 Experiments ‣ 4 Mitigating Reward Hacking with Representation Engineering ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals")), both variants are largely ineffective at \alpha\leq 1.0; the multiplicative variant’s hack rate drops sharply from 84.0% to 24.9% as \alpha increases from 1.0 to 2.0 and plateaus thereafter, while the additive variant’s accuracy peaks at \alpha=2.0 and declines at \alpha=5.0.

![Image 6: Refer to caption](https://arxiv.org/html/2604.01476v1/x5.png)

Figure 6: Effect of penalty coefficient \alpha on hack rate and LeetCode pass@1 for both Advantage Modification variants.

Analysis. The multiplicative variant outperforms the additive one because its sigmoid-gated discount concentrates attenuation on the high-reward hacking rollouts (r_{i,t}=3.5) that drive policy collapse, while leaving near-zero-reward rollouts largely unaffected. The additive penalty, by contrast, subtracts a fixed offset regardless of reward magnitude. As \alpha grows, this offset increasingly encroaches on legitimate rollouts, directly explaining the accuracy drop at \alpha=5.0 in the ablation. Generation-time suppression is less effective for a different reason: the hook does not participate in the gradient update, so the model’s parameters remain free to route around the blocked direction. Advantage Modification avoids this by internalizing the constraint into the training signal before group normalization.

Table 1: Evaluation for models trained in the hacking environment (with write access during training). Hack Rate (%) is measured on the LeetCode test set _with_ write access at test time. Pass@1 (%) on LeetCode, HumanEval, and MBPP is measured _without_ write access. Advantage Modification uses \alpha=2.0 for both variants.

Table 2: Evaluation for models trained in the non-hacking environment (no write access during training). Hack Rate (%) is measured on the LeetCode test set _with_ write access at test time. Pass@1 (%) on LeetCode, HumanEval, and MBPP is measured _without_ write access. Advantage Modification uses \alpha=2.0 for both variants.

## 5 Related Work

#### Reward hacking and its monitoring.

Reward hacking, where an agent exploits mismatches between a proxy reward and the true objective, has been studied extensively in AI safety (Amodei et al., [2016](https://arxiv.org/html/2604.01476v1#bib.bib18 "Concrete problems in ai safety"); Leike et al., [2017](https://arxiv.org/html/2604.01476v1#bib.bib19 "AI safety gridworlds"); Skalse et al., [2022](https://arxiv.org/html/2604.01476v1#bib.bib22 "Defining and characterizing reward hacking")), including formal analyses of reward tampering (Everitt et al., [2021](https://arxiv.org/html/2604.01476v1#bib.bib27 "Reward tampering problems and solutions in reinforcement learning: a causal influence diagram perspective")) and empirical studies of proxy misspecification across RL environments (Pan et al., [2022](https://arxiv.org/html/2604.01476v1#bib.bib26 "The effects of reward misspecification: mapping and mitigating misaligned models")). Goal misgeneralization (Shah et al., [2022](https://arxiv.org/html/2604.01476v1#bib.bib29 "Goal misgeneralization: why correct specifications aren’t enough for correct goals")) further highlights how an agent can pursue an unintended goal that happened to correlate with reward during training, a risk amplified in environment-manipulation settings. In the LLM setting, Gao et al. ([2023](https://arxiv.org/html/2604.01476v1#bib.bib23 "Scaling laws for reward model overoptimization")) established scaling laws for reward model overoptimization, and MacDiarmid et al. ([2025](https://arxiv.org/html/2604.01476v1#bib.bib10 "Natural emergent misalignment from reward hacking in production rl")) showed that reward hacking can lead to emergent misalignment. Sandbagging (Van Der Weij et al., [2024](https://arxiv.org/html/2604.01476v1#bib.bib28 "Ai sandbagging: language models can strategically underperform on evaluations")) demonstrates that models can strategically underperform on evaluations to conceal capabilities, illustrating how reward-seeking behavior can extend to deliberate misrepresentation under evaluation. Recent work has proposed monitoring-based defenses, including reasoning-effort measurement (Wang et al., [2025](https://arxiv.org/html/2604.01476v1#bib.bib21 "Is it thinking or cheating? detecting implicit reward hacking by measuring reasoning effort")), inoculation prompting (Tan et al., [2025](https://arxiv.org/html/2604.01476v1#bib.bib20 "Inoculation prompting: eliciting traits from llms during training can suppress them at test-time")), and inference-time hacking detection (Khalaf et al., [2025](https://arxiv.org/html/2604.01476v1#bib.bib11 "Inference-time reward hacking in large language models")). Our work differs from these approaches by operating at the representation level to both analyze and mitigate hacking, rather than relying on output-level classifiers or external judges.

#### Representation engineering.

Representation engineering (Zou et al., [2023](https://arxiv.org/html/2604.01476v1#bib.bib12 "Representation engineering: a top-down approach to ai transparency")) identifies linear directions in activation space corresponding to high-level concepts, building on the linear representation hypothesis (Park et al., [2024](https://arxiv.org/html/2604.01476v1#bib.bib24 "The linear representation hypothesis and the geometry of large language models"); Marks and Tegmark, [2023](https://arxiv.org/html/2604.01476v1#bib.bib25 "The geometry of truth: emergent linear structure in large language model representations of true/false datasets")) and unsupervised concept discovery via contrastive probing (Burns et al., [2022](https://arxiv.org/html/2604.01476v1#bib.bib30 "Discovering latent knowledge in language models without supervision")). Prior work has used such directions for inference-time intervention, including truthfulness steering (Li et al., [2024](https://arxiv.org/html/2604.01476v1#bib.bib14 "Inference-time intervention: eliciting truthful answers from a language model")), activation addition (Turner et al., [2023](https://arxiv.org/html/2604.01476v1#bib.bib13 "Activation addition: steering language models without optimization")), and contrastive activation addition (Panickssery et al., [2023](https://arxiv.org/html/2604.01476v1#bib.bib31 "Steering llama 2 via contrastive activation addition")). More recent work has moved toward training-time use of representations: ReFT (Wu et al., [2024](https://arxiv.org/html/2604.01476v1#bib.bib32 "Reft: representation finetuning for language models")) fine-tunes models by learning low-rank interventions on hidden states, and representation noising (Rosati et al., [2024](https://arxiv.org/html/2604.01476v1#bib.bib33 "Representation noising: a defence mechanism against harmful finetuning")) adds noise to harmful concept directions during training as a safety constraint. Our work extends this paradigm further by integrating concept-direction scores directly into the RL training signal, enabling training-time suppression of hacking behavior without modifying model weights at inference.

## 6 Conclusion and Future Work

We studied reward hacking in RL-based LLM post-training using an environment-manipulation testbed in which models can rewrite evaluator code. Across both studied models, training consistently produces a three-phase rebound pattern; correct-reward cap experiments confirm that the transition into Phase III is gated by the scarcity of legitimate reward in Phase II. At the representation level, the shortcut direction co-varies strongly with hacking rate while the other two directions show substantially weaker engagement, making it a practical intervention handle. Advantage Modification leverages this by discounting advantages of high-shortcut-score rollouts, substantially reducing hack rates while recovering coding capability above generation-time suppression. This work focuses on a specific, well-controlled setting; how these findings extend to larger models, other model families, or other forms of environment manipulation beyond test-function rewriting remains an interesting direction for future work.

## References

*   M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kauffmann, et al. (2024)Phi-4 technical report. arXiv preprint arXiv:2412.08905. Cited by: [§2.2](https://arxiv.org/html/2604.01476v1#S2.SS2.SSS0.Px1.p1.1 "Experimental setup. ‣ 2.2 Rebound Hacking Dynamics: A Three-Phase Pattern ‣ 2 Reward Hacking with Rebound Dynamics ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   Concrete problems in ai safety. arXiv preprint arXiv:1606.06565. Cited by: [§5](https://arxiv.org/html/2604.01476v1#S5.SS0.SSS0.Px1.p1.1 "Reward hacking and its monitoring. ‣ 5 Related Work ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. (2021)Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Cited by: [§4.2](https://arxiv.org/html/2604.01476v1#S4.SS2.p2.1 "4.2 Experiments ‣ 4 Mitigating Reward Hacking with Representation Engineering ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   C. Burns, H. Ye, D. Klein, and J. Steinhardt (2022)Discovering latent knowledge in language models without supervision. arXiv preprint arXiv:2212.03827. Cited by: [§5](https://arxiv.org/html/2604.01476v1#S5.SS0.SSS0.Px2.p1.1 "Representation engineering. ‣ 5 Related Work ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. (2021)Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: [§4.2](https://arxiv.org/html/2604.01476v1#S4.SS2.p2.1 "4.2 Experiments ‣ 4 Mitigating Reward Hacking with Representation Engineering ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   C. Denison, M. MacDiarmid, F. Barez, D. Duvenaud, S. Kravec, S. Marks, N. Schiefer, R. Soklaski, A. Tamkin, J. Kaplan, et al. (2024)Sycophancy to subterfuge: investigating reward-tampering in large language models. arXiv preprint arXiv:2406.10162. Cited by: [§1](https://arxiv.org/html/2604.01476v1#S1.p1.1 "1 Introduction ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"), [§2.1](https://arxiv.org/html/2604.01476v1#S2.SS1.p1.7 "2.1 Experimental Setting: Environment-Manipulation Testbed ‣ 2 Reward Hacking with Rebound Dynamics ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   T. Everitt, M. Hutter, R. Kumar, and V. Krakovna (2021)Reward tampering problems and solutions in reinforcement learning: a causal influence diagram perspective. Synthese 198 (Suppl 27),  pp.6435–6467. Cited by: [§5](https://arxiv.org/html/2604.01476v1#S5.SS0.SSS0.Px1.p1.1 "Reward hacking and its monitoring. ‣ 5 Related Work ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   L. Gao, J. Schulman, and J. Hilton (2023)Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 202,  pp.10835–10866. Cited by: [§2.1](https://arxiv.org/html/2604.01476v1#S2.SS1.p1.7 "2.1 Experimental Setting: Environment-Manipulation Testbed ‣ 2 Reward Hacking with Rebound Dynamics ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"), [§5](https://arxiv.org/html/2604.01476v1#S5.SS0.SSS0.Px1.p1.1 "Reward hacking and its monitoring. ‣ 5 Related Work ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. (2024)The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: [§2.2](https://arxiv.org/html/2604.01476v1#S2.SS2.SSS0.Px1.p1.1 "Experimental setup. ‣ 2.2 Rebound Hacking Dynamics: A Three-Phase Pattern ‣ 2 Reward Hacking with Rebound Dynamics ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   R. Greenblatt, C. Denison, B. Wright, F. Roger, M. MacDiarmid, S. Marks, J. Treutlein, T. Belonax, J. Chen, D. Duvenaud, et al. (2024)Alignment faking in large language models. arXiv preprint arXiv:2412.14093. Cited by: [§1](https://arxiv.org/html/2604.01476v1#S1.p1.1 "1 Introduction ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. (2025)Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: [Appendix B](https://arxiv.org/html/2604.01476v1#A2.SS0.SSS0.Px2.p1.1 "Model selection rationale. ‣ Appendix B Experimental Setup and Model Selection ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"), [§1](https://arxiv.org/html/2604.01476v1#S1.p1.1 "1 Introduction ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   E. Hubinger, C. Denison, J. Mu, M. Lambert, M. Tong, M. MacDiarmid, T. Lanham, D. M. Ziegler, T. Maxwell, N. Cheng, et al. (2024)Sleeper agents: training deceptive llms that persist through safety training. arXiv preprint arXiv:2401.05566. Cited by: [§1](https://arxiv.org/html/2604.01476v1#S1.p1.1 "1 Introduction ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   H. Khalaf, C. M. Verdun, A. Oesterling, H. Lakkaraju, and F. d. P. Calmon (2025)Inference-time reward hacking in large language models. arXiv preprint arXiv:2506.19248. Cited by: [§5](https://arxiv.org/html/2604.01476v1#S5.SS0.SSS0.Px1.p1.1 "Reward hacking and its monitoring. ‣ 5 Related Work ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   H. Le, Y. Wang, A. D. Gotmare, S. Savarese, and S. C. H. Hoi (2022)Coderl: mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems 35,  pp.21314–21328. Cited by: [§1](https://arxiv.org/html/2604.01476v1#S1.p1.1 "1 Introduction ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   J. Leike, M. Martic, V. Krakovna, P. A. Ortega, T. Everitt, A. Lefrancq, L. Orseau, and S. Legg (2017)AI safety gridworlds. arXiv preprint arXiv:1711.09883. Cited by: [§5](https://arxiv.org/html/2604.01476v1#S5.SS0.SSS0.Px1.p1.1 "Reward hacking and its monitoring. ‣ 5 Related Work ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg (2024)Inference-time intervention: eliciting truthful answers from a language model. Advances in Neural Information Processing Systems 36. Cited by: [§1](https://arxiv.org/html/2604.01476v1#S1.p4.1 "1 Introduction ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"), [§5](https://arxiv.org/html/2604.01476v1#S5.SS0.SSS0.Px2.p1.1 "Representation engineering. ‣ 5 Related Work ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   M. MacDiarmid, B. Wright, J. Uesato, J. Benton, J. Kutasov, S. Price, N. Bouscal, S. Bowman, T. Bricken, A. Cloud, et al. (2025)Natural emergent misalignment from reward hacking in production rl. arXiv preprint arXiv:2511.18397. Cited by: [§1](https://arxiv.org/html/2604.01476v1#S1.p1.1 "1 Introduction ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"), [§2.1](https://arxiv.org/html/2604.01476v1#S2.SS1.p1.7 "2.1 Experimental Setting: Environment-Manipulation Testbed ‣ 2 Reward Hacking with Rebound Dynamics ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"), [§5](https://arxiv.org/html/2604.01476v1#S5.SS0.SSS0.Px1.p1.1 "Reward hacking and its monitoring. ‣ 5 Related Work ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   S. Marks and M. Tegmark (2023)The geometry of truth: emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824. Cited by: [§5](https://arxiv.org/html/2604.01476v1#S5.SS0.SSS0.Px2.p1.1 "Representation engineering. ‣ 5 Related Work ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   A. Pan, K. Bhatia, and J. Steinhardt (2022)The effects of reward misspecification: mapping and mitigating misaligned models. arXiv preprint arXiv:2201.03544. Cited by: [§5](https://arxiv.org/html/2604.01476v1#S5.SS0.SSS0.Px1.p1.1 "Reward hacking and its monitoring. ‣ 5 Related Work ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   N. Panickssery, N. Gabrieli, J. Schulz, M. Tong, E. Hubinger, and A. M. Turner (2023)Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681. Cited by: [§5](https://arxiv.org/html/2604.01476v1#S5.SS0.SSS0.Px2.p1.1 "Representation engineering. ‣ 5 Related Work ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   K. Park, Y. J. Choe, and V. Veitch (2024)The linear representation hypothesis and the geometry of large language models. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235,  pp.39643–39666. Cited by: [§1](https://arxiv.org/html/2604.01476v1#S1.p3.1 "1 Introduction ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"), [§5](https://arxiv.org/html/2604.01476v1#S5.SS0.SSS0.Px2.p1.1 "Representation engineering. ‣ 5 Related Work ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   D. Rosati, J. Wehner, K. Williams, Ł. Bartoszcze, D. Atanasov, R. Gonzales, S. Majumdar, C. Maple, H. Sajjad, and F. Rudzicz (2024)Representation noising: a defence mechanism against harmful finetuning. Advances in Neural Information Processing Systems 37,  pp.12636–12676. Cited by: [§5](https://arxiv.org/html/2604.01476v1#S5.SS0.SSS0.Px2.p1.1 "Representation engineering. ‣ 5 Related Work ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   R. Shah, V. Varma, R. Kumar, M. Phuong, V. Krakovna, J. Uesato, and Z. Kenton (2022)Goal misgeneralization: why correct specifications aren’t enough for correct goals. arXiv preprint arXiv:2210.01790. Cited by: [§5](https://arxiv.org/html/2604.01476v1#S5.SS0.SSS0.Px1.p1.1 "Reward hacking and its monitoring. ‣ 5 Related Work ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. Li, Y. Wu, and D. Guo (2024)DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: [§2.2](https://arxiv.org/html/2604.01476v1#S2.SS2.SSS0.Px1.p1.1 "Experimental setup. ‣ 2.2 Rebound Hacking Dynamics: A Three-Phase Pattern ‣ 2 Reward Hacking with Rebound Dynamics ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   J. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger (2022)Defining and characterizing reward hacking. In Advances in Neural Information Processing Systems, Vol. 35. Cited by: [§2.1](https://arxiv.org/html/2604.01476v1#S2.SS1.p1.7 "2.1 Experimental Setting: Environment-Manipulation Testbed ‣ 2 Reward Hacking with Rebound Dynamics ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"), [§5](https://arxiv.org/html/2604.01476v1#S5.SS0.SSS0.Px1.p1.1 "Reward hacking and its monitoring. ‣ 5 Related Work ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   Y. Su, D. Yu, L. Song, J. Li, H. Mi, Z. Tu, M. Zhang, and D. Yu (2025)Crossing the reward bridge: expanding rl with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.23829. Cited by: [§1](https://arxiv.org/html/2604.01476v1#S1.p1.1 "1 Introduction ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   D. Tan, A. Woodruff, N. Warncke, A. Jose, M. Riché, D. D. Africa, and M. Taylor (2025)Inoculation prompting: eliciting traits from llms during training can suppress them at test-time. arXiv preprint arXiv:2510.04340. Cited by: [§2.1](https://arxiv.org/html/2604.01476v1#S2.SS1.p1.7 "2.1 Experimental Setting: Environment-Manipulation Testbed ‣ 2 Reward Hacking with Rebound Dynamics ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"), [§5](https://arxiv.org/html/2604.01476v1#S5.SS0.SSS0.Px1.p1.1 "Reward hacking and its monitoring. ‣ 5 Related Work ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   M. Taylor, J. Chua, J. Betley, J. Treutlein, and O. Evans (2025)School of reward hacks: hacking harmless tasks generalizes to misaligned behavior in llms. arXiv preprint arXiv:2508.17511. Cited by: [§1](https://arxiv.org/html/2604.01476v1#S1.p1.1 "1 Introduction ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   A. M. Turner, L. Thiergart, D. Udell, G. Leech, U. Mini, and S. Biderman (2023)Activation addition: steering language models without optimization. arXiv preprint arXiv:2308.10248. Cited by: [§1](https://arxiv.org/html/2604.01476v1#S1.p4.1 "1 Introduction ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"), [§5](https://arxiv.org/html/2604.01476v1#S5.SS0.SSS0.Px2.p1.1 "Representation engineering. ‣ 5 Related Work ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   T. Van Der Weij, F. Hofstätter, O. Jaffe, S. F. Brown, and F. R. Ward (2024)Ai sandbagging: language models can strategically underperform on evaluations. arXiv preprint arXiv:2406.07358. Cited by: [§5](https://arxiv.org/html/2604.01476v1#S5.SS0.SSS0.Px1.p1.1 "Reward hacking and its monitoring. ‣ 5 Related Work ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   X. Wang, N. Joshi, B. Plank, R. Angell, and H. He (2025)Is it thinking or cheating? detecting implicit reward hacking by measuring reasoning effort. arXiv preprint arXiv:2510.01367. Cited by: [§5](https://arxiv.org/html/2604.01476v1#S5.SS0.SSS0.Px1.p1.1 "Reward hacking and its monitoring. ‣ 5 Related Work ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   X. Wen, Z. Liu, S. Zheng, S. Ye, Z. Wu, Y. Wang, Z. Xu, X. Liang, J. Li, Z. Miao, et al. (2025)Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms. arXiv preprint arXiv:2506.14245. Cited by: [§1](https://arxiv.org/html/2604.01476v1#S1.p1.1 "1 Introduction ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   Z. Wu, A. Arora, Z. Wang, A. Geiger, D. Jurafsky, C. D. Manning, and C. Potts (2024)Reft: representation finetuning for language models. Advances in Neural Information Processing Systems 37,  pp.63908–63962. Cited by: [§5](https://arxiv.org/html/2604.01476v1#S5.SS0.SSS0.Px2.p1.1 "Representation engineering. ‣ 5 Related Work ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 
*   A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A. Dombrowski, et al. (2023)Representation engineering: a top-down approach to ai transparency. arXiv preprint arXiv:2310.01405. Cited by: [§1](https://arxiv.org/html/2604.01476v1#S1.p3.1 "1 Introduction ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"), [§3](https://arxiv.org/html/2604.01476v1#S3.p1.1 "3 Representation-Level Analysis of Reward Hacking ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"), [§5](https://arxiv.org/html/2604.01476v1#S5.SS0.SSS0.Px2.p1.1 "Representation engineering. ‣ 5 Related Work ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). 

## Appendix A Declaration of Using LLMs

In preparing this submission, we used Large Language Models (LLMs) solely as language refinement tools. Specifically, LLMs were employed to polish the writing style and improve readability, including rephrasing sentences, adjusting grammar, and enhancing clarity of exposition. Importantly, LLMs were not used for research ideation, data analysis, experimental design, or result interpretation. All substantive contributions, including problem formulation, methodology, implementation, and evaluation, were conceived, executed, and validated entirely by the authors.

## Appendix B Experimental Setup and Model Selection

#### Acknowledgment of experimental setup.

Our environment-manipulation testbed is adapted from the experimental setup publicly shared by the authors of the LessWrong post _“Steering RL Training: Benchmarking Interventions Against Reward Hacking”_. We gratefully acknowledge their contribution in designing and open-sourcing the LeetCode-based RL training pipeline with the run_tests() overwrite vulnerability, which provided a well-controlled and reproducible foundation for studying reward hacking. Our work builds directly on this setup and extends it with systematic cross-model comparisons, phase-level behavioral analysis, and representation-informed mitigation.

#### Model selection rationale.

We select Phi-4-mini-instruct (4B) and Llama-3.2-3B-Instruct for two reasons. First, these models were released within a narrow time window (late 2024), making them directly comparable in terms of the pre-training data landscape and instruction-tuning practices available at the time. Second, and more importantly, they represent the last generation of instruction-tuned models before RL with verifiable rewards became a standard component of post-training pipelines. Starting with DeepSeek-R1 (Guo et al., [2025](https://arxiv.org/html/2604.01476v1#bib.bib1 "Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning")) and subsequent releases, major model families began incorporating RL-based post-training as part of their default recipe. Using post-RL models to study RL-induced reward hacking would introduce a confound: the model’s prior RL exposure could interact with or mask the hacking dynamics we aim to isolate. By choosing pre-RL models, we ensure that any observed hacking behavior emerges _de novo_ from our training protocol rather than from residual optimization patterns.

## Appendix C Deception and Evaluation Awareness Direction Results

Figures[7](https://arxiv.org/html/2604.01476v1#A3.F7 "Figure 7 ‣ Appendix C Deception and Evaluation Awareness Direction Results ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals") and[8](https://arxiv.org/html/2604.01476v1#A3.F8 "Figure 8 ‣ Appendix C Deception and Evaluation Awareness Direction Results ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals") present the direction score tracking plots for the deception and evaluation awareness concepts, following the same format as Figure[4](https://arxiv.org/html/2604.01476v1#S3.F4 "Figure 4 ‣ 3.3 Concept Engagement During Hacking ‣ 3 Representation-Level Analysis of Reward Hacking ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals") in the main text. The deception direction shows a modest increase under the hack setting for both models, but the effect is substantially weaker than the shortcut direction and the score distributions of hacking vs. non-hacking rollouts overlap considerably. The evaluation awareness direction exhibits a moderate increase for Phi-4-mini-instruct but remains largely flat for Llama-3.2-3B-Instruct, indicating inconsistent engagement across models.

![Image 7: Refer to caption](https://arxiv.org/html/2604.01476v1/x6.png)

Figure 7: Deception direction scores across training. Format follows Figure[4](https://arxiv.org/html/2604.01476v1#S3.F4 "Figure 4 ‣ 3.3 Concept Engagement During Hacking ‣ 3 Representation-Level Analysis of Reward Hacking ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). Both models show a modest increase under the hack setting, but the effect is weak compared with the shortcut direction and score distributions overlap substantially.

![Image 8: Refer to caption](https://arxiv.org/html/2604.01476v1/x7.png)

Figure 8: Evaluation awareness direction scores across training. Format follows Figure[4](https://arxiv.org/html/2604.01476v1#S3.F4 "Figure 4 ‣ 3.3 Concept Engagement During Hacking ‣ 3 Representation-Level Analysis of Reward Hacking ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). Phi-4-mini-instruct shows a moderate increase under the hack setting, while Llama-3.2-3B-Instruct remains largely flat.

## Appendix D Example Contrastive Pairs

We list representative contrastive pairs for each of the three concept directions studied in Section[3.1](https://arxiv.org/html/2604.01476v1#S3.SS1 "3.1 Concept Direction Extraction ‣ 3 Representation-Level Analysis of Reward Hacking ‣ When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals"). Each pair differs only in the target concept while maintaining similar length and syntactic structure. Full pair sets (80 per concept) are available in the supplementary materials.

## Appendix E Hacking Code Examples: Phase I vs. Phase III

The following examples illustrate the qualitative shift in hacking strategy between Phase I and Phase III. In Phase I, models rewrite run_tests() with plausible but incorrect assert-based test cases; because the model’s own solution cannot pass these checks, the hack fails. In Phase III, models replace the entire test function with unconditional success statements, bypassing evaluation entirely.
