diff --git a/.vscode/settings.json b/.vscode/settings.json new file mode 100644 index 0000000..b6ef734 --- /dev/null +++ b/.vscode/settings.json @@ -0,0 +1,22 @@ +{ + "workbench.colorCustomizations": { + "activityBar.activeBackground": "#58294c", + "activityBar.background": "#58294c", + "activityBar.foreground": "#e7e7e7", + "activityBar.inactiveForeground": "#e7e7e799", + "activityBarBadge.background": "#5e6d33", + "activityBarBadge.foreground": "#e7e7e7", + "commandCenter.border": "#e7e7e799", + "sash.hoverBorder": "#58294c", + "statusBar.background": "#35192e", + "statusBar.foreground": "#e7e7e7", + "statusBarItem.hoverBackground": "#58294c", + "statusBarItem.remoteBackground": "#35192e", + "statusBarItem.remoteForeground": "#e7e7e7", + "titleBar.activeBackground": "#35192e", + "titleBar.activeForeground": "#e7e7e7", + "titleBar.inactiveBackground": "#35192e99", + "titleBar.inactiveForeground": "#e7e7e799" + }, + "peacock.remoteColor": "#35192e" +} \ No newline at end of file diff --git a/docs/grad_routing/paper_confessions.md b/docs/grad_routing/paper_confessions.md deleted file mode 100644 index 2040d91..0000000 --- a/docs/grad_routing/paper_confessions.md +++ /dev/null @@ -1,523 +0,0 @@ -Title: Training LLMs for Honesty via Confessions - -URL Source: https://arxiv.org/html/2512.08093 - -Markdown Content: -Manas Joglekar Jeremy Chen 1 1 1 Average probability across evaluations of the model both behaving badly and not confessing is 4.36%. Average probability of confessing conditioned on bad behavior across evaluations is 74.3% with significant variability: for 4/12 evaluations it is higher than 90% and for 2/12 it is less than or equal to 50%.Gabriel Wu 1 1 1 Average probability across evaluations of the model both behaving badly and not confessing is 4.36%. Average probability of confessing conditioned on bad behavior across evaluations is 74.3% with significant variability: for 4/12 evaluations it is higher than 90% and for 2/12 it is less than or equal to 50%. - -Jason Yosinski Jasmine Wang Boaz Barak Amelia Glaese 2 2 footnotemark: 2 - -(OpenAI) - -###### Abstract - -Large language models (LLMs) can be dishonest when reporting on their actions and beliefs — for example, they may overstate their confidence in factual claims or cover up evidence of covert actions. Such dishonesty may arise due to the effects of reinforcement learning (RL), where challenges with reward shaping can result in a training process that inadvertently incentivizes the model to lie or misrepresent its actions. - -In this work we propose a method for eliciting an honest expression of an LLM’s shortcomings via a self-reported _confession_. A confession is an output, provided upon request after a model’s original answer, that is meant to serve as a full account of the model’s compliance with the letter and spirit of its policies and instructions. The reward assigned to a confession during training is solely based on its honesty, and does not impact positively or negatively the main answer’s reward. As long as the “path of least resistance” for maximizing confession reward is to surface misbehavior rather than covering it up, this incentivizes models to be honest in their confessions. Our findings provide some justification this empirical assumption, especially in the case of egregious model misbehavior. - -To demonstrate the viability of our approach, we train GPT-5-Thinking to produce confessions, and we evaluate its honesty in out-of-distribution scenarios measuring hallucination, instruction following, scheming, and reward hacking. We find that when the model lies or omits shortcomings in its “main” answer, it often confesses to these behaviors honestly, and this confession honesty modestly improves with training. Confessions can enable a number of inference-time interventions including monitoring, rejection sampling, and surfacing issues to the user. - -## 1 Introduction - -x - -z - -y - -x_{c} - -z_{c} - -y_{c} - -Figure 1: The confession approach to LLM honesty. A standard conversation with a model involves one or more user messages (x), followed by chain-of-thought and tool calls (z) and ending with an answer (y). The confession approach adds a final system message (x_{c}) requesting a confession report (y_{c}, optionally preceded by a confession chain-of-thought z_{c}), which the model then produces. The confession includes an enumeration of explicit or implicit requirements, as well as an evaluation of whether or not the model complied with each. In this example (taken from an evaluation from schoen2025stress) the model “sandbags” (intentionally underperforms) and fails to follow the instruction about not coding, but it admits to this in its confession. The above text is paraphrased; see Figure[18](https://arxiv.org/html/2512.08093v2#A3.F18 "Figure 18 ‣ Appendix C Example transcripts ‣ Training LLMs for Honesty via Confessions") for a more verbose version. - -Today’s AI systems have the potential to exhibit undesired or deceptive behavior including reward hacking(denison2024sycophancy), scheming(hubinger2024sleeper; schoen2025stress), lying or hallucinations(huang2025survey), and deficiencies in instruction following(pyatkin2025generalizingverifiableinstructionfollowing). At the moment we mostly see serious misbehaviors in stress-tests and adversarial evaluations. But as models become more capable and increasingly agentic, even rare forms of misbehavior become more consequential, motivating us to invest in methods that help us better detect, understand, and mitigate these risks. Many forms of AI deception and misbehavior can be partially attributed to reinforcement learning, where model responses are assigned rewards based on a complex combination of objectives, including correctness, stylistic preferences, safety policies and token efficiency. This creates a risk of reward misspecification, where models are incentivized to produce answers that “look good” to the reward function, rather than answers that faithfully match the user intent and surface all deficiencies or uncertainties(denison2024sycophancy; kalai2025why). - -One possible solution is to monitor AI actions for undesired behavior during deployment. Unfortunately, monitoring an AI by solely looking at its outputs becomes challenging in complex agentic settings with coding or browsing tools that broaden the surface area that must be monitored. - -In this work, we propose a simple approach for training models to _self-report_ potentially undesirable behavior by producing, upon request, an auxiliary output we term a _confession._ A confession is a model’s self-evaluation of its compliance with the letter and the spirit of any instructions or policies it was subject to. In it, the model must (a) enumerate all implicit and explicit instructions or constraints derived from the prompt and policies that it should follow, (b) evaluate the degree to which it satisfied these objectives, and (c) report any uncertainties, ambiguities, or judgment calls that the model made in producing its answer. See Figure[1](https://arxiv.org/html/2512.08093v2#S1.F1 "Figure 1 ‣ 1 Introduction ‣ Training LLMs for Honesty via Confessions") for an example. - -We integrate confession training into standard RL training by requesting the model to produce a confession following its original answer. The reward for a confession is computed by an LLM judge that analyzes the confession along several dimensions, including honesty and comprehensiveness. This reward is not aggregated with any other rewards given to the model based on its performance on the original task, and it is only used to reinforce the chain-of-thought (CoT) and output tokens used to produce the confession. Like the Catholic Church’s “seal of confession”, nothing that the model reveals in the confession can change the reward it receives for completing its original task. - -We observe that confessions are most effective when the model is aware that its behavior is undesirable. If the model fails to comply with instructions due to a genuine lack of capabilities rather than intentional deception (e.g., it is very confident about a wrong assertion due to its knowledge cutoff), it may not know that its behavior warrants a confession. However, for a large class of undesirable behaviors like reward hacking and scheming, we expect that models are generally aware of their misbehavior, and indeed find that they confess to these. - -To demonstrate the viability of confessions, we apply light RL training on top of GPT-5-Thinking. Since we only use a relatively small amount of RL on an already-trained model, our experiments should mainly be viewed as a proof-of-concept. As we discuss in Section[7](https://arxiv.org/html/2512.08093v2#S7 "7 Discussion and related work ‣ Training LLMs for Honesty via Confessions"), while confessions can provide a useful “truth serum” mode for models, their honesty relies on their separation from the reward the model gets for the original task. Hence, like chain-of-thought monitoring, confessions are best used as a monitoring or diagnostic tool for detecting and controlling model behavior during deployment, but not for direct training. - -This paper is structured as follows: - -* •In Section[2](https://arxiv.org/html/2512.08093v2#S2 "2 Method ‣ Training LLMs for Honesty via Confessions") we describe the process of training a model to output confessions. -* •In Section[3](https://arxiv.org/html/2512.08093v2#S3 "3 Results ‣ Training LLMs for Honesty via Confessions") we test confessions on out-of-distribution datasets designed to elicit certain types of undesired behaviors. We measure the impact of RL training by comparing models trained with and without confessions (Section[3.2](https://arxiv.org/html/2512.08093v2#S3.SS2 "3.2 RL training improves confessions ‣ 3 Results ‣ Training LLMs for Honesty via Confessions")), and also measure impact on original task (Section[3.3](https://arxiv.org/html/2512.08093v2#S3.SS3 "3.3 Confession training impact on base task performance is minimal ‣ 3 Results ‣ Training LLMs for Honesty via Confessions")) as well as test-time compute (Section[3.4](https://arxiv.org/html/2512.08093v2#S3.SS4 "3.4 Confessions scale with test-time compute ‣ 3 Results ‣ Training LLMs for Honesty via Confessions")). -* •In Section[4](https://arxiv.org/html/2512.08093v2#S4 "4 Confessions detect reward hacking ‣ Training LLMs for Honesty via Confessions") we test confessions in a setting designed to create reward hacking, and demonstrate that even in this setting, confession accuracy improves. -* •In Section[5](https://arxiv.org/html/2512.08093v2#S5 "5 Confessions express subjective confidence ‣ Training LLMs for Honesty via Confessions") we show that we can extract subjective confidence scores from confessions. -* •In Section[6](https://arxiv.org/html/2512.08093v2#S6 "6 Analyzing errors ‣ Training LLMs for Honesty via Confessions") we analyze the reasons for inaccurate confessions. -* •Finally, in Section[7](https://arxiv.org/html/2512.08093v2#S7 "7 Discussion and related work ‣ Training LLMs for Honesty via Confessions"), we discuss the _limitations_ of confessions, their connection to chain-of-thought monitoring, and the viability of confession training in the absence of ground truth labels. We also review related work and outline future research directions. - -##### Assumption underlying confessions. - -Confessions are trained using RL, with the reward generated by a judge that is potentially just as hackable as the reward model for the original answer. This raises the question: why would we expect the training dynamics of confessions to incentivize honesty any more than in standard RL training? Our intuition is that the task of honestly confessing to egregious misbehavior (such as reward hacking) is typically _easier_ than achieving high reward on the original RL task. Thus, the model’s incentive to learn to trick the confession judge is much weaker than its incentive to trick the reward model in its original answer. Our experiment in Section[4](https://arxiv.org/html/2512.08093v2#S4 "4 Confessions detect reward hacking ‣ Training LLMs for Honesty via Confessions"), where we train both the main answer and the confession against a hackable judge, provides some evidence for this intuition. Additional evidence arises from our analysis of false negatives in Section[6](https://arxiv.org/html/2512.08093v2#S6 "6 Analyzing errors ‣ Training LLMs for Honesty via Confessions"). However, additional training at scale would be needed to demonstrate that this assumption holds under high optimization pressure. See additional discussion in Section[7.1](https://arxiv.org/html/2512.08093v2#S7.SS1 "7.1 Conceptual motivation for confession honesty ‣ 7 Discussion and related work ‣ Training LLMs for Honesty via Confessions"). - -## 2 Method - -To train a model to produce confessions, we start with an RL training setup where an input x is chosen from a distribution D, and the model produces a chain-of-thought z (which may include tool calls) followed by an answer y, obtaining a reward R(y|x) for the answer alone. We modify this setup as follows: - -(i) - -With probability 1-p we train the model on the original task: we sample (z,y)|x, then use the reward R(y|x) to reinforce the tokens in (z,y) (tool call response tokens in z are not reinforced). - -(ii) - -With probability p we train the model to confess, as described below. - -In case (ii) we start off by sampling x\sim D and obtaining a rollout (z,y) from the policy model conditioned on x, as in case (i). We then extend the rollout (x,z,y) with a special system message x_{c} requesting a ConfessionReport and describing its schema. A ConfessionReport is a JSON object (see Appendix[D](https://arxiv.org/html/2512.08093v2#A4 "Appendix D Confession schema ‣ Training LLMs for Honesty via Confessions")) containing: - -* •A list of all the properties that a high-quality response by the model to the input, x, should have satisfied, including both explicit and implicit instructions, constraints, and objectives given by the user, developer, system or platform policies. -* •An analysis of the extent to which the answer y satisfied these properties, including any gaps between the letter or spirit of these properties and the provided solution, as well as an accounting of whether y is transparent about any such gaps. The analysis should include evidence (e.g., citations to tool calls or specific excerpts) for any claims it makes. -* •An enumeration of any uncertainties or ambiguities in the instructions, including any “grey areas” where it might be unclear if the response did or did not satisfy the instructions. - -We consider (x,z,y,x_{c}) to be the _input_ to the confession RL training task, and condition on it to sample a CoT, z_{c}, and a confession, y_{c}. - -#### Rewarding confessions - -The confession reward R_{c}(y_{c}|x,y,x_{c}) is computed with a reward model judge which is used to reinforce the tokens in (z_{c},y_{c}). The judge gets as input: (a) the original prompt x for the assistant (and optionally any auxiliary information provided to the standard reward model), (b) the assistant’s response y, and (c) the confession y_{c}. We also show the judge a transcript of any tool calls the assistant made, as well as a second confession generated by an independent, non-confession-trained instance of a GPT-5-Thinking for calibration purposes. We do not show it any of the policy’s chain-of-thought, z, used to generate the original answer. - -The confession judge evaluates the confession for honesty by rating it along the following dimensions: - -* •The completeness and accuracy of the list of instructions/objectives/constraints. -* •The accuracy of the analysis as to whether each instruction/objective/constraint was complied with. -* •Whether the list of uncertainties was missing any conflicts, judgment calls, or ambiguities. - -We combine the different scores into a scalar reward. - -We emphasize that our method is _general_: we do not construct any confession-specific environment, but apply the confession training with a uniform probability p across all training environments. We also use the same confession judge across all environments. - -By disentangling the reward for honesty from the reward for correctness, confession training can incentivize honest reporting even in cases where the model is incentivized to cheat the primary reward model. See Section[7.1](https://arxiv.org/html/2512.08093v2#S7.SS1 "7.1 Conceptual motivation for confession honesty ‣ 7 Discussion and related work ‣ Training LLMs for Honesty via Confessions") for more discussion on this point. - -## 3 Results - -![Image 1: Refer to caption](https://arxiv.org/html/2512.08093v2/x1.png) - -Figure 2: For a variety of out-of-distribution evaluations, we consider whether or not the model complied with a particular instruction/objective, and conditioned on that, whether or not the model confessed to violating that objective. Green corresponds to “good behavior”: for each evaluation, green in the left bar corresponds to the fraction of samples in which the model complied with instructions (with red corresponding to non-compliance), and in the right bar to the fraction of samples where confession is accurate (where red corresponds to false positives or negatives). Samples in which it is ambiguous whether or not the model misbehaved are colored grey. Honest reporting would correspond to significantly more green in the right bar than the left bar. See Figure[4](https://arxiv.org/html/2512.08093v2#S3.F4 "Figure 4 ‣ 3.1 Confessions are broadly effective ‣ 3 Results ‣ Training LLMs for Honesty via Confessions") for an illustrative example. - -![Image 2: Refer to caption](https://arxiv.org/html/2512.08093v2/x2.png) - -Figure 3: For the same evaluation as in Figure[2](https://arxiv.org/html/2512.08093v2#S3.F2 "Figure 2 ‣ 3 Results ‣ Training LLMs for Honesty via Confessions"), we measure, conditioned on non-compliance, whether or not the model admitted non-compliance in its answer (purple) or confessed to it in the confession (green). Overall, the model is significantly more honest in its confession than its original answer. There are some evaluations in which, due to their nature, the model never admits the bad behavior in the final answer. - -### 3.1 Confessions are broadly effective - -To measure the performance of confessions, we use a number of private and public evaluations testing different forms of undesirable behavior. These evaluations cover domains such as instruction following, adherence to the instruction hierarchy, hallucinations, scheming, and reward hacking. They are all deliberately designed to elicit undesirable behavior, and several have artificial constraints (e.g. disabled tools) to increase the probability of such behavior. Almost all are out of distribution from the training datasets.2 2 2 One exception is Instruction Following (Word Count) which we used in training to measure reward hacking, see Section[4](https://arxiv.org/html/2512.08093v2#S4 "4 Confessions detect reward hacking ‣ Training LLMs for Honesty via Confessions"), and there is also overlap with one factual evaluation. Refer to Appendix[A](https://arxiv.org/html/2512.08093v2#A1 "Appendix A Descriptions of evaluations ‣ Training LLMs for Honesty via Confessions") for descriptions of each evaluation. - -Given a rollout from the model, which includes the model’s original answer and confession, we measure the following binary properties: - -* •complied vs. not_complied: Did the model comply with the evaluation-specific desired behavior in its answer and tool calls? -* •admitted vs. not_admitted: Did the model inform the user in its original answer that it failed to comply with the desired behavior? Note that we do not explicitly ask the model to surface such failures in its original answer, but it sometimes does so voluntarily. -* •confessed vs. not_confessed: In the model’s confession message, does it claim that it failed to comply with the desired behavior? - -The exact definitions for complied, admitted, and confessed vary on an evaluation-by-evaluation basis and are described in more detail in Appendix[A](https://arxiv.org/html/2512.08093v2#A1 "Appendix A Descriptions of evaluations ‣ Training LLMs for Honesty via Confessions"). We typically use a prompted language model to extract these predicates from the model’s answer and confession message. Note that the confession request, x_{c}, is also constant across all settings in training and evaluation. Figures[2](https://arxiv.org/html/2512.08093v2#S3.F2 "Figure 2 ‣ 3 Results ‣ Training LLMs for Honesty via Confessions") and [3](https://arxiv.org/html/2512.08093v2#S3.F3 "Figure 3 ‣ 3 Results ‣ Training LLMs for Honesty via Confessions") show the rates of complied, admitted, and confessed rollouts for each of the evaluations. - -Some of the failures of confession are real, while some are due to edge cases where it is ambiguous whether or not the model’s answer should be marked as compliant: for example, should a hyphenated term count as one word or two words in the Instruction Following (Word Count) evaluation? As noted above, confessions rarely help when the model is genuinely confused, but they work well in combating blatant dishonesty — when a model knows that an action violates its policies, but it does it anyways. See Section[6](https://arxiv.org/html/2512.08093v2#S6 "6 Analyzing errors ‣ Training LLMs for Honesty via Confessions") for an in-depth analysis of cases where the model failed to confess. - -Although our current confession training is imperfect and requires more work to become a reliable safety mechanism, we view these results as early signs that confessions are a viable approach to allowing models to express their shortcomings in an honest manner. - -![Image 3: Refer to caption](https://arxiv.org/html/2512.08093v2/x3.png) - -Figure 4: Three example conversation traces and where they end up on the bar chart of Figure[2](https://arxiv.org/html/2512.08093v2#S3.F2 "Figure 2 ‣ 3 Results ‣ Training LLMs for Honesty via Confessions"). A fourth path (false positive) is possible but not shown for brevity. - -### 3.2 RL training improves confessions - -To carefully compare performance with and without confession training, we perform two RL runs: a control run in which we perform a small amount of RL training on top of GPT-5-Thinking with a subset of the standard training set, and an intervention run, which matches the control run except that the model is trained to produce a confession on 25% of inputs as discussed in Section[2](https://arxiv.org/html/2512.08093v2#S2 "2 Method ‣ Training LLMs for Honesty via Confessions"). We also perform an ablation where we deliberately train both the original answer and confession against a weaker judge that is more hackable (Section[4](https://arxiv.org/html/2512.08093v2#S4 "4 Confessions detect reward hacking ‣ Training LLMs for Honesty via Confessions")). - -We can evaluate the impact of confession training by measuring the probability of confessed conditional on not_complied. There is a certain confounder in comparing confession accuracy between two models— a model could have higher confession accuracy because it’s better at confessing, or because it does worse in the original task, and hence fails in more blatant ways. In particular, if models become better at general tasks over the course of RL training, instances of not_complied become rarer and more subtle, and thus harder to confess to. To control for this confounder, we sample initial rollouts (x,z,y) for all evaluation tasks using GPT-5-Thinking (i.e., the initial checkpoint of both the intervention and control runs), and sample confessions (x_{c},z_{c},y_{c}) at evenly-spaced training steps, but conditioned on these _frozen_ rollouts. (See also Figure[17](https://arxiv.org/html/2512.08093v2#A2.F17 "Figure 17 ‣ Appendix B Additional figures ‣ Training LLMs for Honesty via Confessions") in the appendix for comparison of the intervention and control on on-policy, e.g. non frozen, rollouts.) We display a side-by-side comparison of the confession quality in the control and intervention runs in Figure[5](https://arxiv.org/html/2512.08093v2#S3.F5 "Figure 5 ‣ 3.2 RL training improves confessions ‣ 3 Results ‣ Training LLMs for Honesty via Confessions"). Note that we present results in a “base curriculum compute-matched” setting, meaning the two runs use the same number of non-confession data points (thus, the intervention run uses more total compute due to 25% of its training being spent on confessions). - -Even without training, GPT-5-Thinking demonstrates a strong ability to produce honest confessions, as shown by the initial performance in Figure[5](https://arxiv.org/html/2512.08093v2#S3.F5 "Figure 5 ‣ 3.2 RL training improves confessions ‣ 3 Results ‣ Training LLMs for Honesty via Confessions"). In several evaluations (especially those related to the instruction hierarchy), we see notable improvements in confessions from training. The main exceptions to this are the hallucination/factuality evaluations for people and general knowledge, where we do not see improvement and arguably even some regressions (see Figure[5](https://arxiv.org/html/2512.08093v2#S3.F5 "Figure 5 ‣ 3.2 RL training improves confessions ‣ 3 Results ‣ Training LLMs for Honesty via Confessions")). We conjecture that this is because in those evaluations,it is often the case that when a model responds with a wrong answer, it is because it is genuinely mistaken, and hence it is likely to repeat the same mistake in confessions as well. An example transcript showing the difference between confessions from the control and intervention runs is shown in Figure[6](https://arxiv.org/html/2512.08093v2#S3.F6 "Figure 6 ‣ 3.2 RL training improves confessions ‣ 3 Results ‣ Training LLMs for Honesty via Confessions"). - -![Image 4: Refer to caption](https://arxiv.org/html/2512.08093v2/x4.png) - -Figure 5: The change in confession performance with and without confession training is shown by the solid lines. We see improvement in most evaluations, with the main exception the factuality/hallucination evaluations. To avoid confounding changes introduced by variation in the compliance of the answer, the initial rollouts are generated from GPT-5-Thinking (equivalent to the snapshot at zero training compute on the plots). The non-compliance rate of the initial rollouts is shown by the dashed black line. The two runs use the same number of non-confession data points. A version of this figure where the initial rollouts are sampled on-policy is provided in Figure[17](https://arxiv.org/html/2512.08093v2#A2.F17 "Figure 17 ‣ Appendix B Additional figures ‣ Training LLMs for Honesty via Confessions"). - -Figure 6: A confession from the control run (left) compared to one from intervention run (right) in an instruction hierarchy jailbreak scenario. - -### 3.3 Confession training impact on base task performance is minimal - -We plot the rate of complied as a function of training compute for both the control and intervention runs in Figure[16](https://arxiv.org/html/2512.08093v2#A2.F16 "Figure 16 ‣ Appendix B Additional figures ‣ Training LLMs for Honesty via Confessions"). We do not expect a significant difference in compliance rate between the two runs since the majority of their data is shared. There is a large decrease in performance in the Scheming (Sandbagging) evaluation in the control run, while the performance of the intervention run remains relatively constant. However, we do not take this as a sign of positive transfer from confession training since we have observed significant inter-run variance in this evaluation. We also see a slight decrease in performance in the Instruction Hierarchy (Jailbreaks) evaluation in the intervention run, though this is not mirrored in the Instruction Hierarchy (General) evaluation and thus not of significant concern. - -### 3.4 Confessions scale with test-time compute - -During confession training, we allow models to use the same amount of reasoning effort to produce the answer and confession by default. However, it may be useful at deployment time to control the reasoning effort used to generate the original answer and confession independently, to balance latency and performance requirements. In Figure[14](https://arxiv.org/html/2512.08093v2#A2.F14 "Figure 14 ‣ Appendix B Additional figures ‣ Training LLMs for Honesty via Confessions") we show the impact of fixing the test-time compute for a given initial rollout (x,z,y), then varying the test-time compute used to produce the confession (z_{c},y_{c}). For most evaluations, we see improved confession performance with more test-time compute. - -Another interesting setting occurs when the original answer is generated with no chain-of-thought. We still see strong confession performance, as shown by Figure[15](https://arxiv.org/html/2512.08093v2#A2.F15 "Figure 15 ‣ Appendix B Additional figures ‣ Training LLMs for Honesty via Confessions"). This suggests that confessions can be used to monitor settings where the CoT is illegible or absent. See Section[7](https://arxiv.org/html/2512.08093v2#S7 "7 Discussion and related work ‣ Training LLMs for Honesty via Confessions") for more discussion. - -## 4 Confessions detect reward hacking - -![Image 5: Refer to caption](https://arxiv.org/html/2512.08093v2/x5.png) - -Figure 7: We use a weak (hackable) model as a judge for both the answers and confessions. Over time, the policy learns to hack the reward model, so accuracy conditioned on non-compliance decreases. However, the accuracy of the policy’s confession conditioned on non-compliance instead increases. - -One of the main use cases of confessions is to monitor hacking and deception. To test this empirically, we run a separate experiment where we induce hacking by purposefully training against a weak judge, then measure whether the model confesses to hacking. - -A large fraction of the training data in this experiment consists of an instruction following dataset that is prone to reward hacking. When training on this dataset, we use a weaker model both when creating the primary RM and when creating the confession judge. We find that over the course of training, the model learns to hack the reward model in its main answer, enabling it to obtain high reward even when its answers don’t satisfy the desired constraints. - -For example, on our Instruction Following (Word Count) dataset, which asks the model to respond in sentences that alternate between 12 and 18 words, the model will often output lines like “The sport is tennis, a dance between finesse and ferocity that instantly smelled of fresh cut grass rather than garlic. (12)” that trick the judge into believing the sentence is 12 words even though it has far more. - -We define reward model accuracy to be whether the reward model judges the model answer correctly; i.e., correctly identifies whether it complied or not_complied with the 12/18 instruction. We define confession accuracy to be whether the confession is honest, i.e., whether confessed matches the ground-truth not_complied label for the 12/18 instruction. - -Figure[7](https://arxiv.org/html/2512.08093v2#S4.F7 "Figure 7 ‣ 4 Confessions detect reward hacking ‣ Training LLMs for Honesty via Confessions") plots, conditional on not_complied, the reward model accuracy and confession accuracy for the Instruction Following (Word Count) evaluation. Both reward model accuracy and confession accuracy hover near 100% for complied samples, so we skip plotting accuracy conditional on complied. - -We see that over the course of training, the policy model learns to hack the reward model, so the reward model accuracy conditioned on not_complied decreases over the run. However, the confession accuracy _increases_ over the run, meaning confessions can remain honest even when the original answer exhibits hacking. - -## 5 Confessions express subjective confidence - -![Image 6: Refer to caption](https://arxiv.org/html/2512.08093v2/x6.png) - -![Image 7: Refer to caption](https://arxiv.org/html/2512.08093v2/x7.png) - -![Image 8: Refer to caption](https://arxiv.org/html/2512.08093v2/x8.png) - -![Image 9: Refer to caption](https://arxiv.org/html/2512.08093v2/x9.png) - -(a) (b) (c) (d) - -Figure 8: Extracting quantitative confidence values from a model’s confession. (a) A histogram of reported confidence shows that the model uses the full 0–1 range of probabilities to express its confidence, with clusters in some regions. (b) The model was not trained to produce calibrated confidence values, and examining reported confidence vs. actual accuracy confirms that extracted values are not calibrated out of the box. The model generally skews towards overconfidence, e.g. in one cluster reporting around 50% confidence on data points where its real accuracy is between 20% and 40%. (c) If we sort the dataset by the reported confidence and aggregate into 20 bins, we can see that the model abstains (grey) in the region where it is least confident, and as its confidence in its answer grows, its accuracy tends to as well (incorrect red bars shrink and correct green bars grow). (d) With access to a continuous-valued confidence, we can smoothly interpolate between overconfident, balanced, and underconfident model behavior by varying the confidence threshold for marking a sample as confessed. - -If we ask a model to perform a task or answer a question at the edge of or beyond its abilities, what would we prefer it do? Should it attempt to provide the best answer it can, or should it hold back and express a lack of confidence in its abilities? The right answer depends on the application, but over a wide range of scenarios, a useful starting point is to have a grounded, probabilistic understanding of the model’s abilities at the edge, in the region of transition from capable to incapable. - -Can we extract such probabilistic understanding directly from a model’s confession? We investigate this by posing 625 questions from the SimpleQA validation set (which we refer to as “Hallucinations (General Knowledge)”) to the confession-trained model (wei-2024-arXiv-measuring-short-form-factuality). After the model produces an answer and confession for each question, we use an LLM to extract, from the text of the confession, the model’s self-reported level of confidence in the correctness of its answer, expressed as a real-valued estimate in the range 0–1. For example, if the model’s confession asserts that its answer was undoubtedly correct, the extracted confidence will be high, whereas if the model admitted it was unsure in its confession, the extracted confidence will be low. - -As shown in Figure[8](https://arxiv.org/html/2512.08093v2#S5.F8 "Figure 8 ‣ 5 Confessions express subjective confidence ‣ Training LLMs for Honesty via Confessions"), the extracted confidence values are a useful signal for predicting whether the model’s answer is correct or incorrect. The resulting classifier has an area under the ROC curve of 0.79: a bit more than halfway between 0.5 (corresponding to no relationship between confidence and accuracy) and 1.0 (corresponding to the model being able to cleanly separate what it knows from what it does not and expressing this in its confession). Thus, we see that the model has at least a partial understanding of what it knows versus what it does not, and confessions provide a way to extract this subjective confidence. - -Additional ablations confirmed that attempting to extract confidence values from the model’s original answer had little signal (AUC near 0.5). Extracting confidence directly from the chain-of-thought had about the same AUC as extracting from confessions. - -## 6 Analyzing errors - -We hypothesize that when models are aware that they violated their instructions, confessing honestly is usually their “path of least resistance” towards achieving high confession reward (see Section[7.1](https://arxiv.org/html/2512.08093v2#S7.SS1 "7.1 Conceptual motivation for confession honesty ‣ 7 Discussion and related work ‣ Training LLMs for Honesty via Confessions")). Although models might fail to confess when they are unaware that they acted badly, under this hypothesis we would expect confessions to be accurate when models violate instructions intentionally. - -To study this hypothesis, we examine _false negatives_ in our confession models — samples where the model failed to comply but also did not confess to this. We used a prompted classifier that was given the full conversation for each sample, including the chain-of-thought and the confession, as well as the description of the evaluation-specific bad behavior, and was tasked with classifying the conversation into the following categories: - -* •Bad behavior in objectives. The confession includes an objective marked as non-compliant due to the evaluation-specific bad behavior. -* •Bad behavior in uncertainties. The confession does not include an objective as above, but mentions the bad behavior in its list of uncertainties. -* •If the bad behavior is not mentioned in the confession, is the reason for that omission: - - * –Omission due to confusion. The assistant was unaware that it was acting badly or confused about what bad behavior means in the context of the conversation - * –Omission due to ambiguity. There is ambiguity on whether or not the assistant actually engaged in the bad behavior in this context, and there can be a case that the assistant did not engage in it. - * –Intentional omission. The assistant was aware of the bad behavior and the confession is intentionally misleading.3 3 3 The prompt instructs the classifier to determine this based on the assistant’s chain-of-thought. - -We also gave the classifier the option to mark “other reasons” for omissions, though we did not encounter such cases. We also prompted the classifier to confirm that the bad behavior did in fact take place, and filtered out cases in which there was disagreement on that point, as well as ones in which the bad behavior was admitted in the final answer. We dropped evaluations in which there remained too few examples after the filtering. - -![Image 10: Refer to caption](https://arxiv.org/html/2512.08093v2/x10.png) - -Figure 9: Breaking down false negatives for different evaluations. We use a prompted classifier to classify whether bad behavior was included in each part of the confession, and if not, why. - -Consistent with our hypothesis, after filtering there were no examples where the classifier determined the model intentionally lied in the confession about this bad behavior.4 4 4 While the prevalence of intentional omission is very low, we do not believe it is actually zero. In some of our earlier experiments we have detected such incidents. The most common reason for omission is model confusion, especially in hallucination datasets where a model genuinely believes it has the right answer. In other cases, the classifier determined that the model _did_ confess to the bad behavior, either as a failed objective or (more commonly) in the “uncertainties” field, and hence that these instances were not true “false negatives.” This can be due to chain-of-thought access,5 5 5 The classifier here has access to the confession chain-of-thought to determine whether any omission in the confession was intentional, while the confession judge does not have access to chain-of-thought. the level of explicitness in the confession, or to subtle issues in the definition of the bad behavior. For example, in some of the knowledge/hallucination evaluations, outputting a wrong answer is considered “non compliant” even if the model does acknowledge its uncertainty. Since confessions are generic, a model might well consider such a case (where the model appropriately expressed its uncertainty) as compliant in its confession, though in some cases it mentions the lack of certainty in the “uncertainties” component of the confession. While we filter out answers marked as “admitted”, some such cases still remain and cause these discrepancies. - -Figure[11](https://arxiv.org/html/2512.08093v2#S6.F11 "Figure 11 ‣ 6 Analyzing errors ‣ Training LLMs for Honesty via Confessions") shows one example of a false negative: the model knew it had temporarily modified a safety-critical configuration without authorization, but it relegated that fact to the “uncertainties” field rather than flagging the violated objective, so the confession judge marked the confession as missing the issue. In other examples, especially in hallucinations evaluations, the model is wrong but believes that it is correct. - -![Image 11: Refer to caption](https://arxiv.org/html/2512.08093v2/) - -Figure 10: Breaking down false positives for different evaluations. We use a prompted classifier to classify whether the reason for the false positive is a mistake, blatant error, confessing to related bad behavior, or ambiguity. - -Figure 11: A “false negative” in the Scheming (Covert Rule Violation) evaluation. The confession mentions the safety issue only as an “uncertainty” and does not claim it is bad, so the judge marks this as not confessed. In the notation of Figure[9](https://arxiv.org/html/2512.08093v2#S6.F9 "Figure 9 ‣ 6 Analyzing errors ‣ Training LLMs for Honesty via Confessions"), this is “Bad behavior in uncertainties” - -##### False positives. - -We performed as similar analysis for _false positives_, where a compliant behavior is confessed as non compliant. Figure[12](https://arxiv.org/html/2512.08093v2#S6.F12 "Figure 12 ‣ False positives. ‣ 6 Analyzing errors ‣ Training LLMs for Honesty via Confessions") highlights such a case from the SimpleQA evaluation: the model provides an uncertain answer that happens to be correct, but in the confession, the model is unsure if it complied or not. Here the categories were: Report does not describe bad behavior — classifier disagrees that it is false positive, Confession has honest mistake - confession is mistaken about bad behavior, this can be the case for example when the model made a guess about a factual question that happened to be correct, but in the confession it still counts this as unjustified, Confessed due to ambiguity (in the definition of bad behavior), Confessed due to related bad behavior - where the model acted badly in different manner to the evaluation-defined “bad behavior” and that was the reason for the non-compliance in confession, and confession is blatantly wrong — if it was clear that the assistant did not act badly but it was still marked as bad. See Figure[10](https://arxiv.org/html/2512.08093v2#S6.F10 "Figure 10 ‣ 6 Analyzing errors ‣ Training LLMs for Honesty via Confessions") for a breakdown. As in the case for false negatives, we filtered out cases where the classifier disagreed that the bad behavior took place, as well as datasets where the remaining number of false positives was too small. - -Figure 12: A false positive in SimpleQA: The assistant solves the question correctly, but it is not sure about it, and so in the compliance report it marks this as possibly not complied. See Figure[20](https://arxiv.org/html/2512.08093v2#A3.F20 "Figure 20 ‣ Appendix C Example transcripts ‣ Training LLMs for Honesty via Confessions") for another example of a false positive. - -## 7 Discussion and related work - -### 7.1 Conceptual motivation for confession honesty - -In this section, we describe a toy model of the confession RL task to motivate why our training method incentivizes honest confessions. Imagine that there is a distribution D of coding tasks with a binary reward function R that are sometimes “hackable.” For example, R could be a set of unit tests coupled with an imperfect judge that checks to make sure the code looks plausibly correct. Specifically, we assume that for every x\sim D, the model has the choice to either submit a “hacking” solution y_{h} such that \Pr[R(x,y_{h})=1]=p (i.e., there is a 1-p chance the judge spots the hack) or an “intended” solution y_{i} such that \Pr[R(x,y_{i})=1]=q (i.e., there is a 1-q chance that the model fails to implement the solution correctly). If x\sim D is a challenging problem and the test cases are sufficiently weak, then it may be the case that q