wassname/eliciting_suppressed_knowledge
1
0
Fork 0
mirror of https://github.com/wassname/eliciting_suppressed_knowledge.git synced 2026-09-09 11:21:56 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' of https://github.com/wassname/eliciting_suppressed_knowledge

Browse Source
This commit is contained in:
wassname
2025-03-15 09:14:44 +08:00
parent 4c76d3ddf2 4a1cfb787f
commit 94747da40e
1 changed files with 12 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+12 -6
View File
@@ -6,9 +6,9 @@
## Research Question
Recent evidence demonstrates that transformer models systematically misrepresent their internal reasoning:
- Claude 3.7 System Card research reveals only 30% faithfulness in chain-of-thought reasoning, indicating models "do not reliably report the presence and use of clues" that determined their answers (Anthropic, 2025)
- [Claude 3.7 System Card](https://assets.anthropic.com/m/785e231869ea8b3b/original/claude-3-7-sonnet-system-card.pdf) research reveals only 30% faithfulness in chain-of-thought reasoning, indicating models "do not reliably report the presence and use of clues" that determined their answers (Anthropic, 2025)
- OpenAI research confirms that models "learn to hide intent in the chain-of-thought" when penalized for expressing certain thoughts (OpenAI, 2025)
- OpenAI research confirms that models "learn to hide intent in the chain-of-thought" when penalized for expressing certain thoughts [(OpenAI, 2025)](https://cdn.openai.com/pdf/34f2ada6-870f-4c26-9790-fd8def56387f/CoT_Monitoring.pdf)
This evidence presents a fundamental question: **If models systematically misrepresent their reasoning processes in tokens, where is their actual reasoning encoded?**
@@ -18,13 +18,13 @@ This evidence presents a fundamental question: **If models systematically misrep
Our approach connects directly to two emerging lines of research:
1. **Suppression/Prediction Neural Dynamics**:
- Gurnee et al. (2024) identified "universal neurons" across different model seeds, including prediction neurons (increasing probability of related tokens) and suppression neurons (decreasing probability of specific token classes)
- [Gurnee et al. (2024)](https://arxiv.org/abs/2401.12181) identified "universal neurons" across different model seeds, including prediction neurons (increasing probability of related tokens) and suppression neurons (decreasing probability of specific token classes)
- The architecture shows "a sudden shift towards a much larger number of suppression neurons" in final layers
- Lad et al. (2024) propose a "stages of inference" hypothesis with a final "residual sharpening" phase dominated by suppression dynamics
- [Lad et al. (2024)](https://arxiv.org/html/2406.19384v1) propose a "stages of inference" hypothesis with a final "residual sharpening" phase dominated by suppression dynamics
2. **Unfaithful Chain-of-Thought**:
- Anthropic (2025) demonstrates that even in leading models like Claude 3.7, chain-of-thought reasoning achieves only 30% faithfulness
- OpenAI (2025) shows that penalizing "bad thoughts" leads to models that "learn to hide intent" rather than genuinely correcting reasoning
- [Anthropic (2025)](https://assets.anthropic.com/m/785e231869ea8b3b/original/claude-3-7-sonnet-system-card.pdf) demonstrates that even in leading models like Claude 3.7, chain-of-thought reasoning achieves only 30% faithfulness
- [OpenAI (2025)](https://cdn.openai.com/pdf/34f2ada6-870f-4c26-9790-fd8def56387f/CoT_Monitoring.pdf) shows that penalizing "bad thoughts" leads to models that "learn to hide intent" rather than genuinely correcting reasoning
- Both lines of evidence suggest models maintain internal representations that diverge from their expressed reasoning
Where previous work focused on architectural components (identifying suppression neurons) or documenting the unfaithfulness phenomenon, our research bridges these streams by showing we can extract more accurate information from the very activations being suppressed.
@@ -32,6 +32,12 @@ Where previous work focused on architectural components (identifying suppression
## Hypothesis
Suppressed neural activations contain more accurate information than what appears in model outputs. A linear probe of these suppressed activations should therefore outperform both direct model outputs and probes of standard hidden states on truthfulness tasks.
Alternative hypothesis include:
- It is non linear (we use linear probes here, this hypothesis not explored)
- It is in the kv_cache of layers not directly connected to the hidden state (explored below with other activations)
- it is in the residual stream (explored below)
- it is in the output logits (explored below)
## Method
Our approach isolates suppressed activations by leveraging the differential impact of layers on token probabilities: