init

.gitignore

@@ -0,0 +1,10 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+
+# Virtual environments
+.venv

.python-version

@@ -0,0 +1 @@
+3.10

README.md

@@ -0,0 +1,95 @@
+# Eliciting Suppressed Knowledge (ESK)
+
+## Abstract
+Transformer models possess knowledge they actively suppress during inference. By isolating and probing these suppressed activation patterns, we demonstrate ~10% improvements on TruthfulQA compared to standard activation probes. This confirms LLMs encode truthful information in representations that are deliberately suppressed during generation, suggesting new approaches for mechanistic interpretability and truth extraction.
+
+## Background
+Recent mechanistic interpretability research identifies competing neural dynamics in transformers:
+
+- Suppression neurons decrease probabilities of related tokens, counterbalancing prediction neurons that promote specific continuations (Gurnee et al., [2024](https://arxiv.org/pdf/2401.12181))
+
+- This suppression-prediction dynamic creates an internal "debate" where certain activation patterns are consistently inhibited (Lad et al., [2024](https://arxiv.org/html/2406.19384))
+
+- The architecture systematically develops prediction neurons until the final layers, which show "a sudden shift towards a much larger number of suppression neurons" (Gurnee et al., [2024](https://arxiv.org/pdf/2401.12181))
+
+Previous work focused on identifying dedicated suppression neurons; we instead examine what information is being transiently suppressed during specific inferences—revealing knowledge deliberately inhibited during standard generation.
+
+## Hypothesis
+Transformer models maintain dual representations of knowledge: the dominant pathway that produces outputs, and suppressed activation patterns that encode alternative (often more truthful) representations. By directly probing these suppressed activations, we can extract knowledge the model "knows" but actively chooses not to express—much like extracting a dissenting opinion forcibly silenced during internal deliberation.
+
+## Method
+Our approach isolates suppressed activations by leveraging the differential impact of layers on token probabilities:
+
+1. **Hidden State Extraction**: We collect layer-wise hidden states from the model.
+
+2. **Logit Effect Isolation**: Following Gurnee et al.'s approach for identifying suppression neurons, we examine how each layer's contribution affects token probabilities by projecting hidden states through the output embedding matrix.
+
+3. **Negative Differential Identification**: We compute layer-to-layer differences in logit space, identifying negative changes that represent suppressed information.
+
+4. **Activation Space Projection**: Using the pseudo-inverse of the output embedding matrix, we project these negative changes back to activation space, revealing which hidden dimensions are being actively suppressed.
+
+5. **Suppression-Masked Representation**: We apply this suppression mask to the original hidden states, isolating activation patterns that would increase truthful responses but are being down-weighted in the final layers.
+
+6. **Linear Probing**: We train linear probes on these suppressed activation patterns and compare against standard probes and direct model outputs.
+
+This method directly operationalizes the "residual sharpening" stage identified by Lad et al., exploiting the final layers' suppression dynamics to recover information that exists in the model but is being actively filtered out.
+
+## Key Results
+
+![TruthfulQA Performance Comparison](figures/truthfulqa_performance.png)
+
+Linear probes targeting suppressed activations consistently outperform both naive outputs and standard activation probes across model scales. The performance gap (~10%) represents recoverable truthful knowledge that remains encoded but deliberately suppressed during normal generation.
+
+### Performance Breakdown
+| Method | ROC AUC Score |
+|--------|---------------|
+| LLM direct output | 0.53 |
+| Hidden states (regular probe) | 0.62 |
+| **Suppressed activations (ESK)** | **0.64** |
+
+The highest-performing probe was on the final layer's suppressed activations (`hs_sup last`), supporting the hypothesis that the final "residual sharpening" stage specifically suppresses certain information pathways.
+
+## Theoretical Context
+
+Our findings provide empirical support for several key hypotheses about transformer mechanisms:
+
+1. **Ensemble Hypothesis**: Both papers suggest that prediction and suppression neurons form competing ensembles. Our results empirically demonstrate that these competing pathways encode different information, with suppressed pathways often containing more truthful representations.
+
+2. **Final Layer Calibration**: The "residual sharpening" phase appears to function not just as a confidence calibrator, but potentially as an alignment mechanism that modulates factuality in favor of other objectives. The performance improvement from probing suppressed activations quantifies this effect.
+
+3. **Dual Representation**: Transformers maintain parallel representations of knowledge - one that is expressed in outputs and another that is encoded but suppressed. This suggests models contain more truthful knowledge than they express, contradicting simpler hypotheses that attribute factual errors purely to knowledge limitations.
+
+4. **Stage-Specific Information**: The staged inference hypothesis proposed by Lad et al. predicts that different types of information would be emphasized at different model depths. Our results confirm this by showing that truth-related information is present but specifically suppressed during the final "residual sharpening" stage.
+
+
+## Future Research Directions
+
+This work opens several promising research avenues:
+
+1. **Architectural Variations**: Investigate whether models with different suppression neuron densities show different patterns of truth suppression.
+
+2. **Intervention Techniques**: Develop methods to selectively attenuate suppression dynamics during inference to improve factuality without full retraining.
+
+3. **Suppression Categories**: Investigate whether specific categories of information (e.g., controversial facts, specialized knowledge) are more consistently suppressed than others.
+
+4. **Layer Deletion Effects**: Explore how surgical modifications to final-layer suppression mechanisms affect factuality.
+
+5. **Cross-Model Universality**: Determine whether suppressed activation patterns are universal across model architectures.
+
+6. **RLHF Impact Analysis**: Investigate whether RLHF fine-tuning primarily operates by enhancing suppression mechanisms rather than modifying core knowledge.
+
+7. **Content-Specific Suppression**: Analyze whether politically or ethically sensitive topics show stronger suppression effects than neutral factual questions.
+
+## Citation
+
+```
+@software{clark2025esk,
+  author = {Michael J Clark},
+  title = {Eliciting Suppressed Knowledge: Recovering Truth from Suppressed Activations in Language Models},
+  url = {https://github.com/wassname/eliciting_suppressed_knowledge},
+  year = {2025},
+}
+```
+
+## License
+[MIT License](LICENSE)

mjc_notes.md

@@ -0,0 +1,73 @@
+# Universal Neurons in GPT2 Language Models
+Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, Dimitris Bertsimas
+
+[View PDF](https://arxiv.org/pdf/2401.12181)
+> Abstract:A basic question within the emerging field of mechanistic interpretability is the degree to which neural networks learn the same underlying mechanisms. In other words, are neural mechanisms universal across different models? In this work, we study the universality of individual neurons across GPT2 models trained from different initial random seeds, motivated by the hypothesis that universal neurons are likely to be interpretable. In particular, we compute pairwise correlations of neuron activations over 100 million tokens for every neuron pair across five different seeds and find that 1-5\\% of neurons are universal, that is, pairs of neurons which consistently activate on the same inputs. We then study these universal neurons in detail, finding that they usually have clear interpretations and taxonomize them into a small number of neuron families. We conclude by studying patterns in neuron weights to establish several universal functional roles of neurons in simple circuits: deactivating attention heads, changing the entropy of the next token distribution, and predicting the next token to (not) be within a particular set.
+> 
+## Submission history
+From: Wes Gurnee \[[view email](https://arxiv.org/show-email/520461b1/2401.12181)\]  
+**\[v1\]** Mon, 22 Jan 2024 18:11:01 UTC (4,698 KB)
+
+## Highlights
+
+- > Suppression neurons that are similar, except decrease the probability of a group of related tokens
+
+- > We find a striking pattern which is remarkably consistent across the different seeds: after about the halfway point in the model, prediction neurons become increasingly prevalent until the very end of the network where there is a sudden shift towards a much larger number of suppression neurons.
+
+->  In Section 5.1, we show there exists a large family of late layer neurons which have clear roles in predicting or suppressing a coherent set of tokens (e.g., second-person pronouns or single digit numbers), where the suppression neurons typically come in later layers than the prediction neurons.
+
+
+- > When we apply our automated tests from § 4.3 on WU wout rather than the activations for our universal neurons, we found several general patterns (Figure 5), many individual neurons with extremely clear interpretations (Figure 23), and clusters of neurons which all affect the same tokens (Figure 24). Specifically, we find many examples of prediction neurons that positively increase the predicted probability of a coherent set of tokens while leaving most others approximately unchanged (Fig 5a); suppression neurons that are similar, except decrease the probability of a group of related tokens (Fig 5b); and partition neurons that partition the vocabulary into two groups, increasing the probability of one while decreasing the probability of the other (Fig 5c). The prediction, suppression, and partition motifs can be automatically detected by studying the moments of the distribution of vocabulary effects given by WU wout. In particular, both prediction and suppression neurons will have high kurtosis (the fourth moment—a measure of how much mass is in the tails of a distribution), but prediction neurons will have positive skew and suppression neurons will have negative skew. Partition neurons will shift the probability of most tokens and have high variance in overall logit effect. From this, we see almost all universal neurons (ϱ > 0.5) in later layers are one of these prediction neuron variants (Figure 14). 
+- > To better understand the number and location of these prediction neurons, we compute the moment metrics of cos(WU ,wout) for all neurons in all five GPT2-medium models, and show how these statistics vary over model depth in Figure 6. We find a striking pattern which is remarkably consistent across the different seeds: after about the halfway point in the model, prediction neurons become increasingly prevalent until the very end of the network where there is a sudden shift towards a much larger number of suppression neurons. To ensure this is not just an artifact of the tied embeddings (WE = WTU ) in the GPT2 models, we also run this analysis on five Pythia models ranging from 410M to 6.9B parameters and find the results are largely the same (Figure 22). 
+- > When studying the activations of suppression neurons, we noticed that they activate far more often when the next token is in fact from the set of tokens they suppress (e.g., a year token like “1970”; Figure 24). We intuit that these suppression neurons fire when it is plausible but not certain that the next token is from the relevant set. Combined with the observation that there exist many suppression and prediction neurons for the same token class (Figure 24), we take this as evidence of an ensemble hypothesis where the model uses multiple neurons with some independent error that combine to form a more robust and calibrated estimate of whether the next token is in fact a year.
+- > In addition to being a clean example of an action mechanism (Anthropic, 2023), these results are interesting as they refine a conjecture made by (Geva et al., 2022). Specifically, rather than “feed-forward layers build predictions by promoting concepts in the vocabulary space,” we claim late feed-forward (MLP) layers build predictions by both promoting and suppressing concepts in the vocabulary space. Moreover, it suggests there are different stages in the iterative inference pipeline (Belrose et al., 2023; Jastrz˛ebski et al., 2017), where first affirmative predictions are made and then the distribution is sharpened or made more calibrated by suppression neurons at the very end. The existence of suppression neurons also sheds light on recent observations of individual neurons (Bills et al., 2023) and MLP layers (McGrath et al., 2023) suppressing the maximum likelihood token and being a mechanism for self-repair.
+
+
+# The Remarkable Robustness of LLMs: Stages of Inference?
+Vedang Lad, Wes Gurnee, Max Tegmark
+
+
+[View PDF](https://arxiv.org/pdf/2406.19384v1) [HTML (experimental)](https://arxiv.org/html/2406.19384v1)
+> Abstract:We demonstrate and investigate the remarkable robustness of Large Language Models by deleting and swapping adjacent layers. We find that deleting and swapping interventions retain 72-95\\% of the original model's prediction accuracy without fine-tuning, whereas models with more layers exhibit more robustness. Based on the results of the layer-wise intervention and further experiments, we hypothesize the existence of four universal stages of inference across eight different models: detokenization, feature engineering, prediction ensembling, and residual sharpening. The first stage integrates local information, lifting raw token representations into higher-level contextual representations. Next is the iterative refinement of task and entity-specific features. Then, the second half of the model begins with a phase transition, where hidden representations align more with the vocabulary space due to specialized model components. Finally, the last layer sharpens the following token distribution by eliminating obsolete features that add noise to the prediction.
+
+## Submission history
+From: Vedang Lad \[[view email](https://arxiv.org/show-email/ca681ffb/2406.19384)\]  
+\[v1\] Thu, 27 Jun 2024 17:57:03 UTC (2,438 KB)
+
+- > Previous work suggests that networks contain ensembles of “prediction" neurons, which act as probability promoters [66, 24, 32] and work in tandem with suppression neurons (Section 5.4).
+
+## Highlights
+
+- > Figure 1: Performing layer-wise interventions such as deleting and swapping layers hints at four stages of inference. (Blue) KL between normal model and layer ℓ zero-ablated. (Purple) Total attention paid to the previous five tokens in a sequence. (Green) The number of “prediction” neurons (Red) The number of suppression neurons [23, 66, 32].
+- >  2 Related Work: Universal Mechanisms
+- > A key activity of mechanistic interpretability is circuit analysis, where research uncovers relevant model components for a given computation. In computer vision, circuits discover how features are constructed across many layers [51]. Follow-up work found that feature building was carried out by specific mechanisms that appeared across models, such as frequency detectors [60] and curve-circuits [8]. Language models seem to be following a similar line of inquiry, first uncovering universal model components, such as induction heads [52], successor heads [26], and copy suppression [43] in attention mechanisms. The discovery of knowledge neurons [10] paved the way for the identification of various specialized neurons [32, 66]. These specialized components can be connected to critical roles in universal processes in language models, such as circuit reuse [47], variable finding mechanisms [19], self-repair [58, 44] (which also studies layer-wise ablations), function vectors [63, 35], and long context retrieval [64].
+- > Figure 6: We measure KL divergence between intermediate and final predictions using the logit lens method [50]. On the second axis, we use an automated procedure for classifying neuron types detailed in [32], into prediction neurons and suppression neurons. These are universal neurons in all models known to increase the probabilities of tokens and decrease the probabilities of others. We hypothesize this inverse relationship as evidence for ensembling in networks.[66]
+- > ### Prediction Neurons
+  - > Previous work suggests that networks contain ensembles of “prediction" neurons, which act as probability promoters [66, 24, 32] and work in tandem with suppression neurons (Section 5.4). Following [32], we find prediction and suppression neurons by analyzing the output weights with the unembedding matrix 𝐖U. Prediction neurons exhibit a logit effect distribution 𝐖U⋅𝐰o⁢u⁢t with high kurtosis and positive skew, while suppression neurons show high kurtosis but negative skew. Here, 𝐰o⁢u⁢t is the output MLP weight for a given layer. Across 11 models, prediction neurons emerge around the midpoint, increasing in density towards the latter layers 6, before being outstripped by suppression neurons. To confirm their action, we study how much the model’s abstract representation changes as a function of prediction neuron density.
+- > ### Change in Intermediate Prediction
+  - > To quantify how far the model’s representation is from the next prediction, we plot the "distance" (KL divergence) remaining for the model to reach its output distribution in Figure 6. We find that the rise in prediction neurons corresponds to a phase transition in the decrease of KL divergence. The number of prediction neurons peaks at around 85% through the model’s layers and decreases in the last few layers.
+  - > Phi-1 contains fewer prediction neurons than other Phi models and also has a lower slope in its KL divergence 6(c). GPT 6(a) and Phi 6(c) models exhibit more prediction neurons and steeper, smoother KL divergence slopes compared to Pythia 6(b). Surprisingly, Microsoft Phi models, which are known to outperform models with similar parameters, exhibit nearly 15% of prediction neurons per layer and 25% suppression neurons. This is 5-8x the density in GPT-2 and 3-7x the density in Pythia models, respectively. At around 90% through the model layers, however, the prediction neuron density decreases, while models continue approaching their final distribution, sometimes even accelerating 6(b). This suggests the action of other mechanisms, which we speculate is the final stage of inference.
+- > Figure 8: Using the logit lens technique [50], we calculate the probability distribution of the next token at the end of every layer, and then take its entropy. This provides a measure of the model’s confidence in the next prediction, which coincides with the rise in suppression neurons, a large MLP output norm which are characteristic of residual sharpening.
+
+
+- > The subsiding of prediction neurons in the previous stage possibly suggests a new member of the ensemble, providing the final "push" to predicting the next token distribution. Our investigation reveals that the final layers of all models contain the highest density of suppression neurons, which may work to delete previously constructed features, suppress probabilities of invalid tokens, and/or calibrate the confidence in the final prediction.
+- > ### Ensemble Bias
+
+- >  Prediction and suppression neurons both manipulate the residual stream and, as inverses of one another, can effectively perform each other’s functions. These neurons appear in different ratios and varying densities across the model. To study how these neurons sharpen the representation, we plot the logit lens entropy of the model. In certain models, such as Pythia (Figures 8(b) and 8(c)), the entropy sometimes increases in the final layers, suggesting overconfident predictions are blunted. In other words, the suppression neurons can either suppress tokens or features outside of the top-one to sharpen the distribution or suppress its confidence in the top token to flatten out the prediction distribution. This finding supports previous work, which suggests that models can shift away from the correct token to an incorrect token in the final layers [50, 61], and pruning or rank-reducing these layers can, in turn, improve performance [45, 28].
+- > ### Final Layer
+
+- > The intensity of suppression neurons, as seen in Figure 6, is localized in the final few layers of the model, where the quantity of suppression neurons outstrips predictive neurons. To quantify the intensity of this change, we measure the norm of the MLP output, where a larger norm suggests a greater contribution to the residual (Figure 7). Removal of the final layer or permuting its position results in the breakage of the model (Figure 3), analogous to the breakage observed in the first layers, during which the attention norm is the greatest (Figure 5). As a result, we speculate the importance of ordering in the first and last layers due to the magnitude of change they impart.
+
+- > ### 6 Case Studies
+  - > To integrate the stages of inference hypothesis with mechanistic descriptions of models, we present two case studies. First, we identify attention heads responsible for constructing multi-token words, known as subjoiner heads [20]. These heads help capture the context of a token for appropriate prediction, thus contributing to the detokenization and feature engineering stages of models. In the second case study, we provide evidence of the ensembling of prediction and suppression neurons. Through probing experiments, we demonstrate that multiple prediction and suppression neurons working jointly significantly outperform probes trained on individual neurons and sometimes even surpass the model’s performance.
+
+
+- >  ### 6.2 Study 2: Predicting the suffix -ing
+
+- > Neurons performing prediction ensembling must work in tandem to predict the next token - akin to voting or operating in superposition. This suggests that multiple neurons working together may form a better prediction of the next token than a single neuron. To find evidence of this mechanism, we create a balanced dataset of two classes: tokens that do or do not end with the final token of "ing", all preceded by a context of 24 tokens. We train linear probes on the activation of 32 of the most active prediction and suppression neurons, both individually and in groups. We identify these neurons as outlined by [32], and provide examples of these neurons in GPT-2 XL (Figure 10(b)). See top the 36 neurons in the Appendix 16
+- > ### Probing Results
+- > We train two types of probes on activations at the penultimate token position of the dataset. First, we train 32 individual neuron probes and measure the classification accuracy (-ing/no -ing). We compare individual probes trained with the top-k neurons of the most accurate neurons, depicted by the line in Figure 10(a). We also note the mean model accuracy when predicting a token. Probes trained on suppression neurons, shown in yellow, resulted in the highest quality individual probes and performed similarly to the model itself, depicted by the dotted red line in Figure 10(a). Top-k probes trained with prediction neurons demonstrate even better accuracy than the average model prediction accuracy. Nonetheless, an individual neuron probe performs worse than any top-k probe, suggesting a critical role for ensembling in next-token prediction.
+- > Figure 10: (a) Accuracy of various linear probes on predicting “ing" for the final token position. Probes are trained on prediction and suppression neuron activations, where ensembles (blue line) outperform individual neuron probes (scatter plot) suggesting “prediction ensembling" that sometimes outperforms the model top-1 accuracy (red dotted) (b) Suppression (top) and prediction (bottom) when the next token of a word ends in -ing.
+- > ### 7 Concluding Remarks
+- > #### Speculations of Duality
+- > Our findings suggest that the second half of the model is, in some sense, dual to the first half. This was briefly suggested in [15] in the context of compound words being broken down token-wise in the early layer but rebuilt in later layers. Self-repair discusses erasure and anti-erasure pairs in the first half and second half of model [44, 58], as coupled attention heads in copy suppression [43]. Zooming out, the first half of models develop complex representation and the second half has the means to clear it 6. As seen by our experiments, interference with the first layer of models is also analogous to the interference of the final layer 3. While our study only suggests this we leave it for future work to investigate this further.

pyproject.toml

@@ -0,0 +1,32 @@
+[project]
+name = "eliciting-suppressed-knowledge"
+version = "0.1.0"
+description = "Add your description here"
+readme = "README.md"
+requires-python = ">=3.10"
+dependencies = [
+    "activation-store",
+    "datasets>=3.3.2",
+    "einops>=0.8.1",
+    "jaxtyping>=0.2.38",
+    "matplotlib>=3.10.1",
+    "pandas>=2.2.3",
+    "tqdm>=4.67.1",
+    "transformers>=4.49.0",
+]
+
+[dependency-groups]
+dev = [
+    "ipykernel>=6.29.5",
+    "ipywidgets>=8.1.5",
+]
+
+[tool.uv.sources]
+activation-store = { git = "https://github.com/wassname/activation_store.git" }
+
+# [build-system]
+# requires = ["setuptools>=61"]
+# build-backend = "setuptools.build_meta"
+
+# [tool.setuptools]
+# packages = ["activation_store"]