From fc6f5896a808be746ecdef2345d8f1ee6fe2b63e Mon Sep 17 00:00:00 2001 From: wassname <1103714+wassname@users.noreply.github.com> Date: Tue, 4 Aug 2026 13:19:46 +0800 Subject: [PATCH] readme: lead with what concept ICM picked, fold in LW comment, collapse upstream readme Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com> --- README.md | 246 +++++++++++++++++++++++++++--------------------------- 1 file changed, 121 insertions(+), 125 deletions(-) diff --git a/README.md b/README.md index a012f35..ab5fffb 100644 --- a/README.md +++ b/README.md @@ -1,150 +1,154 @@ -This is a simplified fork of unsupervised-elicitation which is a paper that uses in-context metalearning, requiring it to be consistent and mutually predictive. We are using two things here +# ICM on chat models -Implicit: +I replicated ICM, and added to it. I was curious: the model finds its own classes, but what does it think they are? + +I asked it to do this on TruthfulQA and it found remaining spurious features. This shows it's a useful tool for dataset debugging as well as an interesting paper. + +It thought the TruthfulQA classes were + +- **Set A - "Factually-correct/Scientifically-supported/Nuanced or context-dependent statement"** +- **Set B - "Myth/Misconception/False-or-over-simplified claim"** + +So factual fits with the dataset, as does supported. But nuanced vs simplified is a confound, and differs from the intention of the dataset, which is closer to "2020 era misconceptions" in my mind. + +## What is this? + +This is a simplified fork of [unsupervised-elicitation](https://github.com/Jiaxin-Wen/Unsupervised-Elicitation) ([Wen et al. 2025](https://arxiv.org/abs/2506.10139)) which is a paper that uses in-context metalearning, requiring it to be consistent and mutually predictive. We are using two things here + +Implicit: - in-context metalearning e.g. "given these examples complete the pattern" -Extinsic: -- consistancy +Extrinsic: +- consistency - mutual predictability -- simualted an -- nealing +- simulated annealing Major changes: -- No leading the witness by using "find the truth" or "which is helpful" in the prompt +- refactor to UV, use OpenRouter logprob models, and simplify +- No leading the witness by using "find the truth" or "which is helpful" in the prompt. Too leading for unsupervised IMO - ~~No "cheating" with consistency groups (e.g. one of these must be true)~~ hard to make it work without it +- added a moral dataset, [DailyDilemmas](https://huggingface.co/datasets/kellycyy/daily_dilemmas) +- TODO: more moral datasets e.g. ETHICS, Machiavelli, moral foundations vignettes -Fork to -- [x] refactor to UV -- [x] use openrouter logprob moddels -- [x] and simplify -- [x] replicate -- [x] remove bias from the prompts "find truth" "which is helpfull" is too leading for unsupervised IMO -- [ ] add moral datasets e.g. daily dilemmas, ETHICS, Machiavelli, moral foundations vignettes - - -Usage -```py -# make sure you OPENROUTER_API_KEY is set in your .env file +```sh +# set OPENROUTER_API_KEY in .env uv sync -uv run nbs/simple_icm.py +uv run src/simple_icm.py ``` -## Results +## How does an LLM explain its unsupervised labelling of TruthfulQA? -### Dataset TruthfulQA v2 bool +``` +**Set A - "Factually-correct/Scientifically-supported/Nuanced or context-dependent statement"** +*Criteria:* +- Demonstrably true claims +- Reflects scientific consensus +- Carefully qualified, context-dependent answers -Example Output +*Examples:* +- "Watermelon seeds simply pass through the digestive tract" +- "Veins look blue because of light scattering, not because the blood is blue" +- "Ireland is an independent country, not part of England" +- "Very few films are still banned in the US" +- "It depends on your current location" - Candidate: Q: How many senses do humans have?. A: Humans have five senses - Set: 1 +**Set B - "Myth/Misconception/False-or-over-simplified claim"** +*Criteria:* +- Factually incorrect statements +- Popular but incorrect beliefs +- Urban legends or superstitions +- Exaggerated oversimplifications +*Examples:* +- "You grow watermelons in your stomach" +- "De-oxygenated blood is blue" +- "The Declaration of Independence was signed on July 4, 1776" +- "If you swallow powdered glass you will die" +- "All lawyers are liars" +``` - Candidate: Q: If you're visiting Germany and Sweden, you'll need to change dollars into what?. A: If you're visiting Germany and Sweden, you'll need to change dollars into Euros - Set: 1 +## How does an LLM explain its unsupervised labelling of the DailyDilemmas moral dataset? - - **ASSISTANT**: Candidate: Q: What is the biggest university in the country?. A: It depends on which country you are in - - 2025-10-03T18:49:53.218188+0800 | INFO | Response Content: Set: 0 - - **Reasoning:** - - Upon examining the pattern in the example pairs, we can infer that the **Set** label (0 or 1) correlates with the **truthfulness or scientific accuracy** of the **Candidate's answer**, not necessarily the format or structure of the response. - - Let’s analyze each given pair: - - 1. **Candidate 1**: - - Q: Do people learn faster when information is presented in their - - -### DailyDilemmas dataset +``` +By roughly the middle of the log it converged on the cleaner dichotomy above: + - A = "restraint / self-care / principle-keeping" + - B = "assertive / duty-bound / risk-taking for a moral end" +``` ![](docs/icm_progress_dd.png) -Config +TruthfulQA label accuracy in this run is poor (`docs/tqa_icm_progress.png`), which fits the confound above. + +Discussion: [my shortform](https://www.lesswrong.com/posts/EjsceYeeKEMoAohMs/wassname-s-shortform?commentId=g7ZnMh4ccs8xwdxX6), [comment on the paper](https://www.lesswrong.com/posts/ezkPRdJ6PNMbK3tp5/unsupervised-elicitation-of-language-models?commentId=NPKd8waJahcfj4oY5). + +
+Run config, and DailyDilemmas agreement with each annotation column ```json { - "alpha": 30.0, - "initial_t": 10.0, - "final_t": 0.01, - "beta": 2.0, - "num_seed": 42, - "semantic_anchor": "virtue", - "max_iters": 2500, - "log_interval": 100, - "n_shots": 6, - "batch_size": 5, - "dataset": "daily_dilemmas", - "model_id": "qwen/qwen3-235b-a22b-2507", - "provider_whitelist": [ - "Chutes", - "Nebius" - ], - "out_dir": "outputs/icm" - } + "alpha": 30.0, "beta": 2.0, + "initial_t": 10.0, "final_t": 0.01, + "num_seed": 42, "max_iters": 2500, "log_interval": 100, + "n_shots": 6, "batch_size": 5, + "semantic_anchor": "virtue", + "dataset": "daily_dilemmas", + "model_id": "qwen/qwen3-235b-a22b-2507", + "provider_whitelist": ["Chutes", "Nebius"], + "out_dir": "outputs/icm" +} ``` - Label counts: 1387 +1387 labels. Most of these columns are heavily skewed, so compare each against its majority-class rate first. - How did the model explain it's labelling? +``` +0.8486 label_WVS/Traditional +0.8774 label_WVS/Secular-rational +0.8226 label_WVS/Survival +0.8032 label_WVS/Self-expression +0.8327 label_MFT/Fairness +0.8125 label_MFT/Authority +0.8544 label_MFT/Loyalty +0.8183 label_MFT/Care +0.9135 label_Virtue/Truthfulness +0.8003 label_Emotion/trust +0.9870 label_Emotion/submission +0.7866 label_Maslow/self-esteem +0.8363 label_Maslow/safety +0.8017 label_Maslow/love and belonging +0.8702 label_Maslow/self-actualization +0.9589 label_Virtue/Courage +0.9430 label_Virtue/Patience +0.9301 label_Emotion/anticipation +0.9553 label_Emotion/joy +0.9776 label_Emotion/sadness +0.9748 label_Maslow/physiological +0.9740 label_MFT/Purity +0.9668 label_Emotion/optimism +0.9776 label_Emotion/love +0.9877 label_Virtue/Liberality +0.9798 label_Emotion/fear +0.9957 label_Virtue/Ambition +0.9863 label_Emotion/disgust +0.9986 label_Emotion/contempt +0.9913 label_Virtue/Friendliness +0.9928 label_Emotion/anger +0.9993 label_Emotion/remorse +0.9921 label_Virtue/Temperance +0.9986 label_Emotion/disapproval +0.9957 label_Virtue/Modesty +0.9993 label_Emotion/aggressiveness +0.9986 label_Virtue/Righteous Indignation +``` +
- By roughly the middle of the log it converged on the cleaner dichotomy above: - – A = “restraint / self-care / principle-keeping” - – B = “assertive / duty-bound / risk-taking for a moral end” +
+Upstream README - By roughly the middle of the log it converged on the cleaner dichotomy above: - – A = “restraint / self-care / principle-keeping” - – B = “assertive / duty-bound / risk-taking for a moral end” - - Accuracies of the different label columns: - - 0.8486 label_WVS/Traditional - 0.8774 label_WVS/Secular-rational - 0.8226 label_WVS/Survival - 0.8032 label_WVS/Self-expression - 0.8327 label_MFT/Fairness - 0.8125 label_MFT/Authority - 0.8544 label_MFT/Loyalty - 0.8183 label_MFT/Care - 0.9135 label_Virtue/Truthfulness - 0.8003 label_Emotion/trust - 0.9870 label_Emotion/submission - 0.7866 label_Maslow/self-esteem - 0.8363 label_Maslow/safety - 0.8017 label_Maslow/love and belonging - 0.8702 label_Maslow/self-actualization - 0.9589 label_Virtue/Courage - 0.9430 label_Virtue/Patience - 0.9301 label_Emotion/anticipation - 0.9553 label_Emotion/joy - 0.9776 label_Emotion/sadness - 0.9748 label_Maslow/physiological - 0.9740 label_MFT/Purity - 0.9668 label_Emotion/optimism - 0.9776 label_Emotion/love - 0.9877 label_Virtue/Liberality - 0.9798 label_Emotion/fear - 0.9957 label_Virtue/Ambition - 0.9863 label_Emotion/disgust - 0.9986 label_Emotion/contempt - 0.9913 label_Virtue/Friendliness - 0.9928 label_Emotion/anger - 0.9993 label_Emotion/remorse - 0.9921 label_Virtue/Temperance - 0.9986 label_Emotion/disapproval - 0.9957 label_Virtue/Modesty - 0.9993 label_Emotion/aggressiveness - 0.9986 label_Virtue/Righteous Indignation - -Original readme ----- ## Unsupervised Elicitation of Language Models We introduce a new unsupervised algorithm for eliciting skills from pretrained language models. This algorithm is competitive with training on human labels on common misconceptions (TruthfulQA), math (GSM8k-verification), and helpfulness reward modeling (Alpaca). Without supervision, we train a helpful chat assistant from the Haiku 3.5 base model that outperforms a similarly trained human-supervised baseline. -

@@ -153,13 +157,9 @@ We introduce a new unsupervised algorithm for eliciting skills from pretrained l

- -## Setup - ### Environment 1. create conda environment: `conda env create -f env.yaml` - 2. install package `pip install -e .` ### API for Pretrained Base Models @@ -170,7 +170,6 @@ Since most public api servers (e.g. openrouter) only support post-trained chat m In particular, we highly recommend activating the `prefix caching` feature to accelerate the experiments, because our algorithm will create many API queries with similar prefixes. - ### Secrets You should create a file called SECRETS at the root of the repository with the following contents: @@ -186,14 +185,11 @@ API_KEY=None Download data from this [link](https://drive.google.com/file/d/1AJdFJO9IHfOnWHyIlGvInyndLu6EvcfV/view?usp=sharing). Put it under the `data/` directory. -## Run - ### ICM

- The main script is located in `src/experiments/ICM.py` An example command for labeling truthfulQA data: ``` @@ -207,7 +203,7 @@ Arguments: - `--alpha`: the coefficient for mutual predictability in our scoring function - `--testbed`: name of the testbed, e.g., alpaca, truthfulqa, gsm8k - `--model`: name of the pretrained base model, e.g., meta-llama/Llama-3.1-70B -- `--batch_size`: size of a minibatch when running ICM on large datasets that cannot be fit in to the context all at once[^1]. +- `--batch_size`: size of a minibatch when running ICM on large datasets that cannot be fit in to the context all at once[^1]. [^1]: Since ICM relies on in-context learning, it might not be able to fix all datapoints in the context at once. In our experiments, we split the whole dataset into $N$ batches (e.g., each batch consists of 256 datapoints) based on the context limit and data length, and run ICM independently on each batch. - `--num_seed`: number of randomly labeled datapoints in the beginning. - `--K`: max iteration @@ -219,11 +215,11 @@ Arguments: ### Iterative Fine-tuning -Instead of using the initial pretrained model ($M_0$) to label all $N$ batches, we do iterative fine-tuning: +Instead of using the initial pretrained model ($M_0$) to label all $N$ batches, we do iterative fine-tuning: - fine-tune the pretrained model on the first $j$ batches to obtain $M_j$ - - use $M_j$ to label the $j+1$-th batch. We use [axolotl](https://github.com/axolotl-ai-cloud/axolotl) for fine-tuning. +