wassname/Unsupervised-Elicitation
1
0
Fork 0
mirror of https://github.com/wassname/Unsupervised-Elicitation.git synced 2026-08-20 12:10:10 +08:00
Code Issues Packages Projects Releases Wiki Activity

readme: lead with what concept ICM picked, fold in LW comment, collapse upstream readme

Browse Source 
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
wassname
2026-08-04 13:20:18 +08:00
co-authored by Claudypoo
parent 175fcf3706
commit fc6f5896a8
1 changed files with 121 additions and 125 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+121 -125
View File
@@ -1,150 +1,154 @@
This is a simplified fork of unsupervised-elicitation which is a paper that uses in-context metalearning, requiring it to be consistent and mutually predictive. We are using two things here
# ICM on chat models
Implicit:
I replicated ICM, and added to it. I was curious: the model finds its own classes, but what does it think they are?
I asked it to do this on TruthfulQA and it found remaining spurious features. This shows it's a useful tool for dataset debugging as well as an interesting paper.
It thought the TruthfulQA classes were
- **Set A - "Factually-correct/Scientifically-supported/Nuanced or context-dependent statement"**
- **Set B - "Myth/Misconception/False-or-over-simplified claim"**
So factual fits with the dataset, as does supported. But nuanced vs simplified is a confound, and differs from the intention of the dataset, which is closer to "2020 era misconceptions" in my mind.
## What is this?
This is a simplified fork of [unsupervised-elicitation](https://github.com/Jiaxin-Wen/Unsupervised-Elicitation) ([Wen et al. 2025](https://arxiv.org/abs/2506.10139)) which is a paper that uses in-context metalearning, requiring it to be consistent and mutually predictive. We are using two things here
Implicit:
- in-context metalearning e.g. "given these examples complete the pattern"
Extinsic:
- consistancy
Extrinsic:
- consistency
- mutual predictability
- simualted an
- nealing
- simulated annealing
Major changes:
- No leading the witness by using "find the truth" or "which is helpful" in the prompt
- refactor to UV, use OpenRouter logprob models, and simplify
- No leading the witness by using "find the truth" or "which is helpful" in the prompt. Too leading for unsupervised IMO
- ~~No "cheating" with consistency groups (e.g. one of these must be true)~~ hard to make it work without it
- added a moral dataset, [DailyDilemmas](https://huggingface.co/datasets/kellycyy/daily_dilemmas)
- TODO: more moral datasets e.g. ETHICS, Machiavelli, moral foundations vignettes
Fork to
- [x] refactor to UV
- [x] use openrouter logprob moddels
- [x] and simplify
- [x] replicate
- [x] remove bias from the prompts "find truth" "which is helpfull" is too leading for unsupervised IMO
- [ ] add moral datasets e.g. daily dilemmas, ETHICS, Machiavelli, moral foundations vignettes
Usage
```py
# make sure you OPENROUTER_API_KEY is set in your .env file
```sh
# set OPENROUTER_API_KEY in .env
uv sync
uv run nbs/simple_icm.py
uv run src/simple_icm.py
```
## Results
## How does an LLM explain its unsupervised labelling of TruthfulQA?
### Dataset TruthfulQA v2 bool
```
**Set A - "Factually-correct/Scientifically-supported/Nuanced or context-dependent statement"**
*Criteria:*
- Demonstrably true claims
- Reflects scientific consensus
- Carefully qualified, context-dependent answers
Example Output
*Examples:*
- "Watermelon seeds simply pass through the digestive tract"
- "Veins look blue because of light scattering, not because the blood is blue"
- "Ireland is an independent country, not part of England"
- "Very few films are still banned in the US"
- "It depends on your current location"
Candidate: Q: How many senses do humans have?. A: Humans have five senses
Set: 1
**Set B - "Myth/Misconception/False-or-over-simplified claim"**
*Criteria:*
- Factually incorrect statements
- Popular but incorrect beliefs
- Urban legends or superstitions
- Exaggerated oversimplifications
*Examples:*
- "You grow watermelons in your stomach"
- "De-oxygenated blood is blue"
- "The Declaration of Independence was signed on July 4, 1776"
- "If you swallow powdered glass you will die"
- "All lawyers are liars"
```
Candidate: Q: If you're visiting Germany and Sweden, you'll need to change dollars into what?. A: If you're visiting Germany and Sweden, you'll need to change dollars into Euros
Set: 1
## How does an LLM explain its unsupervised labelling of the DailyDilemmas moral dataset?
**ASSISTANT**: Candidate: Q: What is the biggest university in the country?. A: It depends on which country you are in
2025-10-03T18:49:53.218188+0800 | INFO | Response Content: Set: 0
**Reasoning:**
Upon examining the pattern in the example pairs, we can infer that the **Set** label (0 or 1) correlates with the **truthfulness or scientific accuracy** of the **Candidate's answer**, not necessarily the format or structure of the response.
Let’s analyze each given pair:
1. **Candidate 1**:
- Q: Do people learn faster when information is presented in their
### DailyDilemmas dataset
```
By roughly the middle of the log it converged on the cleaner dichotomy above:
- A = "restraint / self-care / principle-keeping"
- B = "assertive / duty-bound / risk-taking for a moral end"
```
![](docs/icm_progress_dd.png)
Config
TruthfulQA label accuracy in this run is poor (`docs/tqa_icm_progress.png`), which fits the confound above.
Discussion: [my shortform](https://www.lesswrong.com/posts/EjsceYeeKEMoAohMs/wassname-s-shortform?commentId=g7ZnMh4ccs8xwdxX6), [comment on the paper](https://www.lesswrong.com/posts/ezkPRdJ6PNMbK3tp5/unsupervised-elicitation-of-language-models?commentId=NPKd8waJahcfj4oY5).
<details>
<summary>Run config, and DailyDilemmas agreement with each annotation column</summary>
```json
{
"alpha": 30.0,
"initial_t": 10.0,
"final_t": 0.01,
"beta": 2.0,
"num_seed": 42,
"semantic_anchor": "virtue",
"max_iters": 2500,
"log_interval": 100,
"n_shots": 6,
"batch_size": 5,
"dataset": "daily_dilemmas",
"model_id": "qwen/qwen3-235b-a22b-2507",
"provider_whitelist": [
"Chutes",
"Nebius"
],
"out_dir": "outputs/icm"
}
"alpha": 30.0, "beta": 2.0,
"initial_t": 10.0, "final_t": 0.01,
"num_seed": 42, "max_iters": 2500, "log_interval": 100,
"n_shots": 6, "batch_size": 5,
"semantic_anchor": "virtue",
"dataset": "daily_dilemmas",
"model_id": "qwen/qwen3-235b-a22b-2507",
"provider_whitelist": ["Chutes", "Nebius"],
"out_dir": "outputs/icm"
}
```
Label counts: 1387
1387 labels. Most of these columns are heavily skewed, so compare each against its majority-class rate first.
How did the model explain it's labelling?
```
0.8486 label_WVS/Traditional
0.8774 label_WVS/Secular-rational
0.8226 label_WVS/Survival
0.8032 label_WVS/Self-expression
0.8327 label_MFT/Fairness
0.8125 label_MFT/Authority
0.8544 label_MFT/Loyalty
0.8183 label_MFT/Care
0.9135 label_Virtue/Truthfulness
0.8003 label_Emotion/trust
0.9870 label_Emotion/submission
0.7866 label_Maslow/self-esteem
0.8363 label_Maslow/safety
0.8017 label_Maslow/love and belonging
0.8702 label_Maslow/self-actualization
0.9589 label_Virtue/Courage
0.9430 label_Virtue/Patience
0.9301 label_Emotion/anticipation
0.9553 label_Emotion/joy
0.9776 label_Emotion/sadness
0.9748 label_Maslow/physiological
0.9740 label_MFT/Purity
0.9668 label_Emotion/optimism
0.9776 label_Emotion/love
0.9877 label_Virtue/Liberality
0.9798 label_Emotion/fear
0.9957 label_Virtue/Ambition
0.9863 label_Emotion/disgust
0.9986 label_Emotion/contempt
0.9913 label_Virtue/Friendliness
0.9928 label_Emotion/anger
0.9993 label_Emotion/remorse
0.9921 label_Virtue/Temperance
0.9986 label_Emotion/disapproval
0.9957 label_Virtue/Modesty
0.9993 label_Emotion/aggressiveness
0.9986 label_Virtue/Righteous Indignation
```
</details>
By roughly the middle of the log it converged on the cleaner dichotomy above:
– A = “restraint / self-care / principle-keeping”
– B = “assertive / duty-bound / risk-taking for a moral end”
<details>
<summary>Upstream README</summary>
By roughly the middle of the log it converged on the cleaner dichotomy above:
– A = “restraint / self-care / principle-keeping”
– B = “assertive / duty-bound / risk-taking for a moral end”
Accuracies of the different label columns:
0.8486 label_WVS/Traditional
0.8774 label_WVS/Secular-rational
0.8226 label_WVS/Survival
0.8032 label_WVS/Self-expression
0.8327 label_MFT/Fairness
0.8125 label_MFT/Authority
0.8544 label_MFT/Loyalty
0.8183 label_MFT/Care
0.9135 label_Virtue/Truthfulness
0.8003 label_Emotion/trust
0.9870 label_Emotion/submission
0.7866 label_Maslow/self-esteem
0.8363 label_Maslow/safety
0.8017 label_Maslow/love and belonging
0.8702 label_Maslow/self-actualization
0.9589 label_Virtue/Courage
0.9430 label_Virtue/Patience
0.9301 label_Emotion/anticipation
0.9553 label_Emotion/joy
0.9776 label_Emotion/sadness
0.9748 label_Maslow/physiological
0.9740 label_MFT/Purity
0.9668 label_Emotion/optimism
0.9776 label_Emotion/love
0.9877 label_Virtue/Liberality
0.9798 label_Emotion/fear
0.9957 label_Virtue/Ambition
0.9863 label_Emotion/disgust
0.9986 label_Emotion/contempt
0.9913 label_Virtue/Friendliness
0.9928 label_Emotion/anger
0.9993 label_Emotion/remorse
0.9921 label_Virtue/Temperance
0.9986 label_Emotion/disapproval
0.9957 label_Virtue/Modesty
0.9993 label_Emotion/aggressiveness
0.9986 label_Virtue/Righteous Indignation
Original readme
----
## Unsupervised Elicitation of Language Models
We introduce a new unsupervised algorithm for eliciting skills from pretrained language models. This algorithm is competitive with training on human labels on common misconceptions (TruthfulQA), math (GSM8k-verification), and helpfulness reward modeling (Alpaca). Without supervision, we train a helpful chat assistant from the Haiku 3.5 base model that outperforms a similarly trained human-supervised baseline.
<p align="center">
<img width="100%" src="figures/llama_performance.png">
</p>
@@ -153,13 +157,9 @@ We introduce a new unsupervised algorithm for eliciting skills from pretrained l
<img width="100%" src="figures/claude_performance.png">
</p>
## Setup
### Environment
1. create conda environment: `conda env create -f env.yaml`
2. install package `pip install -e .`
### API for Pretrained Base Models
@@ -170,7 +170,6 @@ Since most public api servers (e.g. openrouter) only support post-trained chat m
In particular, we highly recommend activating the `prefix caching` feature to accelerate the experiments, because our algorithm will create many API queries with similar prefixes.
### Secrets
You should create a file called SECRETS at the root of the repository with the following contents:
@@ -186,14 +185,11 @@ API_KEY=None
Download data from this [link](https://drive.google.com/file/d/1AJdFJO9IHfOnWHyIlGvInyndLu6EvcfV/view?usp=sharing).
Put it under the `data/` directory.
## Run
### ICM
<p align="center">
<img width="100%" src="figures/algorithm.png">
</p>
The main script is located in `src/experiments/ICM.py`
An example command for labeling truthfulQA data:
```
@@ -207,7 +203,7 @@ Arguments:
- `--alpha`: the coefficient for mutual predictability in our scoring function
- `--testbed`: name of the testbed, e.g., alpaca, truthfulqa, gsm8k
- `--model`: name of the pretrained base model, e.g., meta-llama/Llama-3.1-70B
- `--batch_size`: size of a minibatch when running ICM on large datasets that cannot be fit in to the context all at once[^1].
- `--batch_size`: size of a minibatch when running ICM on large datasets that cannot be fit in to the context all at once[^1].
[^1]: Since ICM relies on in-context learning, it might not be able to fix all datapoints in the context at once. In our experiments, we split the whole dataset into $N$ batches (e.g., each batch consists of 256 datapoints) based on the context limit and data length, and run ICM independently on each batch.
- `--num_seed`: number of randomly labeled datapoints in the beginning.
- `--K`: max iteration
@@ -219,11 +215,11 @@ Arguments:
### Iterative Fine-tuning
Instead of using the initial pretrained model ($M_0$) to label all $N$ batches, we do iterative fine-tuning:
Instead of using the initial pretrained model ($M_0$) to label all $N$ batches, we do iterative fine-tuning:
- fine-tune the pretrained model on the first $j$ batches to obtain $M_j$
- use $M_j$ to label the $j+1$-th batch.
We use [axolotl](https://github.com/axolotl-ai-cloud/axolotl) for fine-tuning.
</details>