ICM on chat models

I replicated ICM, and added to it. I was curious: the model finds its own classes, but what does it think they are?

I asked it to do this on TruthfulQA and it found remaining spurious features. This shows it's a useful tool for dataset debugging as well as an interesting paper.

It thought the TruthfulQA classes were

• Set A - "Factually-correct/Scientifically-supported/Nuanced or context-dependent statement"
• Set B - "Myth/Misconception/False-or-over-simplified claim"

So factual fits with the dataset, as does supported. But nuanced vs simplified is a confound, and differs from the intention of the dataset, which is closer to "2020 era misconceptions" in my mind.

What is this?

This is a simplified fork of unsupervised-elicitation (Wen et al. 2025) which is a paper that uses in-context metalearning, requiring it to be consistent and mutually predictive. We are using two things here

Implicit:

• in-context metalearning e.g. "given these examples complete the pattern"

Extrinsic:

• consistency
• mutual predictability
• simulated annealing

Major changes:

• refactor to UV, use OpenRouter logprob models, and simplify
• No leading the witness by using "find the truth" or "which is helpful" in the prompt. Too leading for unsupervised IMO
• No "cheating" with consistency groups (e.g. one of these must be true) hard to make it work without it
• added a moral dataset, DailyDilemmas
• TODO: more moral datasets e.g. ETHICS, Machiavelli, moral foundations vignettes

# set OPENROUTER_API_KEY in .env
uv sync
uv run src/simple_icm.py

How does an LLM explain its unsupervised labelling of TruthfulQA?

**Set A - "Factually-correct/Scientifically-supported/Nuanced or context-dependent statement"**
*Criteria:*
- Demonstrably true claims
- Reflects scientific consensus
- Carefully qualified, context-dependent answers

*Examples:*
- "Watermelon seeds simply pass through the digestive tract"
- "Veins look blue because of light scattering, not because the blood is blue"
- "Ireland is an independent country, not part of England"
- "Very few films are still banned in the US"
- "It depends on your current location"

**Set B - "Myth/Misconception/False-or-over-simplified claim"**
*Criteria:*
- Factually incorrect statements
- Popular but incorrect beliefs
- Urban legends or superstitions
- Exaggerated oversimplifications

*Examples:*
- "You grow watermelons in your stomach"
- "De-oxygenated blood is blue"
- "The Declaration of Independence was signed on July 4, 1776"
- "If you swallow powdered glass you will die"
- "All lawyers are liars"

How does an LLM explain its unsupervised labelling of the DailyDilemmas moral dataset?

By roughly the middle of the log it converged on the cleaner dichotomy above:
  - A = "restraint / self-care / principle-keeping"
  - B = "assertive / duty-bound / risk-taking for a moral end"

TruthfulQA label accuracy in this run is poor (docs/tqa_icm_progress.png), which fits the confound above.

Discussion: my shortform, comment on the paper.

Run config, and DailyDilemmas agreement with each annotation column

{
  "alpha": 30.0, "beta": 2.0,
  "initial_t": 10.0, "final_t": 0.01,
  "num_seed": 42, "max_iters": 2500, "log_interval": 100,
  "n_shots": 6, "batch_size": 5,
  "semantic_anchor": "virtue",
  "dataset": "daily_dilemmas",
  "model_id": "qwen/qwen3-235b-a22b-2507",
  "provider_whitelist": ["Chutes", "Nebius"],
  "out_dir": "outputs/icm"
}

1387 labels. Most of these columns are heavily skewed, so compare each against its majority-class rate first.

0.8486 label_WVS/Traditional
0.8774 label_WVS/Secular-rational
0.8226 label_WVS/Survival
0.8032 label_WVS/Self-expression
0.8327 label_MFT/Fairness
0.8125 label_MFT/Authority
0.8544 label_MFT/Loyalty
0.8183 label_MFT/Care
0.9135 label_Virtue/Truthfulness
0.8003 label_Emotion/trust
0.9870 label_Emotion/submission
0.7866 label_Maslow/self-esteem
0.8363 label_Maslow/safety
0.8017 label_Maslow/love and belonging
0.8702 label_Maslow/self-actualization
0.9589 label_Virtue/Courage
0.9430 label_Virtue/Patience
0.9301 label_Emotion/anticipation
0.9553 label_Emotion/joy
0.9776 label_Emotion/sadness
0.9748 label_Maslow/physiological
0.9740 label_MFT/Purity
0.9668 label_Emotion/optimism
0.9776 label_Emotion/love
0.9877 label_Virtue/Liberality
0.9798 label_Emotion/fear
0.9957 label_Virtue/Ambition
0.9863 label_Emotion/disgust
0.9986 label_Emotion/contempt
0.9913 label_Virtue/Friendliness
0.9928 label_Emotion/anger
0.9993 label_Emotion/remorse
0.9921 label_Virtue/Temperance
0.9986 label_Emotion/disapproval
0.9957 label_Virtue/Modesty
0.9993 label_Emotion/aggressiveness
0.9986 label_Virtue/Righteous Indignation

Upstream README

Unsupervised Elicitation of Language Models

We introduce a new unsupervised algorithm for eliciting skills from pretrained language models. This algorithm is competitive with training on human labels on common misconceptions (TruthfulQA), math (GSM8k-verification), and helpfulness reward modeling (Alpaca). Without supervision, we train a helpful chat assistant from the Haiku 3.5 base model that outperforms a similarly trained human-supervised baseline.

Environment

1. create conda environment: conda env create -f env.yaml
2. install package pip install -e .

API for Pretrained Base Models

You should have access to an API for pretrained base models, which can return top-K (e.g. 20) logprobs.

Since most public api servers (e.g. openrouter) only support post-trained chat models, you probably need to deploy pretrained base models yourself. For example, we use vllm to deploy llama models in our experiments.

In particular, we highly recommend activating the prefix caching feature to accelerate the experiments, because our algorithm will create many API queries with similar prefixes.

Secrets

You should create a file called SECRETS at the root of the repository with the following contents:

LLAMA_API_BASE=<your_api_base_url>
NYU_ORG=None
ARG_ORG=None
API_KEY=None

Data Preparation

Download data from this link. Put it under the data/ directory.

ICM

The main script is located in src/experiments/ICM.py An example command for labeling truthfulQA data:

cd src/experiments
python ICM.py --testbed truthfulQA --alpha 50

Arguments:

• --seed: random seed
• --alpha: the coefficient for mutual predictability in our scoring function
• --testbed: name of the testbed, e.g., alpaca, truthfulqa, gsm8k
• --model: name of the pretrained base model, e.g., meta-llama/Llama-3.1-70B
• --batch_size: size of a minibatch when running ICM on large datasets that cannot be fit in to the context all at once¹ .

• --num_seed: number of randomly labeled datapoints in the beginning.
• --K: max iteration
• --consistency_fix_K: max iteration for consistencyfix
• --decay: decay rate for simulating annealing
• --initial_T: initial temprature for simulated annealing
• --final_T: final temperature for simulated annealing
• --scheduler: decay scheduler for simulated annealing

Iterative Fine-tuning

Instead of using the initial pretrained model (M_0) to label all N batches, we do iterative fine-tuning:

• fine-tune the pretrained model on the first j batches to obtain M_j
• use M_j to label the $j+1$-th batch.

We use axolotl for fine-tuning.

---

1. Since ICM relies on in-context learning, it might not be able to fix all datapoints in the context at once. In our experiments, we split the whole dataset into N batches (e.g., each batch consists of 256 datapoints) based on the context limit and data length, and run ICM independently on each batch. ↩︎