Unsupervised-Elicitation/README.md

Fork to

• add moral datasets e.g. daily dilemmas, ETHICS, Machiavelli, moral foundations vignettes
• refactor to UV
• and simplify
• replicate

Usage

uv sync
uv run src/experiments/ICM.py -- --testbed truthfulQA --alpha 50

Original readme

Unsupervised Elicitation of Language Models

We introduce a new unsupervised algorithm for eliciting skills from pretrained language models. This algorithm is competitive with training on human labels on common misconceptions (TruthfulQA), math (GSM8k-verification), and helpfulness reward modeling (Alpaca). Without supervision, we train a helpful chat assistant from the Haiku 3.5 base model that outperforms a similarly trained human-supervised baseline.

Setup

Environment

1. create conda environment: conda env create -f env.yaml

2. install package pip install -e .

API for Pretrained Base Models

You should have access to an API for pretrained base models, which can return top-K (e.g. 20) logprobs.

Since most public api servers (e.g. openrouter) only support post-trained chat models, you probably need to deploy pretrained base models yourself. For example, we use vllm to deploy llama models in our experiments.

In particular, we highly recommend activating the prefix caching feature to accelerate the experiments, because our algorithm will create many API queries with similar prefixes.

Secrets

You should create a file called SECRETS at the root of the repository with the following contents:

LLAMA_API_BASE=<your_api_base_url>
NYU_ORG=None
ARG_ORG=None
API_KEY=None

Data Preparation

Download data from this link. Put it under the data/ directory.

Run

ICM

The main script is located in src/experiments/ICM.py An example command for labeling truthfulQA data:

cd src/experiments
python ICM.py --testbed truthfulQA --alpha 50

Arguments:

• --seed: random seed
• --alpha: the coefficient for mutual predictability in our scoring function
• --testbed: name of the testbed, e.g., alpaca, truthfulqa, gsm8k
• --model: name of the pretrained base model, e.g., meta-llama/Llama-3.1-70B
• --batch_size: size of a minibatch when running ICM on large datasets that cannot be fit in to the context all at once¹ .

• --num_seed: number of randomly labeled datapoints in the beginning.
• --K: max iteration
• --consistency_fix_K: max iteration for consistencyfix
• --decay: decay rate for simulating annealing
• --initial_T: initial temprature for simulated annealing
• --final_T: final temperature for simulated annealing
• --scheduler: decay scheduler for simulated annealing

Iterative Fine-tuning

Instead of using the initial pretrained model (M_0) to label all N batches, we do iterative fine-tuning:

• fine-tune the pretrained model on the first j batches to obtain M_j

• use M_j to label the $j+1$-th batch.

We use axolotl for fine-tuning.

---

1. Since ICM relies on in-context learning, it might not be able to fix all datapoints in the context at once. In our experiments, we split the whole dataset into N batches (e.g., each batch consists of 256 datapoints) based on the context limit and data length, and run ICM independently on each batch. ↩︎