SimPO/on_policy_data_gen/README.md

# On-Policy Preference Data Generation

We provide the code to generate on-policy preference data (e.g., [princeton-nlp/llama3-ultrafeedback-armorm](https://huggingface.co/datasets/princeton-nlp/llama3-ultrafeedback-armorm) and [princeton-nlp/gemma2-ultrafeedback-armorm](https://huggingface.co/datasets/princeton-nlp/gemma2-ultrafeedback-armorm)) used in our experiments.

## Requirements

You will need to install the [`vllm`](https://github.com/vllm-project/vllm) package for decoding. Moreover, if you are running decoding with `gemma-2` models, you will need to also install `flashinfer`.

## On-Policy Preference Data Generation Process

1. Generate multiple responses using the language model:

```
python decode.py --data_dir $DATASET_DIR --seed $SEED
```
This will generate one response per prompt under the specified seed. You need to provide a dataset containing prompts (by default, we use `HuggingFaceH4/ultrafeedback_binarized`). You can also set decoding hyperparameters by passing in corresponding arguments (by default, we use a temperature of `0.8` for sampling).

Note that you will need to run the above command under **multiple different** seeds (by default, we use `13, 21, 42, 79, 100`) to obtain different responses for each prompt.

2. Post-process the generations

```
python post_process.py
```

This will combine the generated responses under each seed and filter out samples with identical responses across all seeds.

3. Annotate the preference labels with a reward model

```
python reward_model_annotate.py --reward_model $MODEL
```

This will score the generations using a reward model (by default, we use `RLHFlow/ArmoRM-Llama3-8B-v0.1`) and binarize the dataset by taking the highest-scoring response as the winning response and the lowest-scoring one as the losing.