release

2026-08-12 11:40:25 +08:00 · 2024-05-23 22:53:13 -04:00
commit 586e5e3d0a
12 changed files with 964 additions and 0 deletions
@@ -0,0 +1,88 @@
+# Simple Preference Optimization (SimPO)
+
+This repository contains the code and released models for our paper [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://arxiv.org/abs/2405.14734). We propose a simpler and more effective preference optimization algorithm than DPO (Direct Preference Optimization) without using a reference model. SimPO outperforms DPO and its latest variants across AlpacaEval 2, MT-Bench, and Arena-Hard benchmarks under various settings.
+
+<img src="./SimPO.png" width="1000px"></img>
+
+## 🔗 Quick Links
+- [SimPO: Simple Preference Optimization with a Reference-Free Reward](#simple-preference-optimization-simpo)
+  - [Released Models](#released-models)
+  - [Install Requirements](#install-requirements)
+  - [Training scripts](#training-scripts)
+  - [Evaluation](#evaluation)
+  - [Bugs or Questions?](#bugs-or-questions)
+  - [Citation](#citation)
+
+## Released Models
+
+
+## Install Requirements
+
+Our codebase is built upon the [alignment-handbook repo](https://github.com/huggingface/alignment-handbook). The following steps will guide you through the installation process.
+
+First, create a Python virtual environment using e.g. Conda:
+```shell
+conda create -n handbook python=3.10 && conda activate handbook
+```
+
+Next, install PyTorch `v2.2.2`. Since this is hardware-dependent, we
+direct you to the [PyTorch Installation Page](https://pytorch.org/get-started/locally/).
+
+You can then install the remaining package dependencies of [alignment-handbook](https://github.com/huggingface/alignment-handbook) as follows:
+
+```shell
+git clone https://github.com/huggingface/alignment-handbook.git
+cd ./alignment-handbook/
+python -m pip install .
+```
+
+You will also need Flash Attention 2 installed, which can be done by running:
+
+```shell
+python -m pip install flash-attn --no-build-isolation
+```
+
+## Training Scripts
+
+We provide four training config files for the four training setups reported in our paper:
+
+* Mistral-Base:
+```shell
+ACCELERATE_LOG_LEVEL=info accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml scripts/run_simpo.py training_configs/mistral-7b-base-simpo.yaml
+```
+* Mistral-Instruct:
+```shell
+ACCELERATE_LOG_LEVEL=info accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml scripts/run_simpo.py training_configs/mistral-7b-instruct-simpo.yaml
+```
+* Llama3-Base:
+```shell
+ACCELERATE_LOG_LEVEL=info accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml scripts/run_simpo.py training_configs/llama-3-8b-base-simpo.yaml
+```
+* Llama3-Instruct:
+```shell
+ACCELERATE_LOG_LEVEL=info accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml scripts/run_simpo.py training_configs/llama-3-8b-instruct-simpo.yaml
+```
+
+## Evaluation
+
+We follow the official implementation for evaluation on AlpacaEval 2, Arena-Hard, and MT-Bench, as follows:
+
+* AlpacaEval 2: Please refer to the [AlpacaEval repo](https://github.com/tatsu-lab/alpaca_eval) for evaluation.
+
+* Arena-Hard: Please refer to to the [Arena-Hard-Auto repo](https://github.com/lm-sys/arena-hard-auto) for evaluation.
+
+* MT-Bench: Please refer to the [FastChat repo](https://github.com/lm-sys/FastChat) for evaluation.
+
+## Bugs or Questions?
+If you have any questions related to the code or the paper, feel free to email Yu (yumeng5@virginia.edu). If you encounter any problems when using the code, or want to report a bug, feel free to open an issue! Please try to specify the problem with details so we can help you better and quicker!
+
+## Citation
+Please cite our paper if you find the repo helpful in your work:
+
+```bibtex
+@article{meng2024simpo,
+  title={{SimPO}: Simple Preference Optimization with a Reference-Free Reward},
+  author={Meng, Yu and Xia, Mengzhou and Chen, Danqi},
+  year={2024}
+}
+```