From 8197fe1b1e4fa2b7203b48e3612f81356ba7d6d2 Mon Sep 17 00:00:00 2001 From: Lewis Tunstall Date: Mon, 9 Oct 2023 16:46:56 +0200 Subject: [PATCH] Update readme --- README.md | 46 ++++++++++++++++++--------- chapters/en/_toctree.yml | 4 +-- chapters/en/chapter0/introduction.mdx | 3 +- recipes/dpo/.gitkeep | 0 requirements.txt | 3 -- 5 files changed, 35 insertions(+), 21 deletions(-) create mode 100644 recipes/dpo/.gitkeep delete mode 100644 requirements.txt diff --git a/README.md b/README.md index f6606aa..2e3b5cb 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,36 @@ # The RLHF Handbook -Robust recipes for RLHF +Robust recipes for Reinforcement Learning from Human Feedback (RLHF) and related techniques to steer language models. -## Developing -To view this locally, run the following (need to point to a specific language if previewing only one section, preview/`build_doc` does not use the `--language` flag): -```shell -doc-builder preview rlhf-handbook {docs_dir} --not_python_module -``` -Example `docs_dir` is `~/Documents/HuggingFace/dev/rlhf-handbook/chapters/en` +## What is this? -## Installation -Create a new conda environment with: -```shell -conda create -n rlhf-handbook python=3.10 -``` -Install the limited requirements with -```shell -pip install -r requirements.txt +Just one year ago, chatbots were out of fashion and most people hadn't heard about techniques like RLHF to align language models with human preferences. Then, OpenAI broke the internet with ChatGPT and Meta followed suit by releasing the Llama series of language models which enabled the ML community to build their very own capable chatbots. This has led to a rich ecosystem of datasets and models that have mostly focused on teaching language models to follow instructions through supervised fine-tuning (SFT). + +However, we know from the [InstructGPT](https://huggingface.co/papers/2203.02155) and [Llama2](https://huggingface.co/papers/2307.09288) papers that significant gains in helpfulness and safety can be had by augmenting SFT with human (or AI) preferences. At the same time, combining reinforcement learning with NLP is a fairly novel idea and there are few pubic resources available on how to train these models, what data to collect, and what metrics to measure for best downstream performance. + +The RLHF Handbook aims to fill that gap by providing the community with a series of robust training recipes that span the whole RLHF pipeline. + +## Contents + +The initial release of the handbook will focus on the following techniques: + +* **Supervised fine-tuning:** teach languge language models to follow instructions and tips on how to collect and curate your own training dataset. +* **Reward modeling:** teach language models to distinguish model responses according to human or AI preferences. +* **Rejection sampling:** a simple, but powerful technique to boost the performance of your SFT model. +* **Proximal policy optimization (PPO):** the workhorse of RLHF, but notoriously sensitive to hyperparameters (like all good RL algorithms). +* **Direct preference optimisation (DPO):** a powerful and promising alternative to PPO. + +## Citation + +If you find the content of this repo useful in your work, please cite it as follows: + +```bibtex +@misc{rlhf_handbook2023, + author = {Lewis Tunstall and Edward Beeching and Nathan Lambert and Nazneen Rajani and Thomas Wolf}, + title = {The RLHF Handbook}, + year = {2023}, + publisher = {GitHub}, + journal = {GitHub repository}, + howpublished = {\url{https://github.com/huggingface/rlhf-handbook}} +} ``` \ No newline at end of file diff --git a/chapters/en/_toctree.yml b/chapters/en/_toctree.yml index 7dd7c2e..e8fc7c0 100644 --- a/chapters/en/_toctree.yml +++ b/chapters/en/_toctree.yml @@ -1,4 +1,4 @@ -- title: Unit 0. Welcome to the course! +- title: Unit 0. Welcome to the RLHF Handbook! sections: - local: chapter0/introduction - title: What to expect from the course \ No newline at end of file + title: What is this about? \ No newline at end of file diff --git a/chapters/en/chapter0/introduction.mdx b/chapters/en/chapter0/introduction.mdx index 8d0d381..26f500f 100644 --- a/chapters/en/chapter0/introduction.mdx +++ b/chapters/en/chapter0/introduction.mdx @@ -1,2 +1,3 @@ # Welcome to the RLHF Handbook! -We are going to... \ No newline at end of file + +Stay tuned for more details 🤗 \ No newline at end of file diff --git a/recipes/dpo/.gitkeep b/recipes/dpo/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/requirements.txt b/requirements.txt deleted file mode 100644 index 8244fc6..0000000 --- a/requirements.txt +++ /dev/null @@ -1,3 +0,0 @@ -hf-doc-builder>=0.4.0 -black>=23.3.0 -watchdog>=3.0.0 \ No newline at end of file