diff --git a/.github/workflows/build_documentation.yml b/.github/workflows/build_documentation.yml
new file mode 100644
index 0000000..9e58cb8
--- /dev/null
+++ b/.github/workflows/build_documentation.yml
@@ -0,0 +1,18 @@
+name: Build documentation
+
+on:
+ push:
+ branches:
+ - main
+ - doc-builder*
+ - v*-release
+
+jobs:
+ build:
+ uses: huggingface/doc-builder/.github/workflows/build_main_documentation.yml@main
+ with:
+ commit_sha: ${{ github.sha }}
+ package: peft
+ notebook_folder: peft_docs
+ secrets:
+ token: ${{ secrets.HUGGINGFACE_PUSH }}
\ No newline at end of file
diff --git a/.github/workflows/build_pr_documentation.yml b/.github/workflows/build_pr_documentation.yml
new file mode 100644
index 0000000..7506143
--- /dev/null
+++ b/.github/workflows/build_pr_documentation.yml
@@ -0,0 +1,16 @@
+name: Build PR Documentation
+
+on:
+ pull_request:
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
+ cancel-in-progress: true
+
+jobs:
+ build:
+ uses: huggingface/doc-builder/.github/workflows/build_pr_documentation.yml@main
+ with:
+ commit_sha: ${{ github.event.pull_request.head.sha }}
+ pr_number: ${{ github.event.number }}
+ package: peft
diff --git a/.github/workflows/delete_doc_comment.yml b/.github/workflows/delete_doc_comment.yml
new file mode 100644
index 0000000..e86cc2d
--- /dev/null
+++ b/.github/workflows/delete_doc_comment.yml
@@ -0,0 +1,13 @@
+name: Delete dev documentation
+
+on:
+ pull_request:
+ types: [ closed ]
+
+
+jobs:
+ delete:
+ uses: huggingface/doc-builder/.github/workflows/delete_doc_comment.yml@main
+ with:
+ pr_number: ${{ github.event.number }}
+ package: peft
diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml
new file mode 100644
index 0000000..8ad3e6d
--- /dev/null
+++ b/.github/workflows/stale.yml
@@ -0,0 +1,27 @@
+name: Stale Bot
+
+on:
+ schedule:
+ - cron: "0 15 * * *"
+
+jobs:
+ close_stale_issues:
+ name: Close Stale Issues
+ if: github.repository == 'huggingface/peft'
+ runs-on: ubuntu-latest
+ env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ steps:
+ - uses: actions/checkout@v3
+
+ - name: Setup Python
+ uses: actions/setup-python@v4
+ with:
+ python-version: 3.8
+
+ - name: Install requirements
+ run: |
+ pip install PyGithub
+ - name: Close stale issues
+ run: |
+ python scripts/stale.py
\ No newline at end of file
diff --git a/Makefile b/Makefile
index 61549db..8c8b9d3 100644
--- a/Makefile
+++ b/Makefile
@@ -1,6 +1,6 @@
.PHONY: quality style test docs
-check_dirs := src tests examples
+check_dirs := src tests examples docs
# Check that source code meets quality standards
@@ -8,13 +8,13 @@ check_dirs := src tests examples
quality:
black --check $(check_dirs)
ruff $(check_dirs)
- doc-builder style src tests --max_len 119 --check_only
+ doc-builder style src/peft tests docs/source --max_len 119 --check_only
# Format source code automatically and check is there are any problems left that need manual fixing
style:
black $(check_dirs)
ruff $(check_dirs) --fix
- doc-builder style src tests --max_len 119
+ doc-builder style src/peft tests docs/source --max_len 119
test:
- pytest tests/
\ No newline at end of file
+ pytest -n 3 tests/
\ No newline at end of file
diff --git a/README.md b/README.md
index 3a9726e..ccdcd55 100644
--- a/README.md
+++ b/README.md
@@ -25,10 +25,10 @@ Seamlessly integrated with 🤗 Accelerate for large scale models leveraging Dee
Supported methods:
-1. LoRA: [LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS](https://arxiv.org/pdf/2106.09685.pdf)
+1. LoRA: [LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS](https://arxiv.org/abs/2106.09685)
2. Prefix Tuning: [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://aclanthology.org/2021.acl-long.353/), [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://arxiv.org/pdf/2110.07602.pdf)
-3. P-Tuning: [GPT Understands, Too](https://arxiv.org/pdf/2103.10385.pdf)
-4. Prompt Tuning: [The Power of Scale for Parameter-Efficient Prompt Tuning](https://arxiv.org/pdf/2104.08691.pdf)
+3. P-Tuning: [GPT Understands, Too](https://arxiv.org/abs/2103.10385)
+4. Prompt Tuning: [The Power of Scale for Parameter-Efficient Prompt Tuning](https://arxiv.org/abs/2104.08691)
## Getting started
@@ -64,7 +64,7 @@ Hardware: Single A100 80GB GPU with CPU RAM above 64GB
| bigscience/bloomz-7b1 (7B params) | OOM GPU | 32GB GPU / 3.8GB CPU | 18.1GB GPU / 35GB CPU |
Performance of PEFT-LoRA tuned [`bigscience/T0_3B`](https://huggingface.co/bigscience/T0_3B) on [`ought/raft/twitter_complaints`](https://huggingface.co/datasets/ought/raft/viewer/twitter_complaints) leaderboard.
-A point to note is that we didn't try to sequeeze performance by playing around with input instruction templates, LoRA hyperparams and other training related hyperparams. Also, we didn't use the larger 13B [mt0-xxl](https://huggingface.co/bigscience/mt0-xxl) model.
+A point to note is that we didn't try to squeeze performance by playing around with input instruction templates, LoRA hyperparams and other training related hyperparams. Also, we didn't use the larger 13B [mt0-xxl](https://huggingface.co/bigscience/mt0-xxl) model.
So, we are already seeing comparable performance to SoTA with parameter efficient tuning. Also, the final checkpoint size is just `19MB` in comparison to `11GB` size of the backbone [`bigscience/T0_3B`](https://huggingface.co/bigscience/T0_3B) model.
| Submission Name | Accuracy |
@@ -81,7 +81,7 @@ GPU memory required by different settings during training is given below. The fi
Hardware: Single A100 80GB GPU with CPU RAM above 64GB
-| Model | Full Finetuning | PEFT-LoRA | PEFT-LoRA with Gradient Checkpoitning |
+| Model | Full Finetuning | PEFT-LoRA | PEFT-LoRA with Gradient Checkpointing |
| --------- | ---- | ---- | ---- |
| CompVis/stable-diffusion-v1-4 | 27.5GB GPU / 3.97GB CPU | 15.5GB GPU / 3.84GB CPU | 8.12GB GPU / 3.77GB CPU |
@@ -126,14 +126,14 @@ Try out the 🤗 Gradio Space which should run seamlessly on a T4 instance:
### Parameter Efficient Tuning of LLMs for RLHF components such as Ranker and Policy
-- Here is an exmaple in [trl](https://github.com/lvwerra/trl) library using PEFT+INT8 for tuning policy model: [gpt2-sentiment_peft.py](https://github.com/lvwerra/trl/blob/main/examples/sentiment/scripts/gpt2-sentiment_peft.py)
+- Here is an example in [trl](https://github.com/lvwerra/trl) library using PEFT+INT8 for tuning policy model: [gpt2-sentiment_peft.py](https://github.com/lvwerra/trl/blob/main/examples/sentiment/scripts/gpt2-sentiment_peft.py)
- Example using PEFT for both reward model and policy [ToDo]
### INT8 training of large models in Colab using PEFT LoRA and bits_and_bytes
-- Here is now a demo on how to fine tune [OPT-6.7b](https://huggingface.co/facebook/opt-6.7b) (14GB in fp16) in a Google colab: [](https://colab.research.google.com/drive/1jCkpikz0J2o20FBQmYmAGdiKmJGOMo-o?usp=sharing)
+- Here is now a demo on how to fine tune [OPT-6.7b](https://huggingface.co/facebook/opt-6.7b) (14GB in fp16) in a Google Colab: [](https://colab.research.google.com/drive/1jCkpikz0J2o20FBQmYmAGdiKmJGOMo-o?usp=sharing)
-- Here is now a demo on how to fine tune [whishper-large](openai/whisper-large-v2) (1.5B params) (14GB in fp16) in a Google colab: [](https://colab.research.google.com/drive/1DOkD_5OUjFa0r5Ik3SgywJLJtEo2qLxO?usp=sharing) and [](https://colab.research.google.com/drive/1vhF8yueFqha3Y3CpTHN6q9EVcII9EYzs?usp=sharing)
+- Here is now a demo on how to fine tune [whishper-large](openai/whisper-large-v2) (1.5B params) (14GB in fp16) in a Google Colab: [](https://colab.research.google.com/drive/1DOkD_5OUjFa0r5Ik3SgywJLJtEo2qLxO?usp=sharing) and [](https://colab.research.google.com/drive/1vhF8yueFqha3Y3CpTHN6q9EVcII9EYzs?usp=sharing)
### Save compute and storage even for medium and small models
@@ -143,12 +143,12 @@ performance comparable to full finetuning.
An example of using LoRA for the task of adapting `LayoutLMForTokenClassification` on `FUNSD` dataset is given in `~examples/token_classification/PEFT_LoRA_LayoutLMForTokenClassification_on_FUNSD.py`. We can observe that with only `0.62 %` of parameters being trainable, we achieve performance (F1 0.777) comparable to full finetuning (F1 0.786) (without any hyerparam tuning runs for extracting more performance), and the checkpoint of this is only `2.8MB`. Now, if there are `N` such datasets, just have these PEFT models one for each dataset and save a lot of storage without having to worry about the problem of catastrophic forgetting or overfitting of backbone/base model.
-Another example is fine-tuning [`roberta-large`](https://huggingface.co/roberta-large) on [`MRPC` GLUE](https://huggingface.co/datasets/glue/viewer/mrpc) dataset suing differenct PEFT methods. The notebooks are given in `~examples/sequence_classification`.
+Another example is fine-tuning [`roberta-large`](https://huggingface.co/roberta-large) on [`MRPC` GLUE](https://huggingface.co/datasets/glue/viewer/mrpc) dataset using different PEFT methods. The notebooks are given in `~examples/sequence_classification`.
## PEFT + 🤗 Accelerate
-PEFT models work with 🤗 Accelerate out of the box. Use 🤗 Accelerate for Distributed training on various hardware such as GPUs, Apple Silicon devices etc during training.
+PEFT models work with 🤗 Accelerate out of the box. Use 🤗 Accelerate for Distributed training on various hardware such as GPUs, Apple Silicon devices, etc during training.
Use 🤗 Accelerate for inferencing on consumer hardware with small resources.
### Example of PEFT model training using 🤗 Accelerate's DeepSpeed integration
diff --git a/docs/Makefile b/docs/Makefile
new file mode 100644
index 0000000..8879933
--- /dev/null
+++ b/docs/Makefile
@@ -0,0 +1,19 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS =
+SPHINXBUILD = sphinx-build
+SOURCEDIR = source
+BUILDDIR = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+ @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+ @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
\ No newline at end of file
diff --git a/docs/README.md b/docs/README.md
new file mode 100644
index 0000000..5955736
--- /dev/null
+++ b/docs/README.md
@@ -0,0 +1,267 @@
+
+
+# Generating the documentation
+
+To generate the documentation, you first have to build it. Several packages are necessary to build the doc,
+you can install them with the following command, at the root of the code repository:
+
+```bash
+pip install -e ".[docs]"
+```
+
+Then you need to install our special tool that builds the documentation:
+
+```bash
+pip install git+https://github.com/huggingface/doc-builder
+```
+
+---
+**NOTE**
+
+You only need to generate the documentation to inspect it locally (if you're planning changes and want to
+check how they look before committing for instance). You don't have to commit the built documentation.
+
+---
+
+## Building the documentation
+
+Once you have setup the `doc-builder` and additional packages, you can generate the documentation by
+typing the following command:
+
+```bash
+doc-builder build peft docs/source/ --build_dir ~/tmp/test-build
+```
+
+You can adapt the `--build_dir` to set any temporary folder that you prefer. This command will create it and generate
+the MDX files that will be rendered as the documentation on the main website. You can inspect them in your favorite
+Markdown editor.
+
+## Previewing the documentation
+
+To preview the docs, first install the `watchdog` module with:
+
+```bash
+pip install watchdog
+```
+
+Then run the following command:
+
+```bash
+doc-builder preview {package_name} {path_to_docs}
+```
+
+For example:
+
+```bash
+doc-builder preview peft docs/source
+```
+
+The docs will be viewable at [http://localhost:3000](http://localhost:3000). You can also preview the docs once you have opened a PR. You will see a bot add a comment to a link where the documentation with your changes lives.
+
+---
+**NOTE**
+
+The `preview` command only works with existing doc files. When you add a completely new file, you need to update `_toctree.yml` & restart `preview` command (`ctrl-c` to stop it & call `doc-builder preview ...` again).
+
+---
+
+## Adding a new element to the navigation bar
+
+Accepted files are Markdown (.md or .mdx).
+
+Create a file with its extension and put it in the source directory. You can then link it to the toc-tree by putting
+the filename without the extension in the [`_toctree.yml`](https://github.com/huggingface/peft/blob/main/docs/source/_toctree.yml) file.
+
+## Renaming section headers and moving sections
+
+It helps to keep the old links working when renaming the section header and/or moving sections from one document to another. This is because the old links are likely to be used in Issues, Forums, and Social media and it'd make for a much more superior user experience if users reading those months later could still easily navigate to the originally intended information.
+
+Therefore, we simply keep a little map of moved sections at the end of the document where the original section was. The key is to preserve the original anchor.
+
+So if you renamed a section from: "Section A" to "Section B", then you can add at the end of the file:
+
+```
+Sections that were moved:
+
+[ Section A ]
+```
+and of course, if you moved it to another file, then:
+
+```
+Sections that were moved:
+
+[ Section A ]
+```
+
+Use the relative style to link to the new file so that the versioned docs continue to work.
+
+
+## Writing Documentation - Specification
+
+The `huggingface/peft` documentation follows the
+[Google documentation](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) style for docstrings,
+although we can write them directly in Markdown.
+
+### Adding a new tutorial
+
+Adding a new tutorial or section is done in two steps:
+
+- Add a new file under `./source`. This file can either be ReStructuredText (.rst) or Markdown (.md).
+- Link that file in `./source/_toctree.yml` on the correct toc-tree.
+
+Make sure to put your new file under the proper section. It's unlikely to go in the first section (*Get Started*), so
+depending on the intended targets (beginners, more advanced users, or researchers) it should go in sections two, three, or
+four.
+
+### Writing source documentation
+
+Values that should be put in `code` should either be surrounded by backticks: \`like so\`. Note that argument names
+and objects like True, None, or any strings should usually be put in `code`.
+
+When mentioning a class, function, or method, it is recommended to use our syntax for internal links so that our tool
+adds a link to its documentation with this syntax: \[\`XXXClass\`\] or \[\`function\`\]. This requires the class or
+function to be in the main package.
+
+If you want to create a link to some internal class or function, you need to
+provide its path. For instance: \[\`utils.gather\`\]. This will be converted into a link with
+`utils.gather` in the description. To get rid of the path and only keep the name of the object you are
+linking to in the description, add a ~: \[\`~utils.gather\`\] will generate a link with `gather` in the description.
+
+The same works for methods so you can either use \[\`XXXClass.method\`\] or \[~\`XXXClass.method\`\].
+
+#### Defining arguments in a method
+
+Arguments should be defined with the `Args:` (or `Arguments:` or `Parameters:`) prefix, followed by a line return and
+an indentation. The argument should be followed by its type, with its shape if it is a tensor, a colon, and its
+description:
+
+```
+ Args:
+ n_layers (`int`): The number of layers of the model.
+```
+
+If the description is too long to fit in one line (more than 119 characters in total), another indentation is necessary
+before writing the description after the argument.
+
+Finally, to maintain uniformity if any *one* description is too long to fit on one line, the
+rest of the parameters should follow suit and have an indention before their description.
+
+Here's an example showcasing everything so far:
+
+```
+ Args:
+ gradient_accumulation_steps (`int`, *optional*, default to 1):
+ The number of steps that should pass before gradients are accumulated. A number > 1 should be combined with `Accelerator.accumulate`.
+ cpu (`bool`, *optional*):
+ Whether or not to force the script to execute on CPU. Will ignore GPU available if set to `True` and force the execution on one process only.
+```
+
+For optional arguments or arguments with defaults we follow the following syntax: imagine we have a function with the
+following signature:
+
+```
+def my_function(x: str = None, a: float = 1):
+```
+
+then its documentation should look like this:
+
+```
+ Args:
+ x (`str`, *optional*):
+ This argument controls ... and has a description longer than 119 chars.
+ a (`float`, *optional*, defaults to 1):
+ This argument is used to ... and has a description longer than 119 chars.
+```
+
+Note that we always omit the "defaults to \`None\`" when None is the default for any argument. Also note that even
+if the first line describing your argument type and its default gets long, you can't break it on several lines. You can
+however write as many lines as you want in the indented description (see the example above with `input_ids`).
+
+#### Writing a multi-line code block
+
+Multi-line code blocks can be useful for displaying examples. They are done between two lines of three backticks as usual in Markdown:
+
+
+````
+```python
+# first line of code
+# second line
+# etc
+```
+````
+
+#### Writing a return block
+
+The return block should be introduced with the `Returns:` prefix, followed by a line return and an indentation.
+The first line should be the type of the return, followed by a line return. No need to indent further for the elements
+building the return.
+
+Here's an example of a single value return:
+
+```
+ Returns:
+ `List[int]`: A list of integers in the range [0, 1] --- 1 for a special token, 0 for a sequence token.
+```
+
+Here's an example of a tuple return, comprising several objects:
+
+```
+ Returns:
+ `tuple(torch.FloatTensor)` comprising various elements depending on the configuration ([`BertConfig`]) and inputs:
+ - ** loss** (*optional*, returned when `masked_lm_labels` is provided) `torch.FloatTensor` of shape `(1,)` --
+ Total loss is the sum of the masked language modeling loss and the next sequence prediction (classification) loss.
+ - **prediction_scores** (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`) --
+ Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
+```
+
+## Styling the docstring
+
+We have an automatic script running with the `make style` comment that will make sure that:
+- the docstrings fully take advantage of the line width
+- all code examples are formatted using black, like the code of the Transformers library
+
+This script may have some weird failures if you made a syntax mistake or if you uncover a bug. Therefore, it's
+recommended to commit your changes before running `make style`, so you can revert the changes done by that script
+easily.
+
+## Writing documentation examples
+
+The syntax for Example docstrings can look as follows:
+
+```
+ Example:
+
+ ```python
+ >>> import time
+ >>> from accelerate import Accelerator
+ >>> accelerator = Accelerator()
+ >>> if accelerator.is_main_process:
+ ... time.sleep(2)
+ >>> else:
+ ... print("I'm waiting for the main process to finish its sleep...")
+ >>> accelerator.wait_for_everyone()
+ >>> # Should print on every process at the same time
+ >>> print("Everyone is here")
+ ```
+```
+
+The docstring should give a minimal, clear example of how the respective function
+is to be used in inference and also include the expected (ideally sensible)
+output.
+Often, readers will try out the example before even going through the function
+or class definitions. Therefore, it is of utmost importance that the example
+works as expected.
\ No newline at end of file
diff --git a/docs/source/_config.py b/docs/source/_config.py
new file mode 100644
index 0000000..2974756
--- /dev/null
+++ b/docs/source/_config.py
@@ -0,0 +1,7 @@
+# docstyle-ignore
+INSTALL_CONTENT = """
+# PEFT installation
+! pip install peft accelerate transformers
+# To install from source instead of the last release, comment the command above and uncomment the following one.
+# ! pip install git+https://github.com/huggingface/peft.git
+"""
diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml
new file mode 100644
index 0000000..12f8901
--- /dev/null
+++ b/docs/source/_toctree.yml
@@ -0,0 +1,23 @@
+- title: Get started
+ sections:
+ - local: index
+ title: 🤗 PEFT
+ - local: quicktour
+ title: Quicktour
+ - local: install
+ title: Installation
+
+- title: Task guides
+ sections:
+ - local: task_guides/image_classification_lora
+ title: Image classification using LoRA
+
+- title: Reference
+ sections:
+ - local: package_reference/peft_model
+ title: PEFT model
+ - local: package_reference/config
+ title: Configuration
+ - local: package_reference/tuners
+ title: Tuners
+
diff --git a/docs/source/index.mdx b/docs/source/index.mdx
new file mode 100644
index 0000000..008be12
--- /dev/null
+++ b/docs/source/index.mdx
@@ -0,0 +1,26 @@
+
+
+# PEFT
+
+🤗 PEFT, or Parameter-Efficient Fine-Tuning (PEFT), is a library for efficiently adapting pre-trained language models (PLMs) to various downstream applications without fine-tuning all the model's parameters.
+PEFT methods only fine-tune a small number of (extra) model parameters, significantly decreasing computational and storage costs because fine-tuning large-scale PLMs is prohibitively costly.
+Recent state-of-the-art PEFT techniques achieve performance comparable to that of full fine-tuning.
+
+PEFT is seamlessly integrated with 🤗 Accelerate for large-scale models leveraging DeepSpeed and [Big Model Inference](https://huggingface.co/docs/accelerate/usage_guides/big_modeling).
+
+Supported methods include:
+
+1. LoRA: [LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS](https://arxiv.org/pdf/2106.09685.pdf)
+2. Prefix Tuning: [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://aclanthology.org/2021.acl-long.353/), [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://arxiv.org/pdf/2110.07602.pdf)
+3. P-Tuning: [GPT Understands, Too](https://arxiv.org/pdf/2103.10385.pdf)
+4. Prompt Tuning: [The Power of Scale for Parameter-Efficient Prompt Tuning](https://arxiv.org/pdf/2104.08691.pdf)
diff --git a/docs/source/install.mdx b/docs/source/install.mdx
new file mode 100644
index 0000000..5f5ecff
--- /dev/null
+++ b/docs/source/install.mdx
@@ -0,0 +1,43 @@
+
+
+# Installation
+
+Before you start, you will need to setup your environment, install the appropriate packages, and configure 🤗 PEFT. 🤗 PEFT is tested on **Python 3.7+**.
+
+🤗 PEFT is available on pypi, as well as GitHub:
+
+## pip
+
+To install 🤗 PEFT from pypi:
+
+```bash
+pip install peft
+```
+
+## Source
+
+New features that haven't been released yet are added every day, which also means there may be some bugs. To try them out, install from the GitHub repository:
+
+```bash
+pip install git+https://github.com/huggingface/peft
+```
+
+If you're working on contributing to the library or wish to play with the source code and see live
+results as you run the code, an editable version can be installed from a locally-cloned version of the
+repository:
+
+```bash
+git clone https://github.com/huggingface/peft
+cd peft
+pip install -e .
+```
diff --git a/docs/source/package_reference/config.mdx b/docs/source/package_reference/config.mdx
new file mode 100644
index 0000000..6866685
--- /dev/null
+++ b/docs/source/package_reference/config.mdx
@@ -0,0 +1,18 @@
+# Configuration
+
+The configuration classes stores the configuration of a [`PeftModel`], PEFT adapter models, and the configurations of [`PrefixTuning`], [`PromptTuning`], and [`PromptEncoder`]. They contain methods for saving and loading model configurations from the Hub, specifying the PEFT method to use, type of task to perform, and model configurations like number of layers and number of attention heads.
+
+## PeftConfigMixin
+
+[[autodoc]] utils.config.PeftConfigMixin
+ - all
+
+## PeftConfig
+
+[[autodoc]] PeftConfig
+ - all
+
+## PromptLearningConfig
+
+[[autodoc]] PromptLearningConfig
+ - all
diff --git a/docs/source/package_reference/peft_model.mdx b/docs/source/package_reference/peft_model.mdx
new file mode 100644
index 0000000..f2618ef
--- /dev/null
+++ b/docs/source/package_reference/peft_model.mdx
@@ -0,0 +1,36 @@
+# Models
+
+[`PeftModel`] is the base model class for specifying the base Transformer model and configuration to apply a PEFT method to. The base `PeftModel` contains methods for loading and saving models from the Hub, and supports the [`PromptEncoder`] for prompt learning.
+
+## PeftModel
+
+[[autodoc]] PeftModel
+ - all
+
+## PeftModelForSequenceClassification
+
+A `PeftModel` for sequence classification tasks.
+
+[[autodoc]] PeftModelForSequenceClassification
+ - all
+
+## PeftModelForTokenClassification
+
+A `PeftModel` for token classification tasks.
+
+[[autodoc]] PeftModelForTokenClassification
+ - all
+
+## PeftModelForCausalLM
+
+A `PeftModel` for causal language modeling.
+
+[[autodoc]] PeftModelForCausalLM
+ - all
+
+## PeftModelForSeq2SeqLM
+
+A `PeftModel` for sequence-to-sequence language modeling.
+
+[[autodoc]] PeftModelForSeq2SeqLM
+ - all
diff --git a/docs/source/package_reference/tuners.mdx b/docs/source/package_reference/tuners.mdx
new file mode 100644
index 0000000..2ec0824
--- /dev/null
+++ b/docs/source/package_reference/tuners.mdx
@@ -0,0 +1,35 @@
+# Tuners
+
+Each tuner (or PEFT method) has a configuration and model.
+
+## LoRA
+
+For finetuning a model with LoRA.
+
+[[autodoc]] LoraConfig
+
+[[autodoc]] LoraModel
+
+[[autodoc]] tuners.lora.LoraLayer
+
+[[autodoc]] tuners.lora.Linear
+
+[[autodoc]] tuners.lora.MergedLinear
+
+## P-tuning
+
+[[autodoc]] tuners.p_tuning.PromptEncoderConfig
+
+[[autodoc]] tuners.p_tuning.PromptEncoder
+
+## Prefix tuning
+
+[[autodoc]] tuners.prefix_tuning.PrefixTuningConfig
+
+[[autodoc]] tuners.prefix_tuning.PrefixEncoder
+
+## Prompt tuning
+
+[[autodoc]] tuners.prompt_tuning.PromptTuningConfig
+
+[[autodoc]] tuners.prompt_tuning.PromptEmbedding
\ No newline at end of file
diff --git a/docs/source/quicktour.mdx b/docs/source/quicktour.mdx
new file mode 100644
index 0000000..a625aa8
--- /dev/null
+++ b/docs/source/quicktour.mdx
@@ -0,0 +1,296 @@
+
+
+# Quick tour
+
+Let's have a look at 🤗 PEFT's main features and learn how to set up a `PeftModel` and train it with 🤗 Accelerate's DeepSpeed integration and use it for inference.
+
+## Main use
+
+To use 🤗 PEFT in your script:
+
+1. Each PEFT method is defined by a `PeftConfig` object.
+
+Create a `PeftConfig` object corresponding to your PEFT method (see the [Configuration](package_reference/config) reference for more details) and [`TaskType`], the type of task you're training your model for.
+This example trains the [`bigscience/mt0-large`](https://huggingface.co/bigscience/mt0-large) model with the Low-Rank Adaptation of Large Language Models (LoRA) method. Load the `LoRAConfig`, and specify the `task_type` for sequence-to-sequence language modeling.
+
+```python
+from peft import LoraConfig, TaskType
+
+peft_config = LoraConfig(task_type=TaskType.SEQ_2_SEQ_LM, inference_mode=False, r=8, lora_alpha=32, lora_dropout=0.1)
+```
+
+2. Load the base model you want to fine-tune.
+
+```python
+from transformers import AutoModelForSeq2SeqLM
+
+model_name_or_path = "bigscience/mt0-large"
+tokenizer_name_or_path = "bigscience/mt0-large"
+model = AutoModelForSeq2SeqLM.from_pretrained(model_name_or_path)
+```
+
+3. Preprocess your model if you use [`bitsandbytes`](https://github.com/TimDettmers/bitsandbytes) for `int8` quantized training; otherwise, skip this step.
+
+```python
+from peft import prepare_model_for_int8_training
+
+model = prepare_model_for_int8_training(model)
+```
+
+4. Wrap your model in the `PeftModel` object using the `get_peft_model` function. Also, check the number of trainable parameters of your model.
+
+```python
+from peft import get_peft_model
+
+model = get_peft_model(model, peft_config)
+model.print_trainable_parameters()
+# output: trainable params: 2359296 || all params: 1231940608 || trainable%: 0.19151053100118282
+```
+
+5. Voila 🎉! Now, train the model using the 🤗 Transformers Trainer API, 🤗 Accelerate, or any custom PyTroch training loop (take a look at the end-to-end [example](https://github.com/huggingface/peft/blob/main/examples/conditional_generation/peft_lora_seq2seq.ipynb) of training [`bigscience/mt0-large`](https://huggingface.co/bigscience/mt0-large)).
+
+### Saving/loading a model
+
+1. Save your model using the `save_pretrained` function.
+
+```python
+model.save_pretrained("output_dir")
+# model.push_to_hub("my_awesome_peft_model") also works
+```
+
+This only saves the incremental PEFT weights that were trained.
+For example, [smangrul/twitter_complaints_bigscience_T0_3B_LORA_SEQ_2_SEQ_LM](https://huggingface.co/smangrul/twitter_complaints_bigscience_T0_3B_LORA_SEQ_2_SEQ_LM) is a `bigscience/T0_3B`model finetuned with LoRA on the [`twitter_complaints`](https://huggingface.co/datasets/ought/raft/viewer/twitter_complaints/train) RAFT dataset.
+Notice that it only contains 2 files: `adapter_config.json` and `adapter_model.bin`, with the latter being just 19MB.
+
+2. Load your model using the `from_pretrained` function.
+
+```diff
+ from transformers import AutoModelForSeq2SeqLM
++ from peft import PeftModel, PeftConfig
+
++ peft_model_id = "smangrul/twitter_complaints_bigscience_T0_3B_LORA_SEQ_2_SEQ_LM"
++ config = PeftConfig.from_pretrained(peft_model_id)
+ model = AutoModelForSeq2SeqLM.from_pretrained(config.base_model_name_or_path)
++ model = PeftModel.from_pretrained(model, peft_model_id)
+ tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
+
+ model = model.to(device)
+ model.eval()
+ inputs = tokenizer("Tweet text : @HondaCustSvc Your customer service has been horrible during the recall process. I will never purchase a Honda again. Label :", return_tensors="pt")
+
+ with torch.no_grad():
+ outputs = model.generate(input_ids=inputs["input_ids"].to("cuda"), max_new_tokens=10)
+ print(tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)[0])
+# 'complaint'
+```
+
+## Launching your distributed script
+
+PEFT models work with 🤗 Accelerate out of the box.
+You can use 🤗 Accelerate for distributed training on various hardware such as GPUs, or Apple Silicon devices during training, and for inference on consumer hardware with fewer resources.
+
+### Train with 🤗 Accelerate's DeepSpeed integration
+
+You'll need DeepSpeed version `v0.8.0` for this example. Feel free to check out the full example [script](https://github.com/huggingface/peft/blob/main/examples/conditional_generation/peft_lora_seq2seq_accelerate_ds_zero3_offload.py) for more details!
+
+1. Run `accelerate config --config_file ds_zero3_cpu.yaml` and answer the questionnaire to setup your environment.
+Below are the contents of the config file.
+ ```yaml
+ compute_environment: LOCAL_MACHINE
+ deepspeed_config:
+ gradient_accumulation_steps: 1
+ gradient_clipping: 1.0
+ offload_optimizer_device: cpu
+ offload_param_device: cpu
+ zero3_init_flag: true
+ zero3_save_16bit_model: true
+ zero_stage: 3
+ distributed_type: DEEPSPEED
+ downcast_bf16: 'no'
+ dynamo_backend: 'NO'
+ fsdp_config: {}
+ machine_rank: 0
+ main_training_function: main
+ megatron_lm_config: {}
+ mixed_precision: 'no'
+ num_machines: 1
+ num_processes: 1
+ rdzv_backend: static
+ same_network: true
+ use_cpu: false
+ ```
+2. Run the following command to launch the example script:
+ ```bash
+ accelerate launch --config_file ds_zero3_cpu.yaml examples/peft_lora_seq2seq_accelerate_ds_zero3_offload.py
+ ```
+
+You'll see some output logs that look like this:
+ ```bash
+ GPU Memory before entering the train : 1916
+ GPU Memory consumed at the end of the train (end-begin): 66
+ GPU Peak Memory consumed during the train (max-begin): 7488
+ GPU Total Peak Memory consumed during the train (max): 9404
+ CPU Memory before entering the train : 19411
+ CPU Memory consumed at the end of the train (end-begin): 0
+ CPU Peak Memory consumed during the train (max-begin): 0
+ CPU Total Peak Memory consumed during the train (max): 19411
+ epoch=4: train_ppl=tensor(1.0705, device='cuda:0') train_epoch_loss=tensor(0.0681, device='cuda:0')
+ 100%|████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:27<00:00, 3.92s/it]
+ GPU Memory before entering the eval : 1982
+ GPU Memory consumed at the end of the eval (end-begin): -66
+ GPU Peak Memory consumed during the eval (max-begin): 672
+ GPU Total Peak Memory consumed during the eval (max): 2654
+ CPU Memory before entering the eval : 19411
+ CPU Memory consumed at the end of the eval (end-begin): 0
+ CPU Peak Memory consumed during the eval (max-begin): 0
+ CPU Total Peak Memory consumed during the eval (max): 19411
+ accuracy=100.0
+ eval_preds[:10]=['no complaint', 'no complaint', 'complaint', 'complaint', 'no complaint', 'no complaint', 'no complaint', 'complaint', 'complaint', 'no complaint']
+ dataset['train'][label_column][:10]=['no complaint', 'no complaint', 'complaint', 'complaint', 'no complaint', 'no complaint', 'no complaint', 'complaint', 'complaint', 'no complaint']
+ ```
+
+### Inference with 🤗 Accelerate's Big Model Inference
+An example is provided in `~examples/causal_language_modeling/peft_lora_clm_accelerate_big_model_inference.ipynb`.
+
+## Model Support matrix
+
+### Causal Language Modeling
+| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
+|--------------| ---- | ---- | ---- | ---- |
+| GPT-2 | ✅ | ✅ | ✅ | ✅ |
+| Bloom | ✅ | ✅ | ✅ | ✅ |
+| OPT | ✅ | ✅ | ✅ | ✅ |
+| GPT-Neo | ✅ | ✅ | ✅ | ✅ |
+| GPT-J | ✅ | ✅ | ✅ | ✅ |
+| GPT-NeoX-20B | ✅ | ✅ | ✅ | ✅ |
+| LLaMA | ✅ | ✅ | ✅ | ✅ |
+| ChatGLM | ✅ | ✅ | ✅ | ✅ |
+
+### Conditional Generation
+| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
+| --------- | ---- | ---- | ---- | ---- |
+| T5 | ✅ | ✅ | ✅ | ✅ |
+| BART | ✅ | ✅ | ✅ | ✅ |
+
+### Sequence Classification
+| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
+| --------- | ---- | ---- | ---- | ---- |
+| BERT | ✅ | ✅ | ✅ | ✅ |
+| RoBERTa | ✅ | ✅ | ✅ | ✅ |
+| GPT-2 | ✅ | ✅ | ✅ | ✅ |
+| Bloom | ✅ | ✅ | ✅ | ✅ |
+| OPT | ✅ | ✅ | ✅ | ✅ |
+| GPT-Neo | ✅ | ✅ | ✅ | ✅ |
+| GPT-J | ✅ | ✅ | ✅ | ✅ |
+| Deberta | ✅ | | ✅ | ✅ |
+| Deberta-v2 | ✅ | | ✅ | ✅ |
+
+### Token Classification
+| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
+| --------- | ---- | ---- | ---- | ---- |
+| BERT | ✅ | ✅ | | |
+| RoBERTa | ✅ | ✅ | | |
+| GPT-2 | ✅ | ✅ | | |
+| Bloom | ✅ | ✅ | | |
+| OPT | ✅ | ✅ | | |
+| GPT-Neo | ✅ | ✅ | | |
+| GPT-J | ✅ | ✅ | | |
+| Deberta | ✅ | | | |
+| Deberta-v2 | ✅ | | | |
+
+### Text-to-Image Generation
+
+| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
+| --------- | ---- | ---- | ---- | ---- |
+| Stable Diffusion | ✅ | | | |
+
+
+### Image Classification
+
+| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
+| --------- | ---- | ---- | ---- | ---- |
+| ViT | ✅ | | | |
+| Swin | ✅ | | | |
+
+___Note that we have tested LoRA for [ViT](https://huggingface.co/docs/transformers/model_doc/vit) and [Swin](https://huggingface.co/docs/transformers/model_doc/swin) for fine-tuning on image classification. However, it should be possible to use LoRA for any compatible model [provided](https://huggingface.co/models?pipeline_tag=image-classification&sort=downloads&search=vit) by 🤗 Transformers. Check out the respective
+examples to learn more. If you run into problems, please open an issue.___
+
+The same principle applies to our [segmentation models](https://huggingface.co/models?pipeline_tag=image-segmentation&sort=downloads) as well.
+
+### Semantic Segmentation
+
+| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
+| --------- | ---- | ---- | ---- | ---- |
+| SegFormer | ✅ | | | |
+
+
+## Other caveats
+
+1. Below is an example of using PyTorch FSDP for training. However, it doesn't lead to
+any GPU memory savings. Please refer to issue [[FSDP] FSDP with CPU offload consumes 1.65X more GPU memory when training models with most of the params frozen](https://github.com/pytorch/pytorch/issues/91165).
+
+ ```python
+ from peft.utils.other import fsdp_auto_wrap_policy
+
+
+ if os.environ.get("ACCELERATE_USE_FSDP", None) is not None:
+ accelerator.state.fsdp_plugin.auto_wrap_policy = fsdp_auto_wrap_policy(model)
+
+ model = accelerator.prepare(model)
+ ```
+
+ Example of parameter efficient tuning with [`mt0-xxl`](https://huggingface.co/bigscience/mt0-xxl) base model using 🤗 Accelerate is provided in `~examples/conditional_generation/peft_lora_seq2seq_accelerate_fsdp.py`.
+ a. First, run `accelerate config --config_file fsdp_config.yaml` and answer the questionnaire.
+ Below are the contents of the config file.
+ ```yaml
+ command_file: null
+ commands: null
+ compute_environment: LOCAL_MACHINE
+ deepspeed_config: {}
+ distributed_type: FSDP
+ downcast_bf16: 'no'
+ dynamo_backend: 'NO'
+ fsdp_config:
+ fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
+ fsdp_backward_prefetch_policy: BACKWARD_PRE
+ fsdp_offload_params: true
+ fsdp_sharding_strategy: 1
+ fsdp_state_dict_type: FULL_STATE_DICT
+ fsdp_transformer_layer_cls_to_wrap: T5Block
+ gpu_ids: null
+ machine_rank: 0
+ main_process_ip: null
+ main_process_port: null
+ main_training_function: main
+ megatron_lm_config: {}
+ mixed_precision: 'no'
+ num_machines: 1
+ num_processes: 2
+ rdzv_backend: static
+ same_network: true
+ tpu_name: null
+ tpu_zone: null
+ use_cpu: false
+ ```
+ b. run the below command to launch the example script
+ ```bash
+ accelerate launch --config_file fsdp_config.yaml examples/peft_lora_seq2seq_accelerate_fsdp.py
+ ```
+
+2. When using `P_TUNING` or `PROMPT_TUNING` with `SEQ_2_SEQ` task, remember to remove the `num_virtual_token` virtual prompt predictions from the left side of the model outputs during evaluations.
+
+3. For encoder-decoder models, `P_TUNING` or `PROMPT_TUNING` doesn't support the `generate` functionality of transformers because `generate` strictly requires `decoder_input_ids` but
+`P_TUNING`/`PROMPT_TUNING` append soft prompt embeddings to `input_embeds` to create
+new `input_embeds` to be given to the model. Therefore, `generate` doesn't support this yet.
+
+4. When using ZeRO3 with zero3_init_flag=True, if you find the GPU memory increase with training steps. we might need to set zero3_init_flag=false in accelerate config.yaml. The related issue is [[BUG] memory leak under zero.Init](https://github.com/microsoft/DeepSpeed/issues/2637)
\ No newline at end of file
diff --git a/docs/source/task_guides/image_classification_lora.mdx b/docs/source/task_guides/image_classification_lora.mdx
new file mode 100644
index 0000000..cfbbeca
--- /dev/null
+++ b/docs/source/task_guides/image_classification_lora.mdx
@@ -0,0 +1,429 @@
+
+
+# Image classification using LoRA
+
+This guide demonstrates how to use LoRA, a low-rank approximation technique, to fine-tune an image classification model.
+By using LoRA from 🤗 PEFT, we can reduce the number of trainable parameters in the model to only 0.77% of the original.
+
+LoRA achieves this reduction by adding low-rank "update matrices" to specific blocks of the model, such as the attention
+blocks. During fine-tuning, only these matrices are trained, while the original model parameters are left unchanged.
+At inference time, the update matrices are merged with the original model parameters to produce the final classification result.
+
+For more information on LoRA, please refer to the [original LoRA paper](https://arxiv.org/abs/2106.09685).
+
+## Install dependencies
+
+Install the libraries required for model training:
+
+```bash
+!pip install transformers accelerate evaluate datasets loralib peft -q
+```
+
+Check the versions of all required libraries to make sure you are up to date:
+
+```python
+import transformers
+import accelerate
+import peft
+
+print(f"Transformers version: {transformers.__version__}")
+print(f"Accelerate version: {accelerate.__version__}")
+print(f"PEFT version: {peft.__version__}")
+"Transformers version: 4.27.4"
+"Accelerate version: 0.18.0"
+"PEFT version: 0.2.0"
+```
+
+## Authenticate to share your model
+
+To share the fine-tuned model at the end of the training with the community, authenticate using your 🤗 token.
+You can obtain your token from your [account settings](https://huggingface.co/settings/token).
+
+```python
+from huggingface_hub import notebook_login
+
+notebook_login()
+```
+
+## Select a model checkpoint to fine-tune
+
+Choose a model checkpoint from any of the model architectures supported for [image classification](https://huggingface.co/models?pipeline_tag=image-classification&sort=downloads). When in doubt, refer to
+the [image classification task guide](https://huggingface.co/docs/transformers/v4.27.2/en/tasks/image_classification) in
+🤗 Transformers documentation.
+
+```python
+model_checkpoint = "google/vit-base-patch16-224-in21k"
+```
+
+## Load a dataset
+
+To keep this example's runtime short, let's only load the first 5000 instances from the training set of the [Food-101 dataset](https://huggingface.co/datasets/food101):
+
+```python
+from datasets import load_dataset
+
+dataset = load_dataset("food101", split="train[:5000]")
+```
+
+## Dataset preparation
+
+To prepare the dataset for training and evaluation, create `label2id` and `id2label` dictionaries. These will come in
+handy when performing inference and for metadata information:
+
+```python
+labels = dataset.features["label"].names
+label2id, id2label = dict(), dict()
+for i, label in enumerate(labels):
+ label2id[label] = i
+ id2label[i] = label
+
+id2label[2]
+"baklava"
+```
+
+Next, load the image processor of the model you're fine-tuning:
+
+```python
+from transformers import AutoImageProcessor
+
+image_processor = AutoImageProcessor.from_pretrained(model_checkpoint)
+```
+
+The `image_processor` contains useful information on which size the training and evaluation images should be resized
+to, as well as values that should be used to normalize the pixel values. Using the `image_processor`, prepare transformation
+functions for the datasets. These functions will include data augmentation and pixel scaling:
+
+```python
+from torchvision.transforms import (
+ CenterCrop,
+ Compose,
+ Normalize,
+ RandomHorizontalFlip,
+ RandomResizedCrop,
+ Resize,
+ ToTensor,
+)
+
+normalize = Normalize(mean=image_processor.image_mean, std=image_processor.image_std)
+train_transforms = Compose(
+ [
+ RandomResizedCrop(image_processor.size["height"]),
+ RandomHorizontalFlip(),
+ ToTensor(),
+ normalize,
+ ]
+)
+
+val_transforms = Compose(
+ [
+ Resize(image_processor.size["height"]),
+ CenterCrop(image_processor.size["height"]),
+ ToTensor(),
+ normalize,
+ ]
+)
+
+
+def preprocess_train(example_batch):
+ """Apply train_transforms across a batch."""
+ example_batch["pixel_values"] = [train_transforms(image.convert("RGB")) for image in example_batch["image"]]
+ return example_batch
+
+
+def preprocess_val(example_batch):
+ """Apply val_transforms across a batch."""
+ example_batch["pixel_values"] = [val_transforms(image.convert("RGB")) for image in example_batch["image"]]
+ return example_batch
+```
+
+Split the dataset into training and validation sets:
+
+```python
+splits = dataset.train_test_split(test_size=0.1)
+train_ds = splits["train"]
+val_ds = splits["test"]
+```
+
+Finally, set the transformation functions for the datasets accordingly:
+
+```python
+train_ds.set_transform(preprocess_train)
+val_ds.set_transform(preprocess_val)
+```
+
+## Load and prepare a model
+
+Before loading the model, let's define a helper function to check the total number of parameters a model has, as well
+as how many of them are trainable.
+
+```python
+def print_trainable_parameters(model):
+ trainable_params = 0
+ all_param = 0
+ for _, param in model.named_parameters():
+ all_param += param.numel()
+ if param.requires_grad:
+ trainable_params += param.numel()
+ print(
+ f"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param:.2f}"
+ )
+```
+
+It's important to initialize the original model correctly as it will be used as a base to create the `PeftModel` you'll
+actually fine-tune. Specify the `label2id` and `id2label` so that [`~transformers.AutoModelForImageClassification`] can append a classification
+head to the underlying model, adapted for this dataset. You should see the following output:
+
+```
+Some weights of ViTForImageClassification were not initialized from the model checkpoint at google/vit-base-patch16-224-in21k and are newly initialized: ['classifier.weight', 'classifier.bias']
+```
+
+```python
+from transformers import AutoModelForImageClassification, TrainingArguments, Trainer
+
+model = AutoModelForImageClassification.from_pretrained(
+ model_checkpoint,
+ label2id=label2id,
+ id2label=id2label,
+ ignore_mismatched_sizes=True, # provide this in case you're planning to fine-tune an already fine-tuned checkpoint
+)
+```
+
+Before creating a `PeftModel`, you can check the number of trainable parameters in the original model:
+
+```python
+print_trainable_parameters(model)
+"trainable params: 85876325 || all params: 85876325 || trainable%: 100.00"
+```
+
+Next, use `get_peft_model` to wrap the base model so that "update" matrices are added to the respective places.
+
+```python
+from peft import LoraConfig, get_peft_model
+
+config = LoraConfig(
+ r=16,
+ lora_alpha=16,
+ target_modules=["query", "value"],
+ lora_dropout=0.1,
+ bias="none",
+ modules_to_save=["classifier"],
+)
+lora_model = get_peft_model(model, config)
+print_trainable_parameters(lora_model)
+"trainable params: 667493 || all params: 86466149 || trainable%: 0.77"
+```
+
+Let's unpack what's going on here.
+To use LoRA, you need to specify the target modules in `LoraConfig` so that `get_peft_model()` knows which modules
+inside our model need to be amended with LoRA matrices. In this example, we're only interested in targeting the query and
+value matrices of the attention blocks of the base model. Since the parameters corresponding to these matrices are "named"
+"query" and "value" respectively, we specify them accordingly in the `target_modules` argument of `LoraConfig`.
+
+We also specify `modules_to_save`. After wrapping the base model with `get_peft_model()` along with the `config`, we get
+a new model where only the LoRA parameters are trainable (so-called "update matrices") while the pre-trained parameters
+are kept frozen. However, we want the classifier parameters to be trained too when fine-tuning the base model on our
+custom dataset. To ensure that the classifier parameters are also trained, we specify `modules_to_save`. This also
+ensures that these modules are serialized alongside the LoRA trainable parameters when using utilities like `save_pretrained()`
+and `push_to_hub()`.
+
+Here's what the other parameters mean:
+
+- `r`: The dimension used by the LoRA update matrices.
+- `alpha`: Scaling factor.
+- `bias`: Specifies if the `bias` parameters should be trained. `None` denotes none of the `bias` parameters will be trained.
+
+`r` and `alpha` together control the total number of final trainable parameters when using LoRA, giving you the flexibility
+to balance a trade-off between end performance and compute efficiency.
+
+By looking at the number of trainable parameters, you can see how many parameters we're actually training. Since the goal is
+to achieve parameter-efficient fine-tuning, you should expect to see fewer trainable parameters in the `lora_model`
+in comparison to the original model, which is indeed the case here.
+
+## Define training arguments
+
+For model fine-tuning, use [`~transformers.Trainer`]. It accepts
+several arguments which you can wrap using [`~transformers.TrainingArguments`].
+
+```python
+from transformers import TrainingArguments, Trainer
+
+
+model_name = model_checkpoint.split("/")[-1]
+batch_size = 128
+
+args = TrainingArguments(
+ f"{model_name}-finetuned-lora-food101",
+ remove_unused_columns=False,
+ evaluation_strategy="epoch",
+ save_strategy="epoch",
+ learning_rate=5e-3,
+ per_device_train_batch_size=batch_size,
+ gradient_accumulation_steps=4,
+ per_device_eval_batch_size=batch_size,
+ fp16=True,
+ num_train_epochs=5,
+ logging_steps=10,
+ load_best_model_at_end=True,
+ metric_for_best_model="accuracy",
+ push_to_hub=True,
+ label_names=["labels"],
+)
+```
+
+Compared to non-PEFT methods, you can use a larger batch size since there are fewer parameters to train.
+You can also set a larger learning rate than the normal (1e-5 for example).
+
+This can potentially also reduce the need to conduct expensive hyperparameter tuning experiments.
+
+## Prepare evaluation metric
+
+```python
+import numpy as np
+import evaluate
+
+metric = evaluate.load("accuracy")
+
+
+def compute_metrics(eval_pred):
+ """Computes accuracy on a batch of predictions"""
+ predictions = np.argmax(eval_pred.predictions, axis=1)
+ return metric.compute(predictions=predictions, references=eval_pred.label_ids)
+```
+
+The `compute_metrics` function takes a named tuple as input: `predictions`, which are the logits of the model as Numpy arrays,
+and `label_ids`, which are the ground-truth labels as Numpy arrays.
+
+## Define collation function
+
+A collation function is used by [`~transformers.Trainer`] to gather a batch of training and evaluation examples and prepare them in a
+format that is acceptable by the underlying model.
+
+```python
+import torch
+
+
+def collate_fn(examples):
+ pixel_values = torch.stack([example["pixel_values"] for example in examples])
+ labels = torch.tensor([example["label"] for example in examples])
+ return {"pixel_values": pixel_values, "labels": labels}
+```
+
+## Train and evaluate
+
+Bring everything together - model, training arguments, data, collation function, etc. Then, start the training!
+
+```python
+trainer = Trainer(
+ model,
+ args,
+ train_dataset=train_ds,
+ eval_dataset=val_ds,
+ tokenizer=image_processor,
+ compute_metrics=compute_metrics,
+ data_collator=collate_fn,
+)
+train_results = trainer.train()
+```
+
+In just a few minutes, the fine-tuned model shows 96% validation accuracy even on this small
+subset of the training dataset.
+
+```python
+trainer.evaluate(val_ds)
+{
+ "eval_loss": 0.14475855231285095,
+ "eval_accuracy": 0.96,
+ "eval_runtime": 3.5725,
+ "eval_samples_per_second": 139.958,
+ "eval_steps_per_second": 1.12,
+ "epoch": 5.0,
+}
+```
+
+## Share your model and run inference
+
+Once the fine-tuning is done, share the LoRA parameters with the community like so:
+
+```python
+repo_name = f"sayakpaul/{model_name}-finetuned-lora-food101"
+lora_model.push_to_hub(repo_name)
+```
+
+When calling [`~transformers.PreTrainedModel.push_to_hub`] on the `lora_model`, only the LoRA parameters along with any modules specified in `modules_to_save`
+are saved. Take a look at the [trained LoRA parameters](https://huggingface.co/sayakpaul/vit-base-patch16-224-in21k-finetuned-lora-food101/blob/main/adapter_model.bin).
+You'll see that it's only 2.6 MB! This greatly helps with portability, especially when using a very large model to fine-tune (such as [BLOOM](https://huggingface.co/bigscience/bloom)).
+
+Next, let's see how to load the LoRA updated parameters along with our base model for inference. When you wrap a base model
+with `PeftModel`, modifications are done *in-place*. To mitigate any concerns that might stem from in-place modifications,
+initialize the base model just like you did earlier and construct the inference model.
+
+```python
+from peft import PeftConfig, PeftModel
+
+
+config = PeftConfig.from_pretrained(repo_name)
+model = AutoModelForImageClassification.from_pretrained(
+ config.base_model_name_or_path,
+ label2id=label2id,
+ id2label=id2label,
+ ignore_mismatched_sizes=True, # provide this in case you're planning to fine-tune an already fine-tuned checkpoint
+)
+# Load the LoRA model
+inference_model = PeftModel.from_pretrained(model, repo_name)
+```
+
+Let's now fetch an example image for inference.
+
+```python
+from PIL import Image
+import requests
+
+url = "https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/beignets.jpeg"
+image = Image.open(requests.get(url, stream=True).raw)
+image
+```
+
+
+
+