mirror of
https://github.com/wassname/peft.git
synced 2026-09-09 11:28:32 +08:00
Merge branch 'main' into smangrul/multi-lora-support
This commit is contained in:
Sourab Mangrulkar
@@ -0,0 +1,18 @@
|
||||
name: Build documentation
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
- doc-builder*
|
||||
- v*-release
|
||||
|
||||
jobs:
|
||||
build:
|
||||
uses: huggingface/doc-builder/.github/workflows/build_main_documentation.yml@main
|
||||
with:
|
||||
commit_sha: ${{ github.sha }}
|
||||
package: peft
|
||||
notebook_folder: peft_docs
|
||||
secrets:
|
||||
token: ${{ secrets.HUGGINGFACE_PUSH }}
|
||||
@@ -0,0 +1,16 @@
|
||||
name: Build PR Documentation
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
build:
|
||||
uses: huggingface/doc-builder/.github/workflows/build_pr_documentation.yml@main
|
||||
with:
|
||||
commit_sha: ${{ github.event.pull_request.head.sha }}
|
||||
pr_number: ${{ github.event.number }}
|
||||
package: peft
|
||||
@@ -0,0 +1,13 @@
|
||||
name: Delete dev documentation
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [ closed ]
|
||||
|
||||
|
||||
jobs:
|
||||
delete:
|
||||
uses: huggingface/doc-builder/.github/workflows/delete_doc_comment.yml@main
|
||||
with:
|
||||
pr_number: ${{ github.event.number }}
|
||||
package: peft
|
||||
@@ -0,0 +1,27 @@
|
||||
name: Stale Bot
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: "0 15 * * *"
|
||||
|
||||
jobs:
|
||||
close_stale_issues:
|
||||
name: Close Stale Issues
|
||||
if: github.repository == 'huggingface/peft'
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
|
||||
- name: Setup Python
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: 3.8
|
||||
|
||||
- name: Install requirements
|
||||
run: |
|
||||
pip install PyGithub
|
||||
- name: Close stale issues
|
||||
run: |
|
||||
python scripts/stale.py
|
||||
@@ -1,6 +1,6 @@
|
||||
.PHONY: quality style test docs
|
||||
|
||||
check_dirs := src tests examples
|
||||
check_dirs := src tests examples docs
|
||||
|
||||
# Check that source code meets quality standards
|
||||
|
||||
@@ -8,13 +8,13 @@ check_dirs := src tests examples
|
||||
quality:
|
||||
black --check $(check_dirs)
|
||||
ruff $(check_dirs)
|
||||
doc-builder style src tests --max_len 119 --check_only
|
||||
doc-builder style src/peft tests docs/source --max_len 119 --check_only
|
||||
|
||||
# Format source code automatically and check is there are any problems left that need manual fixing
|
||||
style:
|
||||
black $(check_dirs)
|
||||
ruff $(check_dirs) --fix
|
||||
doc-builder style src tests --max_len 119
|
||||
doc-builder style src/peft tests docs/source --max_len 119
|
||||
|
||||
test:
|
||||
pytest tests/
|
||||
pytest -n 3 tests/
|
||||
@@ -25,10 +25,10 @@ Seamlessly integrated with 🤗 Accelerate for large scale models leveraging Dee
|
||||
|
||||
Supported methods:
|
||||
|
||||
1. LoRA: [LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS](https://arxiv.org/pdf/2106.09685.pdf)
|
||||
1. LoRA: [LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS](https://arxiv.org/abs/2106.09685)
|
||||
2. Prefix Tuning: [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://aclanthology.org/2021.acl-long.353/), [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://arxiv.org/pdf/2110.07602.pdf)
|
||||
3. P-Tuning: [GPT Understands, Too](https://arxiv.org/pdf/2103.10385.pdf)
|
||||
4. Prompt Tuning: [The Power of Scale for Parameter-Efficient Prompt Tuning](https://arxiv.org/pdf/2104.08691.pdf)
|
||||
3. P-Tuning: [GPT Understands, Too](https://arxiv.org/abs/2103.10385)
|
||||
4. Prompt Tuning: [The Power of Scale for Parameter-Efficient Prompt Tuning](https://arxiv.org/abs/2104.08691)
|
||||
|
||||
## Getting started
|
||||
|
||||
@@ -64,7 +64,7 @@ Hardware: Single A100 80GB GPU with CPU RAM above 64GB
|
||||
| bigscience/bloomz-7b1 (7B params) | OOM GPU | 32GB GPU / 3.8GB CPU | 18.1GB GPU / 35GB CPU |
|
||||
|
||||
Performance of PEFT-LoRA tuned [`bigscience/T0_3B`](https://huggingface.co/bigscience/T0_3B) on [`ought/raft/twitter_complaints`](https://huggingface.co/datasets/ought/raft/viewer/twitter_complaints) leaderboard.
|
||||
A point to note is that we didn't try to sequeeze performance by playing around with input instruction templates, LoRA hyperparams and other training related hyperparams. Also, we didn't use the larger 13B [mt0-xxl](https://huggingface.co/bigscience/mt0-xxl) model.
|
||||
A point to note is that we didn't try to squeeze performance by playing around with input instruction templates, LoRA hyperparams and other training related hyperparams. Also, we didn't use the larger 13B [mt0-xxl](https://huggingface.co/bigscience/mt0-xxl) model.
|
||||
So, we are already seeing comparable performance to SoTA with parameter efficient tuning. Also, the final checkpoint size is just `19MB` in comparison to `11GB` size of the backbone [`bigscience/T0_3B`](https://huggingface.co/bigscience/T0_3B) model.
|
||||
|
||||
| Submission Name | Accuracy |
|
||||
@@ -81,7 +81,7 @@ GPU memory required by different settings during training is given below. The fi
|
||||
|
||||
Hardware: Single A100 80GB GPU with CPU RAM above 64GB
|
||||
|
||||
| Model | Full Finetuning | PEFT-LoRA | PEFT-LoRA with Gradient Checkpoitning |
|
||||
| Model | Full Finetuning | PEFT-LoRA | PEFT-LoRA with Gradient Checkpointing |
|
||||
| --------- | ---- | ---- | ---- |
|
||||
| CompVis/stable-diffusion-v1-4 | 27.5GB GPU / 3.97GB CPU | 15.5GB GPU / 3.84GB CPU | 8.12GB GPU / 3.77GB CPU |
|
||||
|
||||
@@ -126,14 +126,14 @@ Try out the 🤗 Gradio Space which should run seamlessly on a T4 instance:
|
||||
|
||||
|
||||
### Parameter Efficient Tuning of LLMs for RLHF components such as Ranker and Policy
|
||||
- Here is an exmaple in [trl](https://github.com/lvwerra/trl) library using PEFT+INT8 for tuning policy model: [gpt2-sentiment_peft.py](https://github.com/lvwerra/trl/blob/main/examples/sentiment/scripts/gpt2-sentiment_peft.py)
|
||||
- Here is an example in [trl](https://github.com/lvwerra/trl) library using PEFT+INT8 for tuning policy model: [gpt2-sentiment_peft.py](https://github.com/lvwerra/trl/blob/main/examples/sentiment/scripts/gpt2-sentiment_peft.py)
|
||||
- Example using PEFT for both reward model and policy [ToDo]
|
||||
|
||||
### INT8 training of large models in Colab using PEFT LoRA and bits_and_bytes
|
||||
|
||||
- Here is now a demo on how to fine tune [OPT-6.7b](https://huggingface.co/facebook/opt-6.7b) (14GB in fp16) in a Google colab: [](https://colab.research.google.com/drive/1jCkpikz0J2o20FBQmYmAGdiKmJGOMo-o?usp=sharing)
|
||||
- Here is now a demo on how to fine tune [OPT-6.7b](https://huggingface.co/facebook/opt-6.7b) (14GB in fp16) in a Google Colab: [](https://colab.research.google.com/drive/1jCkpikz0J2o20FBQmYmAGdiKmJGOMo-o?usp=sharing)
|
||||
|
||||
- Here is now a demo on how to fine tune [whishper-large](openai/whisper-large-v2) (1.5B params) (14GB in fp16) in a Google colab: [](https://colab.research.google.com/drive/1DOkD_5OUjFa0r5Ik3SgywJLJtEo2qLxO?usp=sharing) and [](https://colab.research.google.com/drive/1vhF8yueFqha3Y3CpTHN6q9EVcII9EYzs?usp=sharing)
|
||||
- Here is now a demo on how to fine tune [whishper-large](openai/whisper-large-v2) (1.5B params) (14GB in fp16) in a Google Colab: [](https://colab.research.google.com/drive/1DOkD_5OUjFa0r5Ik3SgywJLJtEo2qLxO?usp=sharing) and [](https://colab.research.google.com/drive/1vhF8yueFqha3Y3CpTHN6q9EVcII9EYzs?usp=sharing)
|
||||
|
||||
### Save compute and storage even for medium and small models
|
||||
|
||||
@@ -143,12 +143,12 @@ performance comparable to full finetuning.
|
||||
|
||||
An example of using LoRA for the task of adapting `LayoutLMForTokenClassification` on `FUNSD` dataset is given in `~examples/token_classification/PEFT_LoRA_LayoutLMForTokenClassification_on_FUNSD.py`. We can observe that with only `0.62 %` of parameters being trainable, we achieve performance (F1 0.777) comparable to full finetuning (F1 0.786) (without any hyerparam tuning runs for extracting more performance), and the checkpoint of this is only `2.8MB`. Now, if there are `N` such datasets, just have these PEFT models one for each dataset and save a lot of storage without having to worry about the problem of catastrophic forgetting or overfitting of backbone/base model.
|
||||
|
||||
Another example is fine-tuning [`roberta-large`](https://huggingface.co/roberta-large) on [`MRPC` GLUE](https://huggingface.co/datasets/glue/viewer/mrpc) dataset suing differenct PEFT methods. The notebooks are given in `~examples/sequence_classification`.
|
||||
Another example is fine-tuning [`roberta-large`](https://huggingface.co/roberta-large) on [`MRPC` GLUE](https://huggingface.co/datasets/glue/viewer/mrpc) dataset using different PEFT methods. The notebooks are given in `~examples/sequence_classification`.
|
||||
|
||||
|
||||
## PEFT + 🤗 Accelerate
|
||||
|
||||
PEFT models work with 🤗 Accelerate out of the box. Use 🤗 Accelerate for Distributed training on various hardware such as GPUs, Apple Silicon devices etc during training.
|
||||
PEFT models work with 🤗 Accelerate out of the box. Use 🤗 Accelerate for Distributed training on various hardware such as GPUs, Apple Silicon devices, etc during training.
|
||||
Use 🤗 Accelerate for inferencing on consumer hardware with small resources.
|
||||
|
||||
### Example of PEFT model training using 🤗 Accelerate's DeepSpeed integration
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
# Minimal makefile for Sphinx documentation
|
||||
#
|
||||
|
||||
# You can set these variables from the command line.
|
||||
SPHINXOPTS =
|
||||
SPHINXBUILD = sphinx-build
|
||||
SOURCEDIR = source
|
||||
BUILDDIR = _build
|
||||
|
||||
# Put it first so that "make" without argument is like "make help".
|
||||
help:
|
||||
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
|
||||
|
||||
.PHONY: help Makefile
|
||||
|
||||
# Catch-all target: route all unknown targets to Sphinx using the new
|
||||
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
|
||||
%: Makefile
|
||||
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
|
||||
+267
@@ -0,0 +1,267 @@
|
||||
<!---
|
||||
Copyright 2023 The HuggingFace Team. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
-->
|
||||
|
||||
# Generating the documentation
|
||||
|
||||
To generate the documentation, you first have to build it. Several packages are necessary to build the doc,
|
||||
you can install them with the following command, at the root of the code repository:
|
||||
|
||||
```bash
|
||||
pip install -e ".[docs]"
|
||||
```
|
||||
|
||||
Then you need to install our special tool that builds the documentation:
|
||||
|
||||
```bash
|
||||
pip install git+https://github.com/huggingface/doc-builder
|
||||
```
|
||||
|
||||
---
|
||||
**NOTE**
|
||||
|
||||
You only need to generate the documentation to inspect it locally (if you're planning changes and want to
|
||||
check how they look before committing for instance). You don't have to commit the built documentation.
|
||||
|
||||
---
|
||||
|
||||
## Building the documentation
|
||||
|
||||
Once you have setup the `doc-builder` and additional packages, you can generate the documentation by
|
||||
typing the following command:
|
||||
|
||||
```bash
|
||||
doc-builder build peft docs/source/ --build_dir ~/tmp/test-build
|
||||
```
|
||||
|
||||
You can adapt the `--build_dir` to set any temporary folder that you prefer. This command will create it and generate
|
||||
the MDX files that will be rendered as the documentation on the main website. You can inspect them in your favorite
|
||||
Markdown editor.
|
||||
|
||||
## Previewing the documentation
|
||||
|
||||
To preview the docs, first install the `watchdog` module with:
|
||||
|
||||
```bash
|
||||
pip install watchdog
|
||||
```
|
||||
|
||||
Then run the following command:
|
||||
|
||||
```bash
|
||||
doc-builder preview {package_name} {path_to_docs}
|
||||
```
|
||||
|
||||
For example:
|
||||
|
||||
```bash
|
||||
doc-builder preview peft docs/source
|
||||
```
|
||||
|
||||
The docs will be viewable at [http://localhost:3000](http://localhost:3000). You can also preview the docs once you have opened a PR. You will see a bot add a comment to a link where the documentation with your changes lives.
|
||||
|
||||
---
|
||||
**NOTE**
|
||||
|
||||
The `preview` command only works with existing doc files. When you add a completely new file, you need to update `_toctree.yml` & restart `preview` command (`ctrl-c` to stop it & call `doc-builder preview ...` again).
|
||||
|
||||
---
|
||||
|
||||
## Adding a new element to the navigation bar
|
||||
|
||||
Accepted files are Markdown (.md or .mdx).
|
||||
|
||||
Create a file with its extension and put it in the source directory. You can then link it to the toc-tree by putting
|
||||
the filename without the extension in the [`_toctree.yml`](https://github.com/huggingface/peft/blob/main/docs/source/_toctree.yml) file.
|
||||
|
||||
## Renaming section headers and moving sections
|
||||
|
||||
It helps to keep the old links working when renaming the section header and/or moving sections from one document to another. This is because the old links are likely to be used in Issues, Forums, and Social media and it'd make for a much more superior user experience if users reading those months later could still easily navigate to the originally intended information.
|
||||
|
||||
Therefore, we simply keep a little map of moved sections at the end of the document where the original section was. The key is to preserve the original anchor.
|
||||
|
||||
So if you renamed a section from: "Section A" to "Section B", then you can add at the end of the file:
|
||||
|
||||
```
|
||||
Sections that were moved:
|
||||
|
||||
[ <a href="#section-b">Section A</a><a id="section-a"></a> ]
|
||||
```
|
||||
and of course, if you moved it to another file, then:
|
||||
|
||||
```
|
||||
Sections that were moved:
|
||||
|
||||
[ <a href="../new-file#section-b">Section A</a><a id="section-a"></a> ]
|
||||
```
|
||||
|
||||
Use the relative style to link to the new file so that the versioned docs continue to work.
|
||||
|
||||
|
||||
## Writing Documentation - Specification
|
||||
|
||||
The `huggingface/peft` documentation follows the
|
||||
[Google documentation](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) style for docstrings,
|
||||
although we can write them directly in Markdown.
|
||||
|
||||
### Adding a new tutorial
|
||||
|
||||
Adding a new tutorial or section is done in two steps:
|
||||
|
||||
- Add a new file under `./source`. This file can either be ReStructuredText (.rst) or Markdown (.md).
|
||||
- Link that file in `./source/_toctree.yml` on the correct toc-tree.
|
||||
|
||||
Make sure to put your new file under the proper section. It's unlikely to go in the first section (*Get Started*), so
|
||||
depending on the intended targets (beginners, more advanced users, or researchers) it should go in sections two, three, or
|
||||
four.
|
||||
|
||||
### Writing source documentation
|
||||
|
||||
Values that should be put in `code` should either be surrounded by backticks: \`like so\`. Note that argument names
|
||||
and objects like True, None, or any strings should usually be put in `code`.
|
||||
|
||||
When mentioning a class, function, or method, it is recommended to use our syntax for internal links so that our tool
|
||||
adds a link to its documentation with this syntax: \[\`XXXClass\`\] or \[\`function\`\]. This requires the class or
|
||||
function to be in the main package.
|
||||
|
||||
If you want to create a link to some internal class or function, you need to
|
||||
provide its path. For instance: \[\`utils.gather\`\]. This will be converted into a link with
|
||||
`utils.gather` in the description. To get rid of the path and only keep the name of the object you are
|
||||
linking to in the description, add a ~: \[\`~utils.gather\`\] will generate a link with `gather` in the description.
|
||||
|
||||
The same works for methods so you can either use \[\`XXXClass.method\`\] or \[~\`XXXClass.method\`\].
|
||||
|
||||
#### Defining arguments in a method
|
||||
|
||||
Arguments should be defined with the `Args:` (or `Arguments:` or `Parameters:`) prefix, followed by a line return and
|
||||
an indentation. The argument should be followed by its type, with its shape if it is a tensor, a colon, and its
|
||||
description:
|
||||
|
||||
```
|
||||
Args:
|
||||
n_layers (`int`): The number of layers of the model.
|
||||
```
|
||||
|
||||
If the description is too long to fit in one line (more than 119 characters in total), another indentation is necessary
|
||||
before writing the description after the argument.
|
||||
|
||||
Finally, to maintain uniformity if any *one* description is too long to fit on one line, the
|
||||
rest of the parameters should follow suit and have an indention before their description.
|
||||
|
||||
Here's an example showcasing everything so far:
|
||||
|
||||
```
|
||||
Args:
|
||||
gradient_accumulation_steps (`int`, *optional*, default to 1):
|
||||
The number of steps that should pass before gradients are accumulated. A number > 1 should be combined with `Accelerator.accumulate`.
|
||||
cpu (`bool`, *optional*):
|
||||
Whether or not to force the script to execute on CPU. Will ignore GPU available if set to `True` and force the execution on one process only.
|
||||
```
|
||||
|
||||
For optional arguments or arguments with defaults we follow the following syntax: imagine we have a function with the
|
||||
following signature:
|
||||
|
||||
```
|
||||
def my_function(x: str = None, a: float = 1):
|
||||
```
|
||||
|
||||
then its documentation should look like this:
|
||||
|
||||
```
|
||||
Args:
|
||||
x (`str`, *optional*):
|
||||
This argument controls ... and has a description longer than 119 chars.
|
||||
a (`float`, *optional*, defaults to 1):
|
||||
This argument is used to ... and has a description longer than 119 chars.
|
||||
```
|
||||
|
||||
Note that we always omit the "defaults to \`None\`" when None is the default for any argument. Also note that even
|
||||
if the first line describing your argument type and its default gets long, you can't break it on several lines. You can
|
||||
however write as many lines as you want in the indented description (see the example above with `input_ids`).
|
||||
|
||||
#### Writing a multi-line code block
|
||||
|
||||
Multi-line code blocks can be useful for displaying examples. They are done between two lines of three backticks as usual in Markdown:
|
||||
|
||||
|
||||
````
|
||||
```python
|
||||
# first line of code
|
||||
# second line
|
||||
# etc
|
||||
```
|
||||
````
|
||||
|
||||
#### Writing a return block
|
||||
|
||||
The return block should be introduced with the `Returns:` prefix, followed by a line return and an indentation.
|
||||
The first line should be the type of the return, followed by a line return. No need to indent further for the elements
|
||||
building the return.
|
||||
|
||||
Here's an example of a single value return:
|
||||
|
||||
```
|
||||
Returns:
|
||||
`List[int]`: A list of integers in the range [0, 1] --- 1 for a special token, 0 for a sequence token.
|
||||
```
|
||||
|
||||
Here's an example of a tuple return, comprising several objects:
|
||||
|
||||
```
|
||||
Returns:
|
||||
`tuple(torch.FloatTensor)` comprising various elements depending on the configuration ([`BertConfig`]) and inputs:
|
||||
- ** loss** (*optional*, returned when `masked_lm_labels` is provided) `torch.FloatTensor` of shape `(1,)` --
|
||||
Total loss is the sum of the masked language modeling loss and the next sequence prediction (classification) loss.
|
||||
- **prediction_scores** (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`) --
|
||||
Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
|
||||
```
|
||||
|
||||
## Styling the docstring
|
||||
|
||||
We have an automatic script running with the `make style` comment that will make sure that:
|
||||
- the docstrings fully take advantage of the line width
|
||||
- all code examples are formatted using black, like the code of the Transformers library
|
||||
|
||||
This script may have some weird failures if you made a syntax mistake or if you uncover a bug. Therefore, it's
|
||||
recommended to commit your changes before running `make style`, so you can revert the changes done by that script
|
||||
easily.
|
||||
|
||||
## Writing documentation examples
|
||||
|
||||
The syntax for Example docstrings can look as follows:
|
||||
|
||||
```
|
||||
Example:
|
||||
|
||||
```python
|
||||
>>> import time
|
||||
>>> from accelerate import Accelerator
|
||||
>>> accelerator = Accelerator()
|
||||
>>> if accelerator.is_main_process:
|
||||
... time.sleep(2)
|
||||
>>> else:
|
||||
... print("I'm waiting for the main process to finish its sleep...")
|
||||
>>> accelerator.wait_for_everyone()
|
||||
>>> # Should print on every process at the same time
|
||||
>>> print("Everyone is here")
|
||||
```
|
||||
```
|
||||
|
||||
The docstring should give a minimal, clear example of how the respective function
|
||||
is to be used in inference and also include the expected (ideally sensible)
|
||||
output.
|
||||
Often, readers will try out the example before even going through the function
|
||||
or class definitions. Therefore, it is of utmost importance that the example
|
||||
works as expected.
|
||||
@@ -0,0 +1,7 @@
|
||||
# docstyle-ignore
|
||||
INSTALL_CONTENT = """
|
||||
# PEFT installation
|
||||
! pip install peft accelerate transformers
|
||||
# To install from source instead of the last release, comment the command above and uncomment the following one.
|
||||
# ! pip install git+https://github.com/huggingface/peft.git
|
||||
"""
|
||||
@@ -0,0 +1,23 @@
|
||||
- title: Get started
|
||||
sections:
|
||||
- local: index
|
||||
title: 🤗 PEFT
|
||||
- local: quicktour
|
||||
title: Quicktour
|
||||
- local: install
|
||||
title: Installation
|
||||
|
||||
- title: Task guides
|
||||
sections:
|
||||
- local: task_guides/image_classification_lora
|
||||
title: Image classification using LoRA
|
||||
|
||||
- title: Reference
|
||||
sections:
|
||||
- local: package_reference/peft_model
|
||||
title: PEFT model
|
||||
- local: package_reference/config
|
||||
title: Configuration
|
||||
- local: package_reference/tuners
|
||||
title: Tuners
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
<!--Copyright 2023 The HuggingFace Team. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
||||
the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
||||
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations under the License.
|
||||
-->
|
||||
|
||||
# PEFT
|
||||
|
||||
🤗 PEFT, or Parameter-Efficient Fine-Tuning (PEFT), is a library for efficiently adapting pre-trained language models (PLMs) to various downstream applications without fine-tuning all the model's parameters.
|
||||
PEFT methods only fine-tune a small number of (extra) model parameters, significantly decreasing computational and storage costs because fine-tuning large-scale PLMs is prohibitively costly.
|
||||
Recent state-of-the-art PEFT techniques achieve performance comparable to that of full fine-tuning.
|
||||
|
||||
PEFT is seamlessly integrated with 🤗 Accelerate for large-scale models leveraging DeepSpeed and [Big Model Inference](https://huggingface.co/docs/accelerate/usage_guides/big_modeling).
|
||||
|
||||
Supported methods include:
|
||||
|
||||
1. LoRA: [LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS](https://arxiv.org/pdf/2106.09685.pdf)
|
||||
2. Prefix Tuning: [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://aclanthology.org/2021.acl-long.353/), [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://arxiv.org/pdf/2110.07602.pdf)
|
||||
3. P-Tuning: [GPT Understands, Too](https://arxiv.org/pdf/2103.10385.pdf)
|
||||
4. Prompt Tuning: [The Power of Scale for Parameter-Efficient Prompt Tuning](https://arxiv.org/pdf/2104.08691.pdf)
|
||||
@@ -0,0 +1,43 @@
|
||||
<!--Copyright 2023 The HuggingFace Team. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
||||
the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
||||
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations under the License.
|
||||
-->
|
||||
|
||||
# Installation
|
||||
|
||||
Before you start, you will need to setup your environment, install the appropriate packages, and configure 🤗 PEFT. 🤗 PEFT is tested on **Python 3.7+**.
|
||||
|
||||
🤗 PEFT is available on pypi, as well as GitHub:
|
||||
|
||||
## pip
|
||||
|
||||
To install 🤗 PEFT from pypi:
|
||||
|
||||
```bash
|
||||
pip install peft
|
||||
```
|
||||
|
||||
## Source
|
||||
|
||||
New features that haven't been released yet are added every day, which also means there may be some bugs. To try them out, install from the GitHub repository:
|
||||
|
||||
```bash
|
||||
pip install git+https://github.com/huggingface/peft
|
||||
```
|
||||
|
||||
If you're working on contributing to the library or wish to play with the source code and see live
|
||||
results as you run the code, an editable version can be installed from a locally-cloned version of the
|
||||
repository:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/huggingface/peft
|
||||
cd peft
|
||||
pip install -e .
|
||||
```
|
||||
@@ -0,0 +1,18 @@
|
||||
# Configuration
|
||||
|
||||
The configuration classes stores the configuration of a [`PeftModel`], PEFT adapter models, and the configurations of [`PrefixTuning`], [`PromptTuning`], and [`PromptEncoder`]. They contain methods for saving and loading model configurations from the Hub, specifying the PEFT method to use, type of task to perform, and model configurations like number of layers and number of attention heads.
|
||||
|
||||
## PeftConfigMixin
|
||||
|
||||
[[autodoc]] utils.config.PeftConfigMixin
|
||||
- all
|
||||
|
||||
## PeftConfig
|
||||
|
||||
[[autodoc]] PeftConfig
|
||||
- all
|
||||
|
||||
## PromptLearningConfig
|
||||
|
||||
[[autodoc]] PromptLearningConfig
|
||||
- all
|
||||
@@ -0,0 +1,36 @@
|
||||
# Models
|
||||
|
||||
[`PeftModel`] is the base model class for specifying the base Transformer model and configuration to apply a PEFT method to. The base `PeftModel` contains methods for loading and saving models from the Hub, and supports the [`PromptEncoder`] for prompt learning.
|
||||
|
||||
## PeftModel
|
||||
|
||||
[[autodoc]] PeftModel
|
||||
- all
|
||||
|
||||
## PeftModelForSequenceClassification
|
||||
|
||||
A `PeftModel` for sequence classification tasks.
|
||||
|
||||
[[autodoc]] PeftModelForSequenceClassification
|
||||
- all
|
||||
|
||||
## PeftModelForTokenClassification
|
||||
|
||||
A `PeftModel` for token classification tasks.
|
||||
|
||||
[[autodoc]] PeftModelForTokenClassification
|
||||
- all
|
||||
|
||||
## PeftModelForCausalLM
|
||||
|
||||
A `PeftModel` for causal language modeling.
|
||||
|
||||
[[autodoc]] PeftModelForCausalLM
|
||||
- all
|
||||
|
||||
## PeftModelForSeq2SeqLM
|
||||
|
||||
A `PeftModel` for sequence-to-sequence language modeling.
|
||||
|
||||
[[autodoc]] PeftModelForSeq2SeqLM
|
||||
- all
|
||||
@@ -0,0 +1,35 @@
|
||||
# Tuners
|
||||
|
||||
Each tuner (or PEFT method) has a configuration and model.
|
||||
|
||||
## LoRA
|
||||
|
||||
For finetuning a model with LoRA.
|
||||
|
||||
[[autodoc]] LoraConfig
|
||||
|
||||
[[autodoc]] LoraModel
|
||||
|
||||
[[autodoc]] tuners.lora.LoraLayer
|
||||
|
||||
[[autodoc]] tuners.lora.Linear
|
||||
|
||||
[[autodoc]] tuners.lora.MergedLinear
|
||||
|
||||
## P-tuning
|
||||
|
||||
[[autodoc]] tuners.p_tuning.PromptEncoderConfig
|
||||
|
||||
[[autodoc]] tuners.p_tuning.PromptEncoder
|
||||
|
||||
## Prefix tuning
|
||||
|
||||
[[autodoc]] tuners.prefix_tuning.PrefixTuningConfig
|
||||
|
||||
[[autodoc]] tuners.prefix_tuning.PrefixEncoder
|
||||
|
||||
## Prompt tuning
|
||||
|
||||
[[autodoc]] tuners.prompt_tuning.PromptTuningConfig
|
||||
|
||||
[[autodoc]] tuners.prompt_tuning.PromptEmbedding
|
||||
@@ -0,0 +1,296 @@
|
||||
<!--Copyright 2023 The HuggingFace Team. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
||||
the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
||||
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations under the License.
|
||||
-->
|
||||
|
||||
# Quick tour
|
||||
|
||||
Let's have a look at 🤗 PEFT's main features and learn how to set up a `PeftModel` and train it with 🤗 Accelerate's DeepSpeed integration and use it for inference.
|
||||
|
||||
## Main use
|
||||
|
||||
To use 🤗 PEFT in your script:
|
||||
|
||||
1. Each PEFT method is defined by a `PeftConfig` object.
|
||||
|
||||
Create a `PeftConfig` object corresponding to your PEFT method (see the [Configuration](package_reference/config) reference for more details) and [`TaskType`], the type of task you're training your model for.
|
||||
This example trains the [`bigscience/mt0-large`](https://huggingface.co/bigscience/mt0-large) model with the Low-Rank Adaptation of Large Language Models (LoRA) method. Load the `LoRAConfig`, and specify the `task_type` for sequence-to-sequence language modeling.
|
||||
|
||||
```python
|
||||
from peft import LoraConfig, TaskType
|
||||
|
||||
peft_config = LoraConfig(task_type=TaskType.SEQ_2_SEQ_LM, inference_mode=False, r=8, lora_alpha=32, lora_dropout=0.1)
|
||||
```
|
||||
|
||||
2. Load the base model you want to fine-tune.
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForSeq2SeqLM
|
||||
|
||||
model_name_or_path = "bigscience/mt0-large"
|
||||
tokenizer_name_or_path = "bigscience/mt0-large"
|
||||
model = AutoModelForSeq2SeqLM.from_pretrained(model_name_or_path)
|
||||
```
|
||||
|
||||
3. Preprocess your model if you use [`bitsandbytes`](https://github.com/TimDettmers/bitsandbytes) for `int8` quantized training; otherwise, skip this step.
|
||||
|
||||
```python
|
||||
from peft import prepare_model_for_int8_training
|
||||
|
||||
model = prepare_model_for_int8_training(model)
|
||||
```
|
||||
|
||||
4. Wrap your model in the `PeftModel` object using the `get_peft_model` function. Also, check the number of trainable parameters of your model.
|
||||
|
||||
```python
|
||||
from peft import get_peft_model
|
||||
|
||||
model = get_peft_model(model, peft_config)
|
||||
model.print_trainable_parameters()
|
||||
# output: trainable params: 2359296 || all params: 1231940608 || trainable%: 0.19151053100118282
|
||||
```
|
||||
|
||||
5. Voila 🎉! Now, train the model using the 🤗 Transformers Trainer API, 🤗 Accelerate, or any custom PyTroch training loop (take a look at the end-to-end [example](https://github.com/huggingface/peft/blob/main/examples/conditional_generation/peft_lora_seq2seq.ipynb) of training [`bigscience/mt0-large`](https://huggingface.co/bigscience/mt0-large)).
|
||||
|
||||
### Saving/loading a model
|
||||
|
||||
1. Save your model using the `save_pretrained` function.
|
||||
|
||||
```python
|
||||
model.save_pretrained("output_dir")
|
||||
# model.push_to_hub("my_awesome_peft_model") also works
|
||||
```
|
||||
|
||||
This only saves the incremental PEFT weights that were trained.
|
||||
For example, [smangrul/twitter_complaints_bigscience_T0_3B_LORA_SEQ_2_SEQ_LM](https://huggingface.co/smangrul/twitter_complaints_bigscience_T0_3B_LORA_SEQ_2_SEQ_LM) is a `bigscience/T0_3B`model finetuned with LoRA on the [`twitter_complaints`](https://huggingface.co/datasets/ought/raft/viewer/twitter_complaints/train) RAFT dataset.
|
||||
Notice that it only contains 2 files: `adapter_config.json` and `adapter_model.bin`, with the latter being just 19MB.
|
||||
|
||||
2. Load your model using the `from_pretrained` function.
|
||||
|
||||
```diff
|
||||
from transformers import AutoModelForSeq2SeqLM
|
||||
+ from peft import PeftModel, PeftConfig
|
||||
|
||||
+ peft_model_id = "smangrul/twitter_complaints_bigscience_T0_3B_LORA_SEQ_2_SEQ_LM"
|
||||
+ config = PeftConfig.from_pretrained(peft_model_id)
|
||||
model = AutoModelForSeq2SeqLM.from_pretrained(config.base_model_name_or_path)
|
||||
+ model = PeftModel.from_pretrained(model, peft_model_id)
|
||||
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
|
||||
|
||||
model = model.to(device)
|
||||
model.eval()
|
||||
inputs = tokenizer("Tweet text : @HondaCustSvc Your customer service has been horrible during the recall process. I will never purchase a Honda again. Label :", return_tensors="pt")
|
||||
|
||||
with torch.no_grad():
|
||||
outputs = model.generate(input_ids=inputs["input_ids"].to("cuda"), max_new_tokens=10)
|
||||
print(tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)[0])
|
||||
# 'complaint'
|
||||
```
|
||||
|
||||
## Launching your distributed script
|
||||
|
||||
PEFT models work with 🤗 Accelerate out of the box.
|
||||
You can use 🤗 Accelerate for distributed training on various hardware such as GPUs, or Apple Silicon devices during training, and for inference on consumer hardware with fewer resources.
|
||||
|
||||
### Train with 🤗 Accelerate's DeepSpeed integration
|
||||
|
||||
You'll need DeepSpeed version `v0.8.0` for this example. Feel free to check out the full example [script](https://github.com/huggingface/peft/blob/main/examples/conditional_generation/peft_lora_seq2seq_accelerate_ds_zero3_offload.py) for more details!
|
||||
|
||||
1. Run `accelerate config --config_file ds_zero3_cpu.yaml` and answer the questionnaire to setup your environment.
|
||||
Below are the contents of the config file.
|
||||
```yaml
|
||||
compute_environment: LOCAL_MACHINE
|
||||
deepspeed_config:
|
||||
gradient_accumulation_steps: 1
|
||||
gradient_clipping: 1.0
|
||||
offload_optimizer_device: cpu
|
||||
offload_param_device: cpu
|
||||
zero3_init_flag: true
|
||||
zero3_save_16bit_model: true
|
||||
zero_stage: 3
|
||||
distributed_type: DEEPSPEED
|
||||
downcast_bf16: 'no'
|
||||
dynamo_backend: 'NO'
|
||||
fsdp_config: {}
|
||||
machine_rank: 0
|
||||
main_training_function: main
|
||||
megatron_lm_config: {}
|
||||
mixed_precision: 'no'
|
||||
num_machines: 1
|
||||
num_processes: 1
|
||||
rdzv_backend: static
|
||||
same_network: true
|
||||
use_cpu: false
|
||||
```
|
||||
2. Run the following command to launch the example script:
|
||||
```bash
|
||||
accelerate launch --config_file ds_zero3_cpu.yaml examples/peft_lora_seq2seq_accelerate_ds_zero3_offload.py
|
||||
```
|
||||
|
||||
You'll see some output logs that look like this:
|
||||
```bash
|
||||
GPU Memory before entering the train : 1916
|
||||
GPU Memory consumed at the end of the train (end-begin): 66
|
||||
GPU Peak Memory consumed during the train (max-begin): 7488
|
||||
GPU Total Peak Memory consumed during the train (max): 9404
|
||||
CPU Memory before entering the train : 19411
|
||||
CPU Memory consumed at the end of the train (end-begin): 0
|
||||
CPU Peak Memory consumed during the train (max-begin): 0
|
||||
CPU Total Peak Memory consumed during the train (max): 19411
|
||||
epoch=4: train_ppl=tensor(1.0705, device='cuda:0') train_epoch_loss=tensor(0.0681, device='cuda:0')
|
||||
100%|████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:27<00:00, 3.92s/it]
|
||||
GPU Memory before entering the eval : 1982
|
||||
GPU Memory consumed at the end of the eval (end-begin): -66
|
||||
GPU Peak Memory consumed during the eval (max-begin): 672
|
||||
GPU Total Peak Memory consumed during the eval (max): 2654
|
||||
CPU Memory before entering the eval : 19411
|
||||
CPU Memory consumed at the end of the eval (end-begin): 0
|
||||
CPU Peak Memory consumed during the eval (max-begin): 0
|
||||
CPU Total Peak Memory consumed during the eval (max): 19411
|
||||
accuracy=100.0
|
||||
eval_preds[:10]=['no complaint', 'no complaint', 'complaint', 'complaint', 'no complaint', 'no complaint', 'no complaint', 'complaint', 'complaint', 'no complaint']
|
||||
dataset['train'][label_column][:10]=['no complaint', 'no complaint', 'complaint', 'complaint', 'no complaint', 'no complaint', 'no complaint', 'complaint', 'complaint', 'no complaint']
|
||||
```
|
||||
|
||||
### Inference with 🤗 Accelerate's Big Model Inference
|
||||
An example is provided in `~examples/causal_language_modeling/peft_lora_clm_accelerate_big_model_inference.ipynb`.
|
||||
|
||||
## Model Support matrix
|
||||
|
||||
### Causal Language Modeling
|
||||
| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
|
||||
|--------------| ---- | ---- | ---- | ---- |
|
||||
| GPT-2 | ✅ | ✅ | ✅ | ✅ |
|
||||
| Bloom | ✅ | ✅ | ✅ | ✅ |
|
||||
| OPT | ✅ | ✅ | ✅ | ✅ |
|
||||
| GPT-Neo | ✅ | ✅ | ✅ | ✅ |
|
||||
| GPT-J | ✅ | ✅ | ✅ | ✅ |
|
||||
| GPT-NeoX-20B | ✅ | ✅ | ✅ | ✅ |
|
||||
| LLaMA | ✅ | ✅ | ✅ | ✅ |
|
||||
| ChatGLM | ✅ | ✅ | ✅ | ✅ |
|
||||
|
||||
### Conditional Generation
|
||||
| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
|
||||
| --------- | ---- | ---- | ---- | ---- |
|
||||
| T5 | ✅ | ✅ | ✅ | ✅ |
|
||||
| BART | ✅ | ✅ | ✅ | ✅ |
|
||||
|
||||
### Sequence Classification
|
||||
| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
|
||||
| --------- | ---- | ---- | ---- | ---- |
|
||||
| BERT | ✅ | ✅ | ✅ | ✅ |
|
||||
| RoBERTa | ✅ | ✅ | ✅ | ✅ |
|
||||
| GPT-2 | ✅ | ✅ | ✅ | ✅ |
|
||||
| Bloom | ✅ | ✅ | ✅ | ✅ |
|
||||
| OPT | ✅ | ✅ | ✅ | ✅ |
|
||||
| GPT-Neo | ✅ | ✅ | ✅ | ✅ |
|
||||
| GPT-J | ✅ | ✅ | ✅ | ✅ |
|
||||
| Deberta | ✅ | | ✅ | ✅ |
|
||||
| Deberta-v2 | ✅ | | ✅ | ✅ |
|
||||
|
||||
### Token Classification
|
||||
| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
|
||||
| --------- | ---- | ---- | ---- | ---- |
|
||||
| BERT | ✅ | ✅ | | |
|
||||
| RoBERTa | ✅ | ✅ | | |
|
||||
| GPT-2 | ✅ | ✅ | | |
|
||||
| Bloom | ✅ | ✅ | | |
|
||||
| OPT | ✅ | ✅ | | |
|
||||
| GPT-Neo | ✅ | ✅ | | |
|
||||
| GPT-J | ✅ | ✅ | | |
|
||||
| Deberta | ✅ | | | |
|
||||
| Deberta-v2 | ✅ | | | |
|
||||
|
||||
### Text-to-Image Generation
|
||||
|
||||
| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
|
||||
| --------- | ---- | ---- | ---- | ---- |
|
||||
| Stable Diffusion | ✅ | | | |
|
||||
|
||||
|
||||
### Image Classification
|
||||
|
||||
| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
|
||||
| --------- | ---- | ---- | ---- | ---- |
|
||||
| ViT | ✅ | | | |
|
||||
| Swin | ✅ | | | |
|
||||
|
||||
___Note that we have tested LoRA for [ViT](https://huggingface.co/docs/transformers/model_doc/vit) and [Swin](https://huggingface.co/docs/transformers/model_doc/swin) for fine-tuning on image classification. However, it should be possible to use LoRA for any compatible model [provided](https://huggingface.co/models?pipeline_tag=image-classification&sort=downloads&search=vit) by 🤗 Transformers. Check out the respective
|
||||
examples to learn more. If you run into problems, please open an issue.___
|
||||
|
||||
The same principle applies to our [segmentation models](https://huggingface.co/models?pipeline_tag=image-segmentation&sort=downloads) as well.
|
||||
|
||||
### Semantic Segmentation
|
||||
|
||||
| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
|
||||
| --------- | ---- | ---- | ---- | ---- |
|
||||
| SegFormer | ✅ | | | |
|
||||
|
||||
|
||||
## Other caveats
|
||||
|
||||
1. Below is an example of using PyTorch FSDP for training. However, it doesn't lead to
|
||||
any GPU memory savings. Please refer to issue [[FSDP] FSDP with CPU offload consumes 1.65X more GPU memory when training models with most of the params frozen](https://github.com/pytorch/pytorch/issues/91165).
|
||||
|
||||
```python
|
||||
from peft.utils.other import fsdp_auto_wrap_policy
|
||||
|
||||
|
||||
if os.environ.get("ACCELERATE_USE_FSDP", None) is not None:
|
||||
accelerator.state.fsdp_plugin.auto_wrap_policy = fsdp_auto_wrap_policy(model)
|
||||
|
||||
model = accelerator.prepare(model)
|
||||
```
|
||||
|
||||
Example of parameter efficient tuning with [`mt0-xxl`](https://huggingface.co/bigscience/mt0-xxl) base model using 🤗 Accelerate is provided in `~examples/conditional_generation/peft_lora_seq2seq_accelerate_fsdp.py`.
|
||||
a. First, run `accelerate config --config_file fsdp_config.yaml` and answer the questionnaire.
|
||||
Below are the contents of the config file.
|
||||
```yaml
|
||||
command_file: null
|
||||
commands: null
|
||||
compute_environment: LOCAL_MACHINE
|
||||
deepspeed_config: {}
|
||||
distributed_type: FSDP
|
||||
downcast_bf16: 'no'
|
||||
dynamo_backend: 'NO'
|
||||
fsdp_config:
|
||||
fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
|
||||
fsdp_backward_prefetch_policy: BACKWARD_PRE
|
||||
fsdp_offload_params: true
|
||||
fsdp_sharding_strategy: 1
|
||||
fsdp_state_dict_type: FULL_STATE_DICT
|
||||
fsdp_transformer_layer_cls_to_wrap: T5Block
|
||||
gpu_ids: null
|
||||
machine_rank: 0
|
||||
main_process_ip: null
|
||||
main_process_port: null
|
||||
main_training_function: main
|
||||
megatron_lm_config: {}
|
||||
mixed_precision: 'no'
|
||||
num_machines: 1
|
||||
num_processes: 2
|
||||
rdzv_backend: static
|
||||
same_network: true
|
||||
tpu_name: null
|
||||
tpu_zone: null
|
||||
use_cpu: false
|
||||
```
|
||||
b. run the below command to launch the example script
|
||||
```bash
|
||||
accelerate launch --config_file fsdp_config.yaml examples/peft_lora_seq2seq_accelerate_fsdp.py
|
||||
```
|
||||
|
||||
2. When using `P_TUNING` or `PROMPT_TUNING` with `SEQ_2_SEQ` task, remember to remove the `num_virtual_token` virtual prompt predictions from the left side of the model outputs during evaluations.
|
||||
|
||||
3. For encoder-decoder models, `P_TUNING` or `PROMPT_TUNING` doesn't support the `generate` functionality of transformers because `generate` strictly requires `decoder_input_ids` but
|
||||
`P_TUNING`/`PROMPT_TUNING` append soft prompt embeddings to `input_embeds` to create
|
||||
new `input_embeds` to be given to the model. Therefore, `generate` doesn't support this yet.
|
||||
|
||||
4. When using ZeRO3 with zero3_init_flag=True, if you find the GPU memory increase with training steps. we might need to set zero3_init_flag=false in accelerate config.yaml. The related issue is [[BUG] memory leak under zero.Init](https://github.com/microsoft/DeepSpeed/issues/2637)
|
||||
@@ -0,0 +1,429 @@
|
||||
<!--Copyright 2023 The HuggingFace Team. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
||||
the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
||||
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations under the License.
|
||||
-->
|
||||
|
||||
# Image classification using LoRA
|
||||
|
||||
This guide demonstrates how to use LoRA, a low-rank approximation technique, to fine-tune an image classification model.
|
||||
By using LoRA from 🤗 PEFT, we can reduce the number of trainable parameters in the model to only 0.77% of the original.
|
||||
|
||||
LoRA achieves this reduction by adding low-rank "update matrices" to specific blocks of the model, such as the attention
|
||||
blocks. During fine-tuning, only these matrices are trained, while the original model parameters are left unchanged.
|
||||
At inference time, the update matrices are merged with the original model parameters to produce the final classification result.
|
||||
|
||||
For more information on LoRA, please refer to the [original LoRA paper](https://arxiv.org/abs/2106.09685).
|
||||
|
||||
## Install dependencies
|
||||
|
||||
Install the libraries required for model training:
|
||||
|
||||
```bash
|
||||
!pip install transformers accelerate evaluate datasets loralib peft -q
|
||||
```
|
||||
|
||||
Check the versions of all required libraries to make sure you are up to date:
|
||||
|
||||
```python
|
||||
import transformers
|
||||
import accelerate
|
||||
import peft
|
||||
|
||||
print(f"Transformers version: {transformers.__version__}")
|
||||
print(f"Accelerate version: {accelerate.__version__}")
|
||||
print(f"PEFT version: {peft.__version__}")
|
||||
"Transformers version: 4.27.4"
|
||||
"Accelerate version: 0.18.0"
|
||||
"PEFT version: 0.2.0"
|
||||
```
|
||||
|
||||
## Authenticate to share your model
|
||||
|
||||
To share the fine-tuned model at the end of the training with the community, authenticate using your 🤗 token.
|
||||
You can obtain your token from your [account settings](https://huggingface.co/settings/token).
|
||||
|
||||
```python
|
||||
from huggingface_hub import notebook_login
|
||||
|
||||
notebook_login()
|
||||
```
|
||||
|
||||
## Select a model checkpoint to fine-tune
|
||||
|
||||
Choose a model checkpoint from any of the model architectures supported for [image classification](https://huggingface.co/models?pipeline_tag=image-classification&sort=downloads). When in doubt, refer to
|
||||
the [image classification task guide](https://huggingface.co/docs/transformers/v4.27.2/en/tasks/image_classification) in
|
||||
🤗 Transformers documentation.
|
||||
|
||||
```python
|
||||
model_checkpoint = "google/vit-base-patch16-224-in21k"
|
||||
```
|
||||
|
||||
## Load a dataset
|
||||
|
||||
To keep this example's runtime short, let's only load the first 5000 instances from the training set of the [Food-101 dataset](https://huggingface.co/datasets/food101):
|
||||
|
||||
```python
|
||||
from datasets import load_dataset
|
||||
|
||||
dataset = load_dataset("food101", split="train[:5000]")
|
||||
```
|
||||
|
||||
## Dataset preparation
|
||||
|
||||
To prepare the dataset for training and evaluation, create `label2id` and `id2label` dictionaries. These will come in
|
||||
handy when performing inference and for metadata information:
|
||||
|
||||
```python
|
||||
labels = dataset.features["label"].names
|
||||
label2id, id2label = dict(), dict()
|
||||
for i, label in enumerate(labels):
|
||||
label2id[label] = i
|
||||
id2label[i] = label
|
||||
|
||||
id2label[2]
|
||||
"baklava"
|
||||
```
|
||||
|
||||
Next, load the image processor of the model you're fine-tuning:
|
||||
|
||||
```python
|
||||
from transformers import AutoImageProcessor
|
||||
|
||||
image_processor = AutoImageProcessor.from_pretrained(model_checkpoint)
|
||||
```
|
||||
|
||||
The `image_processor` contains useful information on which size the training and evaluation images should be resized
|
||||
to, as well as values that should be used to normalize the pixel values. Using the `image_processor`, prepare transformation
|
||||
functions for the datasets. These functions will include data augmentation and pixel scaling:
|
||||
|
||||
```python
|
||||
from torchvision.transforms import (
|
||||
CenterCrop,
|
||||
Compose,
|
||||
Normalize,
|
||||
RandomHorizontalFlip,
|
||||
RandomResizedCrop,
|
||||
Resize,
|
||||
ToTensor,
|
||||
)
|
||||
|
||||
normalize = Normalize(mean=image_processor.image_mean, std=image_processor.image_std)
|
||||
train_transforms = Compose(
|
||||
[
|
||||
RandomResizedCrop(image_processor.size["height"]),
|
||||
RandomHorizontalFlip(),
|
||||
ToTensor(),
|
||||
normalize,
|
||||
]
|
||||
)
|
||||
|
||||
val_transforms = Compose(
|
||||
[
|
||||
Resize(image_processor.size["height"]),
|
||||
CenterCrop(image_processor.size["height"]),
|
||||
ToTensor(),
|
||||
normalize,
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def preprocess_train(example_batch):
|
||||
"""Apply train_transforms across a batch."""
|
||||
example_batch["pixel_values"] = [train_transforms(image.convert("RGB")) for image in example_batch["image"]]
|
||||
return example_batch
|
||||
|
||||
|
||||
def preprocess_val(example_batch):
|
||||
"""Apply val_transforms across a batch."""
|
||||
example_batch["pixel_values"] = [val_transforms(image.convert("RGB")) for image in example_batch["image"]]
|
||||
return example_batch
|
||||
```
|
||||
|
||||
Split the dataset into training and validation sets:
|
||||
|
||||
```python
|
||||
splits = dataset.train_test_split(test_size=0.1)
|
||||
train_ds = splits["train"]
|
||||
val_ds = splits["test"]
|
||||
```
|
||||
|
||||
Finally, set the transformation functions for the datasets accordingly:
|
||||
|
||||
```python
|
||||
train_ds.set_transform(preprocess_train)
|
||||
val_ds.set_transform(preprocess_val)
|
||||
```
|
||||
|
||||
## Load and prepare a model
|
||||
|
||||
Before loading the model, let's define a helper function to check the total number of parameters a model has, as well
|
||||
as how many of them are trainable.
|
||||
|
||||
```python
|
||||
def print_trainable_parameters(model):
|
||||
trainable_params = 0
|
||||
all_param = 0
|
||||
for _, param in model.named_parameters():
|
||||
all_param += param.numel()
|
||||
if param.requires_grad:
|
||||
trainable_params += param.numel()
|
||||
print(
|
||||
f"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param:.2f}"
|
||||
)
|
||||
```
|
||||
|
||||
It's important to initialize the original model correctly as it will be used as a base to create the `PeftModel` you'll
|
||||
actually fine-tune. Specify the `label2id` and `id2label` so that [`~transformers.AutoModelForImageClassification`] can append a classification
|
||||
head to the underlying model, adapted for this dataset. You should see the following output:
|
||||
|
||||
```
|
||||
Some weights of ViTForImageClassification were not initialized from the model checkpoint at google/vit-base-patch16-224-in21k and are newly initialized: ['classifier.weight', 'classifier.bias']
|
||||
```
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForImageClassification, TrainingArguments, Trainer
|
||||
|
||||
model = AutoModelForImageClassification.from_pretrained(
|
||||
model_checkpoint,
|
||||
label2id=label2id,
|
||||
id2label=id2label,
|
||||
ignore_mismatched_sizes=True, # provide this in case you're planning to fine-tune an already fine-tuned checkpoint
|
||||
)
|
||||
```
|
||||
|
||||
Before creating a `PeftModel`, you can check the number of trainable parameters in the original model:
|
||||
|
||||
```python
|
||||
print_trainable_parameters(model)
|
||||
"trainable params: 85876325 || all params: 85876325 || trainable%: 100.00"
|
||||
```
|
||||
|
||||
Next, use `get_peft_model` to wrap the base model so that "update" matrices are added to the respective places.
|
||||
|
||||
```python
|
||||
from peft import LoraConfig, get_peft_model
|
||||
|
||||
config = LoraConfig(
|
||||
r=16,
|
||||
lora_alpha=16,
|
||||
target_modules=["query", "value"],
|
||||
lora_dropout=0.1,
|
||||
bias="none",
|
||||
modules_to_save=["classifier"],
|
||||
)
|
||||
lora_model = get_peft_model(model, config)
|
||||
print_trainable_parameters(lora_model)
|
||||
"trainable params: 667493 || all params: 86466149 || trainable%: 0.77"
|
||||
```
|
||||
|
||||
Let's unpack what's going on here.
|
||||
To use LoRA, you need to specify the target modules in `LoraConfig` so that `get_peft_model()` knows which modules
|
||||
inside our model need to be amended with LoRA matrices. In this example, we're only interested in targeting the query and
|
||||
value matrices of the attention blocks of the base model. Since the parameters corresponding to these matrices are "named"
|
||||
"query" and "value" respectively, we specify them accordingly in the `target_modules` argument of `LoraConfig`.
|
||||
|
||||
We also specify `modules_to_save`. After wrapping the base model with `get_peft_model()` along with the `config`, we get
|
||||
a new model where only the LoRA parameters are trainable (so-called "update matrices") while the pre-trained parameters
|
||||
are kept frozen. However, we want the classifier parameters to be trained too when fine-tuning the base model on our
|
||||
custom dataset. To ensure that the classifier parameters are also trained, we specify `modules_to_save`. This also
|
||||
ensures that these modules are serialized alongside the LoRA trainable parameters when using utilities like `save_pretrained()`
|
||||
and `push_to_hub()`.
|
||||
|
||||
Here's what the other parameters mean:
|
||||
|
||||
- `r`: The dimension used by the LoRA update matrices.
|
||||
- `alpha`: Scaling factor.
|
||||
- `bias`: Specifies if the `bias` parameters should be trained. `None` denotes none of the `bias` parameters will be trained.
|
||||
|
||||
`r` and `alpha` together control the total number of final trainable parameters when using LoRA, giving you the flexibility
|
||||
to balance a trade-off between end performance and compute efficiency.
|
||||
|
||||
By looking at the number of trainable parameters, you can see how many parameters we're actually training. Since the goal is
|
||||
to achieve parameter-efficient fine-tuning, you should expect to see fewer trainable parameters in the `lora_model`
|
||||
in comparison to the original model, which is indeed the case here.
|
||||
|
||||
## Define training arguments
|
||||
|
||||
For model fine-tuning, use [`~transformers.Trainer`]. It accepts
|
||||
several arguments which you can wrap using [`~transformers.TrainingArguments`].
|
||||
|
||||
```python
|
||||
from transformers import TrainingArguments, Trainer
|
||||
|
||||
|
||||
model_name = model_checkpoint.split("/")[-1]
|
||||
batch_size = 128
|
||||
|
||||
args = TrainingArguments(
|
||||
f"{model_name}-finetuned-lora-food101",
|
||||
remove_unused_columns=False,
|
||||
evaluation_strategy="epoch",
|
||||
save_strategy="epoch",
|
||||
learning_rate=5e-3,
|
||||
per_device_train_batch_size=batch_size,
|
||||
gradient_accumulation_steps=4,
|
||||
per_device_eval_batch_size=batch_size,
|
||||
fp16=True,
|
||||
num_train_epochs=5,
|
||||
logging_steps=10,
|
||||
load_best_model_at_end=True,
|
||||
metric_for_best_model="accuracy",
|
||||
push_to_hub=True,
|
||||
label_names=["labels"],
|
||||
)
|
||||
```
|
||||
|
||||
Compared to non-PEFT methods, you can use a larger batch size since there are fewer parameters to train.
|
||||
You can also set a larger learning rate than the normal (1e-5 for example).
|
||||
|
||||
This can potentially also reduce the need to conduct expensive hyperparameter tuning experiments.
|
||||
|
||||
## Prepare evaluation metric
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
import evaluate
|
||||
|
||||
metric = evaluate.load("accuracy")
|
||||
|
||||
|
||||
def compute_metrics(eval_pred):
|
||||
"""Computes accuracy on a batch of predictions"""
|
||||
predictions = np.argmax(eval_pred.predictions, axis=1)
|
||||
return metric.compute(predictions=predictions, references=eval_pred.label_ids)
|
||||
```
|
||||
|
||||
The `compute_metrics` function takes a named tuple as input: `predictions`, which are the logits of the model as Numpy arrays,
|
||||
and `label_ids`, which are the ground-truth labels as Numpy arrays.
|
||||
|
||||
## Define collation function
|
||||
|
||||
A collation function is used by [`~transformers.Trainer`] to gather a batch of training and evaluation examples and prepare them in a
|
||||
format that is acceptable by the underlying model.
|
||||
|
||||
```python
|
||||
import torch
|
||||
|
||||
|
||||
def collate_fn(examples):
|
||||
pixel_values = torch.stack([example["pixel_values"] for example in examples])
|
||||
labels = torch.tensor([example["label"] for example in examples])
|
||||
return {"pixel_values": pixel_values, "labels": labels}
|
||||
```
|
||||
|
||||
## Train and evaluate
|
||||
|
||||
Bring everything together - model, training arguments, data, collation function, etc. Then, start the training!
|
||||
|
||||
```python
|
||||
trainer = Trainer(
|
||||
model,
|
||||
args,
|
||||
train_dataset=train_ds,
|
||||
eval_dataset=val_ds,
|
||||
tokenizer=image_processor,
|
||||
compute_metrics=compute_metrics,
|
||||
data_collator=collate_fn,
|
||||
)
|
||||
train_results = trainer.train()
|
||||
```
|
||||
|
||||
In just a few minutes, the fine-tuned model shows 96% validation accuracy even on this small
|
||||
subset of the training dataset.
|
||||
|
||||
```python
|
||||
trainer.evaluate(val_ds)
|
||||
{
|
||||
"eval_loss": 0.14475855231285095,
|
||||
"eval_accuracy": 0.96,
|
||||
"eval_runtime": 3.5725,
|
||||
"eval_samples_per_second": 139.958,
|
||||
"eval_steps_per_second": 1.12,
|
||||
"epoch": 5.0,
|
||||
}
|
||||
```
|
||||
|
||||
## Share your model and run inference
|
||||
|
||||
Once the fine-tuning is done, share the LoRA parameters with the community like so:
|
||||
|
||||
```python
|
||||
repo_name = f"sayakpaul/{model_name}-finetuned-lora-food101"
|
||||
lora_model.push_to_hub(repo_name)
|
||||
```
|
||||
|
||||
When calling [`~transformers.PreTrainedModel.push_to_hub`] on the `lora_model`, only the LoRA parameters along with any modules specified in `modules_to_save`
|
||||
are saved. Take a look at the [trained LoRA parameters](https://huggingface.co/sayakpaul/vit-base-patch16-224-in21k-finetuned-lora-food101/blob/main/adapter_model.bin).
|
||||
You'll see that it's only 2.6 MB! This greatly helps with portability, especially when using a very large model to fine-tune (such as [BLOOM](https://huggingface.co/bigscience/bloom)).
|
||||
|
||||
Next, let's see how to load the LoRA updated parameters along with our base model for inference. When you wrap a base model
|
||||
with `PeftModel`, modifications are done *in-place*. To mitigate any concerns that might stem from in-place modifications,
|
||||
initialize the base model just like you did earlier and construct the inference model.
|
||||
|
||||
```python
|
||||
from peft import PeftConfig, PeftModel
|
||||
|
||||
|
||||
config = PeftConfig.from_pretrained(repo_name)
|
||||
model = AutoModelForImageClassification.from_pretrained(
|
||||
config.base_model_name_or_path,
|
||||
label2id=label2id,
|
||||
id2label=id2label,
|
||||
ignore_mismatched_sizes=True, # provide this in case you're planning to fine-tune an already fine-tuned checkpoint
|
||||
)
|
||||
# Load the LoRA model
|
||||
inference_model = PeftModel.from_pretrained(model, repo_name)
|
||||
```
|
||||
|
||||
Let's now fetch an example image for inference.
|
||||
|
||||
```python
|
||||
from PIL import Image
|
||||
import requests
|
||||
|
||||
url = "https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/beignets.jpeg"
|
||||
image = Image.open(requests.get(url, stream=True).raw)
|
||||
image
|
||||
```
|
||||
|
||||
<div class="flex justify-center">
|
||||
<img src="https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/beignets.jpeg" alt="image of beignets"/>
|
||||
</div>
|
||||
|
||||
First, instantiate an `image_processor` from the underlying model repo.
|
||||
|
||||
```python
|
||||
image_processor = AutoImageProcessor.from_pretrained(repo_name)
|
||||
```
|
||||
|
||||
Then, prepare the example for inference.
|
||||
|
||||
```python
|
||||
encoding = image_processor(image.convert("RGB"), return_tensors="pt")
|
||||
```
|
||||
|
||||
Finally, run inference!
|
||||
|
||||
```python
|
||||
with torch.no_grad():
|
||||
outputs = inference_model(**encoding)
|
||||
logits = outputs.logits
|
||||
|
||||
predicted_class_idx = logits.argmax(-1).item()
|
||||
print("Predicted class:", inference_model.config.id2label[predicted_class_idx])
|
||||
"Predicted class: beignets"
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1063,7 +1063,9 @@ def main(args):
|
||||
)
|
||||
text_encoder_state_dict = {f"text_encoder_{k}": v for k, v in text_encoder_state_dict.items()}
|
||||
state_dict.update(text_encoder_state_dict)
|
||||
lora_config["text_encoder_peft_config"] = unwarpped_text_encoder.get_peft_config_as_dict(inference=True)
|
||||
lora_config["text_encoder_peft_config"] = unwarpped_text_encoder.get_peft_config_as_dict(
|
||||
inference=True
|
||||
)
|
||||
|
||||
accelerator.print(state_dict)
|
||||
accelerator.save(state_dict, os.path.join(args.output_dir, f"{args.instance_prompt}_lora.pt"))
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
# Copyright 2023 The HuggingFace Team, the AllenNLP library authors. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
"""
|
||||
Script to close stale issue. Taken in part from the AllenNLP repository.
|
||||
https://github.com/allenai/allennlp.
|
||||
"""
|
||||
from datetime import datetime as dt
|
||||
import os
|
||||
|
||||
from github import Github
|
||||
|
||||
|
||||
LABELS_TO_EXEMPT = [
|
||||
"good first issue",
|
||||
"good second issue",
|
||||
"good difficult issue",
|
||||
"feature request",
|
||||
"new model",
|
||||
"wip",
|
||||
"PRs welcome to address this",
|
||||
]
|
||||
|
||||
|
||||
def main():
|
||||
g = Github(os.environ["GITHUB_TOKEN"])
|
||||
repo = g.get_repo("huggingface/peft")
|
||||
open_issues = repo.get_issues(state="open")
|
||||
|
||||
for issue in open_issues:
|
||||
comments = sorted([comment for comment in issue.get_comments()], key=lambda i: i.created_at, reverse=True)
|
||||
last_comment = comments[0] if len(comments) > 0 else None
|
||||
if (
|
||||
last_comment is not None and last_comment.user.login == "github-actions[bot]"
|
||||
and (dt.utcnow() - issue.updated_at).days > 7
|
||||
and (dt.utcnow() - issue.created_at).days >= 30
|
||||
and not any(label.name.lower() in LABELS_TO_EXEMPT for label in issue.get_labels())
|
||||
):
|
||||
issue.edit(state="closed")
|
||||
elif (
|
||||
(dt.utcnow() - issue.updated_at).days > 23
|
||||
and (dt.utcnow() - issue.created_at).days >= 30
|
||||
and not any(label.name.lower() in LABELS_TO_EXEMPT for label in issue.get_labels())
|
||||
):
|
||||
issue.create_comment(
|
||||
"This issue has been automatically marked as stale because it has not had "
|
||||
"recent activity. If you think this still needs to be addressed "
|
||||
"please comment on this thread.\n\n"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+159
-70
@@ -54,26 +54,26 @@ PEFT_TYPE_TO_MODEL_MAPPING = {
|
||||
|
||||
class PeftModel(PushToHubMixin, torch.nn.Module):
|
||||
"""
|
||||
Parameter-Efficient Fine-Tuning Model. Base model encompassing various Peft methods.
|
||||
Base model encompassing various Peft methods.
|
||||
|
||||
Args:
|
||||
model ([`PreTrainedModel`]): The base transformer model used for Peft.
|
||||
model ([`~transformers.PreTrainedModel`]): The base transformer model used for Peft.
|
||||
peft_config ([`PeftConfig`]): The configuration of the Peft model.
|
||||
|
||||
|
||||
**Attributes**:
|
||||
- **base_model** ([`PreTrainedModel`]) -- The base transformer model used for Peft.
|
||||
- **base_model** ([`~transformers.PreTrainedModel`]) -- The base transformer model used for Peft.
|
||||
- **peft_config** ([`PeftConfig`]) -- The configuration of the Peft model.
|
||||
- **modules_to_save** (`list` of `str`) -- The list of sub-module names to save when
|
||||
saving the model.
|
||||
- **prompt_encoder** ([`PromptEncoder`]) -- The prompt encoder used for Peft if
|
||||
`isinstance(self.peft_config, PromptLearningConfig)`.
|
||||
using [`PromptLearningConfig`].
|
||||
- **prompt_tokens** (`torch.Tensor`) -- The virtual prompt tokens used for Peft if
|
||||
`isinstance(self.peft_config, PromptLearningConfig)`.
|
||||
using [`PromptLearningConfig`].
|
||||
- **transformer_backbone_name** (`str`) -- The name of the transformer
|
||||
backbone in the base model if `isinstance(self.peft_config, PromptLearningConfig)`.
|
||||
backbone in the base model if using [`PromptLearningConfig`].
|
||||
- **word_embeddings** (`torch.nn.Embedding`) -- The word embeddings of the transformer backbone
|
||||
in the base model if `isinstance(self.peft_config, PromptLearningConfig)`.
|
||||
in the base model if using [`PromptLearningConfig`].
|
||||
"""
|
||||
|
||||
def __init__(self, model, peft_config: PeftConfig, adapter_name="default"):
|
||||
@@ -94,14 +94,15 @@ class PeftModel(PushToHubMixin, torch.nn.Module):
|
||||
|
||||
def save_pretrained(self, save_directory, **kwargs):
|
||||
r"""
|
||||
Args:
|
||||
This function saves the adapter model and the adapter configuration files to a directory, so that it can be
|
||||
re-loaded using the `LoraModel.from_pretrained` class method, and also used by the `LoraModel.push_to_hub`
|
||||
reloaded using the [`LoraModel.from_pretrained`] class method, and also used by the [`LoraModel.push_to_hub`]
|
||||
method.
|
||||
|
||||
Args:
|
||||
save_directory (`str`):
|
||||
Directory where the adapter model and configuration files will be saved (will be created if it does not
|
||||
exist).
|
||||
**kwargs:
|
||||
kwargs (additional keyword arguments, *optional*):
|
||||
Additional keyword arguments passed along to the `push_to_hub` method.
|
||||
"""
|
||||
if os.path.isfile(save_directory):
|
||||
@@ -130,17 +131,18 @@ class PeftModel(PushToHubMixin, torch.nn.Module):
|
||||
@classmethod
|
||||
def from_pretrained(cls, model, model_id, adapter_name="default", **kwargs):
|
||||
r"""
|
||||
Instantiate a [`LoraModel`] from a pretrained Lora configuration and weights.
|
||||
|
||||
Args:
|
||||
Instantiate a `PeftModel` from a pretrained Peft configuration and weights.
|
||||
model (`transformers.PreTrainedModel`):
|
||||
The model to be adapted. The model should be initialized with the `from_pretrained` method. from
|
||||
`transformers` library.
|
||||
model_id (`str`):
|
||||
model ([`~transformers.PreTrainedModel`]):
|
||||
The model to be adapted. The model should be initialized with the
|
||||
[`~transformers.PreTrainedModel.from_pretrained`] method from the 🤗 Transformers library.
|
||||
model_id (`str` or `os.PathLike`):
|
||||
The name of the Lora configuration to use. Can be either:
|
||||
- A string, the `model id` of a Lora configuration hosted inside a model repo on
|
||||
huggingface Hub
|
||||
- A path to a directory containing a Lora configuration file saved using the
|
||||
`save_pretrained` method, e.g., ``./my_lora_config_directory/``.
|
||||
- A string, the `model id` of a Lora configuration hosted inside a model repo on the Hugging Face
|
||||
Hub.
|
||||
- A path to a directory containing a Lora configuration file saved using the `save_pretrained`
|
||||
method (`./my_lora_config_directory/`).
|
||||
"""
|
||||
from .mapping import MODEL_TYPE_TO_PEFT_MODEL_MAPPING, PEFT_TYPE_TO_CONFIG_MAPPING
|
||||
|
||||
@@ -349,7 +351,17 @@ class PeftModel(PushToHubMixin, torch.nn.Module):
|
||||
):
|
||||
device_map = kwargs.get("device_map", "auto")
|
||||
max_memory = kwargs.get("max_memory", None)
|
||||
offload_dir = kwargs.get("offload_folder", None)
|
||||
offload_index = kwargs.get("offload_index", None)
|
||||
|
||||
dispatch_model_kwargs = {}
|
||||
# Safety checker for previous `accelerate` versions
|
||||
# `offload_index` was introduced in https://github.com/huggingface/accelerate/pull/873/
|
||||
if "offload_index" in inspect.signature(dispatch_model).parameters:
|
||||
dispatch_model_kwargs["offload_index"] = offload_index
|
||||
|
||||
no_split_module_classes = self._no_split_modules
|
||||
|
||||
if device_map != "sequential":
|
||||
max_memory = get_balanced_memory(
|
||||
self,
|
||||
@@ -361,7 +373,12 @@ class PeftModel(PushToHubMixin, torch.nn.Module):
|
||||
device_map = infer_auto_device_map(
|
||||
self, max_memory=max_memory, no_split_module_classes=no_split_module_classes
|
||||
)
|
||||
dispatch_model(self, device_map=device_map)
|
||||
dispatch_model(
|
||||
self,
|
||||
device_map=device_map,
|
||||
offload_dir=offload_dir,
|
||||
**dispatch_model_kwargs,
|
||||
)
|
||||
hook = AlignDevicesHook(io_same_device=True)
|
||||
if not isinstance(self.peft_config[adapter_name]) == PeftType.LORA:
|
||||
add_hook_to_module(self.base_model.model, hook)
|
||||
@@ -386,25 +403,39 @@ class PeftModelForSequenceClassification(PeftModel):
|
||||
Peft model for sequence classification tasks.
|
||||
|
||||
Args:
|
||||
model ([`PreTrainedModel`]): Base transformer model
|
||||
model ([`~transformers.PreTrainedModel`]): Base transformer model.
|
||||
peft_config ([`PeftConfig`]): Peft config.
|
||||
|
||||
**Attributes**:
|
||||
- **config** ([`PretrainedConfig`]) -- The configuration object of the base model.
|
||||
- **config** ([`~transformers.PretrainedConfig`]) -- The configuration object of the base model.
|
||||
- **cls_layer_name** (`str`) -- The name of the classification layer.
|
||||
|
||||
Example::
|
||||
Example:
|
||||
|
||||
>>> from transformers import AutoModelForSequenceClassification >>> from peft import
|
||||
PeftModelForSequenceClassification, get_peft_config >>> config = {
|
||||
'peft_type': 'PREFIX_TUNING', 'task_type': 'SEQ_CLS', 'inference_mode': False, 'num_virtual_tokens':
|
||||
20, 'token_dim': 768, 'num_transformer_submodules': 1, 'num_attention_heads': 12, 'num_layers': 12,
|
||||
'encoder_hidden_size': 768, 'prefix_projection': False, 'postprocess_past_key_value_function': None
|
||||
}
|
||||
>>> peft_config = get_peft_config(config) >>> model =
|
||||
AutoModelForSequenceClassification.from_pretrained("bert-base-cased") >>> peft_model =
|
||||
PeftModelForSequenceClassification(model, peft_config) >>> peft_model.print_trainable_parameters() trainable
|
||||
params: 370178 || all params: 108680450 || trainable%: 0.3406113979101117
|
||||
```py
|
||||
>>> from transformers import AutoModelForSequenceClassification
|
||||
>>> from peft import PeftModelForSequenceClassification, get_peft_config
|
||||
|
||||
>>> config = {
|
||||
... "peft_type": "PREFIX_TUNING",
|
||||
... "task_type": "SEQ_CLS",
|
||||
... "inference_mode": False,
|
||||
... "num_virtual_tokens": 20,
|
||||
... "token_dim": 768,
|
||||
... "num_transformer_submodules": 1,
|
||||
... "num_attention_heads": 12,
|
||||
... "num_layers": 12,
|
||||
... "encoder_hidden_size": 768,
|
||||
... "prefix_projection": False,
|
||||
... "postprocess_past_key_value_function": None,
|
||||
... }
|
||||
|
||||
>>> peft_config = get_peft_config(config)
|
||||
>>> model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased")
|
||||
>>> peft_model = PeftModelForSequenceClassification(model, peft_config)
|
||||
>>> peft_model.print_trainable_parameters()
|
||||
trainable params: 370178 || all params: 108680450 || trainable%: 0.3406113979101117
|
||||
```
|
||||
"""
|
||||
|
||||
def __init__(self, model, peft_config: PeftConfig, adapter_name="default"):
|
||||
@@ -557,24 +588,39 @@ class PeftModelForSequenceClassification(PeftModel):
|
||||
|
||||
class PeftModelForCausalLM(PeftModel):
|
||||
"""
|
||||
Peft model for Causal LM
|
||||
Peft model for causal language modeling.
|
||||
|
||||
Args:
|
||||
model ([`PreTrainedModel`]): Base transformer model
|
||||
model ([`~transformers.PreTrainedModel`]): Base transformer model.
|
||||
peft_config ([`PeftConfig`]): Peft config.
|
||||
|
||||
|
||||
Example::
|
||||
Example:
|
||||
|
||||
```py
|
||||
>>> from transformers import AutoModelForCausalLM
|
||||
>>> from peft import PeftModelForCausalLM, get_peft_config
|
||||
|
||||
>>> from transformers import AutoModelForCausalLM >>> from peft import PeftModelForCausalLM, get_peft_config
|
||||
>>> config = {
|
||||
'peft_type': 'PREFIX_TUNING', 'task_type': 'CAUSAL_LM', 'inference_mode': False, 'num_virtual_tokens':
|
||||
20, 'token_dim': 1280, 'num_transformer_submodules': 1, 'num_attention_heads': 20, 'num_layers': 36,
|
||||
'encoder_hidden_size': 1280, 'prefix_projection': False, 'postprocess_past_key_value_function': None
|
||||
}
|
||||
>>> peft_config = get_peft_config(config) >>> model = AutoModelForCausalLM.from_pretrained("gpt2-large") >>>
|
||||
peft_model = PeftModelForCausalLM(model, peft_config) >>> peft_model.print_trainable_parameters() trainable
|
||||
params: 1843200 || all params: 775873280 || trainable%: 0.23756456724479544
|
||||
... "peft_type": "PREFIX_TUNING",
|
||||
... "task_type": "CAUSAL_LM",
|
||||
... "inference_mode": False,
|
||||
... "num_virtual_tokens": 20,
|
||||
... "token_dim": 1280,
|
||||
... "num_transformer_submodules": 1,
|
||||
... "num_attention_heads": 20,
|
||||
... "num_layers": 36,
|
||||
... "encoder_hidden_size": 1280,
|
||||
... "prefix_projection": False,
|
||||
... "postprocess_past_key_value_function": None,
|
||||
... }
|
||||
|
||||
>>> peft_config = get_peft_config(config)
|
||||
>>> model = AutoModelForCausalLM.from_pretrained("gpt2-large")
|
||||
>>> peft_model = PeftModelForCausalLM(model, peft_config)
|
||||
>>> peft_model.print_trainable_parameters()
|
||||
trainable params: 1843200 || all params: 775873280 || trainable%: 0.23756456724479544
|
||||
```
|
||||
"""
|
||||
|
||||
def __init__(self, model, peft_config: PeftConfig, adapter_name="default"):
|
||||
@@ -649,7 +695,13 @@ class PeftModelForCausalLM(PeftModel):
|
||||
else:
|
||||
if "input_ids" not in kwargs:
|
||||
raise ValueError("input_ids must be provided for Peft model generation")
|
||||
if kwargs.get("attention_mask", None) is not None:
|
||||
# For gpt2 models, we construct postion_ids on the fly by using attention mask, and position ids need to match input_shape.
|
||||
# for prefix tuning, input shape is determined using `input_ids`. Thus we should not expand 'attention_mask' here
|
||||
# for prompt tuning input_ids is not passed but a concatenated input_embeds is passed. Thus attention_mask needs to be of same size of num_virtual_tokens + input_ids
|
||||
if kwargs.get("attention_mask", None) is not None and self.peft_config.peft_type in [
|
||||
PeftType.PROMPT_TUNING,
|
||||
PeftType.P_TUNING,
|
||||
]:
|
||||
# concat prompt attention mask
|
||||
prefix_attention_mask = torch.ones(
|
||||
kwargs["input_ids"].shape[0], self.peft_config.num_virtual_tokens
|
||||
@@ -678,6 +730,14 @@ class PeftModelForCausalLM(PeftModel):
|
||||
def prepare_inputs_for_generation(self, *args, **kwargs):
|
||||
model_kwargs = self.base_model_prepare_inputs_for_generation(*args, **kwargs)
|
||||
if isinstance(self.peft_config, PromptLearningConfig):
|
||||
if self.peft_config.peft_type == PeftType.PREFIX_TUNING:
|
||||
prefix_attention_mask = torch.ones(
|
||||
model_kwargs["input_ids"].shape[0], self.peft_config.num_virtual_tokens
|
||||
).to(model_kwargs["input_ids"].device)
|
||||
model_kwargs["attention_mask"] = torch.cat(
|
||||
(prefix_attention_mask, model_kwargs["attention_mask"]), dim=1
|
||||
)
|
||||
|
||||
if model_kwargs["past_key_values"] is None and self.peft_config.peft_type == PeftType.PREFIX_TUNING:
|
||||
past_key_values = self.get_prompt(batch_size=model_kwargs["input_ids"].shape[0])
|
||||
model_kwargs["past_key_values"] = past_key_values
|
||||
@@ -694,24 +754,39 @@ class PeftModelForCausalLM(PeftModel):
|
||||
|
||||
class PeftModelForSeq2SeqLM(PeftModel):
|
||||
"""
|
||||
Peft model for Seq2Seq LM
|
||||
Peft model for sequence-to-sequence language modeling.
|
||||
|
||||
Args:
|
||||
model ([`PreTrainedModel`]): Base transformer model
|
||||
model ([`~transformers.PreTrainedModel`]): Base transformer model.
|
||||
peft_config ([`PeftConfig`]): Peft config.
|
||||
|
||||
|
||||
Example::
|
||||
Example:
|
||||
|
||||
```py
|
||||
>>> from transformers import AutoModelForSeq2SeqLM
|
||||
>>> from peft import PeftModelForSeq2SeqLM, get_peft_config
|
||||
|
||||
>>> from transformers import AutoModelForSeq2SeqLM >>> from peft import PeftModelForSeq2SeqLM, get_peft_config
|
||||
>>> config = {
|
||||
'peft_type': 'LORA', 'task_type': 'SEQ_2_SEQ_LM', 'inference_mode': False, 'r': 8, 'target_modules':
|
||||
['q', 'v'], 'lora_alpha': 32, 'lora_dropout': 0.1, 'merge_weights': False, 'fan_in_fan_out': False,
|
||||
'enable_lora': None, 'bias': 'none'
|
||||
}
|
||||
>>> peft_config = get_peft_config(config) >>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base") >>>
|
||||
peft_model = PeftModelForSeq2SeqLM(model, peft_config) >>> peft_model.print_trainable_parameters() trainable
|
||||
params: 884736 || all params: 223843584 || trainable%: 0.3952474242013566
|
||||
... "peft_type": "LORA",
|
||||
... "task_type": "SEQ_2_SEQ_LM",
|
||||
... "inference_mode": False,
|
||||
... "r": 8,
|
||||
... "target_modules": ["q", "v"],
|
||||
... "lora_alpha": 32,
|
||||
... "lora_dropout": 0.1,
|
||||
... "merge_weights": False,
|
||||
... "fan_in_fan_out": False,
|
||||
... "enable_lora": None,
|
||||
... "bias": "none",
|
||||
... }
|
||||
|
||||
>>> peft_config = get_peft_config(config)
|
||||
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
|
||||
>>> peft_model = PeftModelForSeq2SeqLM(model, peft_config)
|
||||
>>> peft_model.print_trainable_parameters()
|
||||
trainable params: 884736 || all params: 223843584 || trainable%: 0.3952474242013566
|
||||
```
|
||||
"""
|
||||
|
||||
def __init__(self, model, peft_config: PeftConfig, adapter_name="default"):
|
||||
@@ -861,28 +936,42 @@ class PeftModelForSeq2SeqLM(PeftModel):
|
||||
|
||||
class PeftModelForTokenClassification(PeftModel):
|
||||
"""
|
||||
Peft model for sequence classification tasks.
|
||||
Peft model for token classification tasks.
|
||||
|
||||
Args:
|
||||
model ([`PreTrainedModel`]): Base transformer model
|
||||
model ([`~transformers.PreTrainedModel`]): Base transformer model.
|
||||
peft_config ([`PeftConfig`]): Peft config.
|
||||
|
||||
**Attributes**:
|
||||
- **config** ([`PretrainedConfig`]) -- The configuration object of the base model.
|
||||
- **config** ([`~transformers.PretrainedConfig`]) -- The configuration object of the base model.
|
||||
- **cls_layer_name** (`str`) -- The name of the classification layer.
|
||||
|
||||
Example::
|
||||
Example:
|
||||
|
||||
>>> from transformers import AutoModelForSequenceClassification >>> from peft import
|
||||
PeftModelForTokenClassification, get_peft_config >>> config = {
|
||||
'peft_type': 'PREFIX_TUNING', 'task_type': 'TOKEN_CLS', 'inference_mode': False, 'num_virtual_tokens':
|
||||
20, 'token_dim': 768, 'num_transformer_submodules': 1, 'num_attention_heads': 12, 'num_layers': 12,
|
||||
'encoder_hidden_size': 768, 'prefix_projection': False, 'postprocess_past_key_value_function': None
|
||||
}
|
||||
>>> peft_config = get_peft_config(config) >>> model =
|
||||
AutoModelForTokenClassification.from_pretrained("bert-base-cased") >>> peft_model =
|
||||
PeftModelForTokenClassification(model, peft_config) >>> peft_model.print_trainable_parameters() trainable
|
||||
params: 370178 || all params: 108680450 || trainable%: 0.3406113979101117
|
||||
```py
|
||||
>>> from transformers import AutoModelForSequenceClassification
|
||||
>>> from peft import PeftModelForTokenClassification, get_peft_config
|
||||
|
||||
>>> config = {
|
||||
... "peft_type": "PREFIX_TUNING",
|
||||
... "task_type": "TOKEN_CLS",
|
||||
... "inference_mode": False,
|
||||
... "num_virtual_tokens": 20,
|
||||
... "token_dim": 768,
|
||||
... "num_transformer_submodules": 1,
|
||||
... "num_attention_heads": 12,
|
||||
... "num_layers": 12,
|
||||
... "encoder_hidden_size": 768,
|
||||
... "prefix_projection": False,
|
||||
... "postprocess_past_key_value_function": None,
|
||||
... }
|
||||
|
||||
>>> peft_config = get_peft_config(config)
|
||||
>>> model = AutoModelForTokenClassification.from_pretrained("bert-base-cased")
|
||||
>>> peft_model = PeftModelForTokenClassification(model, peft_config)
|
||||
>>> peft_model.print_trainable_parameters()
|
||||
trainable params: 370178 || all params: 108680450 || trainable%: 0.3406113979101117
|
||||
```
|
||||
"""
|
||||
|
||||
def __init__(self, model, peft_config: PeftConfig = None, adapter_name="default"):
|
||||
|
||||
+87
-8
@@ -45,10 +45,10 @@ if is_bnb_available():
|
||||
@dataclass
|
||||
class LoraConfig(PeftConfig):
|
||||
"""
|
||||
This is the configuration class to store the configuration of a [`~peft.Lora`].
|
||||
This is the configuration class to store the configuration of a [`LoraModel`].
|
||||
|
||||
Args:
|
||||
r (`int`): Lora attention dimension
|
||||
r (`int`): Lora attention dimension.
|
||||
target_modules (`Union[List[str],str]`): The names of the modules to apply Lora to.
|
||||
lora_alpha (`float`): The alpha parameter for Lora scaling.
|
||||
lora_dropout (`float`): The dropout probability for Lora layers.
|
||||
@@ -56,7 +56,6 @@ class LoraConfig(PeftConfig):
|
||||
Whether to merge the weights of the Lora layers with the base transformer model in `eval` mode.
|
||||
fan_in_fan_out (`bool`): Set this to True if the layer to replace stores weight like (fan_in, fan_out).
|
||||
For example, gpt-2 uses `Conv1D` which stores weights like (fan_in, fan_out) and hence this should be set to `True`.:
|
||||
enable_lora ( `List[bool]`): Used with `lora.MergedLinear`. Usually set to [True, False, True].
|
||||
bias (`str`): Bias type for Lora. Can be 'none', 'all' or 'lora_only'
|
||||
modules_to_save (`List[str]`):List of modules apart from LoRA layers to be set as trainable
|
||||
and saved in the final checkpoint.
|
||||
@@ -88,12 +87,50 @@ class LoraConfig(PeftConfig):
|
||||
"the final layer `classifier/score` are randomly initialized and as such need to be trainable and saved."
|
||||
},
|
||||
)
|
||||
init_lora_weights: bool = field(
|
||||
default=True,
|
||||
metadata={"help": "Whether to initialize the weights of the Lora layers."},
|
||||
)
|
||||
|
||||
def __post_init__(self):
|
||||
self.peft_type = PeftType.LORA
|
||||
|
||||
|
||||
class LoraModel(torch.nn.Module):
|
||||
"""
|
||||
Creates Low Rank Adapter (Lora) model from a pretrained transformers model.
|
||||
|
||||
Args:
|
||||
model ([`~transformers.PreTrainedModel`]): The model to be adapted.
|
||||
config ([`LoraConfig`]): The configuration of the Lora model.
|
||||
|
||||
Returns:
|
||||
`torch.nn.Module`: The Lora model.
|
||||
|
||||
Example:
|
||||
|
||||
```py
|
||||
>>> from transformers import AutoModelForSeq2SeqLM, LoraConfig
|
||||
>>> from peft import LoraModel, LoraConfig
|
||||
|
||||
>>> config = LoraConfig(
|
||||
... peft_type="LORA",
|
||||
... task_type="SEQ_2_SEQ_LM",
|
||||
... r=8,
|
||||
... lora_alpha=32,
|
||||
... target_modules=["q", "v"],
|
||||
... lora_dropout=0.01,
|
||||
... )
|
||||
|
||||
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
|
||||
>>> lora_model = LoraModel(config, model)
|
||||
```
|
||||
|
||||
**Attributes**:
|
||||
- **model** ([`~transformers.PreTrainedModel`]) -- The model to be adapted.
|
||||
- **peft_config** ([`LoraConfig`]): The configuration of the Lora model.
|
||||
"""
|
||||
|
||||
def __init__(self, model, config, adapter_name):
|
||||
super().__init__()
|
||||
self.model = model
|
||||
@@ -129,6 +166,7 @@ class LoraModel(torch.nn.Module):
|
||||
"fan_in_fan_out": lora_config.fan_in_fan_out,
|
||||
"merge_weights": (lora_config.merge_weights or lora_config.inference_mode)
|
||||
and not is_hf_device_map_available,
|
||||
"init_lora_weights": self.peft_config.init_lora_weights,
|
||||
}
|
||||
key_list = [key for key, _ in self.model.named_modules()]
|
||||
for key in key_list:
|
||||
@@ -142,7 +180,13 @@ class LoraModel(torch.nn.Module):
|
||||
parent, target, target_name = _get_submodules(self.model, key)
|
||||
bias = target.bias is not None
|
||||
if isinstance(target, LoraLayer):
|
||||
target.update_layer(adapter_name, lora_config.r, lora_config.lora_alpha, lora_config.lora_dropout)
|
||||
target.update_layer(
|
||||
adapter_name,
|
||||
lora_config.r,
|
||||
lora_config.lora_alpha,
|
||||
lora_config.lora_dropout,
|
||||
lora_config.init_lora_weights,
|
||||
)
|
||||
else:
|
||||
if loaded_in_8bit and isinstance(target, bnb.nn.Linear8bitLt):
|
||||
kwargs.update(
|
||||
@@ -262,6 +306,37 @@ class LoraModel(torch.nn.Module):
|
||||
peft_config.merge_weights = True
|
||||
return peft_config
|
||||
|
||||
def merge_and_unload(self):
|
||||
r"""
|
||||
This method merges the LoRa layers into the base model. This is needed if someone wants to use the base model
|
||||
as a standalone model.
|
||||
"""
|
||||
if self.config.model_type == "gpt2":
|
||||
raise ValueError("GPT2 models are not supported for merging LORA layers")
|
||||
|
||||
if getattr(self.model, "is_loaded_in_8bit", False):
|
||||
raise ValueError("Cannot merge LORA layers when the model is loaded in 8-bit mode")
|
||||
|
||||
key_list = [key for key, _ in self.model.named_modules() if "lora" not in key]
|
||||
for key in key_list:
|
||||
parent, target, target_name = self._get_submodules(key)
|
||||
if isinstance(target, LoraLayer):
|
||||
bias = target.bias is not None
|
||||
new_module = torch.nn.Linear(target.in_features, target.out_features, bias=bias)
|
||||
|
||||
# manually merge if not merged
|
||||
if not target.merged:
|
||||
# merge weights per: https://arxiv.org/pdf/2106.09685.pdf / page 4
|
||||
if target.r > 0:
|
||||
target.weight.data += (
|
||||
transpose(target.lora_B.weight @ target.lora_A.weight, target.fan_in_fan_out)
|
||||
* target.scaling
|
||||
).to(target.weight.dtype)
|
||||
target.merged = True
|
||||
|
||||
self._replace_module(parent, target_name, new_module, target)
|
||||
return self.model
|
||||
|
||||
|
||||
# Below code is based on https://github.com/microsoft/LoRA/blob/main/loralib/layers.py
|
||||
# and modified to work with PyTorch FSDP
|
||||
@@ -312,7 +387,7 @@ class LoraLayer:
|
||||
self.in_features = in_features
|
||||
self.out_features = out_features
|
||||
|
||||
def update_layer(self, adapter_name, r, lora_alpha, lora_dropout):
|
||||
def update_layer(self, adapter_name, r, lora_alpha, lora_dropout, init_lora_weights):
|
||||
self.r[adapter_name] = r
|
||||
self.lora_alpha[adapter_name] = lora_alpha
|
||||
if lora_dropout > 0.0:
|
||||
@@ -328,7 +403,8 @@ class LoraLayer:
|
||||
self.lora_A.update(nn.ModuleDict({adapter_name: nn.Linear(self.in_features, r, bias=False)}))
|
||||
self.lora_B.update(nn.ModuleDict({adapter_name: nn.Linear(r, self.out_features, bias=False)}))
|
||||
self.scaling[adapter_name] = lora_alpha / r
|
||||
self.reset_lora_parameters(adapter_name)
|
||||
if init_lora_weights:
|
||||
self.reset_lora_parameters(adapter_name)
|
||||
self.to(self.weight.device)
|
||||
|
||||
def reset_lora_parameters(self, adapter_name):
|
||||
@@ -352,6 +428,8 @@ class Linear(nn.Linear):
|
||||
merge_weights: bool = True,
|
||||
**kwargs,
|
||||
):
|
||||
init_lora_weights = kwargs.pop("init_lora_weights", True)
|
||||
|
||||
nn.Linear.__init__(self, in_features, out_features, **kwargs)
|
||||
LoraLayer.__init__(self, merge_weights=merge_weights, in_features=in_features, out_features=out_features)
|
||||
# Freezing the pre-trained weight matrix
|
||||
@@ -362,7 +440,7 @@ class Linear(nn.Linear):
|
||||
self.weight.data = self.weight.data.T
|
||||
|
||||
nn.Linear.reset_parameters(self)
|
||||
self.update_layer(adapter_name, r, lora_alpha, lora_dropout)
|
||||
self.update_layer(adapter_name, r, lora_alpha, lora_dropout, init_lora_weights)
|
||||
self.active_adapter = adapter_name
|
||||
|
||||
def merge(self):
|
||||
@@ -445,7 +523,8 @@ if is_bnb_available():
|
||||
# Freezing the pre-trained weight matrix
|
||||
self.weight.requires_grad = False
|
||||
|
||||
self.update_layer(adapter_name, r, lora_alpha, lora_dropout)
|
||||
init_lora_weights = kwargs.pop("init_lora_weights", True)
|
||||
self.update_layer(adapter_name, r, lora_alpha, lora_dropout, init_lora_weights)
|
||||
self.active_adapter = adapter_name
|
||||
|
||||
def forward(self, x: torch.Tensor):
|
||||
|
||||
+28
-17
@@ -31,11 +31,11 @@ class PromptEncoderReparameterizationType(str, enum.Enum):
|
||||
@dataclass
|
||||
class PromptEncoderConfig(PromptLearningConfig):
|
||||
"""
|
||||
This is the configuration class to store the configuration of a [`~peft.PromptEncoder`].
|
||||
This is the configuration class to store the configuration of a [`PromptEncoder`].
|
||||
|
||||
Args:
|
||||
encoder_reparameterization_type
|
||||
(Union[[`PromptEncoderReparameterizationType`], `str`]): The type of reparameterization to use.
|
||||
encoder_reparameterization_type (Union[[`PromptEncoderReparameterizationType`], `str`]):
|
||||
The type of reparameterization to use.
|
||||
encoder_hidden_size (`int`): The hidden size of the prompt encoder.
|
||||
encoder_num_layers (`int`): The number of layers of the prompt encoder.
|
||||
encoder_dropout (`float`): The dropout probability of the prompt encoder.
|
||||
@@ -71,19 +71,30 @@ class PromptEncoder(torch.nn.Module):
|
||||
Args:
|
||||
config ([`PromptEncoderConfig`]): The configuration of the prompt encoder.
|
||||
|
||||
Example::
|
||||
Example:
|
||||
|
||||
>>> from peft import PromptEncoder, PromptEncoderConfig >>> config = PromptEncoderConfig(
|
||||
peft_type="P_TUNING", task_type="SEQ_2_SEQ_LM", num_virtual_tokens=20, token_dim=768,
|
||||
num_transformer_submodules=1, num_attention_heads=12, num_layers=12,
|
||||
encoder_reparameterization_type="MLP", encoder_hidden_size=768
|
||||
)
|
||||
>>> prompt_encoder = PromptEncoder(config)
|
||||
```py
|
||||
>>> from peft import PromptEncoder, PromptEncoderConfig
|
||||
|
||||
>>> config = PromptEncoderConfig(
|
||||
... peft_type="P_TUNING",
|
||||
... task_type="SEQ_2_SEQ_LM",
|
||||
... num_virtual_tokens=20,
|
||||
... token_dim=768,
|
||||
... num_transformer_submodules=1,
|
||||
... num_attention_heads=12,
|
||||
... num_layers=12,
|
||||
... encoder_reparameterization_type="MLP",
|
||||
... encoder_hidden_size=768,
|
||||
... )
|
||||
|
||||
>>> prompt_encoder = PromptEncoder(config)
|
||||
```
|
||||
|
||||
**Attributes**:
|
||||
- **embedding** ([`~torch.nn.Embedding`]) -- The embedding layer of the prompt encoder.
|
||||
- **mlp_head** ([`~torch.nn.Sequential`]) -- The MLP head of the prompt encoder if `inference_mode=False`.
|
||||
- **lstm_head** ([`~torch.nn.LSTM`]) -- The LSTM head of the prompt encoder if `inference_mode=False` and
|
||||
- **embedding** (`torch.nn.Embedding`) -- The embedding layer of the prompt encoder.
|
||||
- **mlp_head** (`torch.nn.Sequential`) -- The MLP head of the prompt encoder if `inference_mode=False`.
|
||||
- **lstm_head** (`torch.nn.LSTM`) -- The LSTM head of the prompt encoder if `inference_mode=False` and
|
||||
`encoder_reparameterization_type="LSTM"`.
|
||||
- **token_dim** (`int`) -- The hidden embedding dimension of the base transformer model.
|
||||
- **input_size** (`int`) -- The input size of the prompt encoder.
|
||||
@@ -91,13 +102,13 @@ class PromptEncoder(torch.nn.Module):
|
||||
- **hidden_size** (`int`) -- The hidden size of the prompt encoder.
|
||||
- **total_virtual_tokens** (`int`): The total number of virtual tokens of the
|
||||
prompt encoder.
|
||||
- **encoder_type** (Union[[`PromptEncoderReparameterizationType`], `str`]):
|
||||
The encoder type of the prompt encoder.
|
||||
- **encoder_type** (Union[[`PromptEncoderReparameterizationType`], `str`]): The encoder type of the prompt
|
||||
encoder.
|
||||
|
||||
|
||||
Input shape: (batch_size, total_virtual_tokens)
|
||||
Input shape: (`batch_size`, `total_virtual_tokens`)
|
||||
|
||||
Output shape: (batch_size, total_virtual_tokens, token_dim)
|
||||
Output shape: (`batch_size`, `total_virtual_tokens`, `token_dim`)
|
||||
"""
|
||||
|
||||
def __init__(self, config):
|
||||
|
||||
@@ -24,7 +24,7 @@ from ..utils import PeftType, PromptLearningConfig
|
||||
@dataclass
|
||||
class PrefixTuningConfig(PromptLearningConfig):
|
||||
"""
|
||||
This is the configuration class to store the configuration of a [`~peft.PrefixEncoder`].
|
||||
This is the configuration class to store the configuration of a [`PrefixEncoder`].
|
||||
|
||||
Args:
|
||||
encoder_hidden_size (`int`): The hidden size of the prompt encoder.
|
||||
@@ -48,30 +48,38 @@ class PrefixTuningConfig(PromptLearningConfig):
|
||||
# with some refactor
|
||||
class PrefixEncoder(torch.nn.Module):
|
||||
r"""
|
||||
The torch.nn model to encode the prefix
|
||||
The `torch.nn` model to encode the prefix.
|
||||
|
||||
Args:
|
||||
config ([`PrefixTuningConfig`]): The configuration of the prefix encoder.
|
||||
|
||||
Example::
|
||||
Example:
|
||||
|
||||
>>> from peft import PrefixEncoder, PrefixTuningConfig >>> config = PrefixTuningConfig(
|
||||
peft_type="PREFIX_TUNING", task_type="SEQ_2_SEQ_LM", num_virtual_tokens=20, token_dim=768,
|
||||
num_transformer_submodules=1, num_attention_heads=12, num_layers=12, encoder_hidden_size=768
|
||||
)
|
||||
>>> prefix_encoder = PrefixEncoder(config)
|
||||
```py
|
||||
>>> from peft import PrefixEncoder, PrefixTuningConfig
|
||||
|
||||
>>> config = PrefixTuningConfig(
|
||||
... peft_type="PREFIX_TUNING",
|
||||
... task_type="SEQ_2_SEQ_LM",
|
||||
... num_virtual_tokens=20,
|
||||
... token_dim=768,
|
||||
... num_transformer_submodules=1,
|
||||
... num_attention_heads=12,
|
||||
... num_layers=12,
|
||||
... encoder_hidden_size=768,
|
||||
... )
|
||||
>>> prefix_encoder = PrefixEncoder(config)
|
||||
```
|
||||
|
||||
**Attributes**:
|
||||
- **embedding** (`torch.nn.Embedding`) --
|
||||
The embedding layer of the prefix encoder.
|
||||
- **transform** (`torch.nn.Sequential`) -- The
|
||||
two-layer MLP to transform the prefix embeddings if `prefix_projection` is `True`.
|
||||
- **embedding** (`torch.nn.Embedding`) -- The embedding layer of the prefix encoder.
|
||||
- **transform** (`torch.nn.Sequential`) -- The two-layer MLP to transform the prefix embeddings if
|
||||
`prefix_projection` is `True`.
|
||||
- **prefix_projection** (`bool`) -- Whether to project the prefix embeddings.
|
||||
|
||||
Input shape: (batch_size, num_virtual_tokens)
|
||||
Input shape: (`batch_size`, `num_virtual_tokens`)
|
||||
|
||||
Output shape: (batch_size, num_virtual_tokens, 2*layers*hidden)
|
||||
Output shape: (`batch_size`, `num_virtual_tokens`, `2*layers*hidden`)
|
||||
"""
|
||||
|
||||
def __init__(self, config):
|
||||
|
||||
@@ -31,14 +31,14 @@ class PromptTuningInit(str, enum.Enum):
|
||||
@dataclass
|
||||
class PromptTuningConfig(PromptLearningConfig):
|
||||
"""
|
||||
This is the configuration class to store the configuration of a [`~peft.PromptEmbedding`].
|
||||
This is the configuration class to store the configuration of a [`PromptEmbedding`].
|
||||
|
||||
Args:
|
||||
prompt_tuning_init (Union[[`PromptTuningInit`], `str`]): The initialization of the prompt embedding.
|
||||
prompt_tuning_init_text ( Optional[`str`]): The text to initialize the prompt embedding.
|
||||
Only used if `prompt_tuning_init` is `TEXT`
|
||||
tokenizer_name_or_path ( Optional[`str`]): The name or path of the tokenizer.
|
||||
Only used if `prompt_tuning_init` is `TEXT`
|
||||
prompt_tuning_init_text (`str`, *optional*):
|
||||
The text to initialize the prompt embedding. Only used if `prompt_tuning_init` is `TEXT`.
|
||||
tokenizer_name_or_path (`str`, *optional*):
|
||||
The name or path of the tokenizer. Only used if `prompt_tuning_init` is `TEXT`.
|
||||
"""
|
||||
|
||||
prompt_tuning_init: Union[PromptTuningInit, str] = field(
|
||||
@@ -71,23 +71,33 @@ class PromptEmbedding(torch.nn.Module):
|
||||
word_embeddings (`torch.nn.Module`): The word embeddings of the base transformer model.
|
||||
|
||||
**Attributes**:
|
||||
**embedding** (`torch.nn.Embedding`) -- The embedding layer of the prompt embedding.
|
||||
- **embedding** (`torch.nn.Embedding`) -- The embedding layer of the prompt embedding.
|
||||
|
||||
Example::
|
||||
Example:
|
||||
|
||||
>>> from peft import PromptEmbedding, PromptTuningConfig >>> config = PromptTuningConfig(
|
||||
peft_type="PROMPT_TUNING", task_type="SEQ_2_SEQ_LM", num_virtual_tokens=20, token_dim=768,
|
||||
num_transformer_submodules=1, num_attention_heads=12, num_layers=12, prompt_tuning_init="TEXT",
|
||||
prompt_tuning_init_text="Predict if sentiment of this review is positive, negative or neutral",
|
||||
tokenizer_name_or_path="t5-base",
|
||||
)
|
||||
>>> # t5_model.shared is the word embeddings of the base model >>> prompt_embedding = PromptEmbedding(config,
|
||||
t5_model.shared)
|
||||
```py
|
||||
>>> from peft import PromptEmbedding, PromptTuningConfig
|
||||
|
||||
>>> config = PromptTuningConfig(
|
||||
... peft_type="PROMPT_TUNING",
|
||||
... task_type="SEQ_2_SEQ_LM",
|
||||
... num_virtual_tokens=20,
|
||||
... token_dim=768,
|
||||
... num_transformer_submodules=1,
|
||||
... num_attention_heads=12,
|
||||
... num_layers=12,
|
||||
... prompt_tuning_init="TEXT",
|
||||
... prompt_tuning_init_text="Predict if sentiment of this review is positive, negative or neutral",
|
||||
... tokenizer_name_or_path="t5-base",
|
||||
... )
|
||||
|
||||
Input Shape: (batch_size, total_virtual_tokens)
|
||||
>>> # t5_model.shared is the word embeddings of the base model
|
||||
>>> prompt_embedding = PromptEmbedding(config, t5_model.shared)
|
||||
```
|
||||
|
||||
Output Shape: (batch_size, total_virtual_tokens, token_dim)
|
||||
Input Shape: (`batch_size`, `total_virtual_tokens`)
|
||||
|
||||
Output Shape: (`batch_size`, `total_virtual_tokens`, `token_dim`)
|
||||
"""
|
||||
|
||||
def __init__(self, config, word_embeddings):
|
||||
|
||||
@@ -43,7 +43,7 @@ class TaskType(str, enum.Enum):
|
||||
class PeftConfigMixin(PushToHubMixin):
|
||||
r"""
|
||||
This is the base configuration class for PEFT adapter models. It contains all the methods that are common to all
|
||||
PEFT adapter models. This class inherits from `transformers.utils.PushToHubMixin` which contains the methods to
|
||||
PEFT adapter models. This class inherits from [`~transformers.utils.PushToHubMixin`] which contains the methods to
|
||||
push your model to the Hub. The method `save_pretrained` will save the configuration of your adapter model in a
|
||||
directory. The method `from_pretrained` will load the configuration of your adapter model from a directory.
|
||||
|
||||
@@ -66,8 +66,8 @@ class PeftConfigMixin(PushToHubMixin):
|
||||
Args:
|
||||
save_directory (`str`):
|
||||
The directory where the configuration will be saved.
|
||||
**kwargs:
|
||||
Additional keyword arguments passed along to the `transformers.utils.PushToHubMixin.push_to_hub`
|
||||
kwargs (additional keyword arguments, *optional*):
|
||||
Additional keyword arguments passed along to the [`~transformers.utils.PushToHubMixin.push_to_hub`]
|
||||
method.
|
||||
"""
|
||||
if os.path.isfile(save_directory):
|
||||
@@ -89,8 +89,8 @@ class PeftConfigMixin(PushToHubMixin):
|
||||
|
||||
Args:
|
||||
pretrained_model_name_or_path (`str`):
|
||||
The directory or the hub-id where the configuration is saved.
|
||||
**kwargs:
|
||||
The directory or the Hub repository id where the configuration is saved.
|
||||
kwargs (additional keyword arguments, *optional*):
|
||||
Additional keyword arguments passed along to the child class initialization.
|
||||
"""
|
||||
path = (
|
||||
@@ -104,7 +104,7 @@ class PeftConfigMixin(PushToHubMixin):
|
||||
try:
|
||||
config_file = hf_hub_download(pretrained_model_name_or_path, CONFIG_NAME, subfolder=subfolder)
|
||||
except Exception:
|
||||
raise ValueError(f"Can't find config.json at '{pretrained_model_name_or_path}'")
|
||||
raise ValueError(f"Can't find '{CONFIG_NAME}' at '{pretrained_model_name_or_path}'")
|
||||
|
||||
loaded_attributes = cls.from_json_file(config_file)
|
||||
|
||||
@@ -134,7 +134,7 @@ class PeftConfigMixin(PushToHubMixin):
|
||||
@dataclass
|
||||
class PeftConfig(PeftConfigMixin):
|
||||
"""
|
||||
This is the base configuration class to store the configuration of a :class:`~peft.PeftModel`.
|
||||
This is the base configuration class to store the configuration of a [`PeftModel`].
|
||||
|
||||
Args:
|
||||
peft_type (Union[[`~peft.utils.config.PeftType`], `str`]): The type of Peft method to use.
|
||||
@@ -151,8 +151,8 @@ class PeftConfig(PeftConfigMixin):
|
||||
@dataclass
|
||||
class PromptLearningConfig(PeftConfig):
|
||||
"""
|
||||
This is the base configuration class to store the configuration of a Union[[`~peft.PrefixTuning`],
|
||||
[`~peft.PromptEncoder`], [`~peft.PromptTuning`]].
|
||||
This is the base configuration class to store the configuration of [`PrefixTuning`], [`PromptEncoder`], or
|
||||
[`PromptTuning`].
|
||||
|
||||
Args:
|
||||
num_virtual_tokens (`int`): The number of virtual tokens to use.
|
||||
|
||||
@@ -36,7 +36,7 @@ def prepare_model_for_int8_training(
|
||||
model, output_embedding_layer_name="lm_head", use_gradient_checkpointing=True, layer_norm_names=["layer_norm"]
|
||||
):
|
||||
r"""
|
||||
This method wrapps the entire protocol for preparing a model before running a training. This includes:
|
||||
This method wraps the entire protocol for preparing a model before running a training. This includes:
|
||||
1- Cast the layernorm in fp32 2- making output embedding layer require grads 3- Add the upcasting of the lm
|
||||
head to fp32
|
||||
|
||||
|
||||
@@ -30,11 +30,19 @@ from peft import (
|
||||
from .testing_common import PeftTestConfigManager
|
||||
|
||||
|
||||
# This has to be in the order: model_id, lora_kwargs, prefix_tuning_kwargs, prompt_encoder_kwargs, prompt_tuning_kwargs
|
||||
PEFT_MODELS_TO_TEST = [
|
||||
("hf-internal-testing/tiny-random-OPTForCausalLM", {"target_modules": ["q_proj", "v_proj"]}, {}, {}, {}),
|
||||
PEFT_DECODER_MODELS_TO_TEST = [
|
||||
"hf-internal-testing/tiny-random-OPTForCausalLM",
|
||||
"hf-internal-testing/tiny-random-GPTNeoXForCausalLM",
|
||||
"hf-internal-testing/tiny-random-GPT2LMHeadModel",
|
||||
"hf-internal-testing/tiny-random-BloomForCausalLM",
|
||||
"hf-internal-testing/tiny-random-gpt_neo",
|
||||
"hf-internal-testing/tiny-random-GPTJForCausalLM",
|
||||
]
|
||||
|
||||
FULL_GRID = {
|
||||
"model_ids": PEFT_DECODER_MODELS_TO_TEST,
|
||||
}
|
||||
|
||||
|
||||
class PeftTestMixin:
|
||||
torch_device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
@@ -48,10 +56,6 @@ class PeftModelTester(unittest.TestCase, PeftTestMixin):
|
||||
We use parametrized.expand for debugging purposes to test each model individually.
|
||||
"""
|
||||
|
||||
@parameterized.expand(PeftTestConfigManager.get_grid_parameters(PEFT_MODELS_TO_TEST))
|
||||
def test_attributes_parametrized(self, test_name, model_id, config_cls, config_kwargs):
|
||||
self._test_model_attr(model_id, config_cls, config_kwargs)
|
||||
|
||||
def _test_model_attr(self, model_id, config_cls, config_kwargs):
|
||||
model = AutoModelForCausalLM.from_pretrained(model_id)
|
||||
config = config_cls(
|
||||
@@ -64,6 +68,10 @@ class PeftModelTester(unittest.TestCase, PeftTestMixin):
|
||||
self.assertTrue(hasattr(model, "from_pretrained"))
|
||||
self.assertTrue(hasattr(model, "push_to_hub"))
|
||||
|
||||
@parameterized.expand(PeftTestConfigManager.get_grid_parameters(FULL_GRID))
|
||||
def test_attributes_parametrized(self, test_name, model_id, config_cls, config_kwargs):
|
||||
self._test_model_attr(model_id, config_cls, config_kwargs)
|
||||
|
||||
def _test_prepare_for_training(self, model_id, config_cls, config_kwargs):
|
||||
model = AutoModelForCausalLM.from_pretrained(model_id).to(self.torch_device)
|
||||
config = config_cls(
|
||||
@@ -105,7 +113,7 @@ class PeftModelTester(unittest.TestCase, PeftTestMixin):
|
||||
|
||||
self.assertTrue(dummy_output.requires_grad)
|
||||
|
||||
@parameterized.expand(PeftTestConfigManager.get_grid_parameters(PEFT_MODELS_TO_TEST))
|
||||
@parameterized.expand(PeftTestConfigManager.get_grid_parameters(FULL_GRID))
|
||||
def test_prepare_for_training_parametrized(self, test_name, model_id, config_cls, config_kwargs):
|
||||
self._test_prepare_for_training(model_id, config_cls, config_kwargs)
|
||||
|
||||
@@ -151,6 +159,80 @@ class PeftModelTester(unittest.TestCase, PeftTestMixin):
|
||||
# check if `config.json` is not present
|
||||
self.assertFalse(os.path.exists(os.path.join(tmp_dirname, "config.json")))
|
||||
|
||||
@parameterized.expand(PeftTestConfigManager.get_grid_parameters(PEFT_MODELS_TO_TEST))
|
||||
@parameterized.expand(PeftTestConfigManager.get_grid_parameters(FULL_GRID))
|
||||
def test_save_pretrained(self, test_name, model_id, config_cls, config_kwargs):
|
||||
self._test_save_pretrained(model_id, config_cls, config_kwargs)
|
||||
|
||||
def _test_merge_layers(self, model_id, config_cls, config_kwargs):
|
||||
model = AutoModelForCausalLM.from_pretrained(model_id)
|
||||
config = config_cls(
|
||||
base_model_name_or_path=model_id,
|
||||
**config_kwargs,
|
||||
)
|
||||
model = get_peft_model(model, config)
|
||||
model = model.to(self.torch_device)
|
||||
|
||||
if config.peft_type != "LORA":
|
||||
with self.assertRaises(AttributeError):
|
||||
model = model.merge_and_unload()
|
||||
elif model.config.model_type == "gpt2":
|
||||
with self.assertRaises(ValueError):
|
||||
model = model.merge_and_unload()
|
||||
else:
|
||||
dummy_input = torch.LongTensor([[1, 2, 3, 2, 1]]).to(self.torch_device)
|
||||
model.eval()
|
||||
logits_lora = model(dummy_input)[0]
|
||||
|
||||
model = model.merge_and_unload()
|
||||
|
||||
logits_merged = model(dummy_input)[0]
|
||||
|
||||
transformers_model = AutoModelForCausalLM.from_pretrained(model_id).to(self.torch_device)
|
||||
|
||||
logits_transformers = transformers_model(dummy_input)[0]
|
||||
|
||||
self.assertTrue(torch.allclose(logits_lora, logits_merged, atol=1e-3, rtol=1e-3))
|
||||
self.assertFalse(torch.allclose(logits_merged, logits_transformers, atol=1e-3, rtol=1e-3))
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp_dirname:
|
||||
model.save_pretrained(tmp_dirname)
|
||||
|
||||
model_from_pretrained = AutoModelForCausalLM.from_pretrained(tmp_dirname).to(self.torch_device)
|
||||
|
||||
logits_merged_from_pretrained = model_from_pretrained(dummy_input)[0]
|
||||
|
||||
self.assertTrue(torch.allclose(logits_merged, logits_merged_from_pretrained, atol=1e-3, rtol=1e-3))
|
||||
|
||||
@parameterized.expand(
|
||||
PeftTestConfigManager.get_grid_parameters(
|
||||
{
|
||||
"model_ids": PEFT_DECODER_MODELS_TO_TEST,
|
||||
"lora_kwargs": {"init_lora_weights": [False], "merge_weights": [False, True]},
|
||||
},
|
||||
)
|
||||
)
|
||||
def test_merge_layers(self, test_name, model_id, config_cls, config_kwargs):
|
||||
self._test_merge_layers(model_id, config_cls, config_kwargs)
|
||||
|
||||
def _test_generate(self, model_id, config_cls, config_kwargs):
|
||||
model = AutoModelForCausalLM.from_pretrained(model_id)
|
||||
config = config_cls(
|
||||
base_model_name_or_path=model_id,
|
||||
**config_kwargs,
|
||||
)
|
||||
model = get_peft_model(model, config)
|
||||
model = model.to(self.torch_device)
|
||||
|
||||
input_ids = torch.LongTensor([[1, 1, 1], [2, 1, 2]]).to(self.torch_device)
|
||||
attention_mask = torch.LongTensor([[1, 1, 1], [1, 0, 1]]).to(self.torch_device)
|
||||
|
||||
# check if `generate` works
|
||||
_ = model.generate(input_ids=input_ids, attention_mask=attention_mask)
|
||||
|
||||
with self.assertRaises(TypeError):
|
||||
# check if `generate` raises an error if no positional arguments are passed
|
||||
_ = model.generate(input_ids, attention_mask=attention_mask)
|
||||
|
||||
@parameterized.expand(PeftTestConfigManager.get_grid_parameters(FULL_GRID))
|
||||
def test_generate(self, test_name, model_id, config_cls, config_kwargs):
|
||||
self._test_generate(model_id, config_cls, config_kwargs)
|
||||
|
||||
+38
-24
@@ -71,33 +71,47 @@ class ClassInstantier(OrderedDict):
|
||||
|
||||
return super().__getitem__(key, *args, **kwargs)
|
||||
|
||||
def get_grid_parameters(self, model_list):
|
||||
def get_grid_parameters(self, grid_parameters, filter_params_func=None):
|
||||
r"""
|
||||
Returns a list of all possible combinations of the parameters in the config classes.
|
||||
"""
|
||||
grid_parameters = []
|
||||
for model_tuple in model_list:
|
||||
model_id, lora_kwargs, prefix_tuning_kwargs, prompt_encoder_kwargs, prompt_tuning_kwargs = model_tuple
|
||||
for key, value in self.items():
|
||||
if key == "lora":
|
||||
# update value[1] if necessary
|
||||
if lora_kwargs is not None:
|
||||
value[1].update(lora_kwargs)
|
||||
elif key == "prefix_tuning":
|
||||
# update value[1] if necessary
|
||||
if prefix_tuning_kwargs is not None:
|
||||
value[1].update(prefix_tuning_kwargs)
|
||||
elif key == "prompt_encoder":
|
||||
# update value[1] if necessary
|
||||
if prompt_encoder_kwargs is not None:
|
||||
value[1].update(prompt_encoder_kwargs)
|
||||
else:
|
||||
# update value[1] if necessary
|
||||
if prompt_tuning_kwargs is not None:
|
||||
value[1].update(prompt_tuning_kwargs)
|
||||
grid_parameters.append((f"test_{model_id}_{key}", model_id, value[0], value[1]))
|
||||
|
||||
return grid_parameters
|
||||
Args:
|
||||
grid_parameters (`dict`):
|
||||
A dictionary containing the parameters to be tested. There should be at least the key "model_ids" which
|
||||
contains a list of model ids to be tested. The other keys should be the name of the config class
|
||||
post-fixed with "_kwargs" and the value should be a dictionary containing the parameters to be tested
|
||||
for that config class.
|
||||
filter_params_func (`callable`, `optional`):
|
||||
A function that takes a list of tuples and returns a list of tuples. This function is used to filter
|
||||
out the tests that needs for example to be skipped.
|
||||
|
||||
Returns:
|
||||
generated_tests (`list`):
|
||||
A list of tuples containing the name of the test, the model id, the config class and the config class
|
||||
kwargs.
|
||||
"""
|
||||
generated_tests = []
|
||||
model_list = grid_parameters["model_ids"]
|
||||
|
||||
for model_id in model_list:
|
||||
for key, value in self.items():
|
||||
if "{}_kwargs".format(key) in grid_parameters:
|
||||
peft_configs = []
|
||||
current_peft_config = value[1].copy()
|
||||
for current_key, current_value in grid_parameters[f"{key}_kwargs"].items():
|
||||
for kwarg in current_value:
|
||||
current_peft_config.update({current_key: kwarg})
|
||||
peft_configs.append(current_peft_config)
|
||||
else:
|
||||
peft_configs = [value[1].copy()]
|
||||
|
||||
for peft_config in peft_configs:
|
||||
generated_tests.append((f"test_{model_id}_{key}", model_id, value[0], peft_config))
|
||||
|
||||
if filter_params_func is not None:
|
||||
generated_tests = filter_params_func(generated_tests)
|
||||
|
||||
return generated_tests
|
||||
|
||||
|
||||
PeftTestConfigManager = ClassInstantier(CLASSES_MAPPING)
|
||||
|
||||
Reference in New Issue
Block a user