mirror of
https://github.com/wassname/peft.git
synced 2026-09-09 11:28:32 +08:00
Merge pull request #26 from huggingface/smangrul/fixes
addressing comments and renaming `pet` to `peft`
This commit is contained in:
Sourab Mangrulkar
@@ -1,9 +1,9 @@
|
||||
<h1 align="center"> <p>🤗 PET</p></h1>
|
||||
<h1 align="center"> <p>🤗 PEFT</p></h1>
|
||||
<h3 align="center">
|
||||
<p>State-of-the-art Parameter-Efficient Tuning (PET) methods</p>
|
||||
<p>State-of-the-art Parameter-Efficient Fine-Tuning (PEFT) methods</p>
|
||||
</h3>
|
||||
|
||||
Parameter-Efficient Tuning (PET) methods enable efficient adaptation of pre-trained language models (PLMs) to various downstream applications without fine-tuning all the model's parameters. Fine-tuning large-scale PLMs is often prohibitively costly. In this regard, PET methods only fine-tune a small number of (extra) model parameters, thereby greatly decreasing the computational and storage costs. Recent State-of-the-Art PET techniques achieve performance comparable to that of full fine-tuning.
|
||||
Parameter-Efficient Fine-Tuning (PEFT) methods enable efficient adaptation of pre-trained language models (PLMs) to various downstream applications without fine-tuning all the model's parameters. Fine-tuning large-scale PLMs is often prohibitively costly. In this regard, PEFT methods only fine-tune a small number of (extra) model parameters, thereby greatly decreasing the computational and storage costs. Recent State-of-the-Art PEFT techniques achieve performance comparable to that of full fine-tuning.
|
||||
|
||||
Seamlessly integrated with 🤗 Accelerate for large scale models leveraging PyTorch FSDP.
|
||||
|
||||
@@ -18,16 +18,16 @@ Supported methods:
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForSeq2SeqLM
|
||||
from pet import get_pet_config, get_pet_model, LoRAConfig, TaskType
|
||||
from peft import get_peft_config, get_peft_model, LoRAConfig, TaskType
|
||||
model_name_or_path = "bigscience/mt0-large"
|
||||
tokenizer_name_or_path = "bigscience/mt0-large"
|
||||
|
||||
pet_config = LoRAConfig(
|
||||
peft_config = LoRAConfig(
|
||||
task_type=TaskType.SEQ_2_SEQ_LM, inference_mode=False, r=8, lora_alpha=32, lora_dropout=0.1
|
||||
)
|
||||
|
||||
model = AutoModelForSeq2SeqLM.from_pretrained(model_name_or_path)
|
||||
model = get_pet_model(model, pet_config)
|
||||
model = get_peft_model(model, peft_config)
|
||||
model.print_trainable_parameters()
|
||||
# output: trainable params: 2359296 || all params: 1231940608 || trainable%: 0.19151053100118282
|
||||
```
|
||||
@@ -37,17 +37,17 @@ model.print_trainable_parameters()
|
||||
### Get comparable performance to full finetuning by adapting LLMs to downstream tasks using consumer hardware
|
||||
|
||||
GPU memory required for adapting LLMs on the few-shot dataset `ought/raft/twitter_complaints`. Here, settings considered
|
||||
are full finetuning, PET-LoRA using plain PyTorch and PET-LoRA using DeepSpeed with CPU Offloading.
|
||||
are full finetuning, PEFT-LoRA using plain PyTorch and PEFT-LoRA using DeepSpeed with CPU Offloading.
|
||||
|
||||
Hardware: Single A100 80GB GPU with CPU RAM above 64GB
|
||||
|
||||
| Model | Full Finetuning | PET-LoRA PyTorch | PET-LoRA DeepSpeed with CPU Offloading |
|
||||
| Model | Full Finetuning | PEFT-LoRA PyTorch | PEFT-LoRA DeepSpeed with CPU Offloading |
|
||||
| --------- | ---- | ---- | ---- |
|
||||
| bigscience/T0_3B (3B params) | 47.14GB GPU / 2.96GB CPU | 14.4GB GPU / 2.96GB CPU | 9.8GB GPU / 17.8GB CPU |
|
||||
| bigscience/mt0-xxl (12B params) | OOM GPU | 56GB GPU / 3GB CPU | 22GB GPU / 52GB CPU |
|
||||
| bigscience/bloomz-7b1 (7B params) | OOM GPU | 32GB GPU / 3.8GB CPU | 18.1GB GPU / 35GB CPU |
|
||||
|
||||
Performance of PET-LoRA tuned `bigscience/T0_3B` on `ought/raft/twitter_complaints` leaderboard.
|
||||
Performance of PEFT-LoRA tuned `bigscience/T0_3B` on `ought/raft/twitter_complaints` leaderboard.
|
||||
A point to note is that we didn't try to sequeeze performance by playing around with input instruction templates, LoRA hyperparams and other training related hyperparams. Also, we didn't use the larger 13B mt0-xxl model.
|
||||
So, we are already seeing comparable performance to SoTA with parameter effcient tuning. Also, the final checkpoint size is just `19MB` in comparison to `11GB` size of the backbone `bigscience/T0_3B` model.
|
||||
|
||||
@@ -57,7 +57,7 @@ So, we are already seeing comparable performance to SoTA with parameter effcient
|
||||
| Flan-T5 | 0.892 |
|
||||
| lora-t0-3b | 0.863 |
|
||||
|
||||
**Therefore, we can see that performance comparable to SoTA is achievable by PET methods with consumer hardware such as 16GB and 24GB GPUs.**
|
||||
**Therefore, we can see that performance comparable to SoTA is achievable by PEFT methods with consumer hardware such as 16GB and 24GB GPUs.**
|
||||
|
||||
### Parameter Efficient Tuning of Diffusion Models
|
||||
|
||||
@@ -65,9 +65,9 @@ GPU memory required by different settings during training are given below. The f
|
||||
|
||||
Hardware: Single A100 80GB GPU with CPU RAM above 64G
|
||||
|
||||
| Model | Full Finetuning | PET-LoRA |
|
||||
| Model | Full Finetuning | PEFT-LoRA | PEFT-LoRA with Gradient Checkpoitning |
|
||||
| --------- | ---- | ---- |
|
||||
| CompVis/stable-diffusion-v1-4 | 27.5GB GPU / 3.97GB CPU | 15.5GB GPU / 3.84GB CPU |
|
||||
| CompVis/stable-diffusion-v1-4 | 27.5GB GPU / 3.97GB CPU | 15.5GB GPU / 3.84GB CPU | 8.12GB GPU / 3.77GB CPU |
|
||||
|
||||
|
||||
**Training**
|
||||
@@ -100,6 +100,7 @@ accelerate launch train_dreambooth.py \
|
||||
--lora_text_encoder_alpha 17 \
|
||||
--learning_rate=1e-4 \
|
||||
--gradient_accumulation_steps=1 \
|
||||
--gradient_checkpointing \
|
||||
--max_train_steps=800
|
||||
```
|
||||
|
||||
@@ -108,22 +109,22 @@ accelerate launch train_dreambooth.py \
|
||||
### Save compute and storage even for medium and small models
|
||||
|
||||
Save storage by avoiding full finetuning of models on each of the downstream tasks/datasets,
|
||||
With PET methods, users only need to store tiny checkpoints in the order of `MBs` all the while retaining
|
||||
With PEFT methods, users only need to store tiny checkpoints in the order of `MBs` all the while retaining
|
||||
performance comparable to full finetuning.
|
||||
|
||||
An example of using LoRA for the task of adaping `LayoutLMForTokenClassification` on `FUNSD` dataset is given in `~examples/token_classification/PET_LoRA_LayoutLMForTokenClassification_on_FUNSD.py`. We can observe that with only `0.62 %` of parameters being trainable, we achieve performance (F1 0.777) comparable to full finetuning (F1 0.786) (without any hyerparam tuning runs for extracting more performance), and the checkpoint of this is only `2.8MB`. Now, if there are `N` such datasets, just have these PET models one for each dataset and save a lot of storage without having to worry about the problem of catastrophic forgetting or overfitting of backbone/base model.
|
||||
An example of using LoRA for the task of adaping `LayoutLMForTokenClassification` on `FUNSD` dataset is given in `~examples/token_classification/PEFT_LoRA_LayoutLMForTokenClassification_on_FUNSD.py`. We can observe that with only `0.62 %` of parameters being trainable, we achieve performance (F1 0.777) comparable to full finetuning (F1 0.786) (without any hyerparam tuning runs for extracting more performance), and the checkpoint of this is only `2.8MB`. Now, if there are `N` such datasets, just have these PEFT models one for each dataset and save a lot of storage without having to worry about the problem of catastrophic forgetting or overfitting of backbone/base model.
|
||||
|
||||
Another example is fine-tuning `roberta-large` on `MRPC` GLUE dataset suing differenct PET methods. The notebooks are given in `~examples/sequence_classification`.
|
||||
Another example is fine-tuning `roberta-large` on `MRPC` GLUE dataset suing differenct PEFT methods. The notebooks are given in `~examples/sequence_classification`.
|
||||
|
||||
|
||||
## PET + 🤗 Accelerate
|
||||
## PEFT + 🤗 Accelerate
|
||||
|
||||
PET models work with 🤗 Accelerate out of the box. Use 🤗 Accelerate for Distributed training on various hardware such as GPUs, Apple Silicon devices etc during training.
|
||||
PEFT models work with 🤗 Accelerate out of the box. Use 🤗 Accelerate for Distributed training on various hardware such as GPUs, Apple Silicon devices etc during training.
|
||||
Use 🤗 Accelerate for inferencing on consumer hardware with small resources.
|
||||
|
||||
### Example of PET model training using 🤗 Accelerate's DeepSpeed integation
|
||||
### Example of PEFT model training using 🤗 Accelerate's DeepSpeed integation
|
||||
|
||||
Currently DeepSpeed requires PR [ZeRO3 handling frozen weights](https://github.com/microsoft/DeepSpeed/pull/2653) to fix [[REQUEST] efficiently deal with frozen weights during training](https://github.com/microsoft/DeepSpeed/issues/2615) issue. Example is provided in `~examples/conditional_generation/pet_lora_seq2seq_accelerate_ds_zero3_offload.py`.
|
||||
Currently DeepSpeed requires PR [ZeRO3 handling frozen weights](https://github.com/microsoft/DeepSpeed/pull/2653) to fix [[REQUEST] efficiently deal with frozen weights during training](https://github.com/microsoft/DeepSpeed/issues/2615) issue. Example is provided in `~examples/conditional_generation/peft_lora_seq2seq_accelerate_ds_zero3_offload.py`.
|
||||
a. First run `accelerate config --config_file ds_zero3_cpu.yaml` and answer the questionaire.
|
||||
Below are the contents of the config file.
|
||||
```
|
||||
@@ -152,7 +153,7 @@ Use 🤗 Accelerate for inferencing on consumer hardware with small resources.
|
||||
```
|
||||
b. run the below command to launch example script
|
||||
```
|
||||
accelerate launch --config_file ds_zero3_cpu.yaml examples/pet_lora_seq2seq_accelerate_ds_zero3_offload.py
|
||||
accelerate launch --config_file ds_zero3_cpu.yaml examples/peft_lora_seq2seq_accelerate_ds_zero3_offload.py
|
||||
```
|
||||
|
||||
c. output logs:
|
||||
@@ -180,9 +181,9 @@ Use 🤗 Accelerate for inferencing on consumer hardware with small resources.
|
||||
dataset['train'][label_column][:10]=['no complaint', 'no complaint', 'complaint', 'complaint', 'no complaint', 'no complaint', 'no complaint', 'complaint', 'complaint', 'no complaint']
|
||||
```
|
||||
|
||||
### Example of PET model inference using 🤗 Accelerate's Big Model Inferencing capabilities
|
||||
### Example of PEFT model inference using 🤗 Accelerate's Big Model Inferencing capabilities
|
||||
|
||||
Example is provided in `~examples/causal_language_modeling/pet_lora_clm_accelerate_big_model_inference.ipynb`.
|
||||
Example is provided in `~examples/causal_language_modeling/peft_lora_clm_accelerate_big_model_inference.ipynb`.
|
||||
|
||||
|
||||
## Models support matrix
|
||||
@@ -235,7 +236,7 @@ Example is provided in `~examples/causal_language_modeling/pet_lora_clm_accelera
|
||||
any GPU memory savings. Please refer issue [[FSDP] FSDP with CPU offload consumes 1.65X more GPU memory when training models with most of the params frozen](https://github.com/pytorch/pytorch/issues/91165).
|
||||
|
||||
```python
|
||||
from pet.utils.other import fsdp_auto_wrap_policy
|
||||
from peft.utils.other import fsdp_auto_wrap_policy
|
||||
|
||||
...
|
||||
|
||||
@@ -245,7 +246,7 @@ any GPU memory savings. Please refer issue [[FSDP] FSDP with CPU offload consume
|
||||
model = accelerator.prepare(model)
|
||||
```
|
||||
|
||||
Example of parameter efficient tuning with `mt0-xxl` base model using 🤗 Accelerate is provided in `~examples/conditional_generation/pet_lora_seq2seq_accelerate_fsdp.py`.
|
||||
Example of parameter efficient tuning with `mt0-xxl` base model using 🤗 Accelerate is provided in `~examples/conditional_generation/peft_lora_seq2seq_accelerate_fsdp.py`.
|
||||
a. First run `accelerate config --config_file fsdp_config.yaml` and answer the questionaire.
|
||||
Below are the contents of the config file.
|
||||
```
|
||||
@@ -280,7 +281,7 @@ any GPU memory savings. Please refer issue [[FSDP] FSDP with CPU offload consume
|
||||
```
|
||||
b. run the below command to launch example script
|
||||
```
|
||||
accelerate launch --config_file fsdp_config.yaml examples/pet_lora_seq2seq_accelerate_fsdp.py
|
||||
accelerate launch --config_file fsdp_config.yaml examples/peft_lora_seq2seq_accelerate_fsdp.py
|
||||
```
|
||||
|
||||
2. When using `P_TUNING` or `PROMPT_TUNING` with `SEQ_2_SEQ` task, remember to remove the `num_virtual_token` virtual prompt predictions from the left side of the model outputs during evaluations.
|
||||
@@ -294,15 +295,15 @@ new `input_embeds` to be given to the model. Therefore, `generate` doesn't suppo
|
||||
2. Add tests
|
||||
3. Add more use cases and examples
|
||||
|
||||
## Citing 🤗 PET
|
||||
## Citing 🤗 PEFT
|
||||
|
||||
If you use 🤗 PET in your publication, please cite it by using the following BibTeX entry.
|
||||
If you use 🤗 PEFT in your publication, please cite it by using the following BibTeX entry.
|
||||
|
||||
```bibtex
|
||||
@Misc{pet,
|
||||
title = {PET: State-of-the-art Parameter-Efficient Tuning (PET) methods},
|
||||
author = {Sourab Mangrulkar},
|
||||
howpublished = {\url{https://github.com/huggingface/pet}},
|
||||
@Misc{peft,
|
||||
title = {PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods},
|
||||
author = {Sourab Mangrulkar, Sylvain Gugger},
|
||||
howpublished = {\url{https://github.com/huggingface/peft}},
|
||||
year = {2022}
|
||||
}
|
||||
```
|
||||
|
||||
+11
-6
@@ -8,7 +8,7 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from transformers import AutoModelForCausalLM\n",
|
||||
"from pet import get_pet_config,get_pet_model, get_pet_model_state_dict, set_pet_model_state_dict, LoRAConfig, TaskType, pet_model_load_and_dispatch\n",
|
||||
"from peft import get_peft_config,get_peft_model, get_peft_model_state_dict, set_peft_model_state_dict, LoraConfig, TaskType, peft_model_load_and_dispatch\n",
|
||||
"import torch\n",
|
||||
"from datasets import load_dataset\n",
|
||||
"import os\n",
|
||||
@@ -21,10 +21,10 @@
|
||||
"device = \"cuda\"\n",
|
||||
"model_name_or_path = \"bigscience/bloomz-7b1\"\n",
|
||||
"tokenizer_name_or_path = \"bigscience/bloomz-7b1\"\n",
|
||||
"pet_config = LoRAConfig(task_type=TaskType.CAUSAL_LM, inference_mode=False, r=8, lora_alpha=32, lora_dropout=0.1)\n",
|
||||
"peft_config = LoraConfig(task_type=TaskType.CAUSAL_LM, inference_mode=False, r=8, lora_alpha=32, lora_dropout=0.1)\n",
|
||||
"\n",
|
||||
"dataset_name = \"twitter_complaints\"\n",
|
||||
"checkpoint_name = \"/home/sourab/\"+f\"{dataset_name}_{model_name_or_path}_{pet_config.pet_type}_{pet_config.task_type}_v1.pt\".replace(\"/\", \"_\")\n",
|
||||
"checkpoint_name = \"/home/sourab/\"+f\"{dataset_name}_{model_name_or_path}_{peft_config.peft_type}_{peft_config.task_type}_v1.pt\".replace(\"/\", \"_\")\n",
|
||||
"text_column = \"Tweet text\"\n",
|
||||
"label_column = \"text_label\"\n",
|
||||
"max_length=64\n",
|
||||
@@ -1273,7 +1273,7 @@
|
||||
"max_memory={0: \"1GIB\", 1: \"1GIB\", 2: \"2GIB\", 3: \"10GIB\", \"cpu\":\"30GB\"}\n",
|
||||
"\n",
|
||||
"model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map=\"auto\", max_memory=max_memory)\n",
|
||||
"pet_model_load_and_dispatch(model, torch.load(checkpoint_name), pet_config, max_memory)\n",
|
||||
"peft_model_load_and_dispatch(model, torch.load(checkpoint_name), peft_config, max_memory)\n",
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
@@ -2178,7 +2178,7 @@
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
@@ -2192,7 +2192,12 @@
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.4"
|
||||
"version": "3.10.5 (v3.10.5:f377153967, Jun 6 2022, 12:36:10) [Clang 13.0.0 (clang-1300.0.29.30)]"
|
||||
},
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
|
||||
}
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
+5
-5
@@ -17,7 +17,7 @@ from transformers import (
|
||||
|
||||
import psutil
|
||||
from datasets import load_dataset
|
||||
from pet import LoRAConfig, TaskType, get_pet_model, get_pet_model_state_dict
|
||||
from peft import LoraConfig, TaskType, get_peft_model, get_peft_model_state_dict
|
||||
from tqdm import tqdm
|
||||
|
||||
|
||||
@@ -110,9 +110,9 @@ def main():
|
||||
accelerator = Accelerator()
|
||||
model_name_or_path = "bigscience/bloomz-7b1"
|
||||
dataset_name = "twitter_complaints"
|
||||
pet_config = LoRAConfig(task_type=TaskType.CAUSAL_LM, inference_mode=False, r=8, lora_alpha=32, lora_dropout=0.1)
|
||||
peft_config = LoraConfig(task_type=TaskType.CAUSAL_LM, inference_mode=False, r=8, lora_alpha=32, lora_dropout=0.1)
|
||||
checkpoint_name = (
|
||||
f"{dataset_name}_{model_name_or_path}_{pet_config.pet_type}_{pet_config.task_type}_v1.pt".replace("/", "_")
|
||||
f"{dataset_name}_{model_name_or_path}_{peft_config.peft_type}_{peft_config.task_type}_v1.pt".replace("/", "_")
|
||||
)
|
||||
text_column = "Tweet text"
|
||||
label_column = "text_label"
|
||||
@@ -217,7 +217,7 @@ def main():
|
||||
|
||||
# creating model
|
||||
model = AutoModelForCausalLM.from_pretrained(model_name_or_path)
|
||||
model = get_pet_model(model, pet_config)
|
||||
model = get_peft_model(model, peft_config)
|
||||
model.print_trainable_parameters()
|
||||
|
||||
# optimizer
|
||||
@@ -343,7 +343,7 @@ def main():
|
||||
pred_df.to_csv(f"data/{dataset_name}/predictions.csv", index=False)
|
||||
|
||||
accelerator.wait_for_everyone()
|
||||
accelerator.save(get_pet_model_state_dict(model, state_dict=accelerator.get_state_dict(model)), checkpoint_name)
|
||||
accelerator.save(get_peft_model_state_dict(model, state_dict=accelerator.get_state_dict(model)), checkpoint_name)
|
||||
accelerator.wait_for_everyone()
|
||||
|
||||
|
||||
+9
-9
@@ -8,7 +8,7 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from transformers import AutoModelForCausalLM\n",
|
||||
"from pet import get_pet_config,get_pet_model, get_pet_model_state_dict, set_pet_model_state_dict, PrefixTuningConfig, TaskType, pet_model_load_and_dispatch, bloom_model_postprocess_past_key_value, PETType\n",
|
||||
"from peft import get_peft_config,get_peft_model, get_peft_model_state_dict, set_peft_model_state_dict, PrefixTuningConfig, TaskType, peft_model_load_and_dispatch, bloom_model_postprocess_past_key_value, PeftType\n",
|
||||
"import torch\n",
|
||||
"from datasets import load_dataset\n",
|
||||
"import os\n",
|
||||
@@ -21,12 +21,12 @@
|
||||
"device = \"cuda\"\n",
|
||||
"model_name_or_path = \"bigscience/bloomz-560m\"\n",
|
||||
"tokenizer_name_or_path = \"bigscience/bloomz-560m\"\n",
|
||||
"pet_config = PrefixTuningConfig(task_type=TaskType.CAUSAL_LM, \n",
|
||||
"peft_config = PrefixTuningConfig(task_type=TaskType.CAUSAL_LM, \n",
|
||||
" num_virtual_tokens=30, \n",
|
||||
" postprocess_past_key_value_function=bloom_model_postprocess_past_key_value)\n",
|
||||
"\n",
|
||||
"dataset_name = \"twitter_complaints\"\n",
|
||||
"checkpoint_name = f\"{dataset_name}_{model_name_or_path}_{pet_config.pet_type}_{pet_config.task_type}_v1.pt\".replace(\"/\", \"_\")\n",
|
||||
"checkpoint_name = f\"{dataset_name}_{model_name_or_path}_{peft_config.peft_type}_{peft_config.task_type}_v1.pt\".replace(\"/\", \"_\")\n",
|
||||
"text_column = \"Tweet text\"\n",
|
||||
"label_column = \"text_label\"\n",
|
||||
"max_length=64\n",
|
||||
@@ -684,7 +684,7 @@
|
||||
"\n",
|
||||
"# creating model\n",
|
||||
"model = AutoModelForCausalLM.from_pretrained(model_name_or_path)\n",
|
||||
"model = get_pet_model(model, pet_config)\n",
|
||||
"model = get_peft_model(model, peft_config)\n",
|
||||
"model.print_trainable_parameters()\n",
|
||||
"\n"
|
||||
]
|
||||
@@ -1097,7 +1097,7 @@
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.pet_config"
|
||||
"model.peft_config"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1999,7 +1999,7 @@
|
||||
],
|
||||
"source": [
|
||||
"# saving model\n",
|
||||
"state_dict = get_pet_model_state_dict(model)\n",
|
||||
"state_dict = get_peft_model_state_dict(model)\n",
|
||||
"torch.save(state_dict, checkpoint_name)\n",
|
||||
"print(state_dict)"
|
||||
]
|
||||
@@ -2044,10 +2044,10 @@
|
||||
"source": [
|
||||
"max_memory={0: \"1GIB\", 1: \"1GIB\", 2: \"2GIB\", 3: \"2GIB\", \"cpu\":\"30GB\"}\n",
|
||||
"\n",
|
||||
"pet_config.inference_mode = True\n",
|
||||
"print(pet_config)\n",
|
||||
"peft_config.inference_mode = True\n",
|
||||
"print(peft_config)\n",
|
||||
"model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map=\"auto\", max_memory=max_memory)\n",
|
||||
"model = pet_model_load_and_dispatch(model, torch.load(checkpoint_name), pet_config, max_memory)\n",
|
||||
"model = peft_model_load_and_dispatch(model, torch.load(checkpoint_name), peft_config, max_memory)\n",
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
+4
-4
@@ -8,7 +8,7 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from transformers import AutoModelForSeq2SeqLM\n",
|
||||
"from pet import get_pet_config,get_pet_model, get_pet_model_state_dict, LoRAConfig, TaskType\n",
|
||||
"from peft import get_peft_config,get_peft_model, get_peft_model_state_dict, LoraConfig, TaskType\n",
|
||||
"import torch\n",
|
||||
"from datasets import load_dataset\n",
|
||||
"import os\n",
|
||||
@@ -40,12 +40,12 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# creating model\n",
|
||||
"pet_config = LoRAConfig(\n",
|
||||
"peft_config = LoraConfig(\n",
|
||||
" task_type=TaskType.SEQ_2_SEQ_LM, inference_mode=False, r=8, lora_alpha=32, lora_dropout=0.1\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"model = AutoModelForSeq2SeqLM.from_pretrained(model_name_or_path)\n",
|
||||
"model = get_pet_model(model, pet_config)\n",
|
||||
"model = get_peft_model(model, peft_config)\n",
|
||||
"model.print_trainable_parameters()\n",
|
||||
"model"
|
||||
]
|
||||
@@ -349,7 +349,7 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# saving model\n",
|
||||
"state_dict = get_pet_model_state_dict(model)\n",
|
||||
"state_dict = get_peft_model_state_dict(model)\n",
|
||||
"torch.save(state_dict, checkpoint_name)\n",
|
||||
"print(state_dict)"
|
||||
]
|
||||
+5
-5
@@ -11,7 +11,7 @@ from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, get_linear_schedu
|
||||
|
||||
import psutil
|
||||
from datasets import load_dataset
|
||||
from pet import LoRAConfig, TaskType, get_pet_model, get_pet_model_state_dict
|
||||
from peft import LoraConfig, TaskType, get_peft_model, get_peft_model_state_dict
|
||||
from tqdm import tqdm
|
||||
|
||||
|
||||
@@ -104,11 +104,11 @@ def main():
|
||||
accelerator = Accelerator()
|
||||
model_name_or_path = "bigscience/T0_3B"
|
||||
dataset_name = "twitter_complaints"
|
||||
pet_config = LoRAConfig(
|
||||
peft_config = LoraConfig(
|
||||
task_type=TaskType.SEQ_2_SEQ_LM, inference_mode=False, r=8, lora_alpha=32, lora_dropout=0.1
|
||||
)
|
||||
checkpoint_name = (
|
||||
f"{dataset_name}_{model_name_or_path}_{pet_config.pet_type}_{pet_config.task_type}_v1.pt".replace("/", "_")
|
||||
f"{dataset_name}_{model_name_or_path}_{peft_config.peft_type}_{peft_config.task_type}_v1.pt".replace("/", "_")
|
||||
)
|
||||
text_column = "Tweet text"
|
||||
label_column = "text_label"
|
||||
@@ -167,7 +167,7 @@ def main():
|
||||
|
||||
# creating model
|
||||
model = AutoModelForSeq2SeqLM.from_pretrained(model_name_or_path)
|
||||
model = get_pet_model(model, pet_config)
|
||||
model = get_peft_model(model, peft_config)
|
||||
model.print_trainable_parameters()
|
||||
|
||||
# optimizer
|
||||
@@ -291,7 +291,7 @@ def main():
|
||||
pred_df.to_csv(f"data/{dataset_name}/predictions.csv", index=False)
|
||||
|
||||
accelerator.wait_for_everyone()
|
||||
accelerator.save(get_pet_model_state_dict(model, state_dict=accelerator.get_state_dict(model)), checkpoint_name)
|
||||
accelerator.save(get_peft_model_state_dict(model, state_dict=accelerator.get_state_dict(model)), checkpoint_name)
|
||||
accelerator.wait_for_everyone()
|
||||
|
||||
|
||||
+5
-5
@@ -6,8 +6,8 @@ from torch.utils.data import DataLoader
|
||||
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, default_data_collator, get_linear_schedule_with_warmup
|
||||
|
||||
from datasets import load_dataset
|
||||
from pet import LoRAConfig, TaskType, get_pet_model, get_pet_model_state_dict
|
||||
from pet.utils.other import fsdp_auto_wrap_policy
|
||||
from peft import LoraConfig, TaskType, get_peft_model, get_peft_model_state_dict
|
||||
from peft.utils.other import fsdp_auto_wrap_policy
|
||||
from tqdm import tqdm
|
||||
|
||||
|
||||
@@ -22,12 +22,12 @@ def main():
|
||||
num_epochs = 1
|
||||
base_path = "temp/data/FinancialPhraseBank-v1.0"
|
||||
|
||||
pet_config = LoRAConfig(
|
||||
peft_config = LoraConfig(
|
||||
task_type=TaskType.SEQ_2_SEQ_LM, inference_mode=False, r=8, lora_alpha=32, lora_dropout=0.1
|
||||
)
|
||||
checkpoint_name = "financial_sentiment_analysis_lora_fsdp_v1.pt"
|
||||
model = AutoModelForSeq2SeqLM.from_pretrained(model_name_or_path)
|
||||
model = get_pet_model(model, pet_config)
|
||||
model = get_peft_model(model, peft_config)
|
||||
accelerator.print(model.print_trainable_parameters())
|
||||
|
||||
dataset = load_dataset(
|
||||
@@ -127,7 +127,7 @@ def main():
|
||||
accelerator.print(f"{dataset['validation'][label_column][:10]=}")
|
||||
accelerator.wait_for_everyone()
|
||||
accelerator.save(
|
||||
get_pet_model_state_dict(model, state_dict=accelerator.get_state_dict(model)), checkpoint_name
|
||||
get_peft_model_state_dict(model, state_dict=accelerator.get_state_dict(model)), checkpoint_name
|
||||
)
|
||||
accelerator.wait_for_everyone()
|
||||
|
||||
+4
-4
@@ -8,7 +8,7 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from transformers import AutoModelForSeq2SeqLM\n",
|
||||
"from pet import get_pet_config,get_pet_model, get_pet_model_state_dict, PrefixTuningConfig, TaskType\n",
|
||||
"from peft import get_peft_config,get_peft_model, get_peft_model_state_dict, PrefixTuningConfig, TaskType\n",
|
||||
"import torch\n",
|
||||
"from datasets import load_dataset\n",
|
||||
"import os\n",
|
||||
@@ -41,12 +41,12 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# creating model\n",
|
||||
"pet_config = PrefixTuningConfig(\n",
|
||||
"peft_config = PrefixTuningConfig(\n",
|
||||
" task_type=TaskType.SEQ_2_SEQ_LM, inference_mode=False, num_virtual_tokens=20\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"model = AutoModelForSeq2SeqLM.from_pretrained(model_name_or_path)\n",
|
||||
"model = get_pet_model(model, pet_config)\n",
|
||||
"model = get_peft_model(model, peft_config)\n",
|
||||
"model.print_trainable_parameters()\n",
|
||||
"model"
|
||||
]
|
||||
@@ -441,7 +441,7 @@
|
||||
],
|
||||
"source": [
|
||||
"# saving model\n",
|
||||
"state_dict = get_pet_model_state_dict(model)\n",
|
||||
"state_dict = get_peft_model_state_dict(model)\n",
|
||||
"torch.save(state_dict, checkpoint_name)\n",
|
||||
"print(state_dict)"
|
||||
]
|
||||
+21
-16
File diff suppressed because one or more lines are too long
@@ -29,7 +29,7 @@ from diffusers.optimization import get_scheduler
|
||||
from diffusers.utils import check_min_version
|
||||
from diffusers.utils.import_utils import is_xformers_available
|
||||
from huggingface_hub import HfFolder, Repository, whoami
|
||||
from pet import LoRAConfig, LoRAModel, get_pet_model_state_dict
|
||||
from peft import LoraConfig, LoraModel, get_peft_model_state_dict
|
||||
from PIL import Image
|
||||
from torchvision import transforms
|
||||
from tqdm.auto import tqdm
|
||||
@@ -151,39 +151,39 @@ def parse_args(input_args=None):
|
||||
parser.add_argument("--train_text_encoder", action="store_true", help="Whether to train the text encoder")
|
||||
|
||||
# lora args
|
||||
parser.add_argument("--use_lora", action="store_true", help="Whether to use LoRA for parameter efficient tuning")
|
||||
parser.add_argument("--lora_r", type=int, default=8, help="LoRA rank, only used if use_lora is True")
|
||||
parser.add_argument("--lora_alpha", type=int, default=32, help="LoRA alpha, only used if use_lora is True")
|
||||
parser.add_argument("--lora_dropout", type=float, default=0.0, help="LoRA dropout, only used if use_lora is True")
|
||||
parser.add_argument("--use_lora", action="store_true", help="Whether to use Lora for parameter efficient tuning")
|
||||
parser.add_argument("--lora_r", type=int, default=8, help="Lora rank, only used if use_lora is True")
|
||||
parser.add_argument("--lora_alpha", type=int, default=32, help="Lora alpha, only used if use_lora is True")
|
||||
parser.add_argument("--lora_dropout", type=float, default=0.0, help="Lora dropout, only used if use_lora is True")
|
||||
parser.add_argument(
|
||||
"--lora_bias",
|
||||
type=str,
|
||||
default="none",
|
||||
help="Bias type for LoRA. Can be 'none', 'all' or 'lora_only', only used if use_lora is True",
|
||||
help="Bias type for Lora. Can be 'none', 'all' or 'lora_only', only used if use_lora is True",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--lora_text_encoder_r",
|
||||
type=int,
|
||||
default=8,
|
||||
help="LoRA rank for text encoder, only used if `use_lora` and `train_text_encoder` are True",
|
||||
help="Lora rank for text encoder, only used if `use_lora` and `train_text_encoder` are True",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--lora_text_encoder_alpha",
|
||||
type=int,
|
||||
default=32,
|
||||
help="LoRA alpha for text encoder, only used if `use_lora` and `train_text_encoder` are True",
|
||||
help="Lora alpha for text encoder, only used if `use_lora` and `train_text_encoder` are True",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--lora_text_encoder_dropout",
|
||||
type=float,
|
||||
default=0.0,
|
||||
help="LoRA dropout for text encoder, only used if `use_lora` and `train_text_encoder` are True",
|
||||
help="Lora dropout for text encoder, only used if `use_lora` and `train_text_encoder` are True",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--lora_text_encoder_bias",
|
||||
type=str,
|
||||
default="none",
|
||||
help="Bias type for LoRA. Can be 'none', 'all' or 'lora_only', only used if use_lora and `train_text_encoder` are True",
|
||||
help="Bias type for Lora. Can be 'none', 'all' or 'lora_only', only used if use_lora and `train_text_encoder` are True",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
@@ -682,14 +682,14 @@ def main(args):
|
||||
)
|
||||
|
||||
if args.use_lora:
|
||||
config = LoRAConfig(
|
||||
config = LoraConfig(
|
||||
r=args.lora_r,
|
||||
lora_alpha=args.lora_alpha,
|
||||
target_modules=UNET_TARGET_MODULES,
|
||||
lora_dropout=args.lora_dropout,
|
||||
bias=args.lora_bias,
|
||||
)
|
||||
unet = LoRAModel(config, unet)
|
||||
unet = LoraModel(config, unet)
|
||||
print_trainable_parameters(unet)
|
||||
print(unet)
|
||||
|
||||
@@ -697,14 +697,14 @@ def main(args):
|
||||
if not args.train_text_encoder:
|
||||
text_encoder.requires_grad_(False)
|
||||
elif args.train_text_encoder and args.use_lora:
|
||||
config = LoRAConfig(
|
||||
config = LoraConfig(
|
||||
r=args.lora_text_encoder_r,
|
||||
lora_alpha=args.lora_text_encoder_alpha,
|
||||
target_modules=TEXT_ENCODER_TARGET_MODULES,
|
||||
lora_dropout=args.lora_text_encoder_dropout,
|
||||
bias=args.lora_text_encoder_bias,
|
||||
)
|
||||
text_encoder = LoRAModel(config, text_encoder)
|
||||
text_encoder = LoraModel(config, text_encoder)
|
||||
print_trainable_parameters(text_encoder)
|
||||
print(text_encoder)
|
||||
|
||||
@@ -717,8 +717,8 @@ def main(args):
|
||||
if args.gradient_checkpointing:
|
||||
unet.enable_gradient_checkpointing()
|
||||
# below fails when using lora so commenting it out
|
||||
# if args.train_text_encoder:
|
||||
# text_encoder.gradient_checkpointing_enable()
|
||||
if args.train_text_encoder and not args.use_lora:
|
||||
text_encoder.gradient_checkpointing_enable()
|
||||
|
||||
# Enable TF32 for faster training on Ampere GPUs,
|
||||
# cf https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices
|
||||
@@ -974,15 +974,15 @@ def main(args):
|
||||
if accelerator.is_main_process:
|
||||
if args.use_lora:
|
||||
lora_config = {}
|
||||
state_dict = get_pet_model_state_dict(unet, state_dict=accelerator.get_state_dict(unet))
|
||||
lora_config["pet_config"] = unet.get_pet_config_as_dict(inference=True)
|
||||
state_dict = get_peft_model_state_dict(unet, state_dict=accelerator.get_state_dict(unet))
|
||||
lora_config["peft_config"] = unet.get_peft_config_as_dict(inference=True)
|
||||
if args.train_text_encoder:
|
||||
text_encoder_state_dict = get_pet_model_state_dict(
|
||||
text_encoder_state_dict = get_peft_model_state_dict(
|
||||
text_encoder, state_dict=accelerator.get_state_dict(text_encoder)
|
||||
)
|
||||
text_encoder_state_dict = {f"text_encoder_{k}": v for k, v in text_encoder_state_dict.items()}
|
||||
state_dict.update(text_encoder_state_dict)
|
||||
lora_config["text_encoder_pet_config"] = text_encoder.get_pet_config_as_dict(inference=True)
|
||||
lora_config["text_encoder_peft_config"] = text_encoder.get_peft_config_as_dict(inference=True)
|
||||
|
||||
accelerator.print(state_dict)
|
||||
accelerator.save(state_dict, os.path.join(args.output_dir, f"{args.instance_prompt}_lora.pt"))
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
"import torch\n",
|
||||
"from torch.optim import AdamW\n",
|
||||
"from torch.utils.data import DataLoader\n",
|
||||
"from pet import get_pet_config,get_pet_model, get_pet_model_state_dict, set_pet_model_state_dict, LoRAConfig, PETType, \\\n",
|
||||
"from peft import get_peft_config,get_peft_model, get_peft_model_state_dict, set_peft_model_state_dict, LoraConfig, PeftType, \\\n",
|
||||
"PrefixTuningConfig, PromptEncoderConfig\n",
|
||||
"\n",
|
||||
"import evaluate\n",
|
||||
@@ -32,7 +32,7 @@
|
||||
"batch_size = 32\n",
|
||||
"model_name_or_path = \"roberta-large\"\n",
|
||||
"task = \"mrpc\"\n",
|
||||
"pet_type = PETType.LORA\n",
|
||||
"peft_type = PeftType.LORA\n",
|
||||
"device = \"cuda\"\n",
|
||||
"num_epochs = 20"
|
||||
]
|
||||
@@ -44,7 +44,7 @@
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pet_config = LoRAConfig(\n",
|
||||
"peft_config = LoraConfig(\n",
|
||||
" task_type=\"SEQ_CLS\",\n",
|
||||
" inference_mode=False,\n",
|
||||
" r=8,\n",
|
||||
@@ -1007,7 +1007,7 @@
|
||||
],
|
||||
"source": [
|
||||
"model = AutoModelForSequenceClassification.from_pretrained(model_name_or_path, return_dict=True)\n",
|
||||
"model = get_pet_model(model, pet_config)\n",
|
||||
"model = get_peft_model(model, peft_config)\n",
|
||||
"model.print_trainable_parameters()\n",
|
||||
"model"
|
||||
]
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
"import torch\n",
|
||||
"from torch.optim import AdamW\n",
|
||||
"from torch.utils.data import DataLoader\n",
|
||||
"from pet import get_pet_config,get_pet_model, get_pet_model_state_dict, set_pet_model_state_dict, LoRAConfig, PETType, \\\n",
|
||||
"from peft import get_peft_config,get_peft_model, get_peft_model_state_dict, set_peft_model_state_dict, PeftType, \\\n",
|
||||
"PrefixTuningConfig, PromptEncoderConfig\n",
|
||||
"\n",
|
||||
"import evaluate\n",
|
||||
@@ -32,7 +32,7 @@
|
||||
"batch_size = 32\n",
|
||||
"model_name_or_path = \"roberta-large\"\n",
|
||||
"task = \"mrpc\"\n",
|
||||
"pet_type = PETType.P_TUNING\n",
|
||||
"peft_type = PeftType.P_TUNING\n",
|
||||
"device = \"cuda\"\n",
|
||||
"num_epochs = 30"
|
||||
]
|
||||
@@ -45,7 +45,7 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"\n",
|
||||
"pet_config = PromptEncoderConfig(\n",
|
||||
"peft_config = PromptEncoderConfig(\n",
|
||||
" task_type=\"SEQ_CLS\",\n",
|
||||
" num_virtual_tokens=20,\n",
|
||||
" encoder_hidden_size=128\n",
|
||||
@@ -775,7 +775,7 @@
|
||||
],
|
||||
"source": [
|
||||
"model = AutoModelForSequenceClassification.from_pretrained(model_name_or_path, return_dict=True)\n",
|
||||
"model = get_pet_model(model, pet_config)\n",
|
||||
"model = get_peft_model(model, peft_config)\n",
|
||||
"model.print_trainable_parameters()\n",
|
||||
"model"
|
||||
]
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
"import torch\n",
|
||||
"from torch.optim import AdamW\n",
|
||||
"from torch.utils.data import DataLoader\n",
|
||||
"from pet import get_pet_config,get_pet_model, get_pet_model_state_dict, set_pet_model_state_dict, LoRAConfig, PETType, \\\n",
|
||||
"from peft import get_peft_config,get_peft_model, get_peft_model_state_dict, set_peft_model_state_dict, LoRAConfig, PeftType, \\\n",
|
||||
"PrefixTuningConfig, PromptEncoderConfig, PromptTuningConfig\n",
|
||||
"\n",
|
||||
"import evaluate\n",
|
||||
@@ -32,7 +32,7 @@
|
||||
"batch_size = 32\n",
|
||||
"model_name_or_path = \"roberta-large\"\n",
|
||||
"task = \"mrpc\"\n",
|
||||
"pet_type = PETType.PROMPT_TUNING\n",
|
||||
"peft_type = PeftType.PROMPT_TUNING\n",
|
||||
"device = \"cuda\"\n",
|
||||
"num_epochs = 20"
|
||||
]
|
||||
@@ -44,7 +44,7 @@
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pet_config = PromptTuningConfig(\n",
|
||||
"peft_config = PromptTuningConfig(\n",
|
||||
" task_type=\"SEQ_CLS\",\n",
|
||||
" num_virtual_tokens=10\n",
|
||||
")\n",
|
||||
@@ -766,7 +766,7 @@
|
||||
],
|
||||
"source": [
|
||||
"model = AutoModelForSequenceClassification.from_pretrained(model_name_or_path, return_dict=True)\n",
|
||||
"model = get_pet_model(model, pet_config)\n",
|
||||
"model = get_peft_model(model, peft_config)\n",
|
||||
"model.print_trainable_parameters()\n",
|
||||
"model"
|
||||
]
|
||||
|
||||
+4
-4
@@ -13,7 +13,7 @@
|
||||
"import torch\n",
|
||||
"from torch.optim import AdamW\n",
|
||||
"from torch.utils.data import DataLoader\n",
|
||||
"from pet import get_pet_config,get_pet_model, get_pet_model_state_dict, set_pet_model_state_dict, LoRAConfig, PETType, \\\n",
|
||||
"from peft import get_peft_config,get_peft_model, get_peft_model_state_dict, set_peft_model_state_dict, PeftType, \\\n",
|
||||
"PrefixTuningConfig, PromptEncoderConfig\n",
|
||||
"\n",
|
||||
"import evaluate\n",
|
||||
@@ -32,7 +32,7 @@
|
||||
"batch_size = 32\n",
|
||||
"model_name_or_path = \"roberta-large\"\n",
|
||||
"task = \"mrpc\"\n",
|
||||
"pet_type = PETType.PREFIX_TUNING\n",
|
||||
"peft_type = PeftType.PREFIX_TUNING\n",
|
||||
"device = \"cuda\"\n",
|
||||
"num_epochs = 20"
|
||||
]
|
||||
@@ -44,7 +44,7 @@
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pet_config = PrefixTuningConfig(\n",
|
||||
"peft_config = PrefixTuningConfig(\n",
|
||||
" task_type=\"SEQ_CLS\",\n",
|
||||
" num_virtual_tokens=20\n",
|
||||
")\n",
|
||||
@@ -766,7 +766,7 @@
|
||||
],
|
||||
"source": [
|
||||
"model = AutoModelForSequenceClassification.from_pretrained(model_name_or_path, return_dict=True)\n",
|
||||
"model = get_pet_model(model, pet_config)\n",
|
||||
"model = get_peft_model(model, peft_config)\n",
|
||||
"model.print_trainable_parameters()\n",
|
||||
"model"
|
||||
]
|
||||
+6
-6
@@ -893,8 +893,8 @@
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from pet import get_pet_config, LoRAModel, get_pet_model, LoRAConfig, TaskType\n",
|
||||
"pet_config = LoRAConfig(\n",
|
||||
"from peft import get_peft_config, LoraModel, get_peft_model, LoraConfig, TaskType\n",
|
||||
"peft_config = LoraConfig(\n",
|
||||
" task_type=TaskType.TOKEN_CLS,\n",
|
||||
" inference_mode=False,\n",
|
||||
" r=16,\n",
|
||||
@@ -902,7 +902,7 @@
|
||||
" lora_dropout=0.1,\n",
|
||||
" bias=\"all\"\n",
|
||||
" )\n",
|
||||
"pet_config"
|
||||
"peft_config"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1395,7 +1395,7 @@
|
||||
"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
|
||||
"\n",
|
||||
"model = LayoutLMForTokenClassification.from_pretrained(\"microsoft/layoutlm-base-uncased\", num_labels=num_labels)\n",
|
||||
"model = get_pet_model(model, pet_config)\n",
|
||||
"model = get_peft_model(model, peft_config)\n",
|
||||
"model.to(device)"
|
||||
]
|
||||
},
|
||||
@@ -3140,8 +3140,8 @@
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from pet import get_pet_model_state_dict\n",
|
||||
"to_return = get_pet_model_state_dict(model)\n"
|
||||
"from peft import get_peft_model_state_dict\n",
|
||||
"to_return = get_peft_model_state_dict(model)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -20,9 +20,9 @@ extras["quality"] = ["black ~= 22.0", "isort >= 5.5.4", "flake8 >= 3.8.3"]
|
||||
extras["dev"] = extras["quality"]
|
||||
|
||||
setup(
|
||||
name="pets",
|
||||
name="peft",
|
||||
version="0.1.0.dev0",
|
||||
description="Parameter-Efficient Tuning (PET)",
|
||||
description="Parameter-Efficient Fine-Tuning (PEFT)",
|
||||
long_description=open("README.md", "r", encoding="utf-8").read(),
|
||||
long_description_content_type="text/markdown",
|
||||
keywords="deep learning",
|
||||
|
||||
@@ -4,17 +4,17 @@
|
||||
|
||||
__version__ = "0.1.0.dev0"
|
||||
|
||||
from .mapping import MODEL_TYPE_TO_PET_MODEL_MAPPING, PET_TYPE_TO_CONFIG_MAPPING, get_pet_config, get_pet_model
|
||||
from .pet_model import (
|
||||
PETModel,
|
||||
PETModelForCausalLM,
|
||||
PETModelForSeq2SeqLM,
|
||||
PETModelForSequenceClassification,
|
||||
PETModelForTokenClassification,
|
||||
from .mapping import MODEL_TYPE_TO_PEFT_MODEL_MAPPING, PEFT_TYPE_TO_CONFIG_MAPPING, get_peft_config, get_peft_model
|
||||
from .peft_model import (
|
||||
PeftModel,
|
||||
PeftModelForCausalLM,
|
||||
PeftModelForSeq2SeqLM,
|
||||
PeftModelForSequenceClassification,
|
||||
PeftModelForTokenClassification,
|
||||
)
|
||||
from .tuners import (
|
||||
LoRAConfig,
|
||||
LoRAModel,
|
||||
LoraConfig,
|
||||
LoraModel,
|
||||
PrefixEncoder,
|
||||
PrefixTuningConfig,
|
||||
PromptEmbedding,
|
||||
@@ -25,13 +25,13 @@ from .tuners import (
|
||||
PromptTuningInit,
|
||||
)
|
||||
from .utils import (
|
||||
PETConfig,
|
||||
PETType,
|
||||
PeftConfig,
|
||||
PeftType,
|
||||
PromptLearningConfig,
|
||||
TaskType,
|
||||
bloom_model_postprocess_past_key_value,
|
||||
get_pet_model_state_dict,
|
||||
pet_model_load_and_dispatch,
|
||||
set_pet_model_state_dict,
|
||||
get_peft_model_state_dict,
|
||||
peft_model_load_and_dispatch,
|
||||
set_peft_model_state_dict,
|
||||
shift_tokens_right,
|
||||
)
|
||||
@@ -0,0 +1,126 @@
|
||||
from .peft_model import (
|
||||
PeftModelForCausalLM,
|
||||
PeftModelForSeq2SeqLM,
|
||||
PeftModelForSequenceClassification,
|
||||
PeftModelForTokenClassification,
|
||||
)
|
||||
from .tuners import LoraConfig, PrefixTuningConfig, PromptEncoderConfig, PromptTuningConfig
|
||||
from .utils import PeftType
|
||||
|
||||
|
||||
MODEL_TYPE_TO_PEFT_MODEL_MAPPING = {
|
||||
"SEQ_CLS": PeftModelForSequenceClassification,
|
||||
"SEQ_2_SEQ_LM": PeftModelForSeq2SeqLM,
|
||||
"CAUSAL_LM": PeftModelForCausalLM,
|
||||
"TOKEN_CLS": PeftModelForTokenClassification,
|
||||
}
|
||||
|
||||
PEFT_TYPE_TO_CONFIG_MAPPING = {
|
||||
"PROMPT_TUNING": PromptTuningConfig,
|
||||
"PREFIX_TUNING": PrefixTuningConfig,
|
||||
"P_TUNING": PromptEncoderConfig,
|
||||
"LORA": LoraConfig,
|
||||
}
|
||||
|
||||
TRANSFORMERS_MODELS_TO_LORA_TARGET_MODULES_MAPPING = {
|
||||
"t5": ["q", "v"],
|
||||
"mt5": ["q", "v"],
|
||||
"bart": ["q_proj", "v_proj"],
|
||||
"gpt2": ["c_attn"],
|
||||
"bloom": ["query_key_value"],
|
||||
"opt": ["q_proj", "v_proj"],
|
||||
"gptj": ["q_proj", "v_proj"],
|
||||
"gpt_neox": ["query_key_value"],
|
||||
"gpt_neo": ["q_proj", "v_proj"],
|
||||
"bert": ["query", "value"],
|
||||
"roberta": ["query", "value"],
|
||||
"xlm-roberta": ["query", "value"],
|
||||
"electra": ["query", "value"],
|
||||
"deberta-v2": ["query_proj", "value_proj"],
|
||||
"deberta": ["in_proj"],
|
||||
"layoutlm": ["query", "value"],
|
||||
}
|
||||
|
||||
|
||||
def get_peft_config(config_dict):
|
||||
"""
|
||||
Returns a Peft config object from a dictionary.
|
||||
|
||||
Args:
|
||||
config_dict (`Dict[str, Any]`):
|
||||
"""
|
||||
|
||||
return PEFT_TYPE_TO_CONFIG_MAPPING[config_dict["peft_type"]](**config_dict)
|
||||
|
||||
|
||||
def _prepare_prompt_learning_config(peft_config, model_config):
|
||||
if peft_config.num_layers is None:
|
||||
if "num_hidden_layers" in model_config:
|
||||
num_layers = model_config["num_hidden_layers"]
|
||||
elif "num_layers" in model_config:
|
||||
num_layers = model_config["num_layers"]
|
||||
elif "n_layer" in model_config:
|
||||
num_layers = model_config["n_layer"]
|
||||
else:
|
||||
raise ValueError("Please specify `num_layers` in `peft_config`")
|
||||
peft_config.num_layers = num_layers
|
||||
|
||||
if peft_config.token_dim is None:
|
||||
if "hidden_size" in model_config:
|
||||
token_dim = model_config["hidden_size"]
|
||||
elif "n_embd" in model_config:
|
||||
token_dim = model_config["n_embd"]
|
||||
elif "d_model" in model_config:
|
||||
token_dim = model_config["d_model"]
|
||||
else:
|
||||
raise ValueError("Please specify `token_dim` in `peft_config`")
|
||||
peft_config.token_dim = token_dim
|
||||
|
||||
if peft_config.num_attention_heads is None:
|
||||
if "num_attention_heads" in model_config:
|
||||
num_attention_heads = model_config["num_attention_heads"]
|
||||
elif "n_head" in model_config:
|
||||
num_attention_heads = model_config["n_head"]
|
||||
elif "num_heads" in model_config:
|
||||
num_attention_heads = model_config["num_heads"]
|
||||
elif "encoder_attention_heads" in model_config:
|
||||
num_attention_heads = model_config["encoder_attention_heads"]
|
||||
else:
|
||||
raise ValueError("Please specify `num_attention_heads` in `peft_config`")
|
||||
peft_config.num_attention_heads = num_attention_heads
|
||||
|
||||
if getattr(peft_config, "encoder_hidden_size", None) is None:
|
||||
setattr(peft_config, "encoder_hidden_size", token_dim)
|
||||
|
||||
return peft_config
|
||||
|
||||
|
||||
def _prepare_lora_config(peft_config, model_config):
|
||||
if peft_config.target_modules is None:
|
||||
if model_config["model_type"] not in TRANSFORMERS_MODELS_TO_LORA_TARGET_MODULES_MAPPING:
|
||||
raise ValueError("Please specify `target_modules` in `peft_config`")
|
||||
peft_config.target_modules = TRANSFORMERS_MODELS_TO_LORA_TARGET_MODULES_MAPPING[model_config["model_type"]]
|
||||
if len(peft_config.target_modules) == 1:
|
||||
peft_config.fan_in_fan_out = True
|
||||
peft_config.enable_lora = [True, False, True]
|
||||
if peft_config.inference_mode:
|
||||
peft_config.merge_weights = True
|
||||
return peft_config
|
||||
|
||||
|
||||
def get_peft_model(model, peft_config):
|
||||
"""
|
||||
Returns a Peft model object from a model and a config.
|
||||
|
||||
Args:
|
||||
model (`transformers.PreTrainedModel`):
|
||||
peft_config (`transformers.PeftConfig`):
|
||||
"""
|
||||
|
||||
model_config = model.config.to_dict()
|
||||
if peft_config.peft_type != PeftType.LORA:
|
||||
peft_config = _prepare_prompt_learning_config(peft_config, model_config)
|
||||
else:
|
||||
peft_config = _prepare_lora_config(peft_config, model_config)
|
||||
|
||||
return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](model, peft_config)
|
||||
@@ -6,41 +6,41 @@ from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss
|
||||
from transformers import PreTrainedModel
|
||||
from transformers.modeling_outputs import SequenceClassifierOutput, TokenClassifierOutput
|
||||
|
||||
from .tuners import LoRAModel, PrefixEncoder, PromptEmbedding, PromptEncoder
|
||||
from .utils import PETConfig, PETType, TaskType, _set_trainable, shift_tokens_right
|
||||
from .tuners import LoraModel, PrefixEncoder, PromptEmbedding, PromptEncoder
|
||||
from .utils import PeftConfig, PeftType, TaskType, _set_trainable, shift_tokens_right
|
||||
|
||||
|
||||
class PETModel(torch.nn.Module):
|
||||
class PeftModel(torch.nn.Module):
|
||||
"""
|
||||
Parameter Efficient Tuning Model. Base model encompassing various PET methods.
|
||||
Parameter-Efficient Fine-Tuning Model. Base model encompassing various Peft methods.
|
||||
|
||||
Args:
|
||||
model (:obj:`PreTrainedModel`): The base transformer model used for PET.
|
||||
pet_config (:obj:`PETConfig`): The configuration of the PET model.
|
||||
model (`PreTrainedModel`): The base transformer model used for Peft.
|
||||
peft_config (`PeftConfig`): The configuration of the Peft model.
|
||||
|
||||
|
||||
Attributes:
|
||||
base_model (:obj:`PreTrainedModel`): The base transformer model used for PET. pet_config (:obj:`PETConfig`):
|
||||
The configuration of the PET model. modules_to_save (:obj:`list` of :obj:`str`): The list of sub-module names
|
||||
to save when saving the model. prompt_encoder (:obj:`PromptEncoder`): The prompt encoder used for PET if
|
||||
`pet_config.pet_type != PETType.LORA`. prompt_tokens (:obj:`torch.Tensor`): The virtual prompt tokens used for
|
||||
PET if `pet_config.pet_type != PETType.LORA`. transformer_backbone_name (:obj:`str`): The name of the
|
||||
base_model (`PreTrainedModel`): The base transformer model used for Peft. peft_config (`PeftConfig`):
|
||||
The configuration of the Peft model. modules_to_save (`list` of `str`): The list of sub-module names
|
||||
to save when saving the model. prompt_encoder (`PromptEncoder`): The prompt encoder used for Peft if
|
||||
`peft_config.peft_type != PeftType.LORA`. prompt_tokens (`torch.Tensor`): The virtual prompt tokens used for
|
||||
Peft if `peft_config.peft_type != PeftType.LORA`. transformer_backbone_name (`str`): The name of the
|
||||
transformer backbone in the base model
|
||||
if `pet_config.pet_type != PETType.LORA`.
|
||||
word_embeddings (:obj:`torch.nn.Embedding`): The word embeddings of the transformer backbone
|
||||
in the base model if `pet_config.pet_type != PETType.LORA`.
|
||||
if `peft_config.peft_type != PeftType.LORA`.
|
||||
word_embeddings (`torch.nn.Embedding`): The word embeddings of the transformer backbone
|
||||
in the base model if `peft_config.peft_type != PeftType.LORA`.
|
||||
"""
|
||||
|
||||
def __init__(self, model, pet_config: PETConfig):
|
||||
def __init__(self, model, peft_config: PeftConfig):
|
||||
super().__init__()
|
||||
self.pet_config = pet_config
|
||||
self.peft_config = peft_config
|
||||
self.base_model = model
|
||||
self.config = self.base_model.config
|
||||
self.modules_to_save = None
|
||||
if pet_config.pet_type != PETType.LORA:
|
||||
if peft_config.peft_type != PeftType.LORA:
|
||||
self._setup_prompt_encoder()
|
||||
else:
|
||||
self.base_model = LoRAModel(pet_config, model)
|
||||
self.base_model = LoraModel(peft_config, model)
|
||||
self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
||||
|
||||
def _setup_prompt_encoder(self):
|
||||
@@ -55,66 +55,66 @@ class PETModel(torch.nn.Module):
|
||||
transformer_backbone = module
|
||||
self.transformer_backbone_name = name
|
||||
num_transformer_submodules += 1
|
||||
self.pet_config.num_transformer_submodules = 2 if self.pet_config.task_type == TaskType.SEQ_2_SEQ_LM else 1
|
||||
self.peft_config.num_transformer_submodules = 2 if self.peft_config.task_type == TaskType.SEQ_2_SEQ_LM else 1
|
||||
|
||||
for named_param, value in list(transformer_backbone.named_parameters()):
|
||||
if value.shape[0] == self.base_model.config.vocab_size:
|
||||
self.word_embeddings = transformer_backbone.get_submodule(named_param.replace(".weight", ""))
|
||||
break
|
||||
|
||||
if self.pet_config.pet_type == PETType.PROMPT_TUNING:
|
||||
prompt_encoder = PromptEmbedding(self.pet_config, self.word_embeddings)
|
||||
elif self.pet_config.pet_type == PETType.P_TUNING:
|
||||
prompt_encoder = PromptEncoder(self.pet_config)
|
||||
elif self.pet_config.pet_type == PETType.PREFIX_TUNING:
|
||||
prompt_encoder = PrefixEncoder(self.pet_config)
|
||||
if self.peft_config.peft_type == PeftType.PROMPT_TUNING:
|
||||
prompt_encoder = PromptEmbedding(self.peft_config, self.word_embeddings)
|
||||
elif self.peft_config.peft_type == PeftType.P_TUNING:
|
||||
prompt_encoder = PromptEncoder(self.peft_config)
|
||||
elif self.peft_config.peft_type == PeftType.PREFIX_TUNING:
|
||||
prompt_encoder = PrefixEncoder(self.peft_config)
|
||||
else:
|
||||
raise ValueError("Not supported")
|
||||
self.prompt_encoder = prompt_encoder
|
||||
self.prompt_tokens = torch.arange(
|
||||
self.pet_config.num_virtual_tokens * self.pet_config.num_transformer_submodules
|
||||
self.peft_config.num_virtual_tokens * self.peft_config.num_transformer_submodules
|
||||
).long()
|
||||
|
||||
def get_prompt_embedding_to_save(self):
|
||||
"""
|
||||
Returns the prompt embedding to save when saving the model. Only applocable when `pet_config.pet_type !=
|
||||
PETType.LORA`.
|
||||
Returns the prompt embedding to save when saving the model. Only applocable when `peft_config.peft_type !=
|
||||
PeftType.LORA`.
|
||||
"""
|
||||
prompt_tokens = self.prompt_tokens.unsqueeze(0).expand(1, -1).to(self.device)
|
||||
if self.pet_config.pet_type == PETType.PREFIX_TUNING:
|
||||
prompt_tokens = prompt_tokens[:, : self.pet_config.num_virtual_tokens]
|
||||
if self.peft_config.peft_type == PeftType.PREFIX_TUNING:
|
||||
prompt_tokens = prompt_tokens[:, : self.peft_config.num_virtual_tokens]
|
||||
prompt_embeddings = self.prompt_encoder(prompt_tokens)
|
||||
return prompt_embeddings[0].detach().cpu()
|
||||
|
||||
def get_prompt(self, batch_size):
|
||||
"""
|
||||
Returns the virtual prompts to use for PET. Only applocable when `pet_config.pet_type != PETType.LORA`.
|
||||
Returns the virtual prompts to use for Peft. Only applocable when `peft_config.peft_type != PeftType.LORA`.
|
||||
"""
|
||||
prompt_tokens = self.prompt_tokens.unsqueeze(0).expand(batch_size, -1).to(self.device)
|
||||
if self.pet_config.pet_type == PETType.PREFIX_TUNING:
|
||||
prompt_tokens = prompt_tokens[:, : self.pet_config.num_virtual_tokens]
|
||||
if self.pet_config.inference_mode:
|
||||
if self.peft_config.peft_type == PeftType.PREFIX_TUNING:
|
||||
prompt_tokens = prompt_tokens[:, : self.peft_config.num_virtual_tokens]
|
||||
if self.peft_config.inference_mode:
|
||||
past_key_values = self.prompt_encoder.embedding.weight.repeat(batch_size, 1, 1)
|
||||
else:
|
||||
past_key_values = self.prompt_encoder(prompt_tokens)
|
||||
past_key_values = past_key_values.view(
|
||||
batch_size,
|
||||
self.pet_config.num_virtual_tokens,
|
||||
self.pet_config.num_layers * 2,
|
||||
self.pet_config.num_attention_heads,
|
||||
self.pet_config.token_dim // self.pet_config.num_attention_heads,
|
||||
self.peft_config.num_virtual_tokens,
|
||||
self.peft_config.num_layers * 2,
|
||||
self.peft_config.num_attention_heads,
|
||||
self.peft_config.token_dim // self.peft_config.num_attention_heads,
|
||||
)
|
||||
if self.pet_config.num_transformer_submodules == 2:
|
||||
if self.peft_config.num_transformer_submodules == 2:
|
||||
past_key_values = torch.cat([past_key_values, past_key_values], dim=2)
|
||||
past_key_values = past_key_values.permute([2, 0, 3, 1, 4]).split(
|
||||
self.pet_config.num_transformer_submodules * 2
|
||||
self.peft_config.num_transformer_submodules * 2
|
||||
)
|
||||
if self.pet_config.postprocess_past_key_value_function is not None:
|
||||
post_process_fn = self.pet_config.postprocess_past_key_value_function
|
||||
if self.peft_config.postprocess_past_key_value_function is not None:
|
||||
post_process_fn = self.peft_config.postprocess_past_key_value_function
|
||||
past_key_values = post_process_fn(past_key_values)
|
||||
return past_key_values
|
||||
else:
|
||||
if self.pet_config.inference_mode:
|
||||
if self.peft_config.inference_mode:
|
||||
prompts = self.prompt_encoder.embedding.weight.repeat(batch_size, 1, 1)
|
||||
else:
|
||||
prompts = self.prompt_encoder(prompt_tokens)
|
||||
@@ -142,34 +142,34 @@ class PETModel(torch.nn.Module):
|
||||
return getattr(self.base_model, name)
|
||||
|
||||
|
||||
class PETModelForSequenceClassification(PETModel):
|
||||
class PeftModelForSequenceClassification(PeftModel):
|
||||
"""
|
||||
PET model for sequence classification tasks.
|
||||
Peft model for sequence classification tasks.
|
||||
|
||||
Args:
|
||||
model (:obj:`PreTrainedModel`): Base transformer model
|
||||
pet_config (:obj:`PETConfig`): PET config.
|
||||
model (`PreTrainedModel`): Base transformer model
|
||||
peft_config (`PeftConfig`): Peft config.
|
||||
|
||||
Attributes:
|
||||
config (:obj:`PretrainedConfig`): The configuration object of the base model. cls_layer_name (:obj:`str`): The
|
||||
config (`PretrainedConfig`): The configuration object of the base model. cls_layer_name (`str`): The
|
||||
name of the classification layer.
|
||||
|
||||
Example::
|
||||
|
||||
>>> from transformers import AutoModelForSequenceClassification >>> from pet import
|
||||
PETModelForSequenceClassification, get_pet_config >>> config = {
|
||||
'pet_type': 'PREFIX_TUNING', 'task_type': 'SEQ_CLS', 'inference_mode': False, 'num_virtual_tokens': 20,
|
||||
>>> from transformers import AutoModelForSequenceClassification >>> from peft import
|
||||
PeftModelForSequenceClassification, get_peft_config >>> config = {
|
||||
'peft_type': 'PREFIX_TUNING', 'task_type': 'SEQ_CLS', 'inference_mode': False, 'num_virtual_tokens': 20,
|
||||
'token_dim': 768, 'num_transformer_submodules': 1, 'num_attention_heads': 12, 'num_layers': 12,
|
||||
'encoder_hidden_size': 768, 'prefix_projection': False, 'postprocess_past_key_value_function': None
|
||||
}
|
||||
>>> pet_config = get_pet_config(config) >>> model =
|
||||
AutoModelForSequenceClassification.from_pretrained("bert-base-cased") >>> pet_model =
|
||||
PETModelForSequenceClassification(model, pet_config) >>> pet_model.print_trainable_parameters() trainable
|
||||
>>> peft_config = get_peft_config(config)
|
||||
>>> model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased")
|
||||
>>> peft_model = PeftModelForSequenceClassification(model, peft_config) >>> peft_model.print_trainable_parameters() trainable
|
||||
params: 370178 || all params: 108680450 || trainable%: 0.3406113979101117
|
||||
"""
|
||||
|
||||
def __init__(self, model, pet_config: PETConfig):
|
||||
super().__init__(model, pet_config)
|
||||
def __init__(self, model, peft_config: PeftConfig):
|
||||
super().__init__(model, peft_config)
|
||||
self.modules_to_save = ["classifier", "score"]
|
||||
|
||||
for name, _ in self.base_model.named_children():
|
||||
@@ -193,7 +193,7 @@ class PETModelForSequenceClassification(PETModel):
|
||||
):
|
||||
return_dict = return_dict if return_dict is not None else self.config.use_return_dict
|
||||
|
||||
if self.pet_config.pet_type == PETType.LORA:
|
||||
if self.peft_config.peft_type == PeftType.LORA:
|
||||
return self.base_model(
|
||||
input_ids=input_ids,
|
||||
attention_mask=attention_mask,
|
||||
@@ -208,7 +208,7 @@ class PETModelForSequenceClassification(PETModel):
|
||||
batch_size = input_ids.shape[0]
|
||||
if attention_mask is not None:
|
||||
# concat prompt attention mask
|
||||
prefix_attention_mask = torch.ones(batch_size, self.pet_config.num_virtual_tokens).to(self.device)
|
||||
prefix_attention_mask = torch.ones(batch_size, self.peft_config.num_virtual_tokens).to(self.device)
|
||||
attention_mask = torch.cat((prefix_attention_mask, attention_mask), dim=1)
|
||||
if kwargs.get("position_ids", None) is not None:
|
||||
warnings.warn("Position ids are not supported for parameter efficient tuning. Ignoring position ids.")
|
||||
@@ -223,13 +223,13 @@ class PETModelForSequenceClassification(PETModel):
|
||||
}
|
||||
)
|
||||
|
||||
if self.pet_config.pet_type == PETType.PREFIX_TUNING:
|
||||
if self.peft_config.peft_type == PeftType.PREFIX_TUNING:
|
||||
return self._prefix_tuning_forward(input_ids=input_ids, **kwargs)
|
||||
else:
|
||||
if kwargs.get("token_type_ids", None) is not None:
|
||||
kwargs["token_type_ids"] = torch.cat(
|
||||
(
|
||||
torch.zeros(batch_size, self.pet_config.num_virtual_tokens).to(self.device),
|
||||
torch.zeros(batch_size, self.peft_config.num_virtual_tokens).to(self.device),
|
||||
kwargs["token_type_ids"],
|
||||
),
|
||||
dim=1,
|
||||
@@ -312,30 +312,32 @@ class PETModelForSequenceClassification(PETModel):
|
||||
)
|
||||
|
||||
|
||||
class PETModelForCausalLM(PETModel):
|
||||
class PeftModelForCausalLM(PeftModel):
|
||||
"""
|
||||
PET model for Causal LM
|
||||
Peft model for Causal LM
|
||||
|
||||
Args:
|
||||
model (:obj:`PreTrainedModel`): Base transformer model
|
||||
pet_config (:obj:`PETConfig`): PET config.
|
||||
model (`PreTrainedModel`): Base transformer model
|
||||
peft_config (`PeftConfig`): Peft config.
|
||||
|
||||
|
||||
Example::
|
||||
|
||||
>>> from transformers import AutoModelForCausalLM >>> from pet import PETModelForCausalLM, get_pet_config >>>
|
||||
>>> from transformers import AutoModelForCausalLM >>> from peft import PeftModelForCausalLM, get_peft_config >>>
|
||||
config = {
|
||||
'pet_type': 'PREFIX_TUNING', 'task_type': 'CAUSAL_LM', 'inference_mode': False, 'num_virtual_tokens':
|
||||
'peft_type': 'PREFIX_TUNING', 'task_type': 'CAUSAL_LM', 'inference_mode': False, 'num_virtual_tokens':
|
||||
20, 'token_dim': 1280, 'num_transformer_submodules': 1, 'num_attention_heads': 20, 'num_layers': 36,
|
||||
'encoder_hidden_size': 1280, 'prefix_projection': False, 'postprocess_past_key_value_function': None
|
||||
}
|
||||
>>> pet_config = get_pet_config(config) >>> model = AutoModelForCausalLM.from_pretrained("gpt2-large") >>>
|
||||
pet_model = PETModelForCausalLM(model, pet_config) >>> pet_model.print_trainable_parameters() trainable params:
|
||||
>>> peft_config = get_peft_config(config)
|
||||
>>> model = AutoModelForCausalLM.from_pretrained("gpt2-large")
|
||||
>>> peft_model = PeftModelForCausalLM(model, peft_config)
|
||||
>>> peft_model.print_trainable_parameters() trainable params:
|
||||
1843200 || all params: 775873280 || trainable%: 0.23756456724479544
|
||||
"""
|
||||
|
||||
def __init__(self, model, pet_config: PETConfig):
|
||||
super().__init__(model, pet_config)
|
||||
def __init__(self, model, peft_config: PeftConfig):
|
||||
super().__init__(model, peft_config)
|
||||
self.base_model_prepare_inputs_for_generation = self.base_model.prepare_inputs_for_generation
|
||||
self.base_model.prepare_inputs_for_generation = self.prepare_inputs_for_generation
|
||||
|
||||
@@ -350,7 +352,7 @@ class PETModelForCausalLM(PETModel):
|
||||
return_dict=None,
|
||||
**kwargs,
|
||||
):
|
||||
if self.pet_config.pet_type == PETType.LORA:
|
||||
if self.peft_config.peft_type == PeftType.LORA:
|
||||
return self.base_model(
|
||||
input_ids=input_ids,
|
||||
attention_mask=attention_mask,
|
||||
@@ -365,7 +367,7 @@ class PETModelForCausalLM(PETModel):
|
||||
batch_size = input_ids.shape[0]
|
||||
if attention_mask is not None:
|
||||
# concat prompt attention mask
|
||||
prefix_attention_mask = torch.ones(batch_size, self.pet_config.num_virtual_tokens).to(self.device)
|
||||
prefix_attention_mask = torch.ones(batch_size, self.peft_config.num_virtual_tokens).to(self.device)
|
||||
attention_mask = torch.cat((prefix_attention_mask, attention_mask), dim=1)
|
||||
|
||||
if kwargs.get("position_ids", None) is not None:
|
||||
@@ -384,7 +386,7 @@ class PETModelForCausalLM(PETModel):
|
||||
}
|
||||
)
|
||||
|
||||
if self.pet_config.pet_type == PETType.PREFIX_TUNING:
|
||||
if self.peft_config.peft_type == PeftType.PREFIX_TUNING:
|
||||
past_key_values = self.get_prompt(batch_size)
|
||||
return self.base_model(input_ids=input_ids, past_key_values=past_key_values, **kwargs)
|
||||
else:
|
||||
@@ -392,21 +394,22 @@ class PETModelForCausalLM(PETModel):
|
||||
inputs_embeds = self.word_embeddings(input_ids)
|
||||
# concat prompt labels
|
||||
if labels is not None:
|
||||
prefix_labels = torch.full((batch_size, self.pet_config.num_virtual_tokens), -100).to(self.device)
|
||||
prefix_labels = torch.full((batch_size, self.peft_config.num_virtual_tokens), -100).to(self.device)
|
||||
kwargs["labels"] = torch.cat((prefix_labels, labels), dim=1)
|
||||
prompts = self.get_prompt(batch_size=batch_size)
|
||||
inputs_embeds = torch.cat((prompts, inputs_embeds), dim=1)
|
||||
return self.base_model(inputs_embeds=inputs_embeds, **kwargs)
|
||||
|
||||
def generate(self, **kwargs):
|
||||
if self.pet_config.pet_type == PETType.LORA:
|
||||
if self.peft_config.peft_type == PeftType.LORA:
|
||||
return self.base_model.generate(**kwargs)
|
||||
else:
|
||||
assert "input_ids" in kwargs, "input_ids must be provided for PET model generation"
|
||||
if "input_ids" not in kwargs:
|
||||
raise ValueError("input_ids must be provided for Peft model generation")
|
||||
if kwargs.get("attention_mask", None) is not None:
|
||||
# concat prompt attention mask
|
||||
prefix_attention_mask = torch.ones(
|
||||
kwargs["input_ids"].shape[0], self.pet_config.num_virtual_tokens
|
||||
kwargs["input_ids"].shape[0], self.peft_config.num_virtual_tokens
|
||||
).to(kwargs["input_ids"].device)
|
||||
kwargs["attention_mask"] = torch.cat((prefix_attention_mask, kwargs["attention_mask"]), dim=1)
|
||||
|
||||
@@ -419,7 +422,7 @@ class PETModelForCausalLM(PETModel):
|
||||
)
|
||||
kwargs["token_type_ids"] = None
|
||||
|
||||
if self.pet_config.pet_type == PETType.PREFIX_TUNING:
|
||||
if self.peft_config.peft_type == PeftType.PREFIX_TUNING:
|
||||
batch_size = kwargs["input_ids"].shape[0]
|
||||
past_key_values = self.get_prompt(batch_size)
|
||||
kwargs["past_key_values"] = past_key_values
|
||||
@@ -433,30 +436,32 @@ class PETModelForCausalLM(PETModel):
|
||||
return model_kwargs
|
||||
|
||||
|
||||
class PETModelForSeq2SeqLM(PETModel):
|
||||
class PeftModelForSeq2SeqLM(PeftModel):
|
||||
"""
|
||||
PET model for Seq2Seq LM
|
||||
Peft model for Seq2Seq LM
|
||||
|
||||
Args:
|
||||
model (:obj:`PreTrainedModel`): Base transformer model
|
||||
pet_config (:obj:`PETConfig`): PET config.
|
||||
model (`PreTrainedModel`): Base transformer model
|
||||
peft_config (`PeftConfig`): Peft config.
|
||||
|
||||
|
||||
Example::
|
||||
|
||||
>>> from transformers import AutoModelForSeq2SeqLM >>> from pet import PETModelForSeq2SeqLM, get_pet_config >>>
|
||||
>>> from transformers import AutoModelForSeq2SeqLM >>> from peft import PeftModelForSeq2SeqLM, get_peft_config >>>
|
||||
config = {
|
||||
'pet_type': 'LORA', 'task_type': 'SEQ_2_SEQ_LM', 'inference_mode': False, 'r': 8, 'target_modules':
|
||||
'peft_type': 'LORA', 'task_type': 'SEQ_2_SEQ_LM', 'inference_mode': False, 'r': 8, 'target_modules':
|
||||
['q', 'v'], 'lora_alpha': 32, 'lora_dropout': 0.1, 'merge_weights': False, 'fan_in_fan_out': False,
|
||||
'enable_lora': None, 'bias': 'none'
|
||||
}
|
||||
>>> pet_config = get_pet_config(config) >>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base") >>>
|
||||
pet_model = PETModelForSeq2SeqLM(model, pet_config) >>> pet_model.print_trainable_parameters() trainable
|
||||
>>> peft_config = get_peft_config(config)
|
||||
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
|
||||
>>> peft_model = PeftModelForSeq2SeqLM(model, peft_config)
|
||||
>>> peft_model.print_trainable_parameters() trainable
|
||||
params: 884736 || all params: 223843584 || trainable%: 0.3952474242013566
|
||||
"""
|
||||
|
||||
def __init__(self, model, pet_config: PETConfig):
|
||||
super().__init__(model, pet_config)
|
||||
def __init__(self, model, peft_config: PeftConfig):
|
||||
super().__init__(model, peft_config)
|
||||
self.base_model_prepare_inputs_for_generation = self.base_model.prepare_inputs_for_generation
|
||||
self.base_model.prepare_inputs_for_generation = self.prepare_inputs_for_generation
|
||||
self.base_model_prepare_encoder_decoder_kwargs_for_generation = (
|
||||
@@ -480,7 +485,7 @@ class PETModelForSeq2SeqLM(PETModel):
|
||||
return_dict=None,
|
||||
**kwargs,
|
||||
):
|
||||
if self.pet_config.pet_type == PETType.LORA:
|
||||
if self.peft_config.peft_type == PeftType.LORA:
|
||||
return self.base_model(
|
||||
input_ids=input_ids,
|
||||
attention_mask=attention_mask,
|
||||
@@ -498,7 +503,7 @@ class PETModelForSeq2SeqLM(PETModel):
|
||||
batch_size = input_ids.shape[0]
|
||||
if decoder_attention_mask is not None:
|
||||
# concat prompt attention mask
|
||||
prefix_attention_mask = torch.ones(batch_size, self.pet_config.num_virtual_tokens).to(self.device)
|
||||
prefix_attention_mask = torch.ones(batch_size, self.peft_config.num_virtual_tokens).to(self.device)
|
||||
decoder_attention_mask = torch.cat((prefix_attention_mask, decoder_attention_mask), dim=1)
|
||||
|
||||
if kwargs.get("position_ids", None) is not None:
|
||||
@@ -518,7 +523,7 @@ class PETModelForSeq2SeqLM(PETModel):
|
||||
}
|
||||
)
|
||||
|
||||
if self.pet_config.pet_type == PETType.PREFIX_TUNING:
|
||||
if self.peft_config.peft_type == PeftType.PREFIX_TUNING:
|
||||
past_key_values = self.get_prompt(batch_size)
|
||||
return self.base_model(
|
||||
input_ids=input_ids, decoder_input_ids=decoder_input_ids, past_key_values=past_key_values, **kwargs
|
||||
@@ -534,25 +539,25 @@ class PETModelForSeq2SeqLM(PETModel):
|
||||
|
||||
if attention_mask is not None:
|
||||
# concat prompt attention mask
|
||||
prefix_attention_mask = torch.ones(batch_size, self.pet_config.num_virtual_tokens).to(self.device)
|
||||
prefix_attention_mask = torch.ones(batch_size, self.peft_config.num_virtual_tokens).to(self.device)
|
||||
kwargs["attention_mask"] = torch.cat((prefix_attention_mask, attention_mask), dim=1)
|
||||
# concat prompt labels
|
||||
if labels is not None:
|
||||
prefix_labels = torch.full((batch_size, self.pet_config.num_virtual_tokens), -100).to(self.device)
|
||||
prefix_labels = torch.full((batch_size, self.peft_config.num_virtual_tokens), -100).to(self.device)
|
||||
kwargs["labels"] = torch.cat((prefix_labels, labels), dim=1)
|
||||
prompts = self.get_prompt(batch_size=batch_size)
|
||||
inputs_embeds = torch.cat((prompts[:, : self.pet_config.num_virtual_tokens], inputs_embeds), dim=1)
|
||||
inputs_embeds = torch.cat((prompts[:, : self.peft_config.num_virtual_tokens], inputs_embeds), dim=1)
|
||||
decoder_inputs_embeds = torch.cat(
|
||||
(prompts[:, self.pet_config.num_virtual_tokens :], decoder_inputs_embeds), dim=1
|
||||
(prompts[:, self.peft_config.num_virtual_tokens :], decoder_inputs_embeds), dim=1
|
||||
)
|
||||
return self.base_model(inputs_embeds=inputs_embeds, decoder_inputs_embeds=decoder_inputs_embeds, **kwargs)
|
||||
|
||||
def generate(self, **kwargs):
|
||||
if self.pet_config.pet_type == PETType.LORA:
|
||||
if self.peft_config.peft_type == PeftType.LORA:
|
||||
return self.base_model.generate(**kwargs)
|
||||
else:
|
||||
assert "input_ids" in kwargs, "input_ids must be provided for PET model generation"
|
||||
|
||||
if "input_ids" not in kwargs:
|
||||
raise ValueError("input_ids must be provided for Peft model generation")
|
||||
if kwargs.get("position_ids", None) is not None:
|
||||
warnings.warn("Position ids are not supported for parameter efficient tuning. Ignoring position ids.")
|
||||
kwargs["position_ids"] = None
|
||||
@@ -562,7 +567,7 @@ class PETModelForSeq2SeqLM(PETModel):
|
||||
)
|
||||
kwargs["token_type_ids"] = None
|
||||
|
||||
if self.pet_config.pet_type == PETType.PREFIX_TUNING:
|
||||
if self.peft_config.peft_type == PeftType.PREFIX_TUNING:
|
||||
batch_size = kwargs["input_ids"].shape[0]
|
||||
past_key_values = self.get_prompt(batch_size)
|
||||
kwargs["past_key_values"] = past_key_values
|
||||
@@ -585,34 +590,35 @@ class PETModelForSeq2SeqLM(PETModel):
|
||||
return model_kwargs
|
||||
|
||||
|
||||
class PETModelForTokenClassification(PETModel):
|
||||
class PeftModelForTokenClassification(PeftModel):
|
||||
"""
|
||||
PET model for sequence classification tasks.
|
||||
Peft model for sequence classification tasks.
|
||||
|
||||
Args:
|
||||
model (:obj:`PreTrainedModel`): Base transformer model
|
||||
pet_config (:obj:`PETConfig`): PET config.
|
||||
model (`PreTrainedModel`): Base transformer model
|
||||
peft_config (`PeftConfig`): Peft config.
|
||||
|
||||
Attributes:
|
||||
config (:obj:`PretrainedConfig`): The configuration object of the base model. cls_layer_name (:obj:`str`): The
|
||||
config (`PretrainedConfig`): The configuration object of the base model. cls_layer_name (`str`): The
|
||||
name of the classification layer.
|
||||
|
||||
Example::
|
||||
|
||||
>>> from transformers import AutoModelForSequenceClassification >>> from pet import
|
||||
PETModelForTokenClassification, get_pet_config >>> config = {
|
||||
'pet_type': 'PREFIX_TUNING', 'task_type': 'TOKEN_CLS', 'inference_mode': False, 'num_virtual_tokens':
|
||||
>>> from transformers import AutoModelForSequenceClassification >>> from peft import
|
||||
PeftModelForTokenClassification, get_peft_config >>> config = {
|
||||
'peft_type': 'PREFIX_TUNING', 'task_type': 'TOKEN_CLS', 'inference_mode': False, 'num_virtual_tokens':
|
||||
20, 'token_dim': 768, 'num_transformer_submodules': 1, 'num_attention_heads': 12, 'num_layers': 12,
|
||||
'encoder_hidden_size': 768, 'prefix_projection': False, 'postprocess_past_key_value_function': None
|
||||
}
|
||||
>>> pet_config = get_pet_config(config) >>> model =
|
||||
AutoModelForSequenceClassification.from_pretrained("bert-base-cased") >>> pet_model =
|
||||
PETModelForSequenceClassification(model, pet_config) >>> pet_model.print_trainable_parameters() trainable
|
||||
>>> peft_config = get_peft_config(config) >>> model =
|
||||
AutoModelForTokenClassification.from_pretrained("bert-base-cased")
|
||||
>>> peft_model = PeftModelForTokenClassification(model, peft_config)
|
||||
>>> peft_model.print_trainable_parameters() trainable
|
||||
params: 370178 || all params: 108680450 || trainable%: 0.3406113979101117
|
||||
"""
|
||||
|
||||
def __init__(self, model, pet_config: PETConfig):
|
||||
super().__init__(model, pet_config)
|
||||
def __init__(self, model, peft_config: PeftConfig):
|
||||
super().__init__(model, peft_config)
|
||||
self.modules_to_save = ["classifier", "score"]
|
||||
|
||||
for name, _ in self.base_model.named_children():
|
||||
@@ -636,7 +642,7 @@ class PETModelForTokenClassification(PETModel):
|
||||
):
|
||||
return_dict = return_dict if return_dict is not None else self.config.use_return_dict
|
||||
|
||||
if self.pet_config.pet_type == PETType.LORA:
|
||||
if self.peft_config.peft_type == PeftType.LORA:
|
||||
return self.base_model(
|
||||
input_ids=input_ids,
|
||||
attention_mask=attention_mask,
|
||||
@@ -651,7 +657,7 @@ class PETModelForTokenClassification(PETModel):
|
||||
batch_size = input_ids.shape[0]
|
||||
if attention_mask is not None:
|
||||
# concat prompt attention mask
|
||||
prefix_attention_mask = torch.ones(batch_size, self.pet_config.num_virtual_tokens).to(self.device)
|
||||
prefix_attention_mask = torch.ones(batch_size, self.peft_config.num_virtual_tokens).to(self.device)
|
||||
attention_mask = torch.cat((prefix_attention_mask, attention_mask), dim=1)
|
||||
if kwargs.get("position_ids", None) is not None:
|
||||
warnings.warn("Position ids are not supported for parameter efficient tuning. Ignoring position ids.")
|
||||
@@ -666,13 +672,13 @@ class PETModelForTokenClassification(PETModel):
|
||||
}
|
||||
)
|
||||
|
||||
if self.pet_config.pet_type == PETType.PREFIX_TUNING:
|
||||
if self.peft_config.peft_type == PeftType.PREFIX_TUNING:
|
||||
return self._prefix_tuning_forward(input_ids=input_ids, **kwargs)
|
||||
else:
|
||||
if kwargs.get("token_type_ids", None) is not None:
|
||||
kwargs["token_type_ids"] = torch.cat(
|
||||
(
|
||||
torch.zeros(batch_size, self.pet_config.num_virtual_tokens).to(self.device),
|
||||
torch.zeros(batch_size, self.peft_config.num_virtual_tokens).to(self.device),
|
||||
kwargs["token_type_ids"],
|
||||
),
|
||||
dim=1,
|
||||
@@ -2,7 +2,7 @@
|
||||
# There's no way to ignore "F401 '...' imported but unused" warnings in this
|
||||
# module, but to preserve other warnings. So, don't check this module at all
|
||||
|
||||
from .lora import LoRAConfig, LoRAModel
|
||||
from .lora import LoraConfig, LoraModel
|
||||
from .p_tuning import PromptEncoder, PromptEncoderConfig, PromptEncoderReparameterizationType
|
||||
from .prefix_tuning import PrefixEncoder, PrefixTuningConfig
|
||||
from .prompt_tuning import PromptEmbedding, PromptTuningConfig, PromptTuningInit
|
||||
@@ -12,92 +12,92 @@ from transformers.pytorch_utils import Conv1D
|
||||
import loralib as lora # noqa: F401
|
||||
from loralib import mark_only_lora_as_trainable
|
||||
|
||||
from ..utils import PETConfig, PETType
|
||||
from ..utils import PeftConfig, PeftType, transpose
|
||||
|
||||
|
||||
@dataclass
|
||||
class LoRAConfig(PETConfig):
|
||||
class LoraConfig(PeftConfig):
|
||||
"""
|
||||
This is the configuration class to store the configuration of a :class:`~pet.LoRA`.
|
||||
This is the configuration class to store the configuration of a :class:`~peft.Lora`.
|
||||
|
||||
Args:
|
||||
r: (:obj:`init`): LoRA attention dimension
|
||||
target_modules (:obj: list of :obj: str): The names of the modules to apply LoRA to.
|
||||
lora_alpha (:obj: float): The alpha parameter for LoRA scaling.
|
||||
lora_dropout (:obj: float): The dropout probability for LoRA layers.
|
||||
merge_weights (:obj: bool):
|
||||
Whether to merge the weights of the LoRA layers with the base transformer model in `eval` mode.
|
||||
fan_in_fan_out (:obj: bool): Set this to True if the layer to replace stores weight like (fan_in, fan_out)
|
||||
enable_lora (:obj: list of :obj: bool): Used with `lora.MergedLinear`.
|
||||
bias (:obj: str): Bias type for LoRA. Can be 'none', 'all' or 'lora_only'
|
||||
r: (int): Lora attention dimension
|
||||
target_modules ( list of str): The names of the modules to apply Lora to.
|
||||
lora_alpha ( float): The alpha parameter for Lora scaling.
|
||||
lora_dropout ( float): The dropout probability for Lora layers.
|
||||
merge_weights ( bool):
|
||||
Whether to merge the weights of the Lora layers with the base transformer model in `eval` mode.
|
||||
fan_in_fan_out ( bool): Set this to True if the layer to replace stores weight like (fan_in, fan_out)
|
||||
enable_lora ( list of bool): Used with `lora.MergedLinear`.
|
||||
bias ( str): Bias type for Lora. Can be 'none', 'all' or 'lora_only'
|
||||
"""
|
||||
|
||||
r: int = field(default=8, metadata={"help": "LoRA attention dimension"})
|
||||
target_modules: Optional[list] = field(default=None, metadata={"help": "List of modules to replace with LoRA"})
|
||||
lora_alpha: int = field(default=None, metadata={"help": "LoRA alpha"})
|
||||
lora_dropout: float = field(default=None, metadata={"help": "LoRA dropout"})
|
||||
r: int = field(default=8, metadata={"help": "Lora attention dimension"})
|
||||
target_modules: Optional[list] = field(default=None, metadata={"help": "List of modules to replace with Lora"})
|
||||
lora_alpha: int = field(default=None, metadata={"help": "Lora alpha"})
|
||||
lora_dropout: float = field(default=None, metadata={"help": "Lora dropout"})
|
||||
merge_weights: bool = field(
|
||||
default=False, metadata={"help": "Merge weights of the original model and the LoRA model"}
|
||||
default=False, metadata={"help": "Merge weights of the original model and the Lora model"}
|
||||
)
|
||||
fan_in_fan_out: bool = field(
|
||||
default=False,
|
||||
metadata={"help": "Set this to True if the layer to replace stores weight like (fan_in, fan_out)"},
|
||||
)
|
||||
enable_lora: Optional[list[bool]] = field(default=None, metadata={"help": "Used with `lora.MergedLinear`."})
|
||||
bias: str = field(default="none", metadata={"help": "Bias type for LoRA. Can be 'none', 'all' or 'lora_only'"})
|
||||
bias: str = field(default="none", metadata={"help": "Bias type for Lora. Can be 'none', 'all' or 'lora_only'"})
|
||||
|
||||
def __post_init__(self):
|
||||
self.pet_type = PETType.LORA
|
||||
self.peft_type = PeftType.LORA
|
||||
|
||||
|
||||
class LoRAModel(torch.nn.Module):
|
||||
class LoraModel(torch.nn.Module):
|
||||
"""
|
||||
Creates Low Rank Adapter (LoRA) model from a pretrained transformers model.
|
||||
Creates Low Rank Adapter (Lora) model from a pretrained transformers model.
|
||||
|
||||
Args:
|
||||
model (:obj:`transformers.PreTrainedModel`): The model to be adapted.
|
||||
config (:obj:`LoRAConfig`): The configuration of the LoRA model.
|
||||
model (`transformers.PreTrainedModel`): The model to be adapted.
|
||||
config (`LoraConfig`): The configuration of the Lora model.
|
||||
|
||||
Returns:
|
||||
:obj:`torch.nn.Module`: The LoRA model.
|
||||
`torch.nn.Module`: The Lora model.
|
||||
|
||||
Example::
|
||||
|
||||
>>> from transformers import AutoModelForSeq2SeqLM, LoRAConfig >>> from pet import LoRAModel, LoRAConfig >>>
|
||||
config = LoRAConfig(
|
||||
pet_type="LORA", task_type="SEQ_2_SEQ_LM", r=8, lora_alpha=32, target_modules=["q", "v"],
|
||||
>>> from transformers import AutoModelForSeq2SeqLM, LoraConfig >>> from peft import LoraModel, LoraConfig >>>
|
||||
config = LoraConfig(
|
||||
peft_type="LORA", task_type="SEQ_2_SEQ_LM", r=8, lora_alpha=32, target_modules=["q", "v"],
|
||||
lora_dropout=0.01, )
|
||||
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base") >>> lora_model = LoRAModel(config, model)
|
||||
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base") >>> lora_model = LoraModel(config, model)
|
||||
|
||||
Attributes:
|
||||
model (:obj:`transformers.PreTrainedModel`): The model to be adapted. config (:obj:`LoRAConfig`): The
|
||||
configuration of the LoRA model.
|
||||
model (`transformers.PreTrainedModel`): The model to be adapted. config (`LoraConfig`): The
|
||||
configuration of the Lora model.
|
||||
"""
|
||||
|
||||
def __init__(self, config, model):
|
||||
super().__init__()
|
||||
self.pet_config = config
|
||||
self.peft_config = config
|
||||
self.model = model
|
||||
self._find_and_replace()
|
||||
mark_only_lora_as_trainable(self.model, self.pet_config.bias)
|
||||
mark_only_lora_as_trainable(self.model, self.peft_config.bias)
|
||||
|
||||
def _find_and_replace(self):
|
||||
kwargs = {
|
||||
"r": self.pet_config.r,
|
||||
"lora_alpha": self.pet_config.lora_alpha,
|
||||
"lora_dropout": self.pet_config.lora_dropout,
|
||||
"fan_in_fan_out": self.pet_config.fan_in_fan_out,
|
||||
"merge_weights": self.pet_config.merge_weights,
|
||||
"r": self.peft_config.r,
|
||||
"lora_alpha": self.peft_config.lora_alpha,
|
||||
"lora_dropout": self.peft_config.lora_dropout,
|
||||
"fan_in_fan_out": self.peft_config.fan_in_fan_out,
|
||||
"merge_weights": self.peft_config.merge_weights,
|
||||
}
|
||||
key_list = [key for key, _ in self.model.named_modules()]
|
||||
for key in key_list:
|
||||
if any(key.endswith(target_key) for target_key in self.pet_config.target_modules):
|
||||
if any(key.endswith(target_key) for target_key in self.peft_config.target_modules):
|
||||
parent, target, target_name = self._get_submodules(key)
|
||||
bias = target.bias is not None
|
||||
if isinstance(target, torch.nn.Linear) and self.pet_config.enable_lora is None:
|
||||
if isinstance(target, torch.nn.Linear) and self.peft_config.enable_lora is None:
|
||||
new_module = Linear(target.in_features, target.out_features, bias=bias, **kwargs)
|
||||
elif self.pet_config.enable_lora is not None:
|
||||
kwargs.update({"enable_lora": self.pet_config.enable_lora})
|
||||
elif self.peft_config.enable_lora is not None:
|
||||
kwargs.update({"enable_lora": self.peft_config.enable_lora})
|
||||
if isinstance(target, Conv1D):
|
||||
in_features, out_features = target.weight.shape
|
||||
else:
|
||||
@@ -137,8 +137,8 @@ class LoRAModel(torch.nn.Module):
|
||||
def modules_to_save(self):
|
||||
return None
|
||||
|
||||
def get_pet_config_as_dict(self, inference: bool = False):
|
||||
config = {k: v.value if isinstance(v, Enum) else v for k, v in asdict(self.pet_config).items()}
|
||||
def get_peft_config_as_dict(self, inference: bool = False):
|
||||
config = {k: v.value if isinstance(v, Enum) else v for k, v in asdict(self.peft_config).items()}
|
||||
if inference:
|
||||
config["inference_mode"] = True
|
||||
return config
|
||||
@@ -151,7 +151,7 @@ class LoRAModel(torch.nn.Module):
|
||||
# Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
# Licensed under the MIT License (MIT). See LICENSE in the repo root for license information.
|
||||
# ------------------------------------------------------------------------------------------
|
||||
class LoRALayer:
|
||||
class LoraLayer:
|
||||
def __init__(
|
||||
self,
|
||||
r: int,
|
||||
@@ -171,8 +171,8 @@ class LoRALayer:
|
||||
self.merge_weights = merge_weights
|
||||
|
||||
|
||||
class Linear(nn.Linear, LoRALayer):
|
||||
# LoRA implemented in a dense layer
|
||||
class Linear(nn.Linear, LoraLayer):
|
||||
# Lora implemented in a dense layer
|
||||
def __init__(
|
||||
self,
|
||||
in_features: int,
|
||||
@@ -185,7 +185,7 @@ class Linear(nn.Linear, LoRALayer):
|
||||
**kwargs,
|
||||
):
|
||||
nn.Linear.__init__(self, in_features, out_features, **kwargs)
|
||||
LoRALayer.__init__(self, r=r, lora_alpha=lora_alpha, lora_dropout=lora_dropout, merge_weights=merge_weights)
|
||||
LoraLayer.__init__(self, r=r, lora_alpha=lora_alpha, lora_dropout=lora_dropout, merge_weights=merge_weights)
|
||||
|
||||
self.fan_in_fan_out = fan_in_fan_out
|
||||
# Actual trainable parameters
|
||||
@@ -207,37 +207,32 @@ class Linear(nn.Linear, LoRALayer):
|
||||
nn.init.zeros_(self.lora_B.weight)
|
||||
|
||||
def train(self, mode: bool = True):
|
||||
def T(w):
|
||||
return w.T if self.fan_in_fan_out else w
|
||||
|
||||
nn.Linear.train(self, mode)
|
||||
self.lora_A.train(mode)
|
||||
self.lora_B.train(mode)
|
||||
if self.merge_weights and self.merged:
|
||||
# Make sure that the weights are not merged
|
||||
if self.r > 0:
|
||||
self.weight.data -= T(self.lora_B.weight @ self.lora_A.weight) * self.scaling
|
||||
self.weight.data -= (
|
||||
transpose(self.lora_B.weight @ self.lora_A.weight, self.fan_in_fan_out) * self.scaling
|
||||
)
|
||||
self.merged = False
|
||||
|
||||
def eval(self):
|
||||
def T(w):
|
||||
return w.T if self.fan_in_fan_out else w
|
||||
|
||||
nn.Linear.eval(self)
|
||||
self.lora_A.eval()
|
||||
self.lora_B.eval()
|
||||
if self.merge_weights and not self.merged:
|
||||
# Merge the weights and mark it
|
||||
if self.r > 0:
|
||||
self.weight.data += T(self.lora_B.weight @ self.lora_A.weight) * self.scaling
|
||||
self.weight.data += (
|
||||
transpose(self.lora_B.weight @ self.lora_A.weight, self.fan_in_fan_out) * self.scaling
|
||||
)
|
||||
self.merged = True
|
||||
|
||||
def forward(self, x: torch.Tensor):
|
||||
def T(w):
|
||||
return w.T if self.fan_in_fan_out else w
|
||||
|
||||
if self.r > 0 and not self.merged:
|
||||
result = F.linear(x, T(self.weight), bias=self.bias)
|
||||
result = F.linear(x, transpose(self.weight, self.fan_in_fan_out), bias=self.bias)
|
||||
if self.r > 0:
|
||||
result += self.lora_B(self.lora_A(self.lora_dropout(x))) * self.scaling
|
||||
return result
|
||||
@@ -245,8 +240,8 @@ class Linear(nn.Linear, LoRALayer):
|
||||
return F.linear(x, T(self.weight), bias=self.bias)
|
||||
|
||||
|
||||
class MergedLinear(nn.Linear, LoRALayer):
|
||||
# LoRA implemented in a dense layer
|
||||
class MergedLinear(nn.Linear, LoraLayer):
|
||||
# Lora implemented in a dense layer
|
||||
def __init__(
|
||||
self,
|
||||
in_features: int,
|
||||
@@ -260,8 +255,9 @@ class MergedLinear(nn.Linear, LoRALayer):
|
||||
**kwargs,
|
||||
):
|
||||
nn.Linear.__init__(self, in_features, out_features, **kwargs)
|
||||
LoRALayer.__init__(self, r=r, lora_alpha=lora_alpha, lora_dropout=lora_dropout, merge_weights=merge_weights)
|
||||
assert out_features % len(enable_lora) == 0, "The length of enable_lora must divide out_features"
|
||||
LoraLayer.__init__(self, r=r, lora_alpha=lora_alpha, lora_dropout=lora_dropout, merge_weights=merge_weights)
|
||||
if out_features % len(enable_lora) != 0:
|
||||
raise ValueError("The length of enable_lora must divide out_features")
|
||||
self.enable_lora = enable_lora
|
||||
self.fan_in_fan_out = fan_in_fan_out
|
||||
# Actual trainable parameters
|
||||
@@ -299,9 +295,6 @@ class MergedLinear(nn.Linear, LoRALayer):
|
||||
return result.view((*x.shape[:-1], self.out_features))
|
||||
|
||||
def train(self, mode: bool = True):
|
||||
def T(w):
|
||||
return w.T if self.fan_in_fan_out else w
|
||||
|
||||
nn.Linear.train(self, mode)
|
||||
self.lora_A.train(mode)
|
||||
self.lora_B.train(mode)
|
||||
@@ -313,13 +306,10 @@ class MergedLinear(nn.Linear, LoRALayer):
|
||||
self.lora_B.weight.data.unsqueeze(-1),
|
||||
groups=sum(self.enable_lora),
|
||||
).squeeze(0)
|
||||
self.weight.data -= self.zero_pad(T(delta_w * self.scaling))
|
||||
self.weight.data -= self.zero_pad(transpose(delta_w * self.scaling, self.fan_in_fan_out))
|
||||
self.merged = False
|
||||
|
||||
def eval(self):
|
||||
def T(w):
|
||||
return w.T if self.fan_in_fan_out else w
|
||||
|
||||
nn.Linear.eval(self)
|
||||
self.lora_A.eval()
|
||||
self.lora_B.eval()
|
||||
@@ -331,17 +321,14 @@ class MergedLinear(nn.Linear, LoRALayer):
|
||||
self.lora_B.weight.data.unsqueeze(-1),
|
||||
groups=sum(self.enable_lora),
|
||||
).squeeze(0)
|
||||
self.weight.data += self.zero_pad(T(delta_w * self.scaling))
|
||||
self.weight.data += self.zero_pad(transpose(delta_w * self.scaling, self.fan_in_fan_out))
|
||||
self.merged = True
|
||||
|
||||
def forward(self, x: torch.Tensor):
|
||||
def T(w):
|
||||
return w.T if self.fan_in_fan_out else w
|
||||
|
||||
if self.merged:
|
||||
return F.linear(x, T(self.weight), bias=self.bias)
|
||||
return F.linear(x, transpose(self.weight, self.fan_in_fan_out), bias=self.bias)
|
||||
else:
|
||||
result = F.linear(x, T(self.weight), bias=self.bias)
|
||||
result = F.linear(x, transpose(self.weight, self.fan_in_fan_out), bias=self.bias)
|
||||
if self.r > 0:
|
||||
after_A = self.lora_A(self.lora_dropout(x))
|
||||
after_B = self.lora_B(after_A.transpose(-2, -1)).transpose(-2, -1)
|
||||
@@ -4,7 +4,7 @@ from typing import Union
|
||||
|
||||
import torch
|
||||
|
||||
from ..utils import PETType, PromptLearningConfig
|
||||
from ..utils import PeftType, PromptLearningConfig
|
||||
|
||||
|
||||
class PromptEncoderReparameterizationType(str, enum.Enum):
|
||||
@@ -15,15 +15,15 @@ class PromptEncoderReparameterizationType(str, enum.Enum):
|
||||
@dataclass
|
||||
class PromptEncoderConfig(PromptLearningConfig):
|
||||
"""
|
||||
This is the configuration class to store the configuration of a :class:`~pet.PromptEncoder`.
|
||||
This is the configuration class to store the configuration of a :class:`~peft.PromptEncoder`.
|
||||
|
||||
Args:
|
||||
encoder_reparameterization_type
|
||||
(:obj:Union[:class:`PromptEncoderReparameterizationType`, :obj:`str`]): The type of reparameterization to
|
||||
(Union[:class:`PromptEncoderReparameterizationType`, `str`]): The type of reparameterization to
|
||||
use.
|
||||
encoder_hidden_size (:obj:`int`): The hidden size of the prompt encoder.
|
||||
encoder_num_layers (:obj:`int`): The number of layers of the prompt encoder.
|
||||
encoder_dropout (:obj:`float`): The dropout probability of the prompt encoder.
|
||||
encoder_hidden_size (`int`): The hidden size of the prompt encoder.
|
||||
encoder_num_layers (`int`): The number of layers of the prompt encoder.
|
||||
encoder_dropout (`float`): The dropout probability of the prompt encoder.
|
||||
"""
|
||||
|
||||
encoder_reparameterization_type: Union[str, PromptEncoderReparameterizationType] = field(
|
||||
@@ -44,7 +44,7 @@ class PromptEncoderConfig(PromptLearningConfig):
|
||||
)
|
||||
|
||||
def __post_init__(self):
|
||||
self.pet_type = PETType.P_TUNING
|
||||
self.peft_type = PeftType.P_TUNING
|
||||
|
||||
|
||||
# Based on https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/modules/common/prompt_encoder.py
|
||||
@@ -58,8 +58,8 @@ class PromptEncoder(torch.nn.Module):
|
||||
|
||||
Example::
|
||||
|
||||
>>> from pet import PromptEncoder, PromptEncoderConfig >>> config = PromptEncoderConfig(
|
||||
pet_type="P_TUNING", task_type="SEQ_2_SEQ_LM", num_virtual_tokens=20, token_dim=768,
|
||||
>>> from peft import PromptEncoder, PromptEncoderConfig >>> config = PromptEncoderConfig(
|
||||
peft_type="P_TUNING", task_type="SEQ_2_SEQ_LM", num_virtual_tokens=20, token_dim=768,
|
||||
num_transformer_submodules=1, num_attention_heads=12, num_layers=12,
|
||||
encoder_reparameterization_type="MLP", encoder_hidden_size=768
|
||||
)
|
||||
@@ -70,11 +70,11 @@ class PromptEncoder(torch.nn.Module):
|
||||
(:class:`~torch.nn.Sequential`): The MLP head of the prompt encoder if `inference_mode=False`. lstm_head
|
||||
(:class:`~torch.nn.LSTM`):
|
||||
The LSTM head of the prompt encoder if `inference_mode=False` and `encoder_reparameterization_type="LSTM"`.
|
||||
token_dim (:obj:`int`): The hidden embedding dimension of the base transformer model. input_size (:obj:`int`):
|
||||
The input size of the prompt encoder. output_size (:obj:`int`): The output size of the prompt encoder.
|
||||
hidden_size (:obj:`int`): The hidden size of the prompt encoder. total_virtual_tokens (:obj:`int`): The total
|
||||
token_dim (`int`): The hidden embedding dimension of the base transformer model. input_size (`int`):
|
||||
The input size of the prompt encoder. output_size (`int`): The output size of the prompt encoder.
|
||||
hidden_size (`int`): The hidden size of the prompt encoder. total_virtual_tokens (`int`): The total
|
||||
number of virtual tokens of the prompt encoder. encoder_type
|
||||
(:obj:Union[:class:`PromptEncoderReparameterizationType`, :obj:`str`]):
|
||||
(Union[:class:`PromptEncoderReparameterizationType`, `str`]):
|
||||
The encoder type of the prompt encoder.
|
||||
|
||||
|
||||
@@ -3,17 +3,17 @@ from typing import Callable, Optional
|
||||
|
||||
import torch
|
||||
|
||||
from ..utils import PETType, PromptLearningConfig
|
||||
from ..utils import PeftType, PromptLearningConfig
|
||||
|
||||
|
||||
@dataclass
|
||||
class PrefixTuningConfig(PromptLearningConfig):
|
||||
"""
|
||||
This is the configuration class to store the configuration of a :class:`~pet.PrefixEncoder`.
|
||||
This is the configuration class to store the configuration of a :class:`~peft.PrefixEncoder`.
|
||||
|
||||
Args:
|
||||
encoder_hidden_size (:obj: int): The hidden size of the prompt encoder.
|
||||
prefix_projection (:obj: bool): Whether to project the prefix embeddings.
|
||||
encoder_hidden_size ( int): The hidden size of the prompt encoder.
|
||||
prefix_projection ( bool): Whether to project the prefix embeddings.
|
||||
postprocess_past_key_value_function (:
|
||||
obj: Optional[Callable]): The function to postprocess the past key value.
|
||||
"""
|
||||
@@ -32,7 +32,7 @@ class PrefixTuningConfig(PromptLearningConfig):
|
||||
)
|
||||
|
||||
def __post_init__(self):
|
||||
self.pet_type = PETType.PREFIX_TUNING
|
||||
self.peft_type = PeftType.PREFIX_TUNING
|
||||
|
||||
|
||||
# Based on https://github.com/THUDM/P-tuning-v2/blob/main/model/prefix_encoder.py
|
||||
@@ -46,18 +46,18 @@ class PrefixEncoder(torch.nn.Module):
|
||||
|
||||
Example::
|
||||
|
||||
>>> from pet import PrefixEncoder, PrefixTuningConfig >>> config = PrefixTuningConfig(
|
||||
pet_type="PREFIX_TUNING", task_type="SEQ_2_SEQ_LM", num_virtual_tokens=20, token_dim=768,
|
||||
>>> from peft import PrefixEncoder, PrefixTuningConfig >>> config = PrefixTuningConfig(
|
||||
peft_type="PREFIX_TUNING", task_type="SEQ_2_SEQ_LM", num_virtual_tokens=20, token_dim=768,
|
||||
num_transformer_submodules=1, num_attention_heads=12, num_layers=12, encoder_hidden_size=768
|
||||
)
|
||||
>>> prefix_encoder = PrefixEncoder(config)
|
||||
|
||||
|
||||
Attributes:
|
||||
embedding (:obj:`torch.nn.Embedding`): The embedding layer of the prefix encoder. trans
|
||||
(:obj:`torch.nn.Sequential`): The two-layer MLP to transform the prefix embeddings
|
||||
if :obj:`prefix_projection` is :obj:`True`.
|
||||
prefix_projection (:obj:`bool`): Whether to project the prefix embeddings.
|
||||
embedding (`torch.nn.Embedding`): The embedding layer of the prefix encoder. trans
|
||||
(`torch.nn.Sequential`): The two-layer MLP to transform the prefix embeddings
|
||||
if `prefix_projection` is `True`.
|
||||
prefix_projection (`bool`): Whether to project the prefix embeddings.
|
||||
|
||||
Input shape: (batch_size, num_virtual_tokens)
|
||||
|
||||
@@ -5,7 +5,7 @@ from typing import Optional, Union
|
||||
|
||||
import torch
|
||||
|
||||
from ..utils import PETType, PromptLearningConfig
|
||||
from ..utils import PeftType, PromptLearningConfig
|
||||
|
||||
|
||||
class PromptTuningInit(str, enum.Enum):
|
||||
@@ -16,14 +16,13 @@ class PromptTuningInit(str, enum.Enum):
|
||||
@dataclass
|
||||
class PromptTuningConfig(PromptLearningConfig):
|
||||
"""
|
||||
This is the configuration class to store the configuration of a :class:`~pet.PromptEmbedding`.
|
||||
This is the configuration class to store the configuration of a :class:`~peft.PromptEmbedding`.
|
||||
|
||||
Args:
|
||||
prompt_tuning_init (:
|
||||
obj:Union[:class:`PromptTuningInit`, :obj:`str`]): The initialization of the prompt embedding.
|
||||
prompt_tuning_init_text (:obj: Optional[:obj:`str`]): The text to initialize the prompt embedding.
|
||||
prompt_tuning_init (Union[:class:`PromptTuningInit`, `str`]): The initialization of the prompt embedding.
|
||||
prompt_tuning_init_text ( Optional[`str`]): The text to initialize the prompt embedding.
|
||||
Only used if `prompt_tuning_init` is `TEXT`
|
||||
tokenizer_name_or_path (:obj: Optional[:obj:`str`]): The name or path of the tokenizer.
|
||||
tokenizer_name_or_path ( Optional[`str`]): The name or path of the tokenizer.
|
||||
Only used if `prompt_tuning_init` is `TEXT`
|
||||
"""
|
||||
|
||||
@@ -45,7 +44,7 @@ class PromptTuningConfig(PromptLearningConfig):
|
||||
)
|
||||
|
||||
def __post_init__(self):
|
||||
self.pet_type = PETType.PROMPT_TUNING
|
||||
self.peft_type = PeftType.PROMPT_TUNING
|
||||
|
||||
|
||||
class PromptEmbedding(torch.nn.Module):
|
||||
@@ -54,15 +53,15 @@ class PromptEmbedding(torch.nn.Module):
|
||||
|
||||
Args:
|
||||
config (:class:`PromptTuningConfig`): The configuration of the prompt embedding.
|
||||
word_embeddings (:obj:`torch.nn.Module`): The word embeddings of the base transformer model.
|
||||
word_embeddings (`torch.nn.Module`): The word embeddings of the base transformer model.
|
||||
|
||||
Attributes:
|
||||
embedding (:obj:`torch.nn.Embedding`): The embedding layer of the prompt embedding.
|
||||
embedding (`torch.nn.Embedding`): The embedding layer of the prompt embedding.
|
||||
|
||||
Example::
|
||||
|
||||
>>> from pet import PromptEmbedding, PromptTuningConfig >>> config = PromptTuningConfig(
|
||||
pet_type="PROMPT_TUNING", task_type="SEQ_2_SEQ_LM", num_virtual_tokens=20, token_dim=768,
|
||||
>>> from peft import PromptEmbedding, PromptTuningConfig >>> config = PromptTuningConfig(
|
||||
peft_type="PROMPT_TUNING", task_type="SEQ_2_SEQ_LM", num_virtual_tokens=20, token_dim=768,
|
||||
num_transformer_submodules=1, num_attention_heads=12, num_layers=12, prompt_tuning_init="TEXT",
|
||||
prompt_tuning_init_text="Predict if sentiment of this review is positive, negative or neutral",
|
||||
tokenizer_name_or_path="t5-base",
|
||||
@@ -2,6 +2,6 @@
|
||||
# There's no way to ignore "F401 '...' imported but unused" warnings in this
|
||||
# module, but to preserve other warnings. So, don't check this module at all
|
||||
|
||||
from .config import PETConfig, PETType, PromptLearningConfig, TaskType
|
||||
from .other import _set_trainable, bloom_model_postprocess_past_key_value, shift_tokens_right
|
||||
from .save_and_load import get_pet_model_state_dict, pet_model_load_and_dispatch, set_pet_model_state_dict
|
||||
from .config import PeftConfig, PeftType, PromptLearningConfig, TaskType
|
||||
from .other import _set_trainable, bloom_model_postprocess_past_key_value, shift_tokens_right, transpose
|
||||
from .save_and_load import get_peft_model_state_dict, peft_model_load_and_dispatch, set_peft_model_state_dict
|
||||
@@ -3,7 +3,7 @@ from dataclasses import dataclass, field
|
||||
from typing import Optional, Union
|
||||
|
||||
|
||||
class PETType(str, enum.Enum):
|
||||
class PeftType(str, enum.Enum):
|
||||
PROMPT_TUNING = "PROMPT_TUNING"
|
||||
P_TUNING = "P_TUNING"
|
||||
PREFIX_TUNING = "PREFIX_TUNING"
|
||||
@@ -18,33 +18,33 @@ class TaskType(str, enum.Enum):
|
||||
|
||||
|
||||
@dataclass
|
||||
class PETConfig:
|
||||
class PeftConfig:
|
||||
"""
|
||||
This is the base configuration class to store the configuration of a :class:`~pet.PETModel`.
|
||||
This is the base configuration class to store the configuration of a :class:`~peft.PeftModel`.
|
||||
|
||||
Args:
|
||||
pet_type (:obj:Union[:class:`~pet.utils.config.PETType`, :obj:`str`]): The type of PET method to use.
|
||||
task_type (:obj:Union[:class:`~pet.utils.config.TaskType`, :obj:`str`]): The type of task to perform.
|
||||
inference_mode (:obj:`bool`, defaults to :obj:`False`): Whether to use the PET model in inference mode.
|
||||
peft_type (Union[:class:`~peft.utils.config.PeftType`, `str`]): The type of Peft method to use.
|
||||
task_type (Union[:class:`~peft.utils.config.TaskType`, `str`]): The type of task to perform.
|
||||
inference_mode (`bool`, defaults to `False`): Whether to use the Peft model in inference mode.
|
||||
"""
|
||||
|
||||
pet_type: Union[str, PETType] = field(default=None, metadata={"help": "PET type"})
|
||||
peft_type: Union[str, PeftType] = field(default=None, metadata={"help": "Peft type"})
|
||||
task_type: Union[str, TaskType] = field(default=None, metadata={"help": "Task type"})
|
||||
inference_mode: bool = field(default=False, metadata={"help": "Whether to use inference mode"})
|
||||
|
||||
|
||||
@dataclass
|
||||
class PromptLearningConfig(PETConfig):
|
||||
class PromptLearningConfig(PeftConfig):
|
||||
"""
|
||||
This is the base configuration class to store the configuration of a :obj:Union[:class:`~pet.PrefixTuning`,
|
||||
:class:`~pet.PromptEncoder`, :class:`~pet.PromptTuning`].
|
||||
This is the base configuration class to store the configuration of a Union[:class:`~peft.PrefixTuning`,
|
||||
:class:`~peft.PromptEncoder`, :class:`~peft.PromptTuning`].
|
||||
|
||||
Args:
|
||||
num_virtual_tokens (:obj:`int`): The number of virtual tokens to use.
|
||||
token_dim (:obj:`int`): The hidden embedding dimension of the base transformer model.
|
||||
num_transformer_submodules (:obj:`int`): The number of transformer submodules in the base transformer model.
|
||||
num_attention_heads (:obj:`int`): The number of attention heads in the base transformer model.
|
||||
num_layers (:obj:`int`): The number of layers in the base transformer model.
|
||||
num_virtual_tokens (`int`): The number of virtual tokens to use.
|
||||
token_dim (`int`): The hidden embedding dimension of the base transformer model.
|
||||
num_transformer_submodules (`int`): The number of transformer submodules in the base transformer model.
|
||||
num_attention_heads (`int`): The number of attention heads in the base transformer model.
|
||||
num_layers (`int`): The number of layers in the base transformer model.
|
||||
"""
|
||||
|
||||
num_virtual_tokens: int = field(default=None, metadata={"help": "Number of virtual tokens"})
|
||||
@@ -21,9 +21,9 @@ def shift_tokens_right(input_ids: torch.Tensor, pad_token_id: int, decoder_start
|
||||
Shift input ids one token to the right.
|
||||
|
||||
Args:
|
||||
input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): input ids
|
||||
pad_token_id (:obj:`int`): The id of the `padding` token.
|
||||
decoder_start_token_id (:obj:`int`): The id of the `start` token.
|
||||
input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): input ids
|
||||
pad_token_id (`int`): The id of the `padding` token.
|
||||
decoder_start_token_id (`int`): The id of the `start` token.
|
||||
"""
|
||||
shifted_input_ids = input_ids.new_zeros(input_ids.shape)
|
||||
shifted_input_ids[:, 1:] = input_ids[:, :-1].clone()
|
||||
@@ -77,3 +77,7 @@ def fsdp_auto_wrap_policy(model):
|
||||
|
||||
auto_wrap_policy = functools.partial(_or_policy, policies=[lambda_policy, transformer_wrap_policy])
|
||||
return auto_wrap_policy
|
||||
|
||||
|
||||
def transpose(weight, fan_in_fan_out):
|
||||
return weight.T if fan_in_fan_out else weight
|
||||
@@ -1,12 +1,12 @@
|
||||
from .config import PETType
|
||||
from .config import PeftType
|
||||
|
||||
|
||||
def get_pet_model_state_dict(model, state_dict=None):
|
||||
def get_peft_model_state_dict(model, state_dict=None):
|
||||
"""
|
||||
Get the state dict of the PET model.
|
||||
Get the state dict of the Peft model.
|
||||
|
||||
Args:
|
||||
model (:obj:`PETModel`): The PET model. When using torch.nn.DistributedDataParallel, DeepSpeed or FSDP,
|
||||
model (`PeftModel`): The Peft model. When using torch.nn.DistributedDataParallel, DeepSpeed or FSDP,
|
||||
the model should be teh underlying model/unwrapped model (i.e. model.module).
|
||||
state_dict (:
|
||||
obj:`dict`, `optional`): The state dict of the model. If not provided, the state dict of the model
|
||||
@@ -14,11 +14,11 @@ def get_pet_model_state_dict(model, state_dict=None):
|
||||
"""
|
||||
if state_dict is None:
|
||||
state_dict = model.state_dict()
|
||||
if model.pet_config.pet_type == PETType.LORA:
|
||||
# to_return = lora_state_dict(model, bias=model.pet_config.bias)
|
||||
if model.peft_config.peft_type == PeftType.LORA:
|
||||
# to_return = lora_state_dict(model, bias=model.peft_config.bias)
|
||||
# adapted from `https://github.com/microsoft/LoRA/blob/main/loralib/utils.py`
|
||||
# to directly with the state dict which is necessary when using DeepSpeed or FSDP
|
||||
bias = model.pet_config.bias
|
||||
bias = model.peft_config.bias
|
||||
if bias == "none":
|
||||
to_return = {k: state_dict[k] for k in state_dict if "lora_" in k}
|
||||
elif bias == "all":
|
||||
@@ -44,31 +44,31 @@ def get_pet_model_state_dict(model, state_dict=None):
|
||||
return to_return
|
||||
|
||||
|
||||
def set_pet_model_state_dict(model, pet_model_state_dict):
|
||||
def set_peft_model_state_dict(model, peft_model_state_dict):
|
||||
"""
|
||||
Set the state dict of the PET model.
|
||||
Set the state dict of the Peft model.
|
||||
|
||||
Args:
|
||||
model (:obj:`PETModel`): The PET model.
|
||||
pet_model_state_dict (:obj:`dict`): The state dict of the PET model.
|
||||
model (`PeftModel`): The Peft model.
|
||||
peft_model_state_dict (`dict`): The state dict of the Peft model.
|
||||
"""
|
||||
|
||||
model.load_state_dict(pet_model_state_dict, strict=False)
|
||||
if model.pet_config.pet_type != PETType.LORA:
|
||||
model.load_state_dict(peft_model_state_dict, strict=False)
|
||||
if model.peft_config.peft_type != PeftType.LORA:
|
||||
model.prompt_encoder.embedding.load_state_dict(
|
||||
{"weight": pet_model_state_dict["prompt_embeddings"]}, strict=True
|
||||
{"weight": peft_model_state_dict["prompt_embeddings"]}, strict=True
|
||||
)
|
||||
return model
|
||||
|
||||
|
||||
def pet_model_load_and_dispatch(model, pet_model_state_dict, pet_config, max_memory=None):
|
||||
def peft_model_load_and_dispatch(model, peft_model_state_dict, peft_config, max_memory=None):
|
||||
"""
|
||||
Load the PET model state dict and dispatch the model to the correct device.
|
||||
Load the Peft model state dict and dispatch the model to the correct device.
|
||||
|
||||
Args:
|
||||
model (:obj:`PETModel`): The Pre-trained base model which has already been sharded and dispatched
|
||||
model (`PeftModel`): The Pre-trained base model which has already been sharded and dispatched
|
||||
using `accelerate` functionalities.
|
||||
pet_model_state_dict (:obj:`dict`): The state dict of the PET model.
|
||||
peft_model_state_dict (`dict`): The state dict of the Peft model.
|
||||
max_memory (`Dict`, *optional*):
|
||||
A dictionary device identifier to maximum memory. Will default to the maximum memory available for each GPU
|
||||
and the available CPU RAM if unset.
|
||||
@@ -76,16 +76,16 @@ def pet_model_load_and_dispatch(model, pet_model_state_dict, pet_config, max_mem
|
||||
from accelerate import dispatch_model, infer_auto_device_map
|
||||
from accelerate.hooks import AlignDevicesHook, add_hook_to_module, remove_hook_from_submodules
|
||||
|
||||
from ..mapping import get_pet_model
|
||||
from ..mapping import get_peft_model
|
||||
|
||||
remove_hook_from_submodules(model)
|
||||
model = get_pet_model(model, pet_config)
|
||||
model = get_peft_model(model, peft_config)
|
||||
model.print_trainable_parameters()
|
||||
set_pet_model_state_dict(model, pet_model_state_dict)
|
||||
set_peft_model_state_dict(model, peft_model_state_dict)
|
||||
device_map = infer_auto_device_map(model, max_memory=max_memory, no_split_module_classes=model._no_split_modules)
|
||||
model = dispatch_model(model, device_map=device_map)
|
||||
hook = AlignDevicesHook(io_same_device=True)
|
||||
if model.pet_config.pet_type == PETType.LORA:
|
||||
if model.peft_config.peft_type == PeftType.LORA:
|
||||
add_hook_to_module(model.base_model.model, hook)
|
||||
else:
|
||||
remove_hook_from_submodules(model.prompt_encoder)
|
||||
@@ -1,126 +0,0 @@
|
||||
from .pet_model import (
|
||||
PETModelForCausalLM,
|
||||
PETModelForSeq2SeqLM,
|
||||
PETModelForSequenceClassification,
|
||||
PETModelForTokenClassification,
|
||||
)
|
||||
from .tuners import LoRAConfig, PrefixTuningConfig, PromptEncoderConfig, PromptTuningConfig
|
||||
from .utils import PETType
|
||||
|
||||
|
||||
MODEL_TYPE_TO_PET_MODEL_MAPPING = {
|
||||
"SEQ_CLS": PETModelForSequenceClassification,
|
||||
"SEQ_2_SEQ_LM": PETModelForSeq2SeqLM,
|
||||
"CAUSAL_LM": PETModelForCausalLM,
|
||||
"TOKEN_CLS": PETModelForTokenClassification,
|
||||
}
|
||||
|
||||
PET_TYPE_TO_CONFIG_MAPPING = {
|
||||
"PROMPT_TUNING": PromptTuningConfig,
|
||||
"PREFIX_TUNING": PrefixTuningConfig,
|
||||
"P_TUNING": PromptEncoderConfig,
|
||||
"LORA": LoRAConfig,
|
||||
}
|
||||
|
||||
TRANSFORMERS_MODELS_TO_LORA_TARGET_MODULES_MAPPING = {
|
||||
"t5": ["q", "v"],
|
||||
"mt5": ["q", "v"],
|
||||
"bart": ["q_proj", "v_proj"],
|
||||
"gpt2": ["c_attn"],
|
||||
"bloom": ["query_key_value"],
|
||||
"opt": ["q_proj", "v_proj"],
|
||||
"gptj": ["q_proj", "v_proj"],
|
||||
"gpt_neox": ["query_key_value"],
|
||||
"gpt_neo": ["q_proj", "v_proj"],
|
||||
"bert": ["query", "value"],
|
||||
"roberta": ["query", "value"],
|
||||
"xlm-roberta": ["query", "value"],
|
||||
"electra": ["query", "value"],
|
||||
"deberta-v2": ["query_proj", "value_proj"],
|
||||
"deberta": ["in_proj"],
|
||||
"layoutlm": ["query", "value"],
|
||||
}
|
||||
|
||||
|
||||
def get_pet_config(config_dict):
|
||||
"""
|
||||
Returns a PET config object from a dictionary.
|
||||
|
||||
Args:
|
||||
config_dict (:obj:`Dict[str, Any]`):
|
||||
"""
|
||||
|
||||
return PET_TYPE_TO_CONFIG_MAPPING[config_dict["pet_type"]](**config_dict)
|
||||
|
||||
|
||||
def _prepare_prompt_learning_config(pet_config, model_config):
|
||||
if pet_config.num_layers is None:
|
||||
if "num_hidden_layers" in model_config:
|
||||
num_layers = model_config["num_hidden_layers"]
|
||||
elif "num_layers" in model_config:
|
||||
num_layers = model_config["num_layers"]
|
||||
elif "n_layer" in model_config:
|
||||
num_layers = model_config["n_layer"]
|
||||
else:
|
||||
raise ValueError("Please specify `num_layers` in `pet_config`")
|
||||
pet_config.num_layers = num_layers
|
||||
|
||||
if pet_config.token_dim is None:
|
||||
if "hidden_size" in model_config:
|
||||
token_dim = model_config["hidden_size"]
|
||||
elif "n_embd" in model_config:
|
||||
token_dim = model_config["n_embd"]
|
||||
elif "d_model" in model_config:
|
||||
token_dim = model_config["d_model"]
|
||||
else:
|
||||
raise ValueError("Please specify `token_dim` in `pet_config`")
|
||||
pet_config.token_dim = token_dim
|
||||
|
||||
if pet_config.num_attention_heads is None:
|
||||
if "num_attention_heads" in model_config:
|
||||
num_attention_heads = model_config["num_attention_heads"]
|
||||
elif "n_head" in model_config:
|
||||
num_attention_heads = model_config["n_head"]
|
||||
elif "num_heads" in model_config:
|
||||
num_attention_heads = model_config["num_heads"]
|
||||
elif "encoder_attention_heads" in model_config:
|
||||
num_attention_heads = model_config["encoder_attention_heads"]
|
||||
else:
|
||||
raise ValueError("Please specify `num_attention_heads` in `pet_config`")
|
||||
pet_config.num_attention_heads = num_attention_heads
|
||||
|
||||
if getattr(pet_config, "encoder_hidden_size", None) is None:
|
||||
setattr(pet_config, "encoder_hidden_size", token_dim)
|
||||
|
||||
return pet_config
|
||||
|
||||
|
||||
def _prepare_lora_config(pet_config, model_config):
|
||||
if pet_config.target_modules is None:
|
||||
if model_config["model_type"] not in TRANSFORMERS_MODELS_TO_LORA_TARGET_MODULES_MAPPING:
|
||||
raise ValueError("Please specify `target_modules` in `pet_config`")
|
||||
pet_config.target_modules = TRANSFORMERS_MODELS_TO_LORA_TARGET_MODULES_MAPPING[model_config["model_type"]]
|
||||
if len(pet_config.target_modules) == 1:
|
||||
pet_config.fan_in_fan_out = True
|
||||
pet_config.enable_lora = [True, False, True]
|
||||
if pet_config.inference_mode:
|
||||
pet_config.merge_weights = True
|
||||
return pet_config
|
||||
|
||||
|
||||
def get_pet_model(model, pet_config):
|
||||
"""
|
||||
Returns a PET model object from a model and a config.
|
||||
|
||||
Args:
|
||||
model (:obj:`transformers.PreTrainedModel`):
|
||||
pet_config (:obj:`transformers.PETConfig`):
|
||||
"""
|
||||
|
||||
model_config = model.config.to_dict()
|
||||
if pet_config.pet_type != PETType.LORA:
|
||||
pet_config = _prepare_prompt_learning_config(pet_config, model_config)
|
||||
else:
|
||||
pet_config = _prepare_lora_config(pet_config, model_config)
|
||||
|
||||
return MODEL_TYPE_TO_PET_MODEL_MAPPING[pet_config.task_type](model, pet_config)
|
||||
Reference in New Issue
Block a user