mirror of
https://github.com/wassname/peft.git
synced 2026-09-25 13:50:20 +08:00
69 lines
2.0 KiB
Markdown
69 lines
2.0 KiB
Markdown
# 🤗 PET
|
|
Parameter-Efficient Tuning. Intergrated with 🤗 Accelerate to scale seamlessly to large models using PyTorch FSDP.
|
|
|
|
Supported methods:
|
|
|
|
1. LoRA
|
|
2. Prefix Tuning
|
|
3. P-Tuning
|
|
4. Prompt Tuning
|
|
|
|
## Getting started
|
|
|
|
```python
|
|
from transformers import AutoModelForSeq2SeqLM
|
|
from pet import get_pet_config,get_pet_model
|
|
model_name_or_path = "bigscience/mt0-large"
|
|
tokenizer_name_or_path = "bigscience/mt0-large"
|
|
|
|
config = {
|
|
"pet_type":"LORA",
|
|
"task_type":"SEQ_2_SEQ_LM",
|
|
"r": 8,
|
|
"lora_alpha": 32,
|
|
"lora_dropout": 0.1
|
|
}
|
|
pet_config = get_pet_config(config)
|
|
|
|
model = AutoModelForSeq2SeqLM.from_pretrained(model_name_or_path)
|
|
model = get_pet_model(model, pet_config)
|
|
model.print_trainable_parameters()
|
|
# output:
|
|
```
|
|
|
|
## Models support matrix
|
|
|
|
### Sequence Classification
|
|
| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
|
|
| --------- | ---- | ---- | ---- | ---- |
|
|
| BERT | ✅ | ✅ | ✅ | ✅ |
|
|
| RoBERTa | ✅ | ✅ | ✅ | ✅ |
|
|
| GPT-2 | ✅ | ✅ | ✅ | ✅ |
|
|
| Bloom | ✅ | ✅ | ✅ | ✅ |
|
|
| OPT | ✅ | ✅ | ✅ | ✅ |
|
|
| GPT-Neo | ✅ | ✅ | ✅ | ✅ |
|
|
| GPT-J | ✅ | ✅ | ✅ | ✅ |
|
|
| Deberta | ✅ | | | |
|
|
| Deberta-v2 | ✅ | | | |
|
|
|
|
### Causal Language Modeling
|
|
| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
|
|
| --------- | ---- | ---- | ---- | ---- |
|
|
| GPT-2 | ✅ | ✅ | ✅ | ✅ |
|
|
| Bloom | ✅ | ✅ | ✅ | ✅ |
|
|
| OPT | ✅ | ✅ | ✅ | ✅ |
|
|
| GPT-Neo | ✅ | ✅ | ✅ | ✅ |
|
|
| GPT-J | ✅ | ✅ | ✅ | ✅ |
|
|
|
|
### Conditional Generation
|
|
| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
|
|
| --------- | ---- | ---- | ---- | ---- |
|
|
| T5 | ✅ | ✅ | ✅ | ✅ |
|
|
| BART | ✅ | ✅ | ✅ | ✅ |
|
|
|
|
|
|
## Caveats:
|
|
1. Doesn't work currently with DeeSpeed ZeRO Stage-3. Extending support with DeeSpeed ZeRO Stage-3 is in backlog.
|
|
|
|
|