mirror of
https://github.com/wassname/peft.git
synced 2026-09-25 13:50:20 +08:00
46 lines
1.5 KiB
Markdown
46 lines
1.5 KiB
Markdown
# 🤗 PET
|
|
Parameter-Efficient Tuning. Intergrated with 🤗 Accelerate to scale seamlessly to large models using PyTorch FSDP.
|
|
|
|
Supported methods:
|
|
|
|
1. LoRA
|
|
2. Prefix Tuning
|
|
3. P-Tuning
|
|
4. Prompt Tuning
|
|
|
|
## Models support matrix
|
|
|
|
### Sequence Classification
|
|
| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
|
|
| --------- | ---- | ---- | ---- | ---- |
|
|
| BERT | ✅ | ✅ | ✅ | ✅ |
|
|
| RoBERTa | ✅ | ✅ | ✅ | ✅ |
|
|
| GPT-2 | ✅ | ✅ | ✅ | ✅ |
|
|
| Bloom | ✅ | ✅ | ✅ | ✅ |
|
|
| OPT | ✅ | ✅ | ✅ | ✅ |
|
|
| GPT-Neo | ✅ | ✅ | ✅ | ✅ |
|
|
| GPT-J | ✅ | ✅ | ✅ | ✅ |
|
|
| Deberta | ✅ | | | |
|
|
| Deberta-v2 | ✅ | | | |
|
|
|
|
### Causal Language Modeling
|
|
| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
|
|
| --------- | ---- | ---- | ---- | ---- |
|
|
| GPT-2 | ✅ | ✅ | ✅ | ✅ |
|
|
| Bloom | ✅ | ✅ | ✅ | ✅ |
|
|
| OPT | ✅ | ✅ | ✅ | ✅ |
|
|
| GPT-Neo | ✅ | ✅ | ✅ | ✅ |
|
|
| GPT-J | ✅ | ✅ | ✅ | ✅ |
|
|
|
|
### Conditional Generation
|
|
| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
|
|
| --------- | ---- | ---- | ---- | ---- |
|
|
| T5 | ✅ | ✅ | ✅ | ✅ |
|
|
| BART | ✅ | ✅ | ✅ | ✅ |
|
|
|
|
|
|
## Caveats:
|
|
1. Doesn't work currently with DeeSpeed ZeRO Stage-3. Extending support with DeeSpeed ZeRO Stage-3 is in backlog.
|
|
|
|
|