wassname/peft
1
0
Fork 0
mirror of https://github.com/wassname/peft.git synced 2026-09-12 12:40:18 +08:00
Code Issues Packages Projects Releases Wiki Activity

refactor, lora support and utils for saving and loading

Browse Source
This commit is contained in:
Sourab Mangrulkar
2022-11-30 18:22:12 +05:30
parent 23aecc4f69
commit 751baf8aa7
9 changed files with 259 additions and 111 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+25 -25
View File
@@ -3,40 +3,40 @@ Parameter-Efficient Tuning. Intergrated with 🤗 Accelerate to scale seamlessly
Supported methods:
1. Prefix Tuning
2. P-Tuning
3. Prompt Tuning
4. LoRA [in backlog]
1. LoRA
2. Prefix Tuning
3. P-Tuning
4. Prompt Tuning
## Models support matrix
### Sequence Classification
| | Prefix Tuning | P-Tuning | Prompt Tuning | LoRA |
| --------- | ---- | ---- | ---- | ---- |
| BERT | ✅ | ✅ | ✅ | |
| RoBERTa | ✅ | ✅ | ✅ | |
| GPT-2 | ✅ | ✅ | ✅ | |
| Bloom | ✅ | ✅ | ✅ | |
| OPT | ✅ | ✅ | ✅ | |
| GPT-Neo | ✅ | ✅ | ✅ | |
| GPT-J | ✅ | ✅ | ✅ | |
| Deberta | | | | |
| Deberta-v2 | | | | |
| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
| --------- | ---- | ---- | ---- | ---- |
| BERT | ✅ | ✅ | ✅ | ✅ |
| RoBERTa | ✅ | ✅ | ✅ | ✅ |
| GPT-2 | ✅ | ✅ | ✅ | ✅ |
| Bloom | ✅ | ✅ | ✅ | ✅ |
| OPT | ✅ | ✅ | ✅ | ✅ |
| GPT-Neo | ✅ | ✅ | ✅ | ✅ |
| GPT-J | ✅ | ✅ | ✅ | ✅ |
| Deberta | ✅ | | | |
| Deberta-v2 | ✅ | | | |
### Causal Language Modeling
| | Prefix Tuning | P-Tuning | Prompt Tuning | LoRA |
| --------- | ---- | ---- | ---- | ---- |
| GPT-2 | ✅ | ✅ | ✅ | |
| Bloom | ✅ | ✅ | ✅ | |
| OPT | ✅ | ✅ | ✅ | |
| GPT-Neo | ✅ | ✅ | ✅ | |
| GPT-J | ✅ | ✅ | ✅ | |
| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
| --------- | ---- | ---- | ---- | ---- |
| GPT-2 | ✅ | ✅ | ✅ | ✅ |
| Bloom | ✅ | ✅ | ✅ | ✅ |
| OPT | ✅ | ✅ | ✅ | ✅ |
| GPT-Neo | ✅ | ✅ | ✅ | ✅ |
| GPT-J | ✅ | ✅ | ✅ | ✅ |
### Conditional Generation
| | Prefix Tuning | P-Tuning | Prompt Tuning | LoRA |
| Model | LoRA | Prefix Tuning | P-Tuning | Prompt Tuning |
| --------- | ---- | ---- | ---- | ---- |
| T5 | ✅ | ✅ | ✅ | |
| BART | ✅ | ✅ | ✅ | |
| T5 | ✅ | ✅ | ✅ | ✅ |
| BART | ✅ | ✅ | ✅ | ✅ |
## Caveats: