wassname/peft
1
0
Fork 0
mirror of https://github.com/wassname/peft.git synced 2026-09-10 12:20:21 +08:00
Code Issues Packages Projects Releases Wiki Activity

Update README.md

Browse Source 
add one caveat situation for using LoRA + ZeRO 3 setting.
This commit is contained in:
dumpmemory
2023-03-09 22:01:16 +08:00
committed by GitHub
parent 50aaf99da7
commit e85c18f019
1 changed files with 1 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+1
View File
@@ -344,6 +344,7 @@ any GPU memory savings. Please refer issue [[FSDP] FSDP with CPU offload consume
`P_TUNING`/`PROMPT_TUNING` appends soft prompt embeddings to `input_embeds` to create
new `input_embeds` to be given to the model. Therefore, `generate` doesn't support this yet.
4. For causal_language_modeling with LoRA, like GPT2 models, we might need to set zero3_init_flag=false in accelerate config.yaml. The related issue is [[BUG] memory leak under zero.Init](https://github.com/microsoft/DeepSpeed/issues/2637)
## Backlog:
1. Explore and possibly integrate `(IA)^3`
2. Add tests