wassname/peft
1
0
Fork 0
mirror of https://github.com/wassname/peft.git synced 2026-09-09 11:28:32 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #167 from dumpmemory/update-readme-patch-for-zero-init

Browse Source 
Update README.md
This commit is contained in:
Sourab Mangrulkar
2023-03-15 22:33:42 +05:30
committed by GitHub
parent df0e1fb592 354bea8719
commit 64f63a7df2
1 changed files with 1 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+1
View File
@@ -345,6 +345,7 @@ any GPU memory savings. Please refer issue [[FSDP] FSDP with CPU offload consume
`P_TUNING`/`PROMPT_TUNING` appends soft prompt embeddings to `input_embeds` to create
new `input_embeds` to be given to the model. Therefore, `generate` doesn't support this yet.
4. When using ZeRO3 with zero3_init_flag=True, if you find the gpu memory increase with training steps. we might need to set zero3_init_flag=false in accelerate config.yaml. The related issue is [[BUG] memory leak under zero.Init](https://github.com/microsoft/DeepSpeed/issues/2637)
## Backlog:
1. Explore and possibly integrate `(IA)^3`
2. Add tests