From e85c18f019760b9a46f98d30e5e6132ba077d8d4 Mon Sep 17 00:00:00 2001
From: dumpmemory <64742282+dumpmemory@users.noreply.github.com>
Date: Thu, 9 Mar 2023 22:01:16 +0800
Subject: [PATCH 1/2] Update README.md

add one caveat situation for using LoRA + ZeRO 3 setting.
---
 README.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/README.md b/README.md
index 93cf3f1..4ceabee 100644
--- a/README.md
+++ b/README.md
@@ -344,6 +344,7 @@ any GPU memory savings. Please refer issue [[FSDP] FSDP with CPU offload consume
 `P_TUNING`/`PROMPT_TUNING` appends soft prompt embeddings to `input_embeds` to create
 new `input_embeds` to be given to the model. Therefore, `generate` doesn't support this yet.
 
+4. For causal_language_modeling with LoRA, like GPT2 models, we might need to set zero3_init_flag=false in accelerate config.yaml. The related issue is [[BUG] memory leak under zero.Init](https://github.com/microsoft/DeepSpeed/issues/2637)
 ## Backlog:
 1. Explore and possibly integrate `(IA)^3`
 2. Add tests

From 354bea87194d2f224b6d2d26a6278b837fa39f0c Mon Sep 17 00:00:00 2001
From: dumpmemory <64742282+dumpmemory@users.noreply.github.com>
Date: Thu, 9 Mar 2023 22:03:40 +0800
Subject: [PATCH 2/2] Update README.md

---
 README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/README.md b/README.md
index 4ceabee..4a31d82 100644
--- a/README.md
+++ b/README.md
@@ -344,7 +344,7 @@ any GPU memory savings. Please refer issue [[FSDP] FSDP with CPU offload consume
 `P_TUNING`/`PROMPT_TUNING` appends soft prompt embeddings to `input_embeds` to create
 new `input_embeds` to be given to the model. Therefore, `generate` doesn't support this yet.
 
-4. For causal_language_modeling with LoRA, like GPT2 models, we might need to set zero3_init_flag=false in accelerate config.yaml. The related issue is [[BUG] memory leak under zero.Init](https://github.com/microsoft/DeepSpeed/issues/2637)
+4. When using ZeRO3 with zero3_init_flag=True, if you find the gpu memory increase with training steps. we might need to set zero3_init_flag=false in accelerate config.yaml. The related issue is [[BUG] memory leak under zero.Init](https://github.com/microsoft/DeepSpeed/issues/2637)
 ## Backlog:
 1. Explore and possibly integrate `(IA)^3`
 2. Add tests