[fix] new code complete answer and update readme for clarity

model/supervised_finetuning/README.md

@@ -23,7 +23,37 @@ open-asisstant dataset are available it will be added here.
 
 ## Model
 
-TBD
+Normally you should be able to add new models in configs/config.yml
+
+```
+your-model-name:
+  learning_rate: 2e-6
+  model_name: <huggingface model name>
+  weight_decay: 0.01
+  max_length: 812
+  warmup_steps: 600
+  gradient_checkpointing: false
+  gradient_accumulation_steps: 5
+  per_device_train_batch_size: 4
+  per_device_eval_batch_size: 4
+```
+
+```
+python trainer.py --configs defaults your-model-name
+```
+
+However, if the model of your choice doesn't have pad_token, eos_token, sep_token, you have to update utils.py `get_tokenizer` to use the right token.
+
+
+## Deepspeed support
+
+You can edit the configs/zero_config.json and use any stage you wish. The current config uses zero-stage 3. For more details on how to setup the config checkout [this page](https://www.deepspeed.ai/tutorials/zero/)
+
+Once you are satisfy with your deepzero config, you can add --deepspeed flag at the end to trigger deepspeed
+
+```
+python trainer.py --configs defaults your-model-name --deepspeed
+```
 
 ## Results
 

model/supervised_finetuning/custom_datasets/prompt_dialogue.py

@@ -30,17 +30,16 @@ class PromptGeneratedDataset(Dataset):
         answer = ""
         self.pairs = []
         with open(chat_dialogue, "r") as f:
-            corpus = f.read().split('<|endoftext|>')
+            corpus = f.read().split("<|endoftext|>")
             for dialogue in corpus:
                 dialogue = dialogue.strip()
-                if 'Rosey:' in dialogue:
-                    user, bot = dialogue.split('Rosey:', maxsplit=1)
+                if "Rosey:" in dialogue:
+                    user, bot = dialogue.split("Rosey:", maxsplit=1)
                     question = user.split(":", maxsplit=1)[1].strip()
                     answer = bot.strip()
                     if len(answer) and len(question):
                         self.pairs.append((question, answer))
 
-
         if len(question) > 0 and len(answer) > 0:
             self.pairs.append((question, answer))