wassname/alignment-handbook
1
0
Fork 0
mirror of https://github.com/wassname/alignment-handbook.git synced 2026-06-27 16:14:07 +08:00
Code Issues Packages Projects Releases Wiki Activity

Fix the logic that causes an issue with philschmid/gemma-tokenizer-chatml tokenizer (#146)

Browse Source 
The `setup_chat_format()` logic should not be applied to philschmid/gemma-tokenizer-chatml tokenizer, otherwise gemma models are trained w/o proper bos, eos tokens.
This commit is contained in:
kykim0
2024-04-09 08:02:21 -07:00
committed by GitHub
parent 8497caeaf1
commit a83b1f617f
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/run_sft.py
+1 -1
View File
@@ -122,7 +122,7 @@ def main():
model = model_args.model_name_or_path
# For ChatML we need to add special tokens and resize the embedding layer
if "<|im_start|>" in tokenizer.chat_template:
if "<|im_start|>" in tokenizer.chat_template and "gemma-tokenizer-chatml" not in tokenizer.name_or_path:
model = AutoModelForCausalLM.from_pretrained(model_args.model_name_or_path, **model_kwargs)
model, tokenizer = setup_chat_format(model, tokenizer)
model_kwargs = None