wassname/alignment-handbook
1
0
Fork 0
mirror of https://github.com/wassname/alignment-handbook.git synced 2026-07-25 13:00:12 +08:00
Code Issues Packages Projects Releases Wiki Activity

Check that default_chat_template is also None (#83)

Browse Source 
* Check that `default_chat_template` is also None before overwriting chat template

* add unit test to `get_tokenizer` to ensure default behaviour of chat template is not changed

---------

Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>
This commit is contained in:
Nathan Azrak
2024-01-08 17:54:23 +11:00
committed by GitHub
co-authored by lewtun
parent 98fe28fb14
commit c69ae4b8a5
2 changed files with 15 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/alignment/model_utils.py
+1 -1
View File
@@ -73,7 +73,7 @@ def get_tokenizer(model_args: ModelArguments, data_args: DataArguments) -> PreTr
if data_args.chat_template is not None:
tokenizer.chat_template = data_args.chat_template
elif tokenizer.chat_template is None:
elif tokenizer.chat_template is None and tokenizer.default_chat_template is None:
tokenizer.chat_template = DEFAULT_CHAT_TEMPLATE
return tokenizer