wassname/alignment-handbook
1
0
Fork 0
mirror of https://github.com/wassname/alignment-handbook.git synced 2026-08-11 11:14:37 +08:00
Code Issues Packages Projects Releases Wiki Activity

Adding continued_pretraining task (#131)

Browse Source 
* add continued pretraining script

* simplify config; add dataset_config option

* add ds configs in data mixer creator

* use extended sftconfig

* add option to avoid setting chat template

* fix data_configs bug

* add continued pretraining info

* add gpt2-nl recipe for continued pretraining example

* add final newline

* make style

* Update README.md

Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>

* Update README.md

Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>

* Update recipes/gpt2-nl/README.md

Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>

* rename continued pretraining to cpt

* improve README

---------

Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>
This commit is contained in:
Bram Vanroy
2024-03-14 15:15:23 +01:00
committed by GitHub
co-authored by lewtun
parent a9b8a50a27
commit 595023faa4
12 changed files with 415 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/run_sft.py
+1 -1
View File
@@ -85,7 +85,7 @@ def main():
###############
# Load datasets
###############
raw_datasets = get_datasets(data_args, splits=data_args.dataset_splits)
raw_datasets = get_datasets(data_args, splits=data_args.dataset_splits, configs=data_args.dataset_configs)
logger.info(
f"Training on the following datasets and their proportions: {[split + ' : ' + str(dset.num_rows) for split, dset in raw_datasets.items()]}"
)