mirror of https://github.com/wassname/multifit.git synced 2026-08-21 11:18:10 +08:00

T

Piotr Czapla 70c74a1cc5 Refactor tokenization

So that we can instantiate tokenization before we know what dataset we want to use it on. Previously it was tidly copuled.

2019-10-15 04:17:54 +02:00

fastai_contrib

Add OpenFIle preproc to SentencePiece preproc

2019-10-14 17:20:26 +02:00

notebooks

fix spelling eeror in finetune_lm

2019-09-08 08:43:15 +02:00

sotabench

Refactor tokenization

2019-10-15 04:17:54 +02:00

ulmfit

Refactor tokenization

2019-10-15 04:17:54 +02:00

.gitignore

Initial import from fastaiv1

2018-11-08 20:33:10 +01:00

get_preprocessed_wikis.sh

Download pretrained models

2019-02-15 00:02:04 +01:00

prepare_cls.py

jp -> ja

2019-02-13 10:50:01 -03:00

prepare_imdb.sh

New bolerplate code

2019-09-07 16:19:09 +02:00

prepare_mldoc.py

Add code to test & train mldoc classifier

2019-02-10 09:52:58 +01:00

prepare_wiki-en.sh

Add script that fetches and prepres WT-2 WT-103

2018-11-15 23:40:50 +01:00

prepare_wiki.sh

prepare_wiki.sh made executable

2019-01-13 17:30:35 +01:00

prepare_xnli.py

Add prepare_xnli.py

2018-11-19 06:50:50 +00:00

prepare_xnli.sh

Minor tweaks.

2018-11-15 23:41:02 +01:00

README.md

Refactor tokenization

2019-10-15 04:17:54 +02:00

requirements.txt

Remove cupy from requirements.txt

2019-10-14 13:11:52 +02:00

setup.py

Add sotabench scripts

2019-10-15 00:09:18 +02:00

split-cls.py

Add option to remove duplicates

2019-05-14 02:30:25 +02:00

README.md

Retraining a multifit model from wikipedia

The whole training process from wikipedia to mldoc can be run as follows:

python -m ulmfit new multifit_fp16 \
    pretrain-lm train- data/wiki/de-100 - \
    finetune-lm train- data/mldoc/de-1 - \
    classifier train- data/mldoc/de-1

You can evaulate any model with the following command:

python -m ulmfit load data/mldoc/de-1/models/fsp15k/multfit_fp16 classifier validate data/mldoc/de-1

python -m ulmfit new multifit_fp16_nl3 pretrain-lm train- data/wiki/wikitext-103