From d85bd942414c27d6b5f72b2c36d99bebb276610f Mon Sep 17 00:00:00 2001 From: Piotr Czapla Date: Wed, 6 Nov 2019 10:45:18 +0100 Subject: [PATCH] README + requirements --- README.md | 47 +++++++++++++++++++++++++++++++---------------- requirements.txt | 5 +++-- 2 files changed, 34 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 374dd07..9d2c850 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,35 @@ -# Retraining a multifit model from wikipedia +# MultiFiT: Efficient Multi-lingual Language Model Fine-tuning +Code to reproduce the paper "[MultiFiT: Efficient Multi-lingual Language Model Fine-tuning](https://arxiv.org/abs/1909.04761)". -The whole training process from wikipedia to mldoc can be run as follows: -```bash -python -m ulmfit new multifit_fp16 \ - pretrain-lm train- data/wiki/de-100 - \ - finetune-lm train- data/mldoc/de-1 - \ - classifier train- data/mldoc/de-1 +It is a small framework on top of fastai v1.0; the code is compatible with v1.0.47 up to v1.0.59 (the current as of 2019.11.03). The results between fastai versions may differ due to optimizations added to fast.ai. Our models were trained using 1.0.47. + +The framework was rewritten to make it easier to use with the newest fastai; the original code is still available in the ulmfit-multilingual branch. + +We released 7 language models trained on corresponding Wikipedia dumps: + - de_multifit_paper_version + - es_multifit_paper_version + - fr_multifit_paper_version + - it_multifit_paper_version + - ja_multifit_paper_version + - ru_multifit_paper_version + - zh_multifit_paper_version + +Here is an example notebook that shows how this pretarined models can be used to train model on `cls/ja-music`. + +### Fast.ai Integration +You can use the pretrained models with fastai library as follows: ``` -You can evaulate any model with the following command: -```bash -python -m ulmfit load data/mldoc/de-1/models/fsp15k/multfit_fp16 classifier validate data/mldoc/de-1 -``` - - -```bash -python -m ulmfit new multifit_fp16_nl3 pretrain-lm train- data/wiki/wikitext-103 - +exp = ulmfit.from_pretrained("name of the model") +fa_config = exp.pretrain_lm.tokenizer.get_fastai_config(add_open_file_processor=True) +data_lm = (TextList.from_folder(imdb_path, **fa_config) + .filter_by_folder(include=['train', 'test', 'unsup']) + .split_by_rand_pct(0.1) + .label_for_lm() + .databunch(bs=bs)) +learn = exp.finetune_lm.get_learner(data_lm) +# learn is a preconfigured fastai learner with a pretrained model loaded +learn.fit_one_cycle(10) +learn.save_encoder("enc") +... ``` \ No newline at end of file diff --git a/requirements.txt b/requirements.txt index 7ea83d3..81345c2 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,5 @@ fire>=0.1.3 scikit-learn>=0.20 -sacremoses>=0.0.5 -sentencepiece +sacremoses==0.0.35 +sentencepiece==0.1.83 +fastai >= 1.0.43 \ No newline at end of file