diff --git a/README.md b/README.md index 9d2c850..be34c23 100644 --- a/README.md +++ b/README.md @@ -2,9 +2,12 @@ # MultiFiT: Efficient Multi-lingual Language Model Fine-tuning Code to reproduce the paper "[MultiFiT: Efficient Multi-lingual Language Model Fine-tuning](https://arxiv.org/abs/1909.04761)". -It is a small framework on top of fastai v1.0; the code is compatible with v1.0.47 up to v1.0.59 (the current as of 2019.11.03). The results between fastai versions may differ due to optimizations added to fast.ai. Our models were trained using 1.0.47. +Here is a blog post with an introducing to our paper: http://nlp.fast.ai/classification/2019/09/10/multifit.html -The framework was rewritten to make it easier to use with the newest fastai; the original code is still available in the ulmfit-multilingual branch. +This repository contains a small framework on top of fastai v1.0; the code is compatible with v1.0.47 up to v1.0.59 (the current as of 2019.11.03). +The results between fastai versions may differ due to optimizations added to fastai. Our models were trained using 1.0.47. + +The framework was rewritten to make it easier to use with the newest fastai. We released 7 language models trained on corresponding Wikipedia dumps: - de_multifit_paper_version @@ -14,10 +17,32 @@ We released 7 language models trained on corresponding Wikipedia dumps: - ja_multifit_paper_version - ru_multifit_paper_version - zh_multifit_paper_version - -Here is an example notebook that shows how this pretarined models can be used to train model on `cls/ja-music`. + +To fetch the model just use `ulmfit.from_pretreined` function. +Here are some example notebook showing how to train a classifier using a pretrained models. +- [./notebooks/CLS-JA.ipynb](./notebooks/CLS-JA.ipynb) - example of classifier trained on amazon CLS JA music. +- [./notebooks/MLDoc-JA-multifit_fp16.ipynb](./notebooks/MLDoc-JA-multifit_fp16.ipynb) - example of a faster multifit training using fp16 on MDLDoc. -### Fast.ai Integration +## Results +### MLDoc +Document classification results on MLDoc dataset [Schwenk and Li, 2018](https://arxiv.org/abs/1805.09821) + +| Model | de | es | fr | it | ja | ru | zh | +|----------------|-----------|-----------|-----------|-----------|-----------|-----------|------------| +|LASER | 92.70 | 88.75 | 90.80 | 85.93 | 85.15 | 84.65 | 88.98 | +| MultiBERT | 94.0 | 95.15 | 93.20 | 85.82 | 87.48 | 86.85 | 90.72 | +| MultiFiT | **95.90** | **96.07** | **94.77** | **90.25** | **90.03** | **87.65**| **92.52** | + +### Amazon CLS +Sentiment classification results on CLS dataset [Prettenhofer and Stein, 2010](https://dl.acm.org/citation.cfm?doid=2036264.2036277) + +| | DE | FR | JA | +|----------|-----------------------|-----------------------|----------------------| +| MultiBERT| 86.05 / 84.90 / 82.00 | 86.15 / 86.90 / 86.65 | 80.87 / 82.83 / 79.95| +| MultiFiT | 93.19 / 90.54 / 93.00 | 91.25 / 89.55 / 93.40 | 86.29 / 85.75 / 86.59| + + +## How to use it with fastai v1.0 You can use the pretrained models with fastai library as follows: ``` exp = ulmfit.from_pretrained("name of the model") @@ -32,4 +57,19 @@ learn = exp.finetune_lm.get_learner(data_lm) learn.fit_one_cycle(10) learn.save_encoder("enc") ... +``` + +## Reproducing the results +This repository is a rewrite of the original training scripts so it lacks all the scripts used in the paper. +We are working on a port to fastai v2.0 and then we will be adding the scripts that show how to reproduce the results. +In case you need to use the scripts faster you can access the original scripts [here](https://github.com/n-waves/multifit/tree/ulmfit-multilingual-original-scripts). + +## Citation +``` +@article{Eisenschlos2019MultiFit, + title={MultiFiT: Efficient Multi-lingual Language Model Fine-tuning}, + author={Julian Eisenschlos, Sebastian Ruder, Piotr Czapla, Marcin Kardas, Sylvain Gugger, Jeremy Howard} + journal={Proceedings of EMNLP-IJCNLP 2019}, + year={2019} +} ``` \ No newline at end of file diff --git a/notebooks/CLS-DE.ipynb b/notebooks/CLS-DE.ipynb new file mode 100644 index 0000000..26961fa --- /dev/null +++ b/notebooks/CLS-DE.ipynb @@ -0,0 +1,1567 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "env: CUDA_VISIBLE_DEVICES=1\n" + ] + } + ], + "source": [ + "%env CUDA_VISIBLE_DEVICES=1" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "/home/pczapla/workspace/ulmfit-multilingual\n" + ] + } + ], + "source": [ + "%reload_ext autoreload\n", + "%autoreload 2\n", + "%matplotlib inline\n", + "%cd .." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "from fastai.text import *\n", + "from ulmfit.datasets import ULMFiTDataset, Dataset\n", + "import ulmfit" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "import warnings\n", + "warnings.filterwarnings(\"ignore\") # to ignore pytorch 1.3 warnings triggered by older fastai" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# CLS DE music" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "lang='de'" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [], + "source": [ + "exp = ulmfit.from_pretrained(f'{lang}_multifit_paper_version')" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "scrolled": false + }, + "outputs": [ + { + "data": { + "text/plain": [ + "ULMFiTArchitecture(tokenizer_type='sp', max_vocab=15000, lang='de', emb_sz=400, n_hid=1550, n_layers=4, qrnn=True)" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "exp.replace_(name='multifit_paper_version20')\n", + "exp.arch" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from /home/pczapla/.fastai/models/de_multifit_paper_version to data/cls/de-music/models/sp15k\n" + ] + } + ], + "source": [ + "cls_dataset = exp.arch.dataset(Path(f'data/cls/{lang}-music'), exp.pretrain_lm.tokenizer)" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'de'" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "cls_dataset.lang" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "scrolled": false + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Data lm-notst, trn: 31800, val: 200\n", + "Size of vocabulary: 15000\n", + "First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', \"▁&'\"]\n", + "Data cls, trn: 1800, val: 200\n", + "Data tst, trn: 200, val: 2000\n" + ] + }, + { + "data": { + "text/html": [ + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
texttarget
▁ xxbos ▁ xxfld ▁1 ▁ xxmaj ▁in ▁der ▁ xxmaj ▁ special ▁ xxmaj ▁ edition ▁mit ▁fünf ▁zusätzlich en ▁ xxmaj ▁ track s ▁noch ▁viel ▁besser ▁, ▁als ▁ohne hin ▁schon ▁: ▁ xxmaj ▁alan is ▁ xxmaj ▁mo riss ette s ▁neues ▁ xxmaj ▁album ▁ xxfld ▁2 ▁ xxmaj ▁nach ▁vier ▁ xxmaj ▁jahren ▁ist ▁ xxmaj ▁alan is ▁ xxmaj ▁mo riss ette ▁endlich1
▁ xxbos ▁ xxfld ▁1 ▁und ▁ es ▁geht ▁immer ▁noch ▁schlecht er ▁ xxrep ▁5 ▁. ▁ xxfld ▁2 ▁ xxmaj ▁irgend wi e ▁ist ▁ es ▁bei ▁ xxmaj ▁ ram m stein ▁mittlerweil e ▁ein ▁ xxmaj ▁ phänomen ▁im ▁negative n ▁ xxmaj ▁sinne ▁ - ▁&' ▁ xxmaj ▁reise ▁, ▁ xxmaj ▁reise ▁&' ▁war ▁nach ▁&' ▁ xxmaj ▁herz e leid ▁&' ▁, ▁&'0
▁ xxbos ▁ xxfld ▁1 ▁ xxmaj ▁ tol les ▁ debüt - album ▁ ! ▁ xxfld ▁2 ▁ xxmaj ▁die s ▁ist ▁das ▁1995 ▁erschienen e ▁, ▁erste ▁ xxmaj ▁album ▁der ▁ xxmaj ▁bremer ▁ xxmaj ▁stadt musik anten ▁&' ▁ xxmaj ▁mr . ▁ xxmaj ▁ pres ident ▁&' ▁gewesen ▁. ▁ xxmaj ▁nachdem ▁ ich ▁zuvor ▁schon ▁von ▁den ▁ xxmaj ▁single s ▁&' ▁1
▁ xxbos ▁ xxfld ▁1 ▁ xxmaj ▁hoch mut ▁kommt ▁vor ▁dem ▁ xxmaj ▁ fall ▁ xxfld ▁2 ▁ xxmaj ▁wie ▁lange ▁habe ▁ ich ▁doch ▁ge warte t ▁bis ▁ xxmaj ▁ ti es to ▁seit ▁seiner ▁letzten ▁ xxup ▁cd ▁( ▁ xxmaj ▁in ▁ xxmaj ▁ se arch ▁of ▁ xxmaj ▁ s un ris e ▁ xxmaj ▁ asia ▁) ▁eine ▁neue ▁rau s bringt0
▁ xxbos ▁ xxfld ▁1 ▁5 ▁ xxmaj ▁ sterne ▁ reichen ▁nicht ▁aus ▁ xxfld ▁2 ▁ xxmaj ▁ ig n ite ▁ - ▁ xxmaj ▁ our ▁ xxmaj ▁dar ke st ▁ day s ▁ - ▁cd - re vie w ▁1. in tro ▁( ▁ xxmaj ▁ our ▁ xxmaj ▁dar ke st ▁ xxmaj ▁ day s ▁) ▁ xxmaj ▁das ▁ist ▁kein ▁in tro1
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "cls_dataset.load_clas_databunch(bs=exp.finetune_lm.bs).show_batch()" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "scrolled": false + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Setting LM weights seed seed to 0\n", + "Data lm-notst, trn: 31800, val: 200\n", + "Size of vocabulary: 15000\n", + "First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', \"▁&'\"]\n", + "Training args: {'drop_mult': 0.3, 'true_wd': False, 'wd': 1e-07, 'pretrained': False, 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15, 'tie_weights': True, 'out_bias': True}\n", + "Setting LM training seed seed to 0\n", + "Loading pretrained weights: [PosixPath('/home/pczapla/.fastai/models/de_multifit_paper_version/lm_best'), PosixPath('/home/pczapla/.fastai/models/de_multifit_paper_version/itos')]\n", + "Experiment data/cls/de-music/models/sp15k/multifit_paper_version20\n", + "Fitting using 2 cycle fit schedule\n" + ] + }, + { + "data": { + "text/html": [ + "Total time: 04:09

\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
03.8665413.2333270.43562304:09
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "Total time: 2:00:30

\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
03.2773513.0709620.45536606:01
13.0795632.8937130.47745406:01
22.8975322.7684320.49212506:01
32.8091602.6786150.50337006:01
42.7032922.6287080.50908406:01
52.6731652.5869050.51256406:01
62.5892402.5584730.51622706:01
72.5643562.5461480.51939606:01
82.4691832.5268940.52164806:01
92.4575262.5168670.52309506:01
102.4396642.5055250.52531106:01
112.4252642.5035260.52688606:01
122.3274462.4943360.52829706:01
132.3119022.4977680.52818706:01
142.2656952.4976500.52879106:01
152.2239542.5030260.52818706:01
162.2327972.5055260.52791206:01
172.2040112.5053980.52855306:01
182.1836922.5080690.52866306:01
192.1853232.5083340.52873606:01
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from /home/pczapla/.fastai/models/de_multifit_paper_version to data/cls/de-music/models/sp15k/multifit_paper_version20\n", + "this Learner object self-destroyed - it still exists, but no longer usable\n", + "Saving dump to data/cls/de-music/models/sp15k/multifit_paper_version20/finetuning.json\n", + "Language model saved to data/cls/de-music/models/sp15k/multifit_paper_version20\n" + ] + } + ], + "source": [ + "exp.finetune_lm.train_(cls_dataset, num_epochs=20)" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Loading data/cls/de-music/models/sp15k/multifit_paper_version20/classifier.json\n", + "Loading data/cls/de-music/models/sp15k/multifit_paper_version20/finetuning.json\n", + "ULMFiTClassifier Replacing experiment_path 'None' with 'data/cls/de-music/models/sp15k/multifit_paper_version20\n", + "ULMFiTClassifier Replacing dataset_path 'None' with 'data/cls/de-music\n" + ] + }, + { + "data": { + "text/plain": [ + "ULMFiTFinetuning(seed=0, name='multifit_paper_version20', experiment_path=PosixPath('data/cls/de-music/models/sp15k/multifit_paper_version20'), dataset_path=PosixPath('data/cls/de-music'), num_epochs=20, bs=20, bptt=70, drop_mult=0.3, dropout_values={'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}, label_smoothing_eps=0.0, label_smoothing_eps_norm_by_classes=True, use_adam_08=False, true_wd=False, wd=1e-07, clip=0.12, fp16=False, lr=0.001)" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "exp.load_(cls_dataset.cache_path/exp.pretrain_lm.name).finetune_lm" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "scrolled": false + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/de-music/models/sp15k/multifit_paper_version20 to data/cls/de-music/models/sp15k\n", + "Data lm-notst, trn: 31800, val: 200\n", + "Size of vocabulary: 15000\n", + "First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', \"▁&'\"]\n", + "Data cls, trn: 1800, val: 200\n", + "Data tst, trn: 200, val: 2000\n", + "Using Label smoothing with eps = 0.05\n", + "Setting Classifier weights seed seed to 0\n", + "Training args: {'drop_mult': 0.5, 'wd': 0.01, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-music/models/sp15k/multifit_paper_version20/enc_best\n", + "Setting Classifier training seed seed to 0\n", + "Training: data/cls/de-music/models/sp15k/multifit_paper_version20\n" + ] + }, + { + "data": { + "text/html": [ + "Total time: 02:50

\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
00.4825730.4539160.86500000:21
10.4258210.4109710.85500000:21
20.3491410.4828790.83500000:21
30.2833440.4427840.85000000:21
40.2413100.3725200.89500000:21
50.2319020.3988090.88500000:21
60.2198500.3806060.89000000:21
70.2198380.3812020.87000000:21
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Classifier model saved to data/cls/de-music/models/sp15k/multifit_paper_version20\n", + "Saving dump to data/cls/de-music/models/sp15k/multifit_paper_version20/classifier.json\n", + "this Learner object self-destroyed - it still exists, but no longer usable\n" + ] + } + ], + "source": [ + "exp.classifier.train_(seed=0)" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/de-music/models/sp15k/multifit_paper_version20 to data/cls/de-music/models/sp15k\n", + "Data lm-notst, trn: 31800, val: 200\n", + "Size of vocabulary: 15000\n", + "First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', \"▁&'\"]\n", + "Data cls, trn: 1800, val: 200\n", + "Data tst, trn: 200, val: 2000\n", + "Using Label smoothing with eps = 0.05\n", + "Setting Classifier weights seed seed to 1\n", + "Training args: {'drop_mult': 0.5, 'wd': 0.01, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-music/models/sp15k/multifit_paper_version20/enc_best\n", + "Setting Classifier training seed seed to 1\n", + "Training: data/cls/de-music/models/sp15k/multifit_paper_version20seed1\n" + ] + }, + { + "data": { + "text/html": [ + "Total time: 02:52

\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
00.4780580.4207620.85000000:21
10.4171340.4395870.83500000:21
20.3543910.4746610.83000000:21
30.2836890.3655010.90000000:21
40.2467390.3671080.89500000:21
50.2323220.3663190.89500000:21
60.2243530.3567220.88000000:21
70.2147910.3551310.88500000:21
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/de-music/models/sp15k/multifit_paper_version20 to data/cls/de-music/models/sp15k/multifit_paper_version20seed1\n", + "Classifier model saved to data/cls/de-music/models/sp15k/multifit_paper_version20seed1\n", + "Saving dump to data/cls/de-music/models/sp15k/multifit_paper_version20seed1/classifier.json\n", + "this Learner object self-destroyed - it still exists, but no longer usable\n" + ] + } + ], + "source": [ + "exp.classifier.train_(seed=1)" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/de-music/models/sp15k/multifit_paper_version20 to data/cls/de-music/models/sp15k\n", + "Data lm-notst, trn: 31800, val: 200\n", + "Size of vocabulary: 15000\n", + "First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', \"▁&'\"]\n", + "Data cls, trn: 1800, val: 200\n", + "Data tst, trn: 200, val: 2000\n", + "Using Label smoothing with eps = 0.05\n", + "Setting Classifier weights seed seed to 2\n", + "Training args: {'drop_mult': 0.5, 'wd': 0.01, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-music/models/sp15k/multifit_paper_version20/enc_best\n", + "Setting Classifier training seed seed to 2\n", + "Training: data/cls/de-music/models/sp15k/multifit_paper_version20seed2\n" + ] + }, + { + "data": { + "text/html": [ + "Total time: 02:51

\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
00.4823150.4396440.85000000:21
10.4287510.4452480.86000000:21
20.3612430.3842820.88500000:21
30.2751550.5726100.79000000:21
40.2508880.3687250.91500000:21
50.2373720.3472520.90500000:21
60.2306490.3384490.91000000:21
70.2199030.3368910.91000000:21
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/de-music/models/sp15k/multifit_paper_version20 to data/cls/de-music/models/sp15k/multifit_paper_version20seed2\n", + "Classifier model saved to data/cls/de-music/models/sp15k/multifit_paper_version20seed2\n", + "Saving dump to data/cls/de-music/models/sp15k/multifit_paper_version20seed2/classifier.json\n", + "this Learner object self-destroyed - it still exists, but no longer usable\n" + ] + } + ], + "source": [ + "exp.classifier.train_(seed=2)" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/de-music/models/sp15k/multifit_paper_version20 to data/cls/de-music/models/sp15k\n", + "Data lm-notst, trn: 31800, val: 200\n", + "Size of vocabulary: 15000\n", + "First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', \"▁&'\"]\n", + "Data cls, trn: 1800, val: 200\n", + "Data tst, trn: 200, val: 2000\n", + "Using Label smoothing with eps = 0.05\n", + "Setting Classifier weights seed seed to 3\n", + "Training args: {'drop_mult': 0.5, 'wd': 0.01, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-music/models/sp15k/multifit_paper_version20/enc_best\n", + "Setting Classifier training seed seed to 3\n", + "Training: data/cls/de-music/models/sp15k/multifit_paper_version20seed3\n" + ] + }, + { + "data": { + "text/html": [ + "Total time: 02:51

\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
00.4840430.4207900.87000000:21
10.4084240.5371660.82000000:21
20.3400570.3999380.86000000:21
30.2977420.4357660.84500000:21
40.2464390.3952330.86500000:21
50.2329590.3811820.86000000:21
60.2221370.3700060.87500000:21
70.2189510.3676110.87000000:21
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/de-music/models/sp15k/multifit_paper_version20 to data/cls/de-music/models/sp15k/multifit_paper_version20seed3\n", + "Classifier model saved to data/cls/de-music/models/sp15k/multifit_paper_version20seed3\n", + "Saving dump to data/cls/de-music/models/sp15k/multifit_paper_version20seed3/classifier.json\n", + "this Learner object self-destroyed - it still exists, but no longer usable\n" + ] + } + ], + "source": [ + "exp.classifier.train_(seed=3)" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/de-music/models/sp15k/multifit_paper_version20 to data/cls/de-music/models/sp15k\n", + "Data lm-notst, trn: 31800, val: 200\n", + "Size of vocabulary: 15000\n", + "First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', \"▁&'\"]\n", + "Data cls, trn: 1800, val: 200\n", + "Data tst, trn: 200, val: 2000\n", + "Using Label smoothing with eps = 0.05\n", + "Setting Classifier weights seed seed to 4\n", + "Training args: {'drop_mult': 0.5, 'wd': 0.01, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-music/models/sp15k/multifit_paper_version20/enc_best\n", + "Setting Classifier training seed seed to 4\n", + "Training: data/cls/de-music/models/sp15k/multifit_paper_version20seed4\n" + ] + }, + { + "data": { + "text/html": [ + "Total time: 02:53

\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
00.4811960.4594190.84500000:21
10.4327490.4584110.84500000:21
20.3302440.4398640.84000000:21
30.2861050.3663150.89000000:21
40.2613800.4382140.83500000:21
50.2414550.3624420.87000000:21
60.2300320.3726230.88000000:22
70.2204300.3727610.88000000:21
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/de-music/models/sp15k/multifit_paper_version20 to data/cls/de-music/models/sp15k/multifit_paper_version20seed4\n", + "Classifier model saved to data/cls/de-music/models/sp15k/multifit_paper_version20seed4\n", + "Saving dump to data/cls/de-music/models/sp15k/multifit_paper_version20seed4/classifier.json\n", + "this Learner object self-destroyed - it still exists, but no longer usable\n" + ] + } + ], + "source": [ + "exp.classifier.train_(seed=4)" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/de-music/models/sp15k/multifit_paper_version20 to data/cls/de-music/models/sp15k\n", + "Data lm-notst, trn: 31800, val: 200\n", + "Size of vocabulary: 15000\n", + "First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', \"▁&'\"]\n", + "Data cls, trn: 1800, val: 200\n", + "Data tst, trn: 200, val: 2000\n", + "Using Label smoothing with eps = 0.05\n", + "Setting Classifier weights seed seed to 5\n", + "Training args: {'drop_mult': 0.5, 'wd': 0.01, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-music/models/sp15k/multifit_paper_version20/enc_best\n", + "Setting Classifier training seed seed to 5\n", + "Training: data/cls/de-music/models/sp15k/multifit_paper_version20seed5\n" + ] + }, + { + "data": { + "text/html": [ + "Total time: 02:52

\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
00.4698310.4128380.88000000:21
10.4039460.4784760.86000000:21
20.3528680.6061900.77000000:22
30.2842120.4306320.86500000:20
40.2545040.3705660.89500000:21
50.2331220.3674920.90000000:21
60.2225970.3702170.89000000:21
70.2237860.3720480.89000000:21
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/de-music/models/sp15k/multifit_paper_version20 to data/cls/de-music/models/sp15k/multifit_paper_version20seed5\n", + "Classifier model saved to data/cls/de-music/models/sp15k/multifit_paper_version20seed5\n", + "Saving dump to data/cls/de-music/models/sp15k/multifit_paper_version20seed5/classifier.json\n", + "this Learner object self-destroyed - it still exists, but no longer usable\n" + ] + } + ], + "source": [ + "exp.classifier.train_(seed=5)" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/de-music/models/sp15k/multifit_paper_version20 to data/cls/de-music/models/sp15k\n", + "Data lm-notst, trn: 31800, val: 200\n", + "Size of vocabulary: 15000\n", + "First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', \"▁&'\"]\n", + "Data cls, trn: 1800, val: 200\n", + "Data tst, trn: 200, val: 2000\n", + "Using Label smoothing with eps = 0.05\n", + "Setting Classifier weights seed seed to 6\n", + "Training args: {'drop_mult': 0.5, 'wd': 0.01, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-music/models/sp15k/multifit_paper_version20/enc_best\n", + "Setting Classifier training seed seed to 6\n", + "Training: data/cls/de-music/models/sp15k/multifit_paper_version20seed6\n" + ] + }, + { + "data": { + "text/html": [ + "Total time: 02:53

\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
00.4763410.4126990.85500000:21
10.4142600.4650010.85500000:21
20.3458920.3814680.88000000:21
30.2821830.4716230.84500000:21
40.2543310.3645310.88500000:21
50.2300840.3526390.90000000:22
60.2267630.3546340.90500000:21
70.2259350.3459090.92000000:21
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/de-music/models/sp15k/multifit_paper_version20 to data/cls/de-music/models/sp15k/multifit_paper_version20seed6\n", + "Classifier model saved to data/cls/de-music/models/sp15k/multifit_paper_version20seed6\n", + "Saving dump to data/cls/de-music/models/sp15k/multifit_paper_version20seed6/classifier.json\n", + "this Learner object self-destroyed - it still exists, but no longer usable\n" + ] + } + ], + "source": [ + "exp.classifier.train_(seed=6)" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": { + "scrolled": false + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/de-music/models/sp15k/multifit_paper_version20 to data/cls/de-music/models/sp15k\n", + "Data lm-notst, trn: 31800, val: 200\n", + "Size of vocabulary: 15000\n", + "First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', \"▁&'\"]\n", + "Data cls, trn: 1800, val: 200\n", + "Data tst, trn: 200, val: 2000\n", + "Using Label smoothing with eps = 0.05\n", + "Setting Classifier weights seed seed to 7\n", + "Training args: {'drop_mult': 0.5, 'wd': 0.01, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-music/models/sp15k/multifit_paper_version20/enc_best\n", + "Setting Classifier training seed seed to 7\n", + "Training: data/cls/de-music/models/sp15k/multifit_paper_version20seed7\n" + ] + }, + { + "data": { + "text/html": [ + "Total time: 02:52

\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
00.5040870.4518050.82500000:21
10.4311870.4643050.82500000:21
20.3621950.4099770.86000000:21
30.2895050.3869840.88000000:21
40.2468330.4022010.86500000:21
50.2303460.3854520.85500000:21
60.2215950.3817220.88000000:21
70.2278070.3841490.87000000:21
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/de-music/models/sp15k/multifit_paper_version20 to data/cls/de-music/models/sp15k/multifit_paper_version20seed7\n", + "Classifier model saved to data/cls/de-music/models/sp15k/multifit_paper_version20seed7\n", + "Saving dump to data/cls/de-music/models/sp15k/multifit_paper_version20seed7/classifier.json\n", + "this Learner object self-destroyed - it still exists, but no longer usable\n" + ] + } + ], + "source": [ + "exp.classifier.train_(seed=7)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Results" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'test loss': 0.3395114839076996, 'test accuracy': 0.921500027179718, 'name': 'multifit_paper_version20', 'valid loss': 0.3720475435256958, 'valid accuracy': 0.8899999856948853, 'train loss': 0.20120832324028015, 'train accuracy': 1.0}\n" + ] + }, + { + "data": { + "text/html": [ + "

\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
nameseedtest accuracyvalid accuracy
0multifit_paper_version2070.91700.870
2multifit_paper_version2030.90850.870
5multifit_paper_version2040.90850.880
1multifit_paper_version2010.90800.885
6multifit_paper_version2050.92150.890
3multifit_paper_version2020.91600.910
4multifit_paper_version2060.91300.920
\n", + "
" + ], + "text/plain": [ + " name seed test accuracy valid accuracy\n", + "0 multifit_paper_version20 7 0.9170 0.870\n", + "2 multifit_paper_version20 3 0.9085 0.870\n", + "5 multifit_paper_version20 4 0.9085 0.880\n", + "1 multifit_paper_version20 1 0.9080 0.885\n", + "6 multifit_paper_version20 5 0.9215 0.890\n", + "3 multifit_paper_version20 2 0.9160 0.910\n", + "4 multifit_paper_version20 6 0.9130 0.920" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def get_results(exp_path):\n", + " exp = ulmfit.ULMFiT().load_(exp_path, silent=True).classifier\n", + " results = exp.validate(use_cache=False) \n", + " results.update(seed=exp.seed, fp16=exp.fp16)\n", + " return results\n", + "results = [get_results(exp_path) for exp_path in cls_dataset.cache_path.glob(exp.pretrain_lm.name+\"seed*\")]\n", + "results_df = pd.DataFrame.from_records(results)\n", + "results_df.sort_values([\"valid accuracy\"])[[\"name\", \"seed\", \"test accuracy\", \"valid accuracy\"]]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/notebooks/CLS-JA.ipynb b/notebooks/CLS-JA.ipynb new file mode 100644 index 0000000..a4ee055 --- /dev/null +++ b/notebooks/CLS-JA.ipynb @@ -0,0 +1,1786 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "env: CUDA_VISIBLE_DEVICES=0\n" + ] + } + ], + "source": [ + "%env CUDA_VISIBLE_DEVICES=0" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "/home/pczapla/workspace/ulmfit-multilingual\n" + ] + } + ], + "source": [ + "%reload_ext autoreload\n", + "%autoreload 2\n", + "%matplotlib inline\n", + "%cd .." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "from fastai.text import *\n", + "from ulmfit.datasets import ULMFiTDataset, Dataset\n", + "import ulmfit" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "import warnings\n", + "warnings.filterwarnings(\"ignore\") # to ignore pytorch 1.3 warnings triggered by older fastai" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# CLS JA music" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "#!python prepare_cls.py https://storage.googleapis.com/ulmfit/cls" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [], + "source": [ + "exp = ulmfit.from_pretrained('ja_multifit_paper_version')" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "scrolled": false + }, + "outputs": [ + { + "data": { + "text/plain": [ + "ULMFiTArchitecture(tokenizer_type='sp', max_vocab=15000, lang='ja', emb_sz=400, n_hid=1550, n_layers=4, qrnn=True)" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "exp.arch" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from /home/pczapla/.fastai/models/ja_multifit_paper_version to data/cls/ja-music/models/sp15k\n" + ] + } + ], + "source": [ + "cls_dataset = exp.arch.dataset(Path('data/cls/ja-music'), exp.pretrain_lm.tokenizer)" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'ja'" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "cls_dataset.lang" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "scrolled": false + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Data tst, trn: 200, val: 1999\n" + ] + }, + { + "data": { + "text/html": [ + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
texttarget
▁ xxbos ▁ xxfld ▁ 1 ▁一 音 が すべて 、 すべてが 一 音 。 <unk> し み を 包 んで く れる 宇 宙 その もの ▁ xxfld ▁ 2 ▁ xxup ▁ <unk> かな 世 界 が 、 cd を かけた と たん 拡 が る 。 「 打 波 」 だ 。 それは <unk> 気 と <unk> 発 力 をもって はじまり 、 美 しい 小 波 が1
▁ xxbos ▁ xxfld ▁ 1 ▁ 様 々 な バージョン 違 い や レア 音 源 を 聴 き 散 ら か した 上 で 聴 く と 、 どう にも <unk> 求 不 満 になる アルバム 。 イン スト 部 分 が 多 い のも 「 歌 」 を 充 分 聴 きた い 人 間 には 物 足 り ない かな 、 と 。 ▁ xxfld ▁0
▁ xxbos ▁ xxfld ▁ 1 ▁ xxup ▁ 今 度 は 野 球 の 松 坂 ファンを ex ファンに させ ようと してる ▁ xxrep ▁4 ▁・ ▁ xxfld ▁ 2 ▁ xxup ▁第 2 章 になって すべての シングル 、 アルバムの dvd 付 き を 購 入 した 方 。 よく 考 え ましょう 。 ▁ xxup ▁ cd 収 録 の ▁ xxup ▁ w on ▁ ' ▁0
▁ xxbos ▁ xxfld ▁ 1 ▁ 粒 <unk> い の light な d ance ▁number が <unk> 能 出 来 る b est 盤 ▁ xxfld ▁ 2 ▁ sky y は 80 年 代 を pe ak に 大 活 躍 した new - y or k のお <unk> 落 な fu nk ▁ xxmaj ▁ band である 。 ▁ただ 、 fu nk ▁ band とは 言 え1
▁ xxbos ▁ xxfld ▁ 1 ▁ 少 々 物 足 り ない 内 容 ▁ xxfld ▁ 2 ▁ レイ ・ デイヴィス 自 身 が 監 修 した キン クス 待 望 の ボックス ・ セット の 登 場 です 。 p ye 〜 rca 〜 ari sta 〜 lon don 〜 co lum bia 〜 ko nk の 全 時 代 を 網 羅 した 6 枚 組1
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "cls_dataset.load_clas_databunch(bs=exp.finetune_lm.bs).show_batch()" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "scrolled": false + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Setting LM weights seed seed to 0\n", + "Data lm-notst, trn: 31800, val: 200\n", + "Size of vocabulary: 15000\n", + "First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']\n", + "Training args: {'drop_mult': 0.3, 'true_wd': False, 'wd': 1e-07, 'pretrained': False, 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15, 'tie_weights': True, 'out_bias': True}\n", + "Setting LM training seed seed to 0\n", + "Loading pretrained weights: [PosixPath('/home/pczapla/.fastai/models/ja_multifit_paper_version/lm_best'), PosixPath('/home/pczapla/.fastai/models/ja_multifit_paper_version/itos')]\n", + "Experiment data/cls/ja-music/models/sp15k/multifit_paper_version\n", + "Fitting using 2 cycle fit schedule\n" + ] + }, + { + "data": { + "text/html": [ + "Total time: 02:14

\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
05.8155625.1258860.21491402:13
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "Total time: 1:04:16

\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
04.9622174.8311370.23874303:10
14.5590754.4328360.27894303:12
24.1318044.1288010.31251403:12
33.9604203.9254460.33260003:13
43.8335943.7985640.34608603:12
53.6643753.7085470.35454303:13
63.5923033.6348400.36260003:12
73.4488243.5871910.36771403:12
83.4561723.5494200.37462903:12
93.3536903.5184770.37802903:12
103.3293063.5008240.38137103:12
113.2700153.4912820.38220003:14
123.1662143.4752230.38434303:13
133.1558233.4676660.38782903:12
143.1366823.4579260.38971403:13
153.0228803.4637060.38885703:13
163.0586923.4613110.38908603:12
173.0084923.4617080.38885703:12
182.9826083.4620140.38917103:12
192.9888103.4617810.38945703:13
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from /home/pczapla/.fastai/models/ja_multifit_paper_version to data/cls/ja-music/models/sp15k/multifit_paper_version\n", + "this Learner object self-destroyed - it still exists, but no longer usable\n", + "Saving dump to data/cls/ja-music/models/sp15k/multifit_paper_version/finetuning.json\n", + "Language model saved to data/cls/ja-music/models/sp15k/multifit_paper_version\n" + ] + } + ], + "source": [ + "exp.finetune_lm.train_(cls_dataset)" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Loading data/cls/ja-music/models/sp15k/multifit_paper_version/classifier.json\n", + "Loading data/cls/ja-music/models/sp15k/multifit_paper_version/finetuning.json\n", + "ULMFiTClassifier Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_version\n", + "ULMFiTClassifier Replacing dataset_path 'None' with 'data/cls/ja-music\n", + "ULMFiTClassifier Replacing bs '18' with '20\n", + "ULMFiTClassifier Replacing wd '0.01' with '0.1\n", + "ULMFiTClassifier Replacing early_stopping 'None' with 'accuracy\n" + ] + }, + { + "data": { + "text/plain": [ + "ULMFiTFinetuning(seed=0, name='multifit_paper_version', experiment_path=PosixPath('data/cls/ja-music/models/sp15k/multifit_paper_version'), dataset_path=PosixPath('data/cls/ja-music'), num_epochs=20, bs=20, bptt=70, drop_mult=0.3, dropout_values={'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}, label_smoothing_eps=0.0, label_smoothing_eps_norm_by_classes=True, use_adam_08=False, true_wd=False, wd=1e-07, clip=0.12, fp16=False, lr=0.001)" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "exp.load_(cls_dataset.cache_path/exp.pretrain_lm.name).finetune_lm" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "scrolled": false + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k\n", + "Data lm-notst, trn: 31800, val: 200\n", + "Size of vocabulary: 15000\n", + "First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']\n", + "Data cls, trn: 1800, val: 200\n", + "Data tst, trn: 200, val: 1999\n", + "Using Label smoothing with eps = 0.05\n", + "Setting Classifier weights seed seed to 0\n", + "Training args: {'drop_mult': 0.5, 'wd': 0.1, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-music/models/sp15k/multifit_paper_version/enc_best\n", + "Setting Classifier training seed seed to 0\n", + "Training: data/cls/ja-music/models/sp15k/multifit_paper_version\n" + ] + }, + { + "data": { + "text/html": [ + "Total time: 01:38

\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
00.5516920.5442440.75000000:11
10.5148990.6176700.75000000:12
20.4433670.5961440.78500000:11
30.3680910.5676680.79000000:11
40.3157570.6350340.80500000:12
50.2715950.5152270.81500000:11
60.2417170.5233870.82000000:11
70.2377960.5187630.82500000:11
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Better model found at epoch 0 with accuracy value: 0.75.\n", + "Better model found at epoch 2 with accuracy value: 0.7850000262260437.\n", + "Better model found at epoch 3 with accuracy value: 0.7900000214576721.\n", + "Better model found at epoch 4 with accuracy value: 0.8050000071525574.\n", + "Better model found at epoch 5 with accuracy value: 0.8149999976158142.\n", + "Better model found at epoch 6 with accuracy value: 0.8199999928474426.\n", + "Better model found at epoch 7 with accuracy value: 0.824999988079071.\n", + "Classifier model saved to data/cls/ja-music/models/sp15k/multifit_paper_version\n", + "Saving dump to data/cls/ja-music/models/sp15k/multifit_paper_version/classifier.json\n", + "this Learner object self-destroyed - it still exists, but no longer usable\n" + ] + } + ], + "source": [ + "exp.classifier.train_(seed=0)" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k\n", + "Data lm-notst, trn: 31800, val: 200\n", + "Size of vocabulary: 15000\n", + "First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']\n", + "Data cls, trn: 1800, val: 200\n", + "Data tst, trn: 200, val: 1999\n", + "Using Label smoothing with eps = 0.05\n", + "Setting Classifier weights seed seed to 1\n", + "Training args: {'drop_mult': 0.5, 'wd': 0.1, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-music/models/sp15k/multifit_paper_version/enc_best\n", + "Setting Classifier training seed seed to 1\n", + "Training: data/cls/ja-music/models/sp15k/multifit_paper_versionseed1\n" + ] + }, + { + "data": { + "text/html": [ + "Total time: 01:35

\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
00.5534180.4931840.80000000:11
10.5177540.6030110.74500000:11
20.4446490.4510210.85000000:12
30.3781630.4885510.84500000:12
40.2963320.7226080.74500000:11
50.2670310.5282150.81500000:11
60.2488840.5228990.83000000:11
70.2346140.5128670.82500000:12
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Better model found at epoch 0 with accuracy value: 0.800000011920929.\n", + "Better model found at epoch 2 with accuracy value: 0.8500000238418579.\n", + "Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k/multifit_paper_versionseed1\n", + "Classifier model saved to data/cls/ja-music/models/sp15k/multifit_paper_versionseed1\n", + "Saving dump to data/cls/ja-music/models/sp15k/multifit_paper_versionseed1/classifier.json\n", + "this Learner object self-destroyed - it still exists, but no longer usable\n" + ] + } + ], + "source": [ + "exp.classifier.train_(seed=1)" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k\n", + "Data lm-notst, trn: 31800, val: 200\n", + "Size of vocabulary: 15000\n", + "First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']\n", + "Data cls, trn: 1800, val: 200\n", + "Data tst, trn: 200, val: 1999\n", + "Using Label smoothing with eps = 0.05\n", + "Setting Classifier weights seed seed to 2\n", + "Training args: {'drop_mult': 0.5, 'wd': 0.1, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-music/models/sp15k/multifit_paper_version/enc_best\n", + "Setting Classifier training seed seed to 2\n", + "Training: data/cls/ja-music/models/sp15k/multifit_paper_versionseed2\n" + ] + }, + { + "data": { + "text/html": [ + "Total time: 01:35

\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
00.5220990.5879370.78000000:11
10.5010930.7493370.75500000:11
20.4316290.6106930.76500000:12
30.3699560.5184670.82000000:12
40.3000800.6210230.81000000:11
50.2730650.5486950.81000000:11
60.2443430.5211410.82000000:12
70.2285440.5089840.82000000:11
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Better model found at epoch 0 with accuracy value: 0.7799999713897705.\n", + "Better model found at epoch 3 with accuracy value: 0.8199999928474426.\n", + "Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k/multifit_paper_versionseed2\n", + "Classifier model saved to data/cls/ja-music/models/sp15k/multifit_paper_versionseed2\n", + "Saving dump to data/cls/ja-music/models/sp15k/multifit_paper_versionseed2/classifier.json\n", + "this Learner object self-destroyed - it still exists, but no longer usable\n" + ] + } + ], + "source": [ + "exp.classifier.train_(seed=2)" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k\n", + "Data lm-notst, trn: 31800, val: 200\n", + "Size of vocabulary: 15000\n", + "First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']\n", + "Data cls, trn: 1800, val: 200\n", + "Data tst, trn: 200, val: 1999\n", + "Using Label smoothing with eps = 0.05\n", + "Setting Classifier weights seed seed to 3\n", + "Training args: {'drop_mult': 0.5, 'wd': 0.1, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-music/models/sp15k/multifit_paper_version/enc_best\n", + "Setting Classifier training seed seed to 3\n", + "Training: data/cls/ja-music/models/sp15k/multifit_paper_versionseed3\n" + ] + }, + { + "data": { + "text/html": [ + "Total time: 01:36

\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
00.5407890.5302500.77500000:11
10.4945810.5091510.81500000:12
20.4461260.5290510.78500000:11
30.3769280.5827710.77500000:11
40.3228230.4929150.83500000:11
50.2700820.4998380.85000000:12
60.2496190.5120060.84000000:11
70.2404760.5151560.82000000:11
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Better model found at epoch 0 with accuracy value: 0.7749999761581421.\n", + "Better model found at epoch 1 with accuracy value: 0.8149999976158142.\n", + "Better model found at epoch 4 with accuracy value: 0.8349999785423279.\n", + "Better model found at epoch 5 with accuracy value: 0.8500000238418579.\n", + "Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k/multifit_paper_versionseed3\n", + "Classifier model saved to data/cls/ja-music/models/sp15k/multifit_paper_versionseed3\n", + "Saving dump to data/cls/ja-music/models/sp15k/multifit_paper_versionseed3/classifier.json\n", + "this Learner object self-destroyed - it still exists, but no longer usable\n" + ] + } + ], + "source": [ + "exp.classifier.train_(seed=3)" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k\n", + "Data lm-notst, trn: 31800, val: 200\n", + "Size of vocabulary: 15000\n", + "First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']\n", + "Data cls, trn: 1800, val: 200\n", + "Data tst, trn: 200, val: 1999\n", + "Using Label smoothing with eps = 0.05\n", + "Setting Classifier weights seed seed to 4\n", + "Training args: {'drop_mult': 0.5, 'wd': 0.1, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-music/models/sp15k/multifit_paper_version/enc_best\n", + "Setting Classifier training seed seed to 4\n", + "Training: data/cls/ja-music/models/sp15k/multifit_paper_versionseed4\n" + ] + }, + { + "data": { + "text/html": [ + "Total time: 01:38

\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
00.5577350.4749140.80500000:12
10.4978100.5192760.78500000:11
20.4452260.6326910.77500000:12
30.3763290.5235930.81000000:11
40.3155620.4918260.81500000:11
50.2776760.5319700.80500000:11
60.2416530.4918890.84000000:12
70.2408780.5054520.83500000:11
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Better model found at epoch 0 with accuracy value: 0.8050000071525574.\n", + "Better model found at epoch 3 with accuracy value: 0.8100000023841858.\n", + "Better model found at epoch 4 with accuracy value: 0.8149999976158142.\n", + "Better model found at epoch 6 with accuracy value: 0.8399999737739563.\n", + "Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k/multifit_paper_versionseed4\n", + "Classifier model saved to data/cls/ja-music/models/sp15k/multifit_paper_versionseed4\n", + "Saving dump to data/cls/ja-music/models/sp15k/multifit_paper_versionseed4/classifier.json\n", + "this Learner object self-destroyed - it still exists, but no longer usable\n" + ] + } + ], + "source": [ + "exp.classifier.train_(seed=4)" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k\n", + "Data lm-notst, trn: 31800, val: 200\n", + "Size of vocabulary: 15000\n", + "First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']\n", + "Data cls, trn: 1800, val: 200\n", + "Data tst, trn: 200, val: 1999\n", + "Using Label smoothing with eps = 0.05\n", + "Setting Classifier weights seed seed to 5\n", + "Training args: {'drop_mult': 0.5, 'wd': 0.1, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-music/models/sp15k/multifit_paper_version/enc_best\n", + "Setting Classifier training seed seed to 5\n", + "Training: data/cls/ja-music/models/sp15k/multifit_paper_versionseed5\n" + ] + }, + { + "data": { + "text/html": [ + "Total time: 01:35

\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
00.5390670.5241930.82000000:12
10.5048460.5630580.77500000:11
20.4556300.5549390.78500000:11
30.3660070.5358700.81000000:11
40.3022070.5024170.82500000:11
50.2749100.5430570.81500000:12
60.2571350.5172890.81500000:11
70.2346170.5057330.80000000:11
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Better model found at epoch 0 with accuracy value: 0.8199999928474426.\n", + "Better model found at epoch 4 with accuracy value: 0.824999988079071.\n", + "Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k/multifit_paper_versionseed5\n", + "Classifier model saved to data/cls/ja-music/models/sp15k/multifit_paper_versionseed5\n", + "Saving dump to data/cls/ja-music/models/sp15k/multifit_paper_versionseed5/classifier.json\n", + "this Learner object self-destroyed - it still exists, but no longer usable\n" + ] + } + ], + "source": [ + "exp.classifier.train_(seed=5)" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k\n", + "Data lm-notst, trn: 31800, val: 200\n", + "Size of vocabulary: 15000\n", + "First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']\n", + "Data cls, trn: 1800, val: 200\n", + "Data tst, trn: 200, val: 1999\n", + "Using Label smoothing with eps = 0.05\n", + "Setting Classifier weights seed seed to 6\n", + "Training args: {'drop_mult': 0.5, 'wd': 0.1, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-music/models/sp15k/multifit_paper_version/enc_best\n", + "Setting Classifier training seed seed to 6\n", + "Training: data/cls/ja-music/models/sp15k/multifit_paper_versionseed6\n" + ] + }, + { + "data": { + "text/html": [ + "Total time: 01:38

\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
00.5299630.5332600.75000000:11
10.5011830.5181920.78500000:11
20.4296160.6330200.80000000:12
30.3823840.5265540.83000000:11
40.3154020.5208210.82500000:11
50.2747720.5512670.82000000:12
60.2499560.5199600.83000000:12
70.2267640.5164190.82500000:11
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Better model found at epoch 0 with accuracy value: 0.75.\n", + "Better model found at epoch 1 with accuracy value: 0.7850000262260437.\n", + "Better model found at epoch 2 with accuracy value: 0.800000011920929.\n", + "Better model found at epoch 3 with accuracy value: 0.8299999833106995.\n", + "Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k/multifit_paper_versionseed6\n", + "Classifier model saved to data/cls/ja-music/models/sp15k/multifit_paper_versionseed6\n", + "Saving dump to data/cls/ja-music/models/sp15k/multifit_paper_versionseed6/classifier.json\n", + "this Learner object self-destroyed - it still exists, but no longer usable\n" + ] + } + ], + "source": [ + "exp.classifier.train_(seed=6)" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "scrolled": false + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k\n", + "Data lm-notst, trn: 31800, val: 200\n", + "Size of vocabulary: 15000\n", + "First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']\n", + "Data cls, trn: 1800, val: 200\n", + "Data tst, trn: 200, val: 1999\n", + "Using Label smoothing with eps = 0.05\n", + "Setting Classifier weights seed seed to 7\n", + "Training args: {'drop_mult': 0.5, 'wd': 0.1, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-music/models/sp15k/multifit_paper_version/enc_best\n", + "Setting Classifier training seed seed to 7\n", + "Training: data/cls/ja-music/models/sp15k/multifit_paper_versionseed7\n" + ] + }, + { + "data": { + "text/html": [ + "Total time: 01:36

\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
00.5654360.5178270.80500000:11
10.5177890.7100510.79500000:11
20.4564000.5020670.80500000:11
30.3679900.5602700.80000000:11
40.3051950.5857190.80500000:11
50.2768690.4968070.84000000:12
60.2447160.5024820.84500000:11
70.2380590.4969510.84500000:11
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Better model found at epoch 0 with accuracy value: 0.8050000071525574.\n", + "Better model found at epoch 5 with accuracy value: 0.8399999737739563.\n", + "Better model found at epoch 6 with accuracy value: 0.8450000286102295.\n", + "Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k/multifit_paper_versionseed7\n", + "Classifier model saved to data/cls/ja-music/models/sp15k/multifit_paper_versionseed7\n", + "Saving dump to data/cls/ja-music/models/sp15k/multifit_paper_versionseed7/classifier.json\n", + "this Learner object self-destroyed - it still exists, but no longer usable\n" + ] + } + ], + "source": [ + "exp.classifier.train_(seed=7)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Results" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Loading data/cls/ja-music/models/sp15k/multifit_paper_versionseed5/classifier.json\n", + "Loading data/cls/ja-music/models/sp15k/multifit_paper_version/finetuning.json\n", + "ULMFiTFinetuning Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_version\n", + "ULMFiTFinetuning Replacing dataset_path 'None' with 'data/cls/ja-music\n", + "ULMFiTFinetuning Replacing num_epochs '10' with '20\n", + "ULMFiTFinetuning Replacing drop_mult '1.0' with '0.3\n", + "ULMFiTFinetuning Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "ULMFiTFinetuning Replacing true_wd 'True' with 'False\n", + "ULMFiTFinetuning Replacing wd '0.01' with '1e-07\n", + "ULMFiTFinetuning Replacing clip 'None' with '0.12\n", + "ULMFiTArchitecture Replacing tokenizer_type 'f' with 'sp\n", + "ULMFiTArchitecture Replacing max_vocab '60000' with '15000\n", + "ULMFiTArchitecture Replacing lang 'None' with 'ja\n", + "ULMFiTArchitecture Replacing n_hid '1150' with '1550\n", + "ULMFiTArchitecture Replacing n_layers '3' with '4\n", + "ULMFiTArchitecture Replacing qrnn 'False' with 'True\n", + "ULMFiTClassifier Replacing seed '0' with '5\n", + "ULMFiTClassifier Replacing name 'multifit_paper_versionseed5' with 'multifit_paper_version\n", + "ULMFiTClassifier Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_versionseed5\n", + "ULMFiTClassifier Replacing dataset_path 'None' with 'data/cls/ja-music\n", + "ULMFiTClassifier Replacing num_epochs '10' with '8\n", + "ULMFiTClassifier Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "ULMFiTClassifier Replacing wd '0.01' with '0.1\n", + "ULMFiTClassifier Replacing clip 'None' with '0.12\n", + "ULMFiTClassifier Replacing label_smoothing_eps '0.0' with '0.1\n", + "ULMFiTClassifier Replacing label_smoothing_eps_norm_by_classes 'False' with 'True\n", + "Loading data/cls/ja-music/models/sp15k/multifit_paper_versionseed4/classifier.json\n", + "Loading data/cls/ja-music/models/sp15k/multifit_paper_version/finetuning.json\n", + "ULMFiTFinetuning Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_version\n", + "ULMFiTFinetuning Replacing dataset_path 'None' with 'data/cls/ja-music\n", + "ULMFiTFinetuning Replacing num_epochs '10' with '20\n", + "ULMFiTFinetuning Replacing drop_mult '1.0' with '0.3\n", + "ULMFiTFinetuning Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "ULMFiTFinetuning Replacing true_wd 'True' with 'False\n", + "ULMFiTFinetuning Replacing wd '0.01' with '1e-07\n", + "ULMFiTFinetuning Replacing clip 'None' with '0.12\n", + "ULMFiTArchitecture Replacing tokenizer_type 'f' with 'sp\n", + "ULMFiTArchitecture Replacing max_vocab '60000' with '15000\n", + "ULMFiTArchitecture Replacing lang 'None' with 'ja\n", + "ULMFiTArchitecture Replacing n_hid '1150' with '1550\n", + "ULMFiTArchitecture Replacing n_layers '3' with '4\n", + "ULMFiTArchitecture Replacing qrnn 'False' with 'True\n", + "ULMFiTClassifier Replacing seed '0' with '4\n", + "ULMFiTClassifier Replacing name 'multifit_paper_versionseed4' with 'multifit_paper_version\n", + "ULMFiTClassifier Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_versionseed4\n", + "ULMFiTClassifier Replacing dataset_path 'None' with 'data/cls/ja-music\n", + "ULMFiTClassifier Replacing num_epochs '10' with '8\n", + "ULMFiTClassifier Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "ULMFiTClassifier Replacing wd '0.01' with '0.1\n", + "ULMFiTClassifier Replacing clip 'None' with '0.12\n", + "ULMFiTClassifier Replacing label_smoothing_eps '0.0' with '0.1\n", + "ULMFiTClassifier Replacing label_smoothing_eps_norm_by_classes 'False' with 'True\n", + "Loading data/cls/ja-music/models/sp15k/multifit_paper_versionseed3/classifier.json\n", + "Loading data/cls/ja-music/models/sp15k/multifit_paper_version/finetuning.json\n", + "ULMFiTFinetuning Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_version\n", + "ULMFiTFinetuning Replacing dataset_path 'None' with 'data/cls/ja-music\n", + "ULMFiTFinetuning Replacing num_epochs '10' with '20\n", + "ULMFiTFinetuning Replacing drop_mult '1.0' with '0.3\n", + "ULMFiTFinetuning Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "ULMFiTFinetuning Replacing true_wd 'True' with 'False\n", + "ULMFiTFinetuning Replacing wd '0.01' with '1e-07\n", + "ULMFiTFinetuning Replacing clip 'None' with '0.12\n", + "ULMFiTArchitecture Replacing tokenizer_type 'f' with 'sp\n", + "ULMFiTArchitecture Replacing max_vocab '60000' with '15000\n", + "ULMFiTArchitecture Replacing lang 'None' with 'ja\n", + "ULMFiTArchitecture Replacing n_hid '1150' with '1550\n", + "ULMFiTArchitecture Replacing n_layers '3' with '4\n", + "ULMFiTArchitecture Replacing qrnn 'False' with 'True\n", + "ULMFiTClassifier Replacing seed '0' with '3\n", + "ULMFiTClassifier Replacing name 'multifit_paper_versionseed3' with 'multifit_paper_version\n", + "ULMFiTClassifier Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_versionseed3\n", + "ULMFiTClassifier Replacing dataset_path 'None' with 'data/cls/ja-music\n", + "ULMFiTClassifier Replacing num_epochs '10' with '8\n", + "ULMFiTClassifier Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "ULMFiTClassifier Replacing wd '0.01' with '0.1\n", + "ULMFiTClassifier Replacing clip 'None' with '0.12\n", + "ULMFiTClassifier Replacing label_smoothing_eps '0.0' with '0.1\n", + "ULMFiTClassifier Replacing label_smoothing_eps_norm_by_classes 'False' with 'True\n", + "Loading data/cls/ja-music/models/sp15k/multifit_paper_versionseed1/classifier.json\n", + "Loading data/cls/ja-music/models/sp15k/multifit_paper_version/finetuning.json\n", + "ULMFiTFinetuning Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_version\n", + "ULMFiTFinetuning Replacing dataset_path 'None' with 'data/cls/ja-music\n", + "ULMFiTFinetuning Replacing num_epochs '10' with '20\n", + "ULMFiTFinetuning Replacing drop_mult '1.0' with '0.3\n", + "ULMFiTFinetuning Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "ULMFiTFinetuning Replacing true_wd 'True' with 'False\n", + "ULMFiTFinetuning Replacing wd '0.01' with '1e-07\n", + "ULMFiTFinetuning Replacing clip 'None' with '0.12\n", + "ULMFiTArchitecture Replacing tokenizer_type 'f' with 'sp\n", + "ULMFiTArchitecture Replacing max_vocab '60000' with '15000\n", + "ULMFiTArchitecture Replacing lang 'None' with 'ja\n", + "ULMFiTArchitecture Replacing n_hid '1150' with '1550\n", + "ULMFiTArchitecture Replacing n_layers '3' with '4\n", + "ULMFiTArchitecture Replacing qrnn 'False' with 'True\n", + "ULMFiTClassifier Replacing seed '0' with '1\n", + "ULMFiTClassifier Replacing name 'multifit_paper_versionseed1' with 'multifit_paper_version\n", + "ULMFiTClassifier Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_versionseed1\n", + "ULMFiTClassifier Replacing dataset_path 'None' with 'data/cls/ja-music\n", + "ULMFiTClassifier Replacing num_epochs '10' with '8\n", + "ULMFiTClassifier Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "ULMFiTClassifier Replacing wd '0.01' with '0.1\n", + "ULMFiTClassifier Replacing clip 'None' with '0.12\n", + "ULMFiTClassifier Replacing label_smoothing_eps '0.0' with '0.1\n", + "ULMFiTClassifier Replacing label_smoothing_eps_norm_by_classes 'False' with 'True\n", + "Loading data/cls/ja-music/models/sp15k/multifit_paper_versionseed2/classifier.json\n", + "Loading data/cls/ja-music/models/sp15k/multifit_paper_version/finetuning.json\n", + "ULMFiTFinetuning Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_version\n", + "ULMFiTFinetuning Replacing dataset_path 'None' with 'data/cls/ja-music\n", + "ULMFiTFinetuning Replacing num_epochs '10' with '20\n", + "ULMFiTFinetuning Replacing drop_mult '1.0' with '0.3\n", + "ULMFiTFinetuning Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "ULMFiTFinetuning Replacing true_wd 'True' with 'False\n", + "ULMFiTFinetuning Replacing wd '0.01' with '1e-07\n", + "ULMFiTFinetuning Replacing clip 'None' with '0.12\n", + "ULMFiTArchitecture Replacing tokenizer_type 'f' with 'sp\n", + "ULMFiTArchitecture Replacing max_vocab '60000' with '15000\n", + "ULMFiTArchitecture Replacing lang 'None' with 'ja\n", + "ULMFiTArchitecture Replacing n_hid '1150' with '1550\n", + "ULMFiTArchitecture Replacing n_layers '3' with '4\n", + "ULMFiTArchitecture Replacing qrnn 'False' with 'True\n", + "ULMFiTClassifier Replacing seed '0' with '2\n", + "ULMFiTClassifier Replacing name 'multifit_paper_versionseed2' with 'multifit_paper_version\n", + "ULMFiTClassifier Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_versionseed2\n", + "ULMFiTClassifier Replacing dataset_path 'None' with 'data/cls/ja-music\n", + "ULMFiTClassifier Replacing num_epochs '10' with '8\n", + "ULMFiTClassifier Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "ULMFiTClassifier Replacing wd '0.01' with '0.1\n", + "ULMFiTClassifier Replacing clip 'None' with '0.12\n", + "ULMFiTClassifier Replacing label_smoothing_eps '0.0' with '0.1\n", + "ULMFiTClassifier Replacing label_smoothing_eps_norm_by_classes 'False' with 'True\n", + "Loading data/cls/ja-music/models/sp15k/multifit_paper_versionseed6/classifier.json\n", + "Loading data/cls/ja-music/models/sp15k/multifit_paper_version/finetuning.json\n", + "ULMFiTFinetuning Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_version\n", + "ULMFiTFinetuning Replacing dataset_path 'None' with 'data/cls/ja-music\n", + "ULMFiTFinetuning Replacing num_epochs '10' with '20\n", + "ULMFiTFinetuning Replacing drop_mult '1.0' with '0.3\n", + "ULMFiTFinetuning Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "ULMFiTFinetuning Replacing true_wd 'True' with 'False\n", + "ULMFiTFinetuning Replacing wd '0.01' with '1e-07\n", + "ULMFiTFinetuning Replacing clip 'None' with '0.12\n", + "ULMFiTArchitecture Replacing tokenizer_type 'f' with 'sp\n", + "ULMFiTArchitecture Replacing max_vocab '60000' with '15000\n", + "ULMFiTArchitecture Replacing lang 'None' with 'ja\n", + "ULMFiTArchitecture Replacing n_hid '1150' with '1550\n", + "ULMFiTArchitecture Replacing n_layers '3' with '4\n", + "ULMFiTArchitecture Replacing qrnn 'False' with 'True\n", + "ULMFiTClassifier Replacing seed '0' with '6\n", + "ULMFiTClassifier Replacing name 'multifit_paper_versionseed6' with 'multifit_paper_version\n", + "ULMFiTClassifier Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_versionseed6\n", + "ULMFiTClassifier Replacing dataset_path 'None' with 'data/cls/ja-music\n", + "ULMFiTClassifier Replacing num_epochs '10' with '8\n", + "ULMFiTClassifier Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "ULMFiTClassifier Replacing wd '0.01' with '0.1\n", + "ULMFiTClassifier Replacing clip 'None' with '0.12\n", + "ULMFiTClassifier Replacing label_smoothing_eps '0.0' with '0.1\n", + "ULMFiTClassifier Replacing label_smoothing_eps_norm_by_classes 'False' with 'True\n", + "Loading data/cls/ja-music/models/sp15k/multifit_paper_versionseed7/classifier.json\n", + "Loading data/cls/ja-music/models/sp15k/multifit_paper_version/finetuning.json\n", + "ULMFiTFinetuning Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_version\n", + "ULMFiTFinetuning Replacing dataset_path 'None' with 'data/cls/ja-music\n", + "ULMFiTFinetuning Replacing num_epochs '10' with '20\n", + "ULMFiTFinetuning Replacing drop_mult '1.0' with '0.3\n", + "ULMFiTFinetuning Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "ULMFiTFinetuning Replacing true_wd 'True' with 'False\n", + "ULMFiTFinetuning Replacing wd '0.01' with '1e-07\n", + "ULMFiTFinetuning Replacing clip 'None' with '0.12\n", + "ULMFiTArchitecture Replacing tokenizer_type 'f' with 'sp\n", + "ULMFiTArchitecture Replacing max_vocab '60000' with '15000\n", + "ULMFiTArchitecture Replacing lang 'None' with 'ja\n", + "ULMFiTArchitecture Replacing n_hid '1150' with '1550\n", + "ULMFiTArchitecture Replacing n_layers '3' with '4\n", + "ULMFiTArchitecture Replacing qrnn 'False' with 'True\n", + "ULMFiTClassifier Replacing seed '0' with '7\n", + "ULMFiTClassifier Replacing name 'multifit_paper_versionseed7' with 'multifit_paper_version\n", + "ULMFiTClassifier Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_versionseed7\n", + "ULMFiTClassifier Replacing dataset_path 'None' with 'data/cls/ja-music\n", + "ULMFiTClassifier Replacing num_epochs '10' with '8\n", + "ULMFiTClassifier Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}\n", + "ULMFiTClassifier Replacing wd '0.01' with '0.1\n", + "ULMFiTClassifier Replacing clip 'None' with '0.12\n", + "ULMFiTClassifier Replacing label_smoothing_eps '0.0' with '0.1\n", + "ULMFiTClassifier Replacing label_smoothing_eps_norm_by_classes 'False' with 'True\n" + ] + }, + { + "data": { + "text/html": [ + "

\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
nameseedtest accuracyvalid accuracy
4multifit_paper_version20.8484240.820
0multifit_paper_version50.8519260.825
5multifit_paper_version60.8424210.830
1multifit_paper_version40.8704350.840
6multifit_paper_version70.8614310.845
2multifit_paper_version30.8544270.850
3multifit_paper_version10.8609300.850
\n", + "
" + ], + "text/plain": [ + " name seed test accuracy valid accuracy\n", + "4 multifit_paper_version 2 0.848424 0.820\n", + "0 multifit_paper_version 5 0.851926 0.825\n", + "5 multifit_paper_version 6 0.842421 0.830\n", + "1 multifit_paper_version 4 0.870435 0.840\n", + "6 multifit_paper_version 7 0.861431 0.845\n", + "2 multifit_paper_version 3 0.854427 0.850\n", + "3 multifit_paper_version 1 0.860930 0.850" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def get_results(exp_path):\n", + " exp = ulmfit.ULMFiT().load_(exp_path, silent=False).classifier\n", + " results = exp.validate(use_cache=True) \n", + " results.update(seed=exp.seed, fp16=exp.fp16)\n", + " return results\n", + "results = [get_results(exp_path) for exp_path in cls_dataset.cache_path.glob(exp.pretrain_lm.name+\"seed*\")]\n", + "results_df = pd.DataFrame.from_records(results)\n", + "results_df.sort_values([\"valid accuracy\"])[[\"name\", \"seed\", \"test accuracy\", \"valid accuracy\"]]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.2" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/notebooks/JA-multifit_fp16.ipynb b/notebooks/MLDoc-JA-multifit_fp16.ipynb similarity index 100% rename from notebooks/JA-multifit_fp16.ipynb rename to notebooks/MLDoc-JA-multifit_fp16.ipynb