mirror of
https://github.com/wassname/multifit.git
synced 2026-09-09 11:27:26 +08:00
69 KiB
69 KiB
In [1]:
%env CUDA_VISIBLE_DEVICES=0env: CUDA_VISIBLE_DEVICES=0
In [2]:
%reload_ext autoreload
%autoreload 2
%matplotlib inline
%cd ../home/pczapla/workspace/ulmfit-multilingual
In [3]:
from fastai.text import *
from multifit.datasets import ULMFiTDataset, Dataset
import multifitIn [4]:
import warnings
warnings.filterwarnings("ignore") # to ignore pytorch 1.3 warnings triggered by older fastaiIn [5]:
#!python prepare_cls.py https://storage.googleapis.com/ulmfit/clsIn [6]:
exp = multifit.from_pretrained('ja_multifit_paper_version')In [7]:
exp.archOut [7]:
ULMFiTArchitecture(tokenizer_type='sp', max_vocab=15000, lang='ja', emb_sz=400, n_hid=1550, n_layers=4, qrnn=True)
In [8]:
cls_dataset = exp.arch.dataset(Path('data/cls/ja-music'), exp.pretrain_lm.tokenizer)Copy sp model from /home/pczapla/.fastai/models/ja_multifit_paper_version to data/cls/ja-music/models/sp15k
In [9]:
cls_dataset.langOut [9]:
'ja'
In [10]:
cls_dataset.load_clas_databunch(bs=exp.finetune_lm.bs).show_batch()Data tst, trn: 200, val: 1999
| text | target |
|---|---|
| ▁ xxbos ▁ xxfld ▁ 1 ▁一 音 が すべて 、 すべてが 一 音 。 <unk> し み を 包 んで く れる 宇 宙 その もの ▁ xxfld ▁ 2 ▁ xxup ▁ <unk> かな 世 界 が 、 cd を かけた と たん 拡 が る 。 「 打 波 」 だ 。 それは <unk> 気 と <unk> 発 力 をもって はじまり 、 美 しい 小 波 が | 1 |
| ▁ xxbos ▁ xxfld ▁ 1 ▁ 様 々 な バージョン 違 い や レア 音 源 を 聴 き 散 ら か した 上 で 聴 く と 、 どう にも <unk> 求 不 満 になる アルバム 。 イン スト 部 分 が 多 い のも 「 歌 」 を 充 分 聴 きた い 人 間 には 物 足 り ない かな 、 と 。 ▁ xxfld ▁ | 0 |
| ▁ xxbos ▁ xxfld ▁ 1 ▁ xxup ▁ 今 度 は 野 球 の 松 坂 ファンを ex ファンに させ ようと してる ▁ xxrep ▁4 ▁・ ▁ xxfld ▁ 2 ▁ xxup ▁第 2 章 になって すべての シングル 、 アルバムの dvd 付 き を 購 入 した 方 。 よく 考 え ましょう 。 ▁ xxup ▁ cd 収 録 の ▁ xxup ▁ w on ▁ ' ▁ | 0 |
| ▁ xxbos ▁ xxfld ▁ 1 ▁ 粒 <unk> い の light な d ance ▁number が <unk> 能 出 来 る b est 盤 ▁ xxfld ▁ 2 ▁ sky y は 80 年 代 を pe ak に 大 活 躍 した new - y or k のお <unk> 落 な fu nk ▁ xxmaj ▁ band である 。 ▁ただ 、 fu nk ▁ band とは 言 え | 1 |
| ▁ xxbos ▁ xxfld ▁ 1 ▁ 少 々 物 足 り ない 内 容 ▁ xxfld ▁ 2 ▁ レイ ・ デイヴィス 自 身 が 監 修 した キン クス 待 望 の ボックス ・ セット の 登 場 です 。 p ye 〜 rca 〜 ari sta 〜 lon don 〜 co lum bia 〜 ko nk の 全 時 代 を 網 羅 した 6 枚 組 | 1 |
In [11]:
exp.finetune_lm.train_(cls_dataset)Setting LM weights seed seed to 0
Data lm-notst, trn: 31800, val: 200
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']
Training args: {'drop_mult': 0.3, 'true_wd': False, 'wd': 1e-07, 'pretrained': False, 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15, 'tie_weights': True, 'out_bias': True}
Setting LM training seed seed to 0
Loading pretrained weights: [PosixPath('/home/pczapla/.fastai/models/ja_multifit_paper_version/lm_best'), PosixPath('/home/pczapla/.fastai/models/ja_multifit_paper_version/itos')]
Experiment data/cls/ja-music/models/sp15k/multifit_paper_version
Fitting using 2 cycle fit schedule
Total time: 02:14
| epoch | train_loss | valid_loss | accuracy | time |
|---|---|---|---|---|
| 0 | 5.815562 | 5.125886 | 0.214914 | 02:13 |
Total time: 1:04:16
| epoch | train_loss | valid_loss | accuracy | time |
|---|---|---|---|---|
| 0 | 4.962217 | 4.831137 | 0.238743 | 03:10 |
| 1 | 4.559075 | 4.432836 | 0.278943 | 03:12 |
| 2 | 4.131804 | 4.128801 | 0.312514 | 03:12 |
| 3 | 3.960420 | 3.925446 | 0.332600 | 03:13 |
| 4 | 3.833594 | 3.798564 | 0.346086 | 03:12 |
| 5 | 3.664375 | 3.708547 | 0.354543 | 03:13 |
| 6 | 3.592303 | 3.634840 | 0.362600 | 03:12 |
| 7 | 3.448824 | 3.587191 | 0.367714 | 03:12 |
| 8 | 3.456172 | 3.549420 | 0.374629 | 03:12 |
| 9 | 3.353690 | 3.518477 | 0.378029 | 03:12 |
| 10 | 3.329306 | 3.500824 | 0.381371 | 03:12 |
| 11 | 3.270015 | 3.491282 | 0.382200 | 03:14 |
| 12 | 3.166214 | 3.475223 | 0.384343 | 03:13 |
| 13 | 3.155823 | 3.467666 | 0.387829 | 03:12 |
| 14 | 3.136682 | 3.457926 | 0.389714 | 03:13 |
| 15 | 3.022880 | 3.463706 | 0.388857 | 03:13 |
| 16 | 3.058692 | 3.461311 | 0.389086 | 03:12 |
| 17 | 3.008492 | 3.461708 | 0.388857 | 03:12 |
| 18 | 2.982608 | 3.462014 | 0.389171 | 03:12 |
| 19 | 2.988810 | 3.461781 | 0.389457 | 03:13 |
Copy sp model from /home/pczapla/.fastai/models/ja_multifit_paper_version to data/cls/ja-music/models/sp15k/multifit_paper_version this Learner object self-destroyed - it still exists, but no longer usable Saving dump to data/cls/ja-music/models/sp15k/multifit_paper_version/finetuning.json Language model saved to data/cls/ja-music/models/sp15k/multifit_paper_version
In [12]:
exp.load_(cls_dataset.cache_path/exp.pretrain_lm.name).finetune_lmOut [12]:
Loading data/cls/ja-music/models/sp15k/multifit_paper_version/classifier.json Loading data/cls/ja-music/models/sp15k/multifit_paper_version/finetuning.json ULMFiTClassifier Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_version ULMFiTClassifier Replacing dataset_path 'None' with 'data/cls/ja-music ULMFiTClassifier Replacing bs '18' with '20 ULMFiTClassifier Replacing wd '0.01' with '0.1 ULMFiTClassifier Replacing early_stopping 'None' with 'accuracy
ULMFiTFinetuning(seed=0, name='multifit_paper_version', experiment_path=PosixPath('data/cls/ja-music/models/sp15k/multifit_paper_version'), dataset_path=PosixPath('data/cls/ja-music'), num_epochs=20, bs=20, bptt=70, drop_mult=0.3, dropout_values={'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}, label_smoothing_eps=0.0, label_smoothing_eps_norm_by_classes=True, use_adam_08=False, true_wd=False, wd=1e-07, clip=0.12, fp16=False, lr=0.001)In [13]:
exp.classifier.train_(seed=0)Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k
Data lm-notst, trn: 31800, val: 200
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']
Data cls, trn: 1800, val: 200
Data tst, trn: 200, val: 1999
Using Label smoothing with eps = 0.05
Setting Classifier weights seed seed to 0
Training args: {'drop_mult': 0.5, 'wd': 0.1, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-music/models/sp15k/multifit_paper_version/enc_best
Setting Classifier training seed seed to 0
Training: data/cls/ja-music/models/sp15k/multifit_paper_version
Total time: 01:38
| epoch | train_loss | valid_loss | accuracy | time |
|---|---|---|---|---|
| 0 | 0.551692 | 0.544244 | 0.750000 | 00:11 |
| 1 | 0.514899 | 0.617670 | 0.750000 | 00:12 |
| 2 | 0.443367 | 0.596144 | 0.785000 | 00:11 |
| 3 | 0.368091 | 0.567668 | 0.790000 | 00:11 |
| 4 | 0.315757 | 0.635034 | 0.805000 | 00:12 |
| 5 | 0.271595 | 0.515227 | 0.815000 | 00:11 |
| 6 | 0.241717 | 0.523387 | 0.820000 | 00:11 |
| 7 | 0.237796 | 0.518763 | 0.825000 | 00:11 |
Better model found at epoch 0 with accuracy value: 0.75. Better model found at epoch 2 with accuracy value: 0.7850000262260437. Better model found at epoch 3 with accuracy value: 0.7900000214576721. Better model found at epoch 4 with accuracy value: 0.8050000071525574. Better model found at epoch 5 with accuracy value: 0.8149999976158142. Better model found at epoch 6 with accuracy value: 0.8199999928474426. Better model found at epoch 7 with accuracy value: 0.824999988079071. Classifier model saved to data/cls/ja-music/models/sp15k/multifit_paper_version Saving dump to data/cls/ja-music/models/sp15k/multifit_paper_version/classifier.json this Learner object self-destroyed - it still exists, but no longer usable
In [14]:
exp.classifier.train_(seed=1)Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k
Data lm-notst, trn: 31800, val: 200
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']
Data cls, trn: 1800, val: 200
Data tst, trn: 200, val: 1999
Using Label smoothing with eps = 0.05
Setting Classifier weights seed seed to 1
Training args: {'drop_mult': 0.5, 'wd': 0.1, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-music/models/sp15k/multifit_paper_version/enc_best
Setting Classifier training seed seed to 1
Training: data/cls/ja-music/models/sp15k/multifit_paper_versionseed1
Total time: 01:35
| epoch | train_loss | valid_loss | accuracy | time |
|---|---|---|---|---|
| 0 | 0.553418 | 0.493184 | 0.800000 | 00:11 |
| 1 | 0.517754 | 0.603011 | 0.745000 | 00:11 |
| 2 | 0.444649 | 0.451021 | 0.850000 | 00:12 |
| 3 | 0.378163 | 0.488551 | 0.845000 | 00:12 |
| 4 | 0.296332 | 0.722608 | 0.745000 | 00:11 |
| 5 | 0.267031 | 0.528215 | 0.815000 | 00:11 |
| 6 | 0.248884 | 0.522899 | 0.830000 | 00:11 |
| 7 | 0.234614 | 0.512867 | 0.825000 | 00:12 |
Better model found at epoch 0 with accuracy value: 0.800000011920929. Better model found at epoch 2 with accuracy value: 0.8500000238418579. Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k/multifit_paper_versionseed1 Classifier model saved to data/cls/ja-music/models/sp15k/multifit_paper_versionseed1 Saving dump to data/cls/ja-music/models/sp15k/multifit_paper_versionseed1/classifier.json this Learner object self-destroyed - it still exists, but no longer usable
In [15]:
exp.classifier.train_(seed=2)Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k
Data lm-notst, trn: 31800, val: 200
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']
Data cls, trn: 1800, val: 200
Data tst, trn: 200, val: 1999
Using Label smoothing with eps = 0.05
Setting Classifier weights seed seed to 2
Training args: {'drop_mult': 0.5, 'wd': 0.1, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-music/models/sp15k/multifit_paper_version/enc_best
Setting Classifier training seed seed to 2
Training: data/cls/ja-music/models/sp15k/multifit_paper_versionseed2
Total time: 01:35
| epoch | train_loss | valid_loss | accuracy | time |
|---|---|---|---|---|
| 0 | 0.522099 | 0.587937 | 0.780000 | 00:11 |
| 1 | 0.501093 | 0.749337 | 0.755000 | 00:11 |
| 2 | 0.431629 | 0.610693 | 0.765000 | 00:12 |
| 3 | 0.369956 | 0.518467 | 0.820000 | 00:12 |
| 4 | 0.300080 | 0.621023 | 0.810000 | 00:11 |
| 5 | 0.273065 | 0.548695 | 0.810000 | 00:11 |
| 6 | 0.244343 | 0.521141 | 0.820000 | 00:12 |
| 7 | 0.228544 | 0.508984 | 0.820000 | 00:11 |
Better model found at epoch 0 with accuracy value: 0.7799999713897705. Better model found at epoch 3 with accuracy value: 0.8199999928474426. Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k/multifit_paper_versionseed2 Classifier model saved to data/cls/ja-music/models/sp15k/multifit_paper_versionseed2 Saving dump to data/cls/ja-music/models/sp15k/multifit_paper_versionseed2/classifier.json this Learner object self-destroyed - it still exists, but no longer usable
In [16]:
exp.classifier.train_(seed=3)Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k
Data lm-notst, trn: 31800, val: 200
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']
Data cls, trn: 1800, val: 200
Data tst, trn: 200, val: 1999
Using Label smoothing with eps = 0.05
Setting Classifier weights seed seed to 3
Training args: {'drop_mult': 0.5, 'wd': 0.1, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-music/models/sp15k/multifit_paper_version/enc_best
Setting Classifier training seed seed to 3
Training: data/cls/ja-music/models/sp15k/multifit_paper_versionseed3
Total time: 01:36
| epoch | train_loss | valid_loss | accuracy | time |
|---|---|---|---|---|
| 0 | 0.540789 | 0.530250 | 0.775000 | 00:11 |
| 1 | 0.494581 | 0.509151 | 0.815000 | 00:12 |
| 2 | 0.446126 | 0.529051 | 0.785000 | 00:11 |
| 3 | 0.376928 | 0.582771 | 0.775000 | 00:11 |
| 4 | 0.322823 | 0.492915 | 0.835000 | 00:11 |
| 5 | 0.270082 | 0.499838 | 0.850000 | 00:12 |
| 6 | 0.249619 | 0.512006 | 0.840000 | 00:11 |
| 7 | 0.240476 | 0.515156 | 0.820000 | 00:11 |
Better model found at epoch 0 with accuracy value: 0.7749999761581421. Better model found at epoch 1 with accuracy value: 0.8149999976158142. Better model found at epoch 4 with accuracy value: 0.8349999785423279. Better model found at epoch 5 with accuracy value: 0.8500000238418579. Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k/multifit_paper_versionseed3 Classifier model saved to data/cls/ja-music/models/sp15k/multifit_paper_versionseed3 Saving dump to data/cls/ja-music/models/sp15k/multifit_paper_versionseed3/classifier.json this Learner object self-destroyed - it still exists, but no longer usable
In [17]:
exp.classifier.train_(seed=4)Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k
Data lm-notst, trn: 31800, val: 200
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']
Data cls, trn: 1800, val: 200
Data tst, trn: 200, val: 1999
Using Label smoothing with eps = 0.05
Setting Classifier weights seed seed to 4
Training args: {'drop_mult': 0.5, 'wd': 0.1, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-music/models/sp15k/multifit_paper_version/enc_best
Setting Classifier training seed seed to 4
Training: data/cls/ja-music/models/sp15k/multifit_paper_versionseed4
Total time: 01:38
| epoch | train_loss | valid_loss | accuracy | time |
|---|---|---|---|---|
| 0 | 0.557735 | 0.474914 | 0.805000 | 00:12 |
| 1 | 0.497810 | 0.519276 | 0.785000 | 00:11 |
| 2 | 0.445226 | 0.632691 | 0.775000 | 00:12 |
| 3 | 0.376329 | 0.523593 | 0.810000 | 00:11 |
| 4 | 0.315562 | 0.491826 | 0.815000 | 00:11 |
| 5 | 0.277676 | 0.531970 | 0.805000 | 00:11 |
| 6 | 0.241653 | 0.491889 | 0.840000 | 00:12 |
| 7 | 0.240878 | 0.505452 | 0.835000 | 00:11 |
Better model found at epoch 0 with accuracy value: 0.8050000071525574. Better model found at epoch 3 with accuracy value: 0.8100000023841858. Better model found at epoch 4 with accuracy value: 0.8149999976158142. Better model found at epoch 6 with accuracy value: 0.8399999737739563. Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k/multifit_paper_versionseed4 Classifier model saved to data/cls/ja-music/models/sp15k/multifit_paper_versionseed4 Saving dump to data/cls/ja-music/models/sp15k/multifit_paper_versionseed4/classifier.json this Learner object self-destroyed - it still exists, but no longer usable
In [18]:
exp.classifier.train_(seed=5)Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k
Data lm-notst, trn: 31800, val: 200
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']
Data cls, trn: 1800, val: 200
Data tst, trn: 200, val: 1999
Using Label smoothing with eps = 0.05
Setting Classifier weights seed seed to 5
Training args: {'drop_mult': 0.5, 'wd': 0.1, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-music/models/sp15k/multifit_paper_version/enc_best
Setting Classifier training seed seed to 5
Training: data/cls/ja-music/models/sp15k/multifit_paper_versionseed5
Total time: 01:35
| epoch | train_loss | valid_loss | accuracy | time |
|---|---|---|---|---|
| 0 | 0.539067 | 0.524193 | 0.820000 | 00:12 |
| 1 | 0.504846 | 0.563058 | 0.775000 | 00:11 |
| 2 | 0.455630 | 0.554939 | 0.785000 | 00:11 |
| 3 | 0.366007 | 0.535870 | 0.810000 | 00:11 |
| 4 | 0.302207 | 0.502417 | 0.825000 | 00:11 |
| 5 | 0.274910 | 0.543057 | 0.815000 | 00:12 |
| 6 | 0.257135 | 0.517289 | 0.815000 | 00:11 |
| 7 | 0.234617 | 0.505733 | 0.800000 | 00:11 |
Better model found at epoch 0 with accuracy value: 0.8199999928474426. Better model found at epoch 4 with accuracy value: 0.824999988079071. Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k/multifit_paper_versionseed5 Classifier model saved to data/cls/ja-music/models/sp15k/multifit_paper_versionseed5 Saving dump to data/cls/ja-music/models/sp15k/multifit_paper_versionseed5/classifier.json this Learner object self-destroyed - it still exists, but no longer usable
In [19]:
exp.classifier.train_(seed=6)Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k
Data lm-notst, trn: 31800, val: 200
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']
Data cls, trn: 1800, val: 200
Data tst, trn: 200, val: 1999
Using Label smoothing with eps = 0.05
Setting Classifier weights seed seed to 6
Training args: {'drop_mult': 0.5, 'wd': 0.1, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-music/models/sp15k/multifit_paper_version/enc_best
Setting Classifier training seed seed to 6
Training: data/cls/ja-music/models/sp15k/multifit_paper_versionseed6
Total time: 01:38
| epoch | train_loss | valid_loss | accuracy | time |
|---|---|---|---|---|
| 0 | 0.529963 | 0.533260 | 0.750000 | 00:11 |
| 1 | 0.501183 | 0.518192 | 0.785000 | 00:11 |
| 2 | 0.429616 | 0.633020 | 0.800000 | 00:12 |
| 3 | 0.382384 | 0.526554 | 0.830000 | 00:11 |
| 4 | 0.315402 | 0.520821 | 0.825000 | 00:11 |
| 5 | 0.274772 | 0.551267 | 0.820000 | 00:12 |
| 6 | 0.249956 | 0.519960 | 0.830000 | 00:12 |
| 7 | 0.226764 | 0.516419 | 0.825000 | 00:11 |
Better model found at epoch 0 with accuracy value: 0.75. Better model found at epoch 1 with accuracy value: 0.7850000262260437. Better model found at epoch 2 with accuracy value: 0.800000011920929. Better model found at epoch 3 with accuracy value: 0.8299999833106995. Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k/multifit_paper_versionseed6 Classifier model saved to data/cls/ja-music/models/sp15k/multifit_paper_versionseed6 Saving dump to data/cls/ja-music/models/sp15k/multifit_paper_versionseed6/classifier.json this Learner object self-destroyed - it still exists, but no longer usable
In [20]:
exp.classifier.train_(seed=7)Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k
Data lm-notst, trn: 31800, val: 200
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']
Data cls, trn: 1800, val: 200
Data tst, trn: 200, val: 1999
Using Label smoothing with eps = 0.05
Setting Classifier weights seed seed to 7
Training args: {'drop_mult': 0.5, 'wd': 0.1, 'pretrained': False, 'bptt': 70, 'loss_func': FlattenedLoss of LabelSmoothingCrossEntropy(), 'clip': 0.12} config: {'emb_sz': 400, 'n_hid': 1550, 'n_layers': 4, 'pad_token': 1, 'qrnn': True, 'bidir': False, 'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-music/models/sp15k/multifit_paper_version/enc_best
Setting Classifier training seed seed to 7
Training: data/cls/ja-music/models/sp15k/multifit_paper_versionseed7
Total time: 01:36
| epoch | train_loss | valid_loss | accuracy | time |
|---|---|---|---|---|
| 0 | 0.565436 | 0.517827 | 0.805000 | 00:11 |
| 1 | 0.517789 | 0.710051 | 0.795000 | 00:11 |
| 2 | 0.456400 | 0.502067 | 0.805000 | 00:11 |
| 3 | 0.367990 | 0.560270 | 0.800000 | 00:11 |
| 4 | 0.305195 | 0.585719 | 0.805000 | 00:11 |
| 5 | 0.276869 | 0.496807 | 0.840000 | 00:12 |
| 6 | 0.244716 | 0.502482 | 0.845000 | 00:11 |
| 7 | 0.238059 | 0.496951 | 0.845000 | 00:11 |
Better model found at epoch 0 with accuracy value: 0.8050000071525574. Better model found at epoch 5 with accuracy value: 0.8399999737739563. Better model found at epoch 6 with accuracy value: 0.8450000286102295. Copy sp model from data/cls/ja-music/models/sp15k/multifit_paper_version to data/cls/ja-music/models/sp15k/multifit_paper_versionseed7 Classifier model saved to data/cls/ja-music/models/sp15k/multifit_paper_versionseed7 Saving dump to data/cls/ja-music/models/sp15k/multifit_paper_versionseed7/classifier.json this Learner object self-destroyed - it still exists, but no longer usable
In [13]:
def get_results(exp_path):
exp = multifit.ULMFiT().load_(exp_path, silent=False).classifier
results = exp.validate(use_cache=True)
results.update(seed=exp.seed, fp16=exp.fp16)
return results
results = [get_results(exp_path) for exp_path in cls_dataset.cache_path.glob(exp.pretrain_lm.name+"seed*")]
results_df = pd.DataFrame.from_records(results)
results_df.sort_values(["valid accuracy"])[["name", "seed", "test accuracy", "valid accuracy"]]Out [13]:
Loading data/cls/ja-music/models/sp15k/multifit_paper_versionseed5/classifier.json
Loading data/cls/ja-music/models/sp15k/multifit_paper_version/finetuning.json
ULMFiTFinetuning Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_version
ULMFiTFinetuning Replacing dataset_path 'None' with 'data/cls/ja-music
ULMFiTFinetuning Replacing num_epochs '10' with '20
ULMFiTFinetuning Replacing drop_mult '1.0' with '0.3
ULMFiTFinetuning Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
ULMFiTFinetuning Replacing true_wd 'True' with 'False
ULMFiTFinetuning Replacing wd '0.01' with '1e-07
ULMFiTFinetuning Replacing clip 'None' with '0.12
ULMFiTArchitecture Replacing tokenizer_type 'f' with 'sp
ULMFiTArchitecture Replacing max_vocab '60000' with '15000
ULMFiTArchitecture Replacing lang 'None' with 'ja
ULMFiTArchitecture Replacing n_hid '1150' with '1550
ULMFiTArchitecture Replacing n_layers '3' with '4
ULMFiTArchitecture Replacing qrnn 'False' with 'True
ULMFiTClassifier Replacing seed '0' with '5
ULMFiTClassifier Replacing name 'multifit_paper_versionseed5' with 'multifit_paper_version
ULMFiTClassifier Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_versionseed5
ULMFiTClassifier Replacing dataset_path 'None' with 'data/cls/ja-music
ULMFiTClassifier Replacing num_epochs '10' with '8
ULMFiTClassifier Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
ULMFiTClassifier Replacing wd '0.01' with '0.1
ULMFiTClassifier Replacing clip 'None' with '0.12
ULMFiTClassifier Replacing label_smoothing_eps '0.0' with '0.1
ULMFiTClassifier Replacing label_smoothing_eps_norm_by_classes 'False' with 'True
Loading data/cls/ja-music/models/sp15k/multifit_paper_versionseed4/classifier.json
Loading data/cls/ja-music/models/sp15k/multifit_paper_version/finetuning.json
ULMFiTFinetuning Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_version
ULMFiTFinetuning Replacing dataset_path 'None' with 'data/cls/ja-music
ULMFiTFinetuning Replacing num_epochs '10' with '20
ULMFiTFinetuning Replacing drop_mult '1.0' with '0.3
ULMFiTFinetuning Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
ULMFiTFinetuning Replacing true_wd 'True' with 'False
ULMFiTFinetuning Replacing wd '0.01' with '1e-07
ULMFiTFinetuning Replacing clip 'None' with '0.12
ULMFiTArchitecture Replacing tokenizer_type 'f' with 'sp
ULMFiTArchitecture Replacing max_vocab '60000' with '15000
ULMFiTArchitecture Replacing lang 'None' with 'ja
ULMFiTArchitecture Replacing n_hid '1150' with '1550
ULMFiTArchitecture Replacing n_layers '3' with '4
ULMFiTArchitecture Replacing qrnn 'False' with 'True
ULMFiTClassifier Replacing seed '0' with '4
ULMFiTClassifier Replacing name 'multifit_paper_versionseed4' with 'multifit_paper_version
ULMFiTClassifier Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_versionseed4
ULMFiTClassifier Replacing dataset_path 'None' with 'data/cls/ja-music
ULMFiTClassifier Replacing num_epochs '10' with '8
ULMFiTClassifier Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
ULMFiTClassifier Replacing wd '0.01' with '0.1
ULMFiTClassifier Replacing clip 'None' with '0.12
ULMFiTClassifier Replacing label_smoothing_eps '0.0' with '0.1
ULMFiTClassifier Replacing label_smoothing_eps_norm_by_classes 'False' with 'True
Loading data/cls/ja-music/models/sp15k/multifit_paper_versionseed3/classifier.json
Loading data/cls/ja-music/models/sp15k/multifit_paper_version/finetuning.json
ULMFiTFinetuning Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_version
ULMFiTFinetuning Replacing dataset_path 'None' with 'data/cls/ja-music
ULMFiTFinetuning Replacing num_epochs '10' with '20
ULMFiTFinetuning Replacing drop_mult '1.0' with '0.3
ULMFiTFinetuning Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
ULMFiTFinetuning Replacing true_wd 'True' with 'False
ULMFiTFinetuning Replacing wd '0.01' with '1e-07
ULMFiTFinetuning Replacing clip 'None' with '0.12
ULMFiTArchitecture Replacing tokenizer_type 'f' with 'sp
ULMFiTArchitecture Replacing max_vocab '60000' with '15000
ULMFiTArchitecture Replacing lang 'None' with 'ja
ULMFiTArchitecture Replacing n_hid '1150' with '1550
ULMFiTArchitecture Replacing n_layers '3' with '4
ULMFiTArchitecture Replacing qrnn 'False' with 'True
ULMFiTClassifier Replacing seed '0' with '3
ULMFiTClassifier Replacing name 'multifit_paper_versionseed3' with 'multifit_paper_version
ULMFiTClassifier Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_versionseed3
ULMFiTClassifier Replacing dataset_path 'None' with 'data/cls/ja-music
ULMFiTClassifier Replacing num_epochs '10' with '8
ULMFiTClassifier Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
ULMFiTClassifier Replacing wd '0.01' with '0.1
ULMFiTClassifier Replacing clip 'None' with '0.12
ULMFiTClassifier Replacing label_smoothing_eps '0.0' with '0.1
ULMFiTClassifier Replacing label_smoothing_eps_norm_by_classes 'False' with 'True
Loading data/cls/ja-music/models/sp15k/multifit_paper_versionseed1/classifier.json
Loading data/cls/ja-music/models/sp15k/multifit_paper_version/finetuning.json
ULMFiTFinetuning Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_version
ULMFiTFinetuning Replacing dataset_path 'None' with 'data/cls/ja-music
ULMFiTFinetuning Replacing num_epochs '10' with '20
ULMFiTFinetuning Replacing drop_mult '1.0' with '0.3
ULMFiTFinetuning Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
ULMFiTFinetuning Replacing true_wd 'True' with 'False
ULMFiTFinetuning Replacing wd '0.01' with '1e-07
ULMFiTFinetuning Replacing clip 'None' with '0.12
ULMFiTArchitecture Replacing tokenizer_type 'f' with 'sp
ULMFiTArchitecture Replacing max_vocab '60000' with '15000
ULMFiTArchitecture Replacing lang 'None' with 'ja
ULMFiTArchitecture Replacing n_hid '1150' with '1550
ULMFiTArchitecture Replacing n_layers '3' with '4
ULMFiTArchitecture Replacing qrnn 'False' with 'True
ULMFiTClassifier Replacing seed '0' with '1
ULMFiTClassifier Replacing name 'multifit_paper_versionseed1' with 'multifit_paper_version
ULMFiTClassifier Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_versionseed1
ULMFiTClassifier Replacing dataset_path 'None' with 'data/cls/ja-music
ULMFiTClassifier Replacing num_epochs '10' with '8
ULMFiTClassifier Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
ULMFiTClassifier Replacing wd '0.01' with '0.1
ULMFiTClassifier Replacing clip 'None' with '0.12
ULMFiTClassifier Replacing label_smoothing_eps '0.0' with '0.1
ULMFiTClassifier Replacing label_smoothing_eps_norm_by_classes 'False' with 'True
Loading data/cls/ja-music/models/sp15k/multifit_paper_versionseed2/classifier.json
Loading data/cls/ja-music/models/sp15k/multifit_paper_version/finetuning.json
ULMFiTFinetuning Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_version
ULMFiTFinetuning Replacing dataset_path 'None' with 'data/cls/ja-music
ULMFiTFinetuning Replacing num_epochs '10' with '20
ULMFiTFinetuning Replacing drop_mult '1.0' with '0.3
ULMFiTFinetuning Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
ULMFiTFinetuning Replacing true_wd 'True' with 'False
ULMFiTFinetuning Replacing wd '0.01' with '1e-07
ULMFiTFinetuning Replacing clip 'None' with '0.12
ULMFiTArchitecture Replacing tokenizer_type 'f' with 'sp
ULMFiTArchitecture Replacing max_vocab '60000' with '15000
ULMFiTArchitecture Replacing lang 'None' with 'ja
ULMFiTArchitecture Replacing n_hid '1150' with '1550
ULMFiTArchitecture Replacing n_layers '3' with '4
ULMFiTArchitecture Replacing qrnn 'False' with 'True
ULMFiTClassifier Replacing seed '0' with '2
ULMFiTClassifier Replacing name 'multifit_paper_versionseed2' with 'multifit_paper_version
ULMFiTClassifier Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_versionseed2
ULMFiTClassifier Replacing dataset_path 'None' with 'data/cls/ja-music
ULMFiTClassifier Replacing num_epochs '10' with '8
ULMFiTClassifier Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
ULMFiTClassifier Replacing wd '0.01' with '0.1
ULMFiTClassifier Replacing clip 'None' with '0.12
ULMFiTClassifier Replacing label_smoothing_eps '0.0' with '0.1
ULMFiTClassifier Replacing label_smoothing_eps_norm_by_classes 'False' with 'True
Loading data/cls/ja-music/models/sp15k/multifit_paper_versionseed6/classifier.json
Loading data/cls/ja-music/models/sp15k/multifit_paper_version/finetuning.json
ULMFiTFinetuning Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_version
ULMFiTFinetuning Replacing dataset_path 'None' with 'data/cls/ja-music
ULMFiTFinetuning Replacing num_epochs '10' with '20
ULMFiTFinetuning Replacing drop_mult '1.0' with '0.3
ULMFiTFinetuning Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
ULMFiTFinetuning Replacing true_wd 'True' with 'False
ULMFiTFinetuning Replacing wd '0.01' with '1e-07
ULMFiTFinetuning Replacing clip 'None' with '0.12
ULMFiTArchitecture Replacing tokenizer_type 'f' with 'sp
ULMFiTArchitecture Replacing max_vocab '60000' with '15000
ULMFiTArchitecture Replacing lang 'None' with 'ja
ULMFiTArchitecture Replacing n_hid '1150' with '1550
ULMFiTArchitecture Replacing n_layers '3' with '4
ULMFiTArchitecture Replacing qrnn 'False' with 'True
ULMFiTClassifier Replacing seed '0' with '6
ULMFiTClassifier Replacing name 'multifit_paper_versionseed6' with 'multifit_paper_version
ULMFiTClassifier Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_versionseed6
ULMFiTClassifier Replacing dataset_path 'None' with 'data/cls/ja-music
ULMFiTClassifier Replacing num_epochs '10' with '8
ULMFiTClassifier Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
ULMFiTClassifier Replacing wd '0.01' with '0.1
ULMFiTClassifier Replacing clip 'None' with '0.12
ULMFiTClassifier Replacing label_smoothing_eps '0.0' with '0.1
ULMFiTClassifier Replacing label_smoothing_eps_norm_by_classes 'False' with 'True
Loading data/cls/ja-music/models/sp15k/multifit_paper_versionseed7/classifier.json
Loading data/cls/ja-music/models/sp15k/multifit_paper_version/finetuning.json
ULMFiTFinetuning Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_version
ULMFiTFinetuning Replacing dataset_path 'None' with 'data/cls/ja-music
ULMFiTFinetuning Replacing num_epochs '10' with '20
ULMFiTFinetuning Replacing drop_mult '1.0' with '0.3
ULMFiTFinetuning Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
ULMFiTFinetuning Replacing true_wd 'True' with 'False
ULMFiTFinetuning Replacing wd '0.01' with '1e-07
ULMFiTFinetuning Replacing clip 'None' with '0.12
ULMFiTArchitecture Replacing tokenizer_type 'f' with 'sp
ULMFiTArchitecture Replacing max_vocab '60000' with '15000
ULMFiTArchitecture Replacing lang 'None' with 'ja
ULMFiTArchitecture Replacing n_hid '1150' with '1550
ULMFiTArchitecture Replacing n_layers '3' with '4
ULMFiTArchitecture Replacing qrnn 'False' with 'True
ULMFiTClassifier Replacing seed '0' with '7
ULMFiTClassifier Replacing name 'multifit_paper_versionseed7' with 'multifit_paper_version
ULMFiTClassifier Replacing experiment_path 'None' with 'data/cls/ja-music/models/sp15k/multifit_paper_versionseed7
ULMFiTClassifier Replacing dataset_path 'None' with 'data/cls/ja-music
ULMFiTClassifier Replacing num_epochs '10' with '8
ULMFiTClassifier Replacing dropout_values '{}' with '{'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
ULMFiTClassifier Replacing wd '0.01' with '0.1
ULMFiTClassifier Replacing clip 'None' with '0.12
ULMFiTClassifier Replacing label_smoothing_eps '0.0' with '0.1
ULMFiTClassifier Replacing label_smoothing_eps_norm_by_classes 'False' with 'True
| name | seed | test accuracy | valid accuracy | |
|---|---|---|---|---|
| 4 | multifit_paper_version | 2 | 0.848424 | 0.820 |
| 0 | multifit_paper_version | 5 | 0.851926 | 0.825 |
| 5 | multifit_paper_version | 6 | 0.842421 | 0.830 |
| 1 | multifit_paper_version | 4 | 0.870435 | 0.840 |
| 6 | multifit_paper_version | 7 | 0.861431 | 0.845 |
| 2 | multifit_paper_version | 3 | 0.854427 | 0.850 |
| 3 | multifit_paper_version | 1 | 0.860930 | 0.850 |
In [ ]: