# ES ## SP30k LSTM nl 4 ### LM ```` python -m ulmfit lm --dataset-path data/wiki/es-100 --cuda-id=0 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 30000 --lang es --qrnn=False - train 10 --bs=50 --drop_mult=0 Running tokenization Wiki text was split to 96224 articles Wiki text was split to 105 articles Size of vocabulary: 30000 First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que'] Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': None, 'pretrained_model': None, 'drop_mult': 0} dps: [0.25 0.1 0.2 0.02 0.15] Training lm from random weights epoch train_loss valid_loss accuracy 1 3.269541 3.451855 0.387471 2 3.161740 3.423016 0.386158 3 3.187431 3.419638 0.388626 4 3.115763 3.357066 0.393877 5 2.996527 3.291787 0.402488 6 3.021759 3.202183 0.410873 7 2.998267 3.104373 0.422624 8 2.827225 3.006537 0.436010 9 2.784576 2.937735 0.446654 10 2.789913 2.918509 0.450055 data/wiki/es-100/models/sp30k Saving info data/wiki/es-100/models/sp30k/lstm_nl4.m/info.json ```` ### MLDoc ``` python -m ulmfit cls --dataset-path data/mldoc/es-1 --base-lm-path data/wiki/es-100/models/sp30k/lstm_nl4.m --lang=es --name 'nl4' --cuda-id=1 - train 20 --bs 40 --num-cls-epochs=2 Max vocab: 30000 Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4.m Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv Running tokenization... Saving tokenized: cls.trn 13013, cls.val 1445 Running tokenization... Saving tokenized: cls.trn 1000, cls.val 1000 Size of vocabulary: 30000 First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que'] Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/es-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/es-100/models/sp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15] Unknown tokens 0, first 100: [] Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/es-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/es-100/models/sp30k/lstm_nl4.m/../itos')] epoch train_loss valid_loss accuracy 1 2.805415 2.188974 0.537779 epoch train_loss valid_loss accuracy 1 2.429727 1.989691 0.569048 2 2.218828 1.794969 0.603721 3 2.015097 1.644815 0.629609 4 1.877210 1.537773 0.646898 5 1.775648 1.450283 0.660861 6 1.749334 1.377085 0.672146 7 1.601073 1.311101 0.684400 8 1.564420 1.251074 0.694900 9 1.532728 1.197607 0.704779 10 1.391921 1.145408 0.716044 11 1.379958 1.093550 0.726937 12 1.324111 1.048308 0.735890 13 1.344113 1.007926 0.745691 14 1.243085 0.969521 0.754591 15 1.230809 0.937330 0.762675 16 1.162501 0.913408 0.768044 17 1.170092 0.894892 0.773239 18 1.110860 0.884449 0.775603 19 1.115907 0.880448 0.776671 20 1.083033 0.878421 0.776931 /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4.m/info.json Starting classifier training epoch train_loss valid_loss accuracy 1 0.621574 0.391042 0.856000 epoch train_loss valid_loss accuracy 1 0.411668 0.215625 0.935000 epoch train_loss valid_loss accuracy 1 0.340519 0.222422 0.935000 epoch train_loss valid_loss accuracy 1 0.281729 0.192193 0.949000 2 0.262074 0.202975 0.945000 Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4.m Loss and accuracy using (cls_best): [0.1749019, tensor(0.9515)] ``` ## ES optimization ### Smaler vocab 15k #### LM ``` python -m ulmfit lm --dataset-path data/wiki-m/es-100 --cuda-id=0 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 \ ✘ 1 --lang es --qrnn=False - train 10 --bs=50 --drop_mult=0 Max vocab: 15000 Cache dir: data/wiki-m/es-100/models/sp15k Model dir: data/wiki-m/es-100/models/sp15k/lstm_nl4.m Tokenized data loaded Size of vocabulary: 15000 First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} Training lm from random weights epoch train_loss valid_loss accuracy 1 2.961702 3.187043 0.403149 Better model found at epoch 1 with val_loss value: 3.1870434284210205. 2 2.928991 3.170802 0.402026 Better model found at epoch 2 with val_loss value: 3.170802354812622. 3 2.931906 3.128328 0.407816 Better model found at epoch 3 with val_loss value: 3.128328323364258. 4 2.869332 3.072160 0.414345 Better model found at epoch 4 with val_loss value: 3.072160243988037. 5 2.803377 2.997071 0.424847 Better model found at epoch 5 with val_loss value: 2.997070550918579. 6 2.758087 2.927369 0.432256 Better model found at epoch 6 with val_loss value: 2.927368640899658. 7 2.657733 2.825029 0.446440 Better model found at epoch 7 with val_loss value: 2.8250293731689453. 8 2.563273 2.728652 0.459271 Better model found at epoch 8 with val_loss value: 2.7286524772644043. 9 2.475741 2.654844 0.470864 Better model found at epoch 9 with val_loss value: 2.654844045639038. 10 2.428898 2.634355 0.474821 Better model found at epoch 10 with val_loss value: 2.634355306625366. Total time: 17:53:59 data/wiki-m/es-100/models/sp15k Saving info data/wiki-m/es-100/models/sp15k/lstm_nl4.m/info.json ``` #### MLDoc ``` python -m ulmfit cls --dataset-path data/mldoc/es-1 --base-lm-path data/wiki-m/es-100/models/sp15k/lstm_nl4.m --lang=es --name 'nl4' --cuda-id=0 - train 20 --bs 20 --num-cls-epochs=8 Max vocab: 15000 Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/lstm_nl4.m Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv Running tokenization... Saving tokenized: cls.trn 13013, cls.val 1445 Running tokenization... Saving tokenized: cls.trn 1000, cls.val 1000 Size of vocabulary: 15000 First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} Loading pretrained model Unknown tokens 0, first 100: [] Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki-m/es-100/models/sp15k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki-m/es-100/models/sp15k/lstm_nl4.m/../itos')] epoch train_loss valid_loss accuracy 1 2.355042 1.850417 0.589128 Better model found at epoch 1 with val_loss value: 1.850416898727417. Total time: 03:25 epoch train_loss valid_loss accuracy 1 2.087364 1.677666 0.619909 Better model found at epoch 1 with val_loss value: 1.6776657104492188. 2 1.880730 1.522996 0.648134 Better model found at epoch 2 with val_loss value: 1.5229955911636353. 3 1.767530 1.403644 0.668117 Better model found at epoch 3 with val_loss value: 1.4036436080932617. 4 1.659950 1.309353 0.684900 Better model found at epoch 4 with val_loss value: 1.3093526363372803. 5 1.546585 1.232220 0.699358 Better model found at epoch 5 with val_loss value: 1.2322196960449219. 6 1.592862 1.161846 0.713034 Better model found at epoch 6 with val_loss value: 1.1618456840515137. 7 1.444965 1.098108 0.726811 Better model found at epoch 7 with val_loss value: 1.0981075763702393. 8 1.340874 1.029193 0.741337 Better model found at epoch 8 with val_loss value: 1.0291931629180908. 9 1.351407 0.974317 0.753408 Better model found at epoch 9 with val_loss value: 0.9743167757987976. 10 1.231713 0.915328 0.767088 Better model found at epoch 10 with val_loss value: 0.9153280854225159. 11 1.151926 0.852391 0.782414 Better model found at epoch 11 with val_loss value: 0.852391242980957. 12 1.163565 0.794699 0.797228 Better model found at epoch 12 with val_loss value: 0.7946987152099609. 13 1.054929 0.743652 0.810518 Better model found at epoch 13 with val_loss value: 0.74365234375. 14 0.974651 0.695024 0.823344 Better model found at epoch 14 with val_loss value: 0.6950243711471558. 15 0.869718 0.651691 0.834510 Better model found at epoch 15 with val_loss value: 0.6516908407211304. 16 0.889763 0.615112 0.844947 Better model found at epoch 16 with val_loss value: 0.6151121258735657. 17 0.843503 0.590130 0.851694 Better model found at epoch 17 with val_loss value: 0.5901297926902771. 18 0.752870 0.575217 0.855496 Better model found at epoch 18 with val_loss value: 0.5752172470092773. 19 0.807087 0.567187 0.857605 Better model found at epoch 19 with val_loss value: 0.5671872496604919. 20 0.784531 0.566082 0.857827 Better model found at epoch 20 with val_loss value: 0.5660821199417114. Total time: 1:26:48 /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/lstm_nl4.m/info.json Starting classifier training epoch train_loss valid_loss accuracy 1 0.616963 0.275447 0.923000 Better model found at epoch 1 with val_loss value: 0.27544698119163513. Total time: 00:24 epoch train_loss valid_loss accuracy 1 0.425890 0.201535 0.932000 Better model found at epoch 1 with val_loss value: 0.2015346735715866. Total time: 00:27 epoch train_loss valid_loss accuracy 1 0.265563 0.186434 0.951000 Better model found at epoch 1 with val_loss value: 0.18643426895141602. Total time: 00:32 epoch train_loss valid_loss accuracy 1 0.162097 0.180026 0.955000 Better model found at epoch 1 with val_loss value: 0.1800260841846466. 2 0.161748 0.187014 0.957000 3 0.142789 0.166486 0.961000 Better model found at epoch 3 with val_loss value: 0.1664857715368271. 4 0.105920 0.173207 0.963000 5 0.078164 0.184849 0.962000 6 0.070540 0.189451 0.962000 7 0.051490 0.208019 0.959000 8 0.047614 0.193699 0.962000 Total time: 05:20 Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/lstm_nl4.m Loss and accuracy using (cls_best): [0.1623594, tensor(0.9538)] 0.16235940158367157 0.9537500143051147 ``` ### Larger dropout - no luck ``` python -m ulmfit cls --dataset-path data/mldoc/es-1 --base-lm-path data/wiki-m/es-100/models/sp30k/lstm_nl4.m --lang=es --name 'nl4-drop' --cuda-id=0 - train 0 --bs 20 --num-cls-epochs=8 --drop-mul-lm=0.5 --drop-mul-cls=0.8 Max vocab: 30000 Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4-drop.m Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv Running tokenization... Saving tokenized: cls.trn 13013, cls.val 1445 Running tokenization... Saving tokenized: cls.trn 1000, cls.val 1000 Size of vocabulary: 30000 First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que'] Starting classifier training epoch train_loss valid_loss accuracy 1 1.310594 0.903076 0.724000 Better model found at epoch 1 with val_loss value: 0.9030755758285522. Total time: 00:22 epoch train_loss valid_loss accuracy 1 1.209348 0.714140 0.756000 Better model found at epoch 1 with val_loss value: 0.714139997959137. Total time: 00:23 epoch train_loss valid_loss accuracy 1 1.122636 0.627526 0.797000 Better model found at epoch 1 with val_loss value: 0.6275263428688049. Total time: 00:29 epoch train_loss valid_loss accuracy 1 1.102433 0.593338 0.801000 Better model found at epoch 1 with val_loss value: 0.5933384895324707. 2 1.096480 0.543266 0.818000 Better model found at epoch 2 with val_loss value: 0.5432664155960083. 3 1.082919 0.501089 0.837000 Better model found at epoch 3 with val_loss value: 0.5010889172554016. 4 1.069694 0.518807 0.812000 5 1.040208 0.508399 0.825000 6 1.032841 0.512187 0.838000 7 1.031225 0.504557 0.825000 8 1.016486 0.502335 0.837000 Total time: 04:56 Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4-drop.m Loss and accuracy using (cls_best): [0.52473265, tensor(0.8160)] 0.5247326493263245 0.8159999847412109 ```