From 4742a68d28bc8929ae79d0ebcdfd1d82325b4573 Mon Sep 17 00:00:00 2001 From: Piotr Czapla Date: Wed, 27 Feb 2019 20:31:38 +0100 Subject: [PATCH] Some logs from previous experiments --- results/MLDoc.md | 13 +- results/logs/100examples.md | 1017 +++++++++++++++ results/logs/_experiment-log.md | 57 - results/logs/cls.md | 172 --- results/logs/cls/ulmfit on cls.md | 1092 +++++++++++++++++ results/logs/cls/zeroshoot.md | 118 ++ results/logs/de.md | 66 + results/logs/label_smoothing.md | 951 ++++++++++++++ results/logs/{ => mldoc}/noise/de10k-noise.md | 0 results/logs/mldoc/noise/es10k-noise-lstm.md | 370 ++++++ results/logs/{ => mldoc}/noise/es10k-noise.md | 978 +++++++++++++++ results/logs/qrnn-ru.md | 184 ++- results/logs/ru.md | 3 + results/logs/ru/bs_vs_lr.md | 123 ++ results/logs/ru/lm-opti.md | 260 ++++ results/logs/ru/merity4.md | 101 ++ results/logs/ru/merity5.md | 38 + results/logs/ru/wide2.md | 183 +++ results/logs/zeroshot.md | 398 +++++- 19 files changed, 5881 insertions(+), 243 deletions(-) create mode 100644 results/logs/100examples.md delete mode 100644 results/logs/_experiment-log.md delete mode 100644 results/logs/cls.md create mode 100644 results/logs/cls/ulmfit on cls.md create mode 100644 results/logs/cls/zeroshoot.md rename results/logs/{ => mldoc}/noise/de10k-noise.md (100%) create mode 100644 results/logs/mldoc/noise/es10k-noise-lstm.md rename results/logs/{ => mldoc}/noise/es10k-noise.md (55%) create mode 100644 results/logs/ru/bs_vs_lr.md create mode 100644 results/logs/ru/lm-opti.md create mode 100644 results/logs/ru/merity4.md create mode 100644 results/logs/ru/merity5.md create mode 100644 results/logs/ru/wide2.md diff --git a/results/MLDoc.md b/results/MLDoc.md index 5448079..b8a9ff1 100644 --- a/results/MLDoc.md +++ b/results/MLDoc.md @@ -9,8 +9,9 @@ |ULMFiT L30k-100 | | 91.35 | 83.32 | 88.77 | 77.99 | 71.12 | 72.20 | | |ULMFiT L30k | | 95.4 | 95.15 | 93.67 | 88.42 | 89.20 | **87.27** | 90.20 | |ULMFiT sp-fixed | | **95.6** | 94.80 | 94.20 | 88.52 | 88.72 | 86.85 | 90.47 | -|ULMFIT Q15k 1cyc| **94.62** | **95.65** | 95.15 | **94.42** | **89.92** | 89.60 | | 90.78/89.82 | -|ULMFIT Q15k 1c l| | | | | | | | **92.22** | +|ULMFIT Q15k 1cyc| 94.62 | **95.65** | 95.15 | **94.42** | 89.92 | 89.60 | | 90.78/89.82 | +|ULMFIT Q15k 1c l| **94.99** | | 95.64 | 94.34 | **90.32** | 89.67 | 87.67^ | **92.22** | +|ULMFIT Q15k 1cfl| **95.55** | **96.10** | 95.82 | 94.80 | **90.04** | 89.87 | 87.17 | **91.90** | |ULMFIT L30k 1cyc| | **95.85** | **96.32** | **94.82** | 89.87 | **90.45** | **87.94** | 92.02/91.64 | - L30k - LSTM sp30k trained using gradual unfreezing @@ -18,6 +19,8 @@ - ULMFiT sp-fixed - --||-- with fixed tokenization - Q15k 1cyc - QRNN sp15k trained using 1cycle learning rate schedule - L30k 1cyc - LSTM sp30k trained using 1cycle learning rate schedule +- We checked LSTM on sp15k on DE and got 95.53% accuracy which is comparable to QRNN sp15k +- ^ - 16 epochs qrnn_nl4sl-bs500 ## Zero shot approaches - LSTM @@ -77,9 +80,9 @@ | Impr 10k over 10k | 37% | 12% | 39% | 15% | 9% | 23% | | Impr 10k over 1k | 34% | 19% | 29% | 21% | 11% | 26% | | Impr 1k over 1k | 43% | 20% | 30% | 17% | 10% | 16% | - - -All ULMFiT examples above were trained on 1k training data generated by a LASER classification model +| ULMFiT qrnn on 1k LSRen1k | 91.32 | 78.92 | 89.45 | 75.99 | | 82.45 | +| ULMFiT qrnn on 10k LSRen1k| 91.90 | 78.79 | 88.47 | 76.05 | | | + ## Noise resistance diff --git a/results/logs/100examples.md b/results/logs/100examples.md new file mode 100644 index 0000000..602ec9b --- /dev/null +++ b/results/logs/100examples.md @@ -0,0 +1,1017 @@ +# MLDoc classification using 100 examples + +| Language | 8 epochs (1) | 4 epochs (1) | 4 epochs (2) | 8 epochs (2) | +|-------------|--------------------|---------------------|------------------|----------| +| de | 92.37 | 91.79 | 84.60 | 91.27 | +| en | 77.14 | 66.02 | 70.85 | 87.00 | +| es | 89.52 | 87.55 | 80.17 | 89.57 | +| fr | 81.44 | 74.25 | 79.97 | 88.15 | +| it | 81.15 | 69.24 | 74.17 | 77.54 | +| ja | 78.87 | 70.30 | 69.74 | 78.64 | +| zh | 83.57 | 70.47 | 77.39 | 87.17 | + +- (1) - a larger dropout value for output_p=0.7 instead of output_p=0.2, and wd=1e-1 +- (2) - normal dropout but still wd=1e-1 +- (3) - normal dropout but and normal wd=1e-2 + +## QRNN sp15k - normal dropout, normal wd + + + +## QRNN sp15k - normal dropout, wd=1e-1 + +### 8 epochs +```python -m ulmfit eval --glob="mldoc/*-1/models/sp15k/qrnn_nl4.m" --name nl4-100-e8-normal-dp --limit=100 --num-cls-epochs=8 --lr_sched=1cycle --label-smoothing-eps=0.1 --bs=18 +Processing data/mldoc/de-1/models/sp15k/qrnn_nl4.m +de-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m +Training +Dropout {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/de.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) may be ABI-incompatible with PyTorch! +Please use a compiler that is ABI-compatible with GCC 4.9 and above. +See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html. + +See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6 +for instructions on how to install GCC 4.9 or higher. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler)) +Loading pretrained model +Unknown tokens 0, first 100: [] +Bptt 70 +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.346962 1.354738 0.360000 +2 1.083917 1.077360 0.700000 +3 0.919886 0.868146 0.820000 +4 0.820790 0.769731 0.930000 +5 0.743299 0.783199 0.840000 +6 0.705464 0.722734 0.910000 +7 0.673815 0.693874 0.940000 +8 0.640916 0.677240 0.940000 +Total time: 00:22 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m +Loss and accuracy using (cls_best): [0.49304065, tensor(0.9128)] +Processing data/mldoc/en-1/models/sp15k/qrnn_nl4.m +en-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m +Training +Dropout {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/en.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁the', '▁,', 's', '▁.', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Bptt 70 +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.351400 1.369022 0.290000 +2 1.125228 1.163452 0.670000 +3 0.966749 1.370436 0.350000 +4 0.876823 0.914753 0.760000 +5 0.797715 0.889629 0.760000 +6 0.739767 0.833061 0.800000 +7 0.699772 0.787709 0.810000 +8 0.669680 0.758130 0.820000 +Total time: 00:21 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m +Loss and accuracy using (cls_best): [0.55098367, tensor(0.8700)] +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m +Training +Dropout {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv +Limiting data set to: 100 +Data lm, trn: 13013, val: 1445 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Bptt 70 +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.324274 1.354593 0.450000 +2 1.104991 1.030298 0.810000 +3 0.937743 0.761312 0.890000 +4 0.846890 0.772061 0.860000 +5 0.762197 0.816325 0.850000 +6 0.711926 0.747758 0.890000 +7 0.669704 0.714664 0.910000 +8 0.635587 0.706967 0.910000 +Total time: 00:19 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m +Loss and accuracy using (cls_best): [0.50185955, tensor(0.8957)] +Processing data/mldoc/fr-1/models/sp15k/qrnn_nl4.m +fr-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m +Training +Dropout {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/fr.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Bptt 70 +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.311578 1.391519 0.200000 +2 1.070487 1.097862 0.730000 +3 0.908473 0.982642 0.700000 +4 0.847152 0.954593 0.670000 +5 0.773764 0.740223 0.890000 +6 0.720973 0.732911 0.860000 +7 0.680840 0.725618 0.880000 +8 0.646478 0.711748 0.880000 +Total time: 00:22 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m +Loss and accuracy using (cls_best): [0.5118136, tensor(0.8815)] +Processing data/mldoc/it-1/models/sp15k/qrnn_nl4.m +it-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m +Training +Dropout {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/it.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁di', '▁e', "▁&'", "'", '▁il', '▁la', '▁in', 'e'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Bptt 70 +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.342177 1.374349 0.230000 +2 1.180374 1.193104 0.720000 +3 1.057902 0.981448 0.750000 +4 0.964218 1.267683 0.390000 +5 0.865437 0.927968 0.720000 +6 0.801164 0.933495 0.740000 +7 0.753568 0.910974 0.740000 +8 0.710150 0.893166 0.750000 +Total time: 00:15 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m +Loss and accuracy using (cls_best): [0.6821853, tensor(0.7755)] +Processing data/mldoc/ja-1/models/sp15k/qrnn_nl4.m +ja-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m +Training +Dropout {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Bptt 70 +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.390914 1.373147 0.450000 +2 1.179142 1.231387 0.400000 +3 0.984903 1.356539 0.410000 +4 0.934303 1.067314 0.570000 +5 0.862403 0.971283 0.660000 +6 0.789983 0.919483 0.700000 +7 0.738834 0.877783 0.770000 +8 0.698692 0.849805 0.790000 +Total time: 00:24 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m +Loss and accuracy using (cls_best): [0.7138088, tensor(0.7865)] +Processing data/mldoc/zh-1/models/sp15k/qrnn_nl4.m +zh-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m +Training +Dropout {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Bptt 70 +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.281217 1.359122 0.370000 +2 1.047607 1.106353 0.720000 +3 0.879789 0.842300 0.790000 +4 0.796210 0.830705 0.810000 +5 0.733291 0.782483 0.830000 +6 0.681939 0.757719 0.870000 +7 0.642002 0.751473 0.880000 +8 0.611668 0.735970 0.860000 +Total time: 00:20 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m +Loss and accuracy using (cls_best): [0.56022245, tensor(0.8717)] +OrderedDict([('data/mldoc/de-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m', + 0.9127500057220459), + ('data/mldoc/en-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m', + 0.8700000047683716), + ('data/mldoc/es-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m', + 0.8957499861717224), + ('data/mldoc/fr-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m', + 0.8815000057220459), + ('data/mldoc/it-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m', + 0.7754999995231628), + ('data/mldoc/ja-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m', + 0.7864999771118164), + ('data/mldoc/zh-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m', + 0.871749997138977)]) +data/mldoc/de-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m: 0.9127500057220459 +data/mldoc/en-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m: 0.8700000047683716 +data/mldoc/es-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m: 0.8957499861717224 +data/mldoc/fr-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m: 0.8815000057220459 +data/mldoc/it-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m: 0.7754999995231628 +data/mldoc/ja-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m: 0.7864999771118164 +data/mldoc/zh-1/models/sp15k/qrnn_nl4-100-e8-normal-dp.m: 0.871749997138977 +``` + + +### 4 epochs +``` +python -m ulmfit eval --glob="mldoc/*-1/models/sp15k/qrnn_nl4.m" --name nl4-100-e4-normal-dp --limit=100 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1 --bs=18 +Processing data/mldoc/de-1/models/sp15k/qrnn_nl4.m +de-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m +Training +Dropout {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/de.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) may be ABI-incompatible with PyTorch! +Please use a compiler that is ABI-compatible with GCC 4.9 and above. +See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html. + +See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6 +for instructions on how to install GCC 4.9 or higher. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler)) +Loading pretrained model +Unknown tokens 0, first 100: [] +Bptt 70 +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.284350 1.332569 0.280000 +2 1.016471 0.983958 0.840000 +3 0.867660 0.927123 0.910000 +4 0.761539 0.911453 0.840000 +Total time: 00:11 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m +Loss and accuracy using (cls_best): [0.819043, tensor(0.8460)] +Processing data/mldoc/en-1/models/sp15k/qrnn_nl4.m +en-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m +Training +Dropout {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/en.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁the', '▁,', 's', '▁.', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Bptt 70 +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.290013 1.338139 0.370000 +2 1.052497 1.222131 0.380000 +3 0.904921 1.098593 0.440000 +4 0.809407 0.984065 0.680000 +Total time: 00:10 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m +Loss and accuracy using (cls_best): [0.89315945, tensor(0.7085)] +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m +Training +Dropout {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv +Limiting data set to: 100 +Data lm, trn: 13013, val: 1445 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Bptt 70 +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.286507 1.319209 0.390000 +2 1.063101 1.230659 0.450000 +3 0.925757 1.026052 0.600000 +4 0.812203 0.907415 0.800000 +Total time: 00:09 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m +Loss and accuracy using (cls_best): [0.7879555, tensor(0.8018)] +Processing data/mldoc/fr-1/models/sp15k/qrnn_nl4.m +fr-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m +Training +Dropout {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/fr.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Bptt 70 +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.267655 1.295306 0.460000 +2 1.034598 1.207809 0.360000 +3 0.869741 0.965482 0.690000 +4 0.772019 0.871552 0.810000 +Total time: 00:10 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m +Loss and accuracy using (cls_best): [0.7594081, tensor(0.7997)] +Processing data/mldoc/it-1/models/sp15k/qrnn_nl4.m +it-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m +Training +Dropout {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/it.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁di', '▁e', "▁&'", "'", '▁il', '▁la', '▁in', 'e'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Bptt 70 +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.338249 1.345434 0.320000 +2 1.179869 1.195013 0.480000 +3 1.027482 1.109100 0.560000 +4 0.900706 1.021052 0.730000 +Total time: 00:07 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m +Loss and accuracy using (cls_best): [0.8819375, tensor(0.7418)] +Processing data/mldoc/ja-1/models/sp15k/qrnn_nl4.m +ja-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m +Training +Dropout {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Bptt 70 +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.329939 1.365095 0.200000 +2 1.109706 1.175099 0.420000 +3 0.978721 1.141211 0.440000 +4 0.870699 1.028071 0.700000 +Total time: 00:12 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m +Loss and accuracy using (cls_best): [0.9101604, tensor(0.6975)] +Processing data/mldoc/zh-1/models/sp15k/qrnn_nl4.m +zh-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m +Training +Dropout {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Bptt 70 +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.256988 1.290418 0.370000 +2 1.000337 0.967188 0.780000 +3 0.837002 0.955509 0.760000 +4 0.740278 0.937393 0.780000 +Total time: 00:10 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m +Loss and accuracy using (cls_best): [0.8290863, tensor(0.7740)] +OrderedDict([('data/mldoc/de-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m', + 0.8460000157356262), + ('data/mldoc/en-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m', + 0.7085000276565552), + ('data/mldoc/es-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m', + 0.8017500042915344), + ('data/mldoc/fr-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m', + 0.7997499704360962), + ('data/mldoc/it-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m', + 0.7417500019073486), + ('data/mldoc/ja-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m', + 0.6974999904632568), + ('data/mldoc/zh-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m', + 0.7739999890327454)]) +data/mldoc/de-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m: 0.8460000157356262 +data/mldoc/en-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m: 0.7085000276565552 +data/mldoc/es-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m: 0.8017500042915344 +data/mldoc/fr-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m: 0.7997499704360962 +data/mldoc/it-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m: 0.7417500019073486 +data/mldoc/ja-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m: 0.6974999904632568 +data/mldoc/zh-1/models/sp15k/qrnn_nl4-100-e4-normal-dp.m: 0.7739999890327454 +``` + + +## QRNN sp15k - larger dropout +The following dropouts were used: `output_p=0.7` compared to other experimejnts where `output_p=0.2` + +| Model | Accuracy 8 epochs | Accuracy 4 epochs +|-----------------------------|--------------------|---------------------| +| de-1 sp15k/qrnn_nl4-100e8.m | 92.37 | 91.79 | +| en-1 sp15k/qrnn_nl4-100e8.m | 77.14 | 66.02 | +| es-1 sp15k/qrnn_nl4-100e8.m | 89.52 | 87.55 | +| fr-1 sp15k/qrnn_nl4-100e8.m | 81.44 | 74.25 | +| it-1 sp15k/qrnn_nl4-100e8.m | 81.15 | 69.24 | +| ja-1 sp15k/qrnn_nl4-100e8.m | 78.87 | 70.30 | +| zh-1 sp15k/qrnn_nl4-100e8.m | 83.57 | 70.47 | + +### 8 epochs of training +`self.dps = dict(output_p=0.7, hidden_p=0.1, input_p=0.2, embed_p=0.02, weight_p=0.15)` +```` +python -m ulmfit eval --glob="mldoc/*-1/models/sp15k/qrnn_nl4.m" --name nl4-100e8 --limit=100 --num-cls-epochs=8 --lr_sched=1cycle --label-smoothing-eps=0.1 --bs=18 +Processing data/mldoc/de-1/models/sp15k/qrnn_nl4.m +de-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-100e8.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/de.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"] +/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) may be ABI-incompatible with PyTorch! +Please use a compiler that is ABI-compatible with GCC 4.9 and above. +See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html. + +See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6 +for instructions on how to install GCC 4.9 or higher. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler)) +Single training schedule +epoch train_loss valid_loss accuracy +1 1.354484 1.379077 0.280000 +2 1.152421 1.152570 0.620000 +3 0.980928 1.018830 0.660000 +4 0.871912 0.872564 0.820000 +5 0.803019 0.789034 0.900000 +6 0.748509 0.711389 0.940000 +7 0.701289 0.690140 0.950000 +8 0.664756 0.669448 0.950000 +Total time: 00:23 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-100e8.m +Loss and accuracy using (cls_best): [0.48774713, tensor(0.9237)] +Processing data/mldoc/en-1/models/sp15k/qrnn_nl4.m +en-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-100e8.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/en.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁the', '▁,', 's', '▁.', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed'] +Single training schedule +epoch train_loss valid_loss accuracy +1 1.366923 1.380464 0.310000 +2 1.177802 1.236425 0.500000 +3 0.999587 0.912387 0.740000 +4 0.898221 0.990741 0.660000 +5 0.818071 1.066497 0.580000 +6 0.757039 0.921970 0.680000 +7 0.712545 0.956183 0.660000 +8 0.681197 0.911881 0.710000 +Total time: 00:22 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-100e8.m +Loss and accuracy using (cls_best): [0.66596776, tensor(0.7715)] +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-100e8.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv +Limiting data set to: 100 +Data lm, trn: 13013, val: 1445 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Single training schedule +epoch train_loss valid_loss accuracy +1 1.253322 1.366251 0.220000 +2 1.051253 1.117081 0.650000 +3 0.905148 0.928994 0.770000 +4 0.825990 0.855558 0.810000 +5 0.754039 0.828022 0.810000 +6 0.704161 0.752015 0.880000 +7 0.661291 0.739049 0.880000 +8 0.631924 0.731397 0.880000 +Total time: 00:19 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-100e8.m +Loss and accuracy using (cls_best): [0.5359205, tensor(0.8953)] +Processing data/mldoc/fr-1/models/sp15k/qrnn_nl4.m +fr-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-100e8.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/fr.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à'] +Single training schedule +epoch train_loss valid_loss accuracy +1 1.315270 1.368941 0.220000 +2 1.117436 1.137058 0.630000 +3 0.959049 0.955980 0.750000 +4 0.856808 0.749577 0.900000 +5 0.800600 0.859013 0.740000 +6 0.739985 0.873451 0.720000 +7 0.696956 0.820335 0.780000 +8 0.660949 0.784974 0.820000 +Total time: 00:22 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-100e8.m +Loss and accuracy using (cls_best): [0.6222675, tensor(0.8145)] +Processing data/mldoc/it-1/models/sp15k/qrnn_nl4.m +it-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-100e8.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/it.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁di', '▁e', "▁&'", "'", '▁il', '▁la', '▁in', 'e'] +Single training schedule +epoch train_loss valid_loss accuracy +1 1.406870 1.370643 0.250000 +2 1.248155 1.226949 0.620000 +3 1.098265 1.045475 0.650000 +4 0.989543 0.999063 0.660000 +5 0.904690 0.908897 0.750000 +6 0.838460 0.906191 0.770000 +7 0.790075 0.867051 0.810000 +8 0.748298 0.844697 0.820000 +Total time: 00:16 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-100e8.m +Loss and accuracy using (cls_best): [0.6499791, tensor(0.8115)] +Processing data/mldoc/ja-1/models/sp15k/qrnn_nl4.m +ja-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-100e8.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)'] +Single training schedule +epoch train_loss valid_loss accuracy +1 1.284972 1.374997 0.390000 +2 1.112592 1.235691 0.490000 +3 0.990320 1.235275 0.470000 +4 0.884597 0.985890 0.680000 +5 0.814330 1.017439 0.660000 +6 0.756775 0.912423 0.780000 +7 0.757921 0.860843 0.840000 +8 0.731727 0.835498 0.810000 +Total time: 00:24 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-100e8.m +Loss and accuracy using (cls_best): [0.68836695, tensor(0.7887)] +Processing data/mldoc/zh-1/models/sp15k/qrnn_nl4.m +zh-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-100e8.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有'] +Single training schedule +epoch train_loss valid_loss accuracy +1 1.282265 1.340392 0.670000 +2 1.029523 1.092677 0.600000 +3 0.890638 1.027698 0.650000 +4 0.799896 0.919632 0.730000 +5 0.735252 0.840827 0.800000 +6 0.684259 0.791598 0.860000 +7 0.645315 0.794246 0.840000 +8 0.615542 0.782311 0.840000 +Total time: 00:20 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-100e8.m +Loss and accuracy using (cls_best): [0.63012207, tensor(0.8357)] +OrderedDict([('data/mldoc/de-1/models/sp15k/qrnn_nl4-100e8.m', + 0.9237499833106995), + ('data/mldoc/en-1/models/sp15k/qrnn_nl4-100e8.m', + 0.7714999914169312), + ('data/mldoc/es-1/models/sp15k/qrnn_nl4-100e8.m', + 0.8952500224113464), + ('data/mldoc/fr-1/models/sp15k/qrnn_nl4-100e8.m', + 0.8144999742507935), + ('data/mldoc/it-1/models/sp15k/qrnn_nl4-100e8.m', + 0.8115000128746033), + ('data/mldoc/ja-1/models/sp15k/qrnn_nl4-100e8.m', + 0.7887499928474426), + ('data/mldoc/zh-1/models/sp15k/qrnn_nl4-100e8.m', + 0.8357499837875366)]) +data/mldoc/de-1/models/sp15k/qrnn_nl4-100e8.m: 0.9237499833106995 +data/mldoc/en-1/models/sp15k/qrnn_nl4-100e8.m: 0.7714999914169312 +data/mldoc/es-1/models/sp15k/qrnn_nl4-100e8.m: 0.8952500224113464 +data/mldoc/fr-1/models/sp15k/qrnn_nl4-100e8.m: 0.8144999742507935 +data/mldoc/it-1/models/sp15k/qrnn_nl4-100e8.m: 0.8115000128746033 +data/mldoc/ja-1/models/sp15k/qrnn_nl4-100e8.m: 0.7887499928474426 +data/mldoc/zh-1/models/sp15k/qrnn_nl4-100e8.m: 0.8357499837875366 +```` + +### 4 epochs of training + +```` +python -m ulmfit eval --glob="mldoc/*-1/models/sp15k/qrnn_nl4.m" --name nl4-100e4 --limit=100 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1 --bs=18 +Processing data/mldoc/de-1/models/sp15k/qrnn_nl4.m +de-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-100e4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/de.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.7, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) may be ABI-incompatible with PyTorch! +Please use a compiler that is ABI-compatible with GCC 4.9 and above. +See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html. + +See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6 +for instructions on how to install GCC 4.9 or higher. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler)) +Loading pretrained model +Unknown tokens 0, first 100: [] +Bptt 70 +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-100e4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.268574 1.327856 0.330000 +2 1.006592 1.120209 0.470000 +3 0.865163 0.915048 0.860000 +4 0.775797 0.878599 0.910000 +Total time: 00:11 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-100e4.m +Loss and accuracy using (cls_best): [0.7670934, tensor(0.9180)] +Processing data/mldoc/en-1/models/sp15k/qrnn_nl4.m +en-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-100e4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/en.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁the', '▁,', 's', '▁.', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.7, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Bptt 70 +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-100e4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.346742 1.344590 0.400000 +2 1.097295 1.131111 0.570000 +3 0.927231 0.992626 0.650000 +4 0.823589 1.027419 0.600000 +Total time: 00:10 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-100e4.m +Loss and accuracy using (cls_best): [0.8792457, tensor(0.6603)] +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-100e4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv +Limiting data set to: 100 +Data lm, trn: 13013, val: 1445 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.7, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Bptt 70 +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-100e4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.249590 1.328701 0.370000 +2 0.977393 1.031057 0.540000 +3 0.858498 0.886364 0.850000 +4 0.777858 0.868647 0.890000 +Total time: 00:10 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-100e4.m +Loss and accuracy using (cls_best): [0.72082895, tensor(0.8755)] +Processing data/mldoc/fr-1/models/sp15k/qrnn_nl4.m +fr-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-100e4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/fr.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.7, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Bptt 70 +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-100e4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.305505 1.362959 0.200000 +2 1.073055 1.068938 0.700000 +3 0.892900 0.981664 0.620000 +4 0.796118 0.950983 0.700000 +Total time: 00:11 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-100e4.m +Loss and accuracy using (cls_best): [0.8226887, tensor(0.7425)] +Processing data/mldoc/it-1/models/sp15k/qrnn_nl4.m +it-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-100e4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/it.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁di', '▁e', "▁&'", "'", '▁il', '▁la', '▁in', 'e'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.7, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Bptt 70 +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-100e4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.372190 1.350273 0.500000 +2 1.207576 1.173455 0.750000 +3 1.040837 1.107950 0.590000 +4 0.931903 1.119772 0.540000 +Total time: 00:07 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-100e4.m +Loss and accuracy using (cls_best): [0.95755553, tensor(0.6925)] +Processing data/mldoc/ja-1/models/sp15k/qrnn_nl4.m +ja-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-100e4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.7, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Bptt 70 +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-100e4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.300252 1.326448 0.330000 +2 1.175279 1.211864 0.500000 +3 1.019274 1.129269 0.460000 +4 0.895053 1.059557 0.640000 +Total time: 00:11 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-100e4.m +Loss and accuracy using (cls_best): [0.96151954, tensor(0.7030)] +Processing data/mldoc/zh-1/models/sp15k/qrnn_nl4.m +zh-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-100e4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv +Limiting data set to: 100 +Data lm, trn: 13500, val: 1500 +Data clslimit100, trn: 100, val: 100 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.7, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Bptt 70 +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-100e4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.230801 1.302317 0.660000 +2 0.944098 0.992352 0.710000 +3 0.810783 1.026886 0.630000 +4 0.736866 0.985942 0.680000 +Total time: 00:09 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-100e4.m +Loss and accuracy using (cls_best): [0.8833954, tensor(0.7048)] +OrderedDict([('data/mldoc/de-1/models/sp15k/qrnn_nl4-100e4.m', + 0.9179999828338623), + ('data/mldoc/en-1/models/sp15k/qrnn_nl4-100e4.m', + 0.6602500081062317), + ('data/mldoc/es-1/models/sp15k/qrnn_nl4-100e4.m', + 0.8755000233650208), + ('data/mldoc/fr-1/models/sp15k/qrnn_nl4-100e4.m', + 0.7425000071525574), + ('data/mldoc/it-1/models/sp15k/qrnn_nl4-100e4.m', + 0.6924999952316284), + ('data/mldoc/ja-1/models/sp15k/qrnn_nl4-100e4.m', + 0.703000009059906), + ('data/mldoc/zh-1/models/sp15k/qrnn_nl4-100e4.m', + 0.7047500014305115)]) +data/mldoc/de-1/models/sp15k/qrnn_nl4-100e4.m: 0.9179999828338623 +data/mldoc/en-1/models/sp15k/qrnn_nl4-100e4.m: 0.6602500081062317 +data/mldoc/es-1/models/sp15k/qrnn_nl4-100e4.m: 0.8755000233650208 +data/mldoc/fr-1/models/sp15k/qrnn_nl4-100e4.m: 0.7425000071525574 +data/mldoc/it-1/models/sp15k/qrnn_nl4-100e4.m: 0.6924999952316284 +data/mldoc/ja-1/models/sp15k/qrnn_nl4-100e4.m: 0.703000009059906 +data/mldoc/zh-1/models/sp15k/qrnn_nl4-100e4.m: 0.7047500014305115 +```` \ No newline at end of file diff --git a/results/logs/_experiment-log.md b/results/logs/_experiment-log.md deleted file mode 100644 index 0d0d650..0000000 --- a/results/logs/_experiment-log.md +++ /dev/null @@ -1,57 +0,0 @@ - - - -## QRNN sp15k -``` -cd fastai # go to fast ai -git checkout ulfit_multilingual -git pull - -cd ../ulmfit-multilingual # go to ulmfit -git checkout master -git pull - -export CUDA_VISIBLE_DEVICES=1 -LANG=fr -python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0 - -## Jeremy -export CUDA_VISIBLE_DEVICES=2 -LANG=it -python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0 - -export CUDA_VISIBLE_DEVICES=3 -LANG=ru -python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0 -``` - - - -``` -# -export CUDA_VISIBLE_DEVICES=0 -LANG=de -python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0 -# - -## Piotr -export CUDA_VISIBLE_DEVICES=1 -LANG=es -python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0 - -export CUDA_VISIBLE_DEVICES=0 -LANG=zh -python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0 -``` - -# trained -export CUDA_VISIBLE_DEVICES=0 -LANG=ja -python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0 - -done V100 -``` -export CUDA_VISIBLE_DEVICES=0 -LANG=en -python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0 -``` \ No newline at end of file diff --git a/results/logs/cls.md b/results/logs/cls.md deleted file mode 100644 index bf45ebd..0000000 --- a/results/logs/cls.md +++ /dev/null @@ -1,172 +0,0 @@ -# without col merge -```` -python -m ulmfit cls --dataset-path data/cls/${LANG}-books --base-lm-path data/wiki-m/${LANG}-100/models/sp30k/lstm_nl4.m --lang=${LANG} --name 'nl4' - train 20 --bs 20 --num-cls-epochs=8 --lr-sched=single -Max vocab: 30000 -Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k -Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k/lstm_nl4.m -Validation set not found using 10% of trn -Data lm, trn: 33183, val: 3687 -Data cls, trn: 1800, val: 200 -Running tokenization tst... -Data tst, trn: 200, val: 2000 -Size of vocabulary: 30000 -First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁.', "'", 's', '▁la', '▁le', '▁et', '▁l', '▁à'] -Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} -Loading pretrained model -Unknown tokens 0, first 100: [] -Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki-m/fr-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki-m/fr-100/models/sp30k/lstm_nl4.m/../itos')] -epoch train_loss valid_loss accuracy -1 3.356221 2.821012 0.518492 -Total time: 00:21 -epoch train_loss valid_loss accuracy -1 3.041214 2.734799 0.524577 -2 2.919576 2.648412 0.535661 -3 2.822292 2.542236 0.549206 -4 2.721790 2.414110 0.561852 -5 2.596515 2.276732 0.579841 -6 2.453715 2.140479 0.600370 -7 2.333764 2.000186 0.621349 -8 2.231092 1.873927 0.644259 -9 2.101130 1.765473 0.660529 -10 2.006949 1.666797 0.682196 -11 1.905025 1.584023 0.696058 -12 1.820798 1.513958 0.709841 -13 1.751217 1.456632 0.720846 -14 1.689076 1.410359 0.729947 -15 1.646113 1.371438 0.739868 -16 1.594153 1.346142 0.744577 -17 1.564375 1.332298 0.746693 -18 1.536557 1.322925 0.748995 -19 1.532926 1.319159 0.749444 -20 1.525449 1.318028 0.749815 -Total time: 08:51 -/home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k -Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k/lstm_nl4.m/info.json -Single training schedule -epoch train_loss valid_loss accuracy -1 0.588118 0.581087 0.700000 -2 0.504373 0.583527 0.720000 -3 0.412651 0.538866 0.750000 -4 0.295401 0.658459 0.750000 -5 0.212442 1.054068 0.720000 -6 0.126090 1.302099 0.745000 -7 0.078312 1.307932 0.760000 -8 0.050346 1.339740 0.745000 -Total time: 00:35 -Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k/lstm_nl4.m -Loss and accuracy using (cls_best): [1.3513571, tensor(0.7700)] -1.351357102394104 -0.7699999809265137 -```` - -### FR books - -````bash -python -m ulmfit cls --dataset-path data/cls/${LANG}-books --base-lm-path data/wiki-m/${LANG}-100/models/sp30k/lstm_nl4.m --lang=${LANG} --name 'nl4' - train 20 --bs 20 --num-cls-epochs=8 --lr-sched=single -Max vocab: 30000 -Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k -Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k/lstm_nl4.m -Validation set not found using 10% of trn -Running tokenization lm... -Data lm, trn: 33183, val: 3687 -Running tokenization cls... -Data cls, trn: 1800, val: 200 -Running tokenization tst... -Data tst, trn: 200, val: 2000 -Size of vocabulary: 30000 -First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁.', "'", 's', '▁la', '▁le', '▁et', '▁l', '▁à'] -Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} -Loading pretrained model -Unknown tokens 0, first 100: [] -Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki-m/fr-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki-m/fr-100/models/sp30k/lstm_nl4.m/../itos')] -epoch train_loss valid_loss accuracy -1 3.790325 3.409294 0.367234 -Total time: 06:02 -epoch train_loss valid_loss accuracy -1 3.526303 3.326439 0.378936 -2 3.466923 3.226977 0.392378 -3 3.342312 3.111874 0.406997 -4 3.244619 2.992510 0.422330 -5 3.156150 2.877498 0.437467 -6 3.070326 2.762509 0.453874 -7 2.956969 2.651613 0.471552 -8 2.878008 2.535935 0.491058 -9 2.790110 2.438724 0.508560 -10 2.684145 2.323467 0.528415 -11 2.633781 2.231418 0.547093 -12 2.535126 2.143523 0.564889 -13 2.464436 2.055402 0.582077 -14 2.330094 1.989257 0.596582 -15 2.372371 1.924338 0.610048 -16 2.190224 1.866912 0.621738 -17 2.176868 1.834098 0.629221 -18 2.168293 1.809196 0.633879 -19 2.151132 1.797144 0.636382 -20 2.130476 1.793351 0.637044 -Total time: 2:30:05 -/home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k -Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k/lstm_nl4.m/info.json -Single training schedule -epoch train_loss valid_loss accuracy -1 0.314315 0.530879 0.865000 -2 0.336746 0.468635 0.865000 -3 0.255810 0.324242 0.870000 -4 0.149121 0.480570 0.885000 -5 0.093909 0.613743 0.890000 -6 0.091678 0.660452 0.885000 -7 0.049993 0.649642 0.910000 -8 0.034218 0.640008 0.910000 -Total time: 04:19 -Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k/lstm_nl4.m -Loss and accuracy using (cls_best): [0.5418505, tensor(0.9100)] -0.5418505072593689 -0.9100000262260437 -```` - -``` -Loading pretrained model -Unknown tokens 0, first 100: [] -Training lm from: [PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/../itos')] -epoch train_loss valid_loss accuracy -1 4.037580 3.312217 0.364410 -Total time: 01:44 -epoch train_loss valid_loss accuracy -1 3.709982 3.256320 0.371825 -2 3.459413 3.150574 0.386972 -3 3.296628 3.037327 0.402039 -4 3.186458 2.914899 0.418413 -5 3.092632 2.817097 0.431216 -6 2.966957 2.726081 0.442906 -7 2.924824 2.647339 0.453871 -8 2.818279 2.561596 0.466795 -9 2.773893 2.501994 0.475877 -10 2.736084 2.438490 0.485978 -11 2.688937 2.370927 0.496899 -12 2.615245 2.314875 0.506508 -13 2.583292 2.260717 0.515725 -14 2.535631 2.220295 0.522666 -15 2.466035 2.179093 0.530148 -16 2.461427 2.151952 0.535315 -17 2.390641 2.131065 0.538749 -18 2.376235 2.116927 0.541430 -19 2.407630 2.115370 0.542039 -20 2.391378 2.112687 0.542522 -Total time: 46:33 -/home/n-waves/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k -Saving info /home/n-waves/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k/qrnn_nl4.m/info.json -Single training schedule -epoch train_loss valid_loss accuracy -1 0.466671 0.534682 0.745000 -2 0.358965 0.372612 0.875000 -3 0.251557 0.311034 0.900000 -4 0.166484 0.585425 0.865000 -5 0.101803 0.726341 0.900000 -6 0.072025 0.587875 0.885000 -7 0.045328 0.760989 0.890000 -8 0.027765 0.727203 0.890000 -Total time: 01:17 -Saving models at /home/n-waves/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k/qrnn_nl4.m -Loss and accuracy using (cls_best): [0.55982095, tensor(0.8970)] -0.5598209500312805 -0.8970000147819519 -``` \ No newline at end of file diff --git a/results/logs/cls/ulmfit on cls.md b/results/logs/cls/ulmfit on cls.md new file mode 100644 index 0000000..2195f13 --- /dev/null +++ b/results/logs/cls/ulmfit on cls.md @@ -0,0 +1,1092 @@ +# CLS execution logs + +| model name | Accuracy | +|--------------------------------------------|----------| +| data/cls/de-books/models/sp15k/qrnn_nl4.m: | 93.19 | +| data/cls/de-dvd/models/sp15k/qrnn_nl4.m: | 90.54 | +| data/cls/de-music/models/sp15k/qrnn_nl4.m: | 93.00 | +| data/cls/en-books/models/sp15k/qrnn_nl4.m: | 90.75 | +| data/cls/en-dvd/models/sp15k/qrnn_nl4.m: | 89.30 | +| data/cls/en-music/models/sp15k/qrnn_nl4.m: | 89.45 | +| data/cls/fr-books/models/sp15k/qrnn_nl4.m: | 91.25 | +| data/cls/fr-dvd/models/sp15k/qrnn_nl4.m: | 89.55 | +| data/cls/fr-music/models/sp15k/qrnn_nl4.m: | 93.40 | +| data/cls/ja-books/models/sp15k/qrnn_nl4.m: | 86.29 | +| data/cls/ja-dvd/models/sp15k/qrnn_nl4.m: | 85.75 | +| data/cls/ja-music/models/sp15k/qrnn_nl4.m: | 86.59 | + +### All langs books + +``` +python -m ulmfit eval --glob="wiki/*-100/models/sp15k/qrnn_nl4.m" --name nl4 --dataset-template='../cls/${lang}-books' --num-lm-epochs=20 --num-cls-epochs=8 --bs=18 --lr_sched=1cycle --label-smoothing-eps=0.1 +Processing data/wiki/de-100/models/sp15k/qrnn_nl4.m +../cls/de-books +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books/models/sp15k/qrnn_nl4.m +Training +Validation set not found using 10% of trn +Error You have NaN values in column(s) of your dataframe, please fix it. +Processing data/wiki/en-100/models/sp15k/qrnn_nl4.m +../cls/en-books +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/en-books/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/en-books/models/sp15k/qrnn_nl4.m +Training +Validation set not found using 10% of trn +Running tokenization lm... +Data lm, trn: 48600, val: 5400 +Running tokenization cls... +Data cls, trn: 1800, val: 200 +Running tokenization tst... +Data tst, trn: 200, val: 2000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁the', '▁,', 's', '▁.', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) may be ABI-incompatible with PyTorch! +Please use a compiler that is ABI-compatible with GCC 4.9 and above. +See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html. + +See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6 +for instructions on how to install GCC 4.9 or higher. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler)) +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/en-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/en-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 5.411075 4.929663 0.299280 +Total time: 05:18 +epoch train_loss valid_loss accuracy +1 4.911766 4.729969 0.328842 +2 4.749516 4.571162 0.352025 +3 4.574124 4.432505 0.369642 +4 4.496035 4.315759 0.384135 +5 4.402706 4.238598 0.393190 +6 4.336373 4.176495 0.401667 +7 4.305870 4.111050 0.410795 +8 4.251368 4.055121 0.419378 +9 4.215136 4.005383 0.426659 +10 4.242251 3.959000 0.434112 +11 4.128317 3.919106 0.440762 +12 4.102277 3.881066 0.447852 +13 4.109466 3.840265 0.455235 +14 4.074686 3.798793 0.463265 +15 4.034000 3.765705 0.469861 +16 3.984559 3.736501 0.475469 +17 3.961936 3.718403 0.479141 +18 3.986044 3.708190 0.481330 +19 3.934352 3.700921 0.482685 +20 3.977421 3.698027 0.483279 +Total time: 2:33:09 +/home/pczapla/workspace/ulmfit-multilingual/data/cls/en-books/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/en-books/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.511342 0.485253 0.890000 +2 0.476206 0.497634 0.875000 +3 0.423625 0.495749 0.880000 +4 0.383525 0.472684 0.890000 +5 0.371009 0.455094 0.895000 +6 0.359045 0.453902 0.905000 +7 0.354646 0.453879 0.910000 +8 0.348541 0.462702 0.885000 +Total time: 02:30 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/en-books/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.27204505, tensor(0.9060)] +Processing data/wiki/es-100/models/sp15k/qrnn_nl4.m +../cls/es-books +Processing data/wiki/fr-100/models/sp15k/qrnn_nl4.m +../cls/fr-books +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k/qrnn_nl4.m +Training +Validation set not found using 10% of trn +Running tokenization lm... +Data lm, trn: 33183, val: 3687 +Running tokenization cls... +Data cls, trn: 1800, val: 200 +Running tokenization tst... +Data tst, trn: 200, val: 2000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 4.844616 4.303658 0.358888 +Total time: 02:32 +epoch train_loss valid_loss accuracy +1 4.558980 4.236440 0.368490 +2 4.336353 4.111036 0.389355 +3 4.215691 3.997827 0.406318 +4 4.124804 3.892600 0.422174 +5 4.052222 3.802997 0.435620 +6 3.958529 3.729386 0.446996 +7 3.889615 3.646023 0.460472 +8 3.860830 3.590203 0.470407 +9 3.763688 3.521166 0.482480 +10 3.735285 3.465606 0.493439 +11 3.699683 3.411256 0.504397 +12 3.666552 3.347546 0.517700 +13 3.580596 3.308754 0.526673 +14 3.551433 3.255782 0.537871 +15 3.517213 3.215991 0.546388 +16 3.480192 3.188064 0.553690 +17 3.416178 3.162537 0.559454 +18 3.391725 3.151500 0.561885 +19 3.400107 3.144848 0.563330 +20 3.400756 3.143354 0.563702 +Total time: 1:12:34 +/home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.532118 0.548179 0.820000 +2 0.495962 0.509888 0.860000 +3 0.427645 0.489982 0.900000 +4 0.385989 0.513428 0.850000 +5 0.372433 0.446763 0.905000 +6 0.364960 0.467484 0.885000 +7 0.350225 0.455279 0.905000 +8 0.347019 0.466804 0.885000 +Total time: 01:48 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.27588147, tensor(0.9130)] +Processing data/wiki/it-100/models/sp15k/qrnn_nl4.m +../cls/it-books +Processing data/wiki/ja-100/models/sp15k/qrnn_nl4.m +../cls/ja-books +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-books/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-books/models/sp15k/qrnn_nl4.m +Training +Validation set not found using 10% of trn +Error You have NaN values in column(s) of your dataframe, please fix it. +Processing data/wiki/ru-100/models/sp15k/qrnn_nl4.m +../cls/ru-books +Processing data/wiki/zh-100/models/sp15k/qrnn_nl4.m +../cls/zh-books +OrderedDict([('data/cls/en-books/models/sp15k/qrnn_nl4.m', 0.906000018119812), + ('data/cls/fr-books/models/sp15k/qrnn_nl4.m', 0.9129999876022339)]) +data/cls/en-books/models/sp15k/qrnn_nl4.m: 0.906000018119812 +``` +### All datasets +```bash +python -m ulmfit eval --glob="wiki/*-100/models/sp15k/qrnn_nl4.m" --name nl4 --dataset-template='../cls/${lang}-*' --num-lm-epochs=20 --num-cls-epochs=8 --bs=18 --lr_sched=1cycle --label-smoothing-eps=0.1 +Processing data/wiki/de-100/models/sp15k/qrnn_nl4.m +../cls/de-* +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books/models/sp15k/qrnn_nl4.m +Evaluating previously trained model +Validation set not found using 10% of trn +Data lm, trn: 152523, val: 16947 +Data cls, trn: 1800, val: 200 +Data tst, trn: 200, val: 2000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"] +/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) may be ABI-incompatible with PyTorch! +Please use a compiler that is ABI-compatible with GCC 4.9 and above. +See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html. + +See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6 +for instructions on how to install GCC 4.9 or higher. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler)) +Loss and accuracy using (cls_best): [0.23488419, tensor(0.9320)] +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-dvd/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-dvd/models/sp15k/qrnn_nl4.m +Training +Validation set not found using 10% of trn +Running tokenization lm... +Data lm, trn: 85965, val: 9551 +Running tokenization cls... +Data cls, trn: 1800, val: 200 +Running tokenization tst... +Data tst, trn: 200, val: 2000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/de-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/de-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 4.596065 4.161942 0.436606 +Total time: 11:33 +epoch train_loss valid_loss accuracy +1 4.127364 3.947751 0.466350 +2 3.918618 3.776825 0.490808 +3 3.800280 3.655171 0.506950 +4 3.690864 3.582598 0.516176 +5 3.698156 3.535314 0.522662 +6 3.610991 3.498398 0.527869 +7 3.592985 3.466045 0.532311 +8 3.566916 3.444989 0.535283 +9 3.563903 3.422111 0.539163 +10 3.557671 3.400061 0.542508 +11 3.486254 3.377414 0.546246 +12 3.489782 3.356982 0.549719 +13 3.451218 3.333650 0.553665 +14 3.464072 3.312912 0.557414 +15 3.431759 3.292514 0.560929 +16 3.387984 3.274624 0.564236 +17 3.405241 3.262639 0.566367 +18 3.374643 3.253385 0.568177 +19 3.374621 3.248559 0.569108 +20 3.377307 3.247425 0.569270 +Total time: 5:31:07 +/home/pczapla/workspace/ulmfit-multilingual/data/cls/de-dvd/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-dvd/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.519669 0.485357 0.865000 +2 0.479460 0.426823 0.920000 +3 0.431123 0.432494 0.885000 +4 0.386461 0.431496 0.915000 +5 0.365019 0.427543 0.920000 +6 0.355551 0.415073 0.930000 +7 0.346821 0.412974 0.930000 +8 0.347380 0.413793 0.935000 +Total time: 02:57 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-dvd/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.2746336, tensor(0.9055)] +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-music/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-music/models/sp15k/qrnn_nl4.m +Training +Validation set not found using 10% of trn +Running tokenization lm... +Data lm, trn: 57953, val: 6439 +Running tokenization cls... +Data cls, trn: 1800, val: 200 +Running tokenization tst... +Data tst, trn: 200, val: 2000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/de-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/de-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 4.650980 4.152651 0.433679 +Total time: 07:28 +epoch train_loss valid_loss accuracy +1 4.209763 3.965040 0.459827 +2 3.935442 3.798236 0.482987 +3 3.841647 3.667273 0.500565 +4 3.699245 3.578814 0.512209 +5 3.652425 3.512940 0.521151 +6 3.658080 3.467384 0.527574 +7 3.583572 3.427991 0.533427 +8 3.558736 3.393701 0.538799 +9 3.497921 3.360512 0.544095 +10 3.513804 3.333250 0.548685 +11 3.495725 3.303407 0.553937 +12 3.442093 3.278184 0.558270 +13 3.431648 3.249587 0.563309 +14 3.394415 3.224657 0.567926 +15 3.418715 3.201570 0.572469 +16 3.334841 3.180893 0.576305 +17 3.356635 3.166274 0.579287 +18 3.339209 3.156838 0.581144 +19 3.307631 3.151842 0.582141 +20 3.313863 3.150544 0.582361 +Total time: 3:28:27 +/home/pczapla/workspace/ulmfit-multilingual/data/cls/de-music/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-music/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.518283 0.456440 0.880000 +2 0.474427 0.466113 0.865000 +3 0.420782 0.572000 0.850000 +4 0.377621 0.467714 0.865000 +5 0.361156 0.430405 0.915000 +6 0.345733 0.429622 0.890000 +7 0.347064 0.419322 0.910000 +8 0.342849 0.419672 0.900000 +Total time: 02:41 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-music/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.24861218, tensor(0.9300)] +Processing data/wiki/en-100/models/sp15k/qrnn_nl4.m +../cls/en-* +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/en-books/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/en-books/models/sp15k/qrnn_nl4.m +Evaluating previously trained model +Validation set not found using 10% of trn +Data lm, trn: 48600, val: 5400 +Data cls, trn: 1800, val: 200 +Data tst, trn: 200, val: 2000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁the', '▁,', 's', '▁.', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed'] +Loss and accuracy using (cls_best): [0.2731999, tensor(0.9075)] +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/en-dvd/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/en-dvd/models/sp15k/qrnn_nl4.m +Training +Validation set not found using 10% of trn +Running tokenization lm... +Data lm, trn: 30600, val: 3400 +Running tokenization cls... +Data cls, trn: 1800, val: 200 +Running tokenization tst... +Data tst, trn: 200, val: 2000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁the', '▁,', 's', '▁.', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/en-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/en-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 5.459846 4.880299 0.304455 +Total time: 03:40 +epoch train_loss valid_loss accuracy +1 4.914763 4.696301 0.329923 +2 4.679338 4.542424 0.351552 +3 4.545784 4.381050 0.373713 +4 4.435306 4.261026 0.388227 +5 4.381686 4.146891 0.402335 +6 4.339701 4.057630 0.414286 +7 4.193101 3.985903 0.425350 +8 4.188499 3.904590 0.437700 +9 4.141142 3.831017 0.449915 +10 4.073668 3.769752 0.460387 +11 4.034776 3.704058 0.472169 +12 4.017927 3.645482 0.483226 +13 3.939799 3.571460 0.498373 +14 3.954270 3.544665 0.504316 +15 3.891879 3.494558 0.514514 +16 3.816200 3.460811 0.522036 +17 3.817204 3.436978 0.527317 +18 3.837135 3.423382 0.530152 +19 3.786701 3.413906 0.532207 +20 3.794206 3.412058 0.532526 +Total time: 1:43:34 +/home/pczapla/workspace/ulmfit-multilingual/data/cls/en-dvd/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/en-dvd/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.546413 0.469370 0.895000 +2 0.487449 0.455633 0.880000 +3 0.434380 0.471036 0.890000 +4 0.402179 0.457251 0.885000 +5 0.375344 0.444693 0.910000 +6 0.356960 0.440818 0.900000 +7 0.352019 0.427976 0.900000 +8 0.347301 0.427571 0.905000 +Total time: 02:39 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/en-dvd/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.28846368, tensor(0.8930)] +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/en-music/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/en-music/models/sp15k/qrnn_nl4.m +Training +Validation set not found using 10% of trn +Running tokenization lm... +Data lm, trn: 26298, val: 2922 +Running tokenization cls... +Data cls, trn: 1800, val: 200 +Running tokenization tst... +Data tst, trn: 200, val: 2000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁the', '▁,', 's', '▁.', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/en-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/en-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 5.360476 4.782286 0.319278 +Total time: 02:41 +epoch train_loss valid_loss accuracy +1 4.751396 4.553875 0.351461 +2 4.583414 4.389054 0.374384 +3 4.383287 4.250410 0.394109 +4 4.299458 4.122505 0.409391 +5 4.248813 4.014976 0.423715 +6 4.149672 3.927886 0.434895 +7 4.099454 3.838499 0.448373 +8 4.001449 3.762594 0.459935 +9 3.931528 3.687276 0.472542 +10 3.904028 3.625594 0.483216 +11 3.892965 3.560731 0.495610 +12 3.851050 3.500596 0.506427 +13 3.748619 3.447429 0.517146 +14 3.783010 3.407018 0.526054 +15 3.737605 3.356801 0.536732 +16 3.642893 3.315823 0.544464 +17 3.664337 3.291245 0.550013 +18 3.636087 3.277205 0.553098 +19 3.614743 3.269276 0.554796 +20 3.615276 3.267146 0.555189 +Total time: 1:16:19 +/home/pczapla/workspace/ulmfit-multilingual/data/cls/en-music/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/en-music/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.558971 0.567553 0.840000 +2 0.512537 0.503813 0.855000 +3 0.456603 0.525335 0.830000 +4 0.404400 0.528679 0.870000 +5 0.380277 0.508553 0.850000 +6 0.370005 0.473182 0.875000 +7 0.353120 0.475434 0.875000 +8 0.349033 0.476645 0.875000 +Total time: 02:17 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/en-music/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.30121252, tensor(0.8945)] +Processing data/wiki/es-100/models/sp15k/qrnn_nl4.m +../cls/es-* +Processing data/wiki/fr-100/models/sp15k/qrnn_nl4.m +../cls/fr-* +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k/qrnn_nl4.m +Evaluating previously trained model +Validation set not found using 10% of trn +Data lm, trn: 33183, val: 3687 +Data cls, trn: 1800, val: 200 +Data tst, trn: 200, val: 2000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à'] +Loss and accuracy using (cls_best): [0.27542278, tensor(0.9125)] +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-dvd/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-dvd/models/sp15k/qrnn_nl4.m +Training +Validation set not found using 10% of trn +Running tokenization lm... +Data lm, trn: 12023, val: 1335 +Running tokenization cls... +Data cls, trn: 1800, val: 200 +Running tokenization tst... +Data tst, trn: 200, val: 2000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 5.066386 4.331831 0.359093 +Total time: 00:59 +epoch train_loss valid_loss accuracy +1 4.724524 4.284094 0.365222 +2 4.508605 4.191759 0.380180 +3 4.362468 4.066796 0.399366 +4 4.244251 3.960160 0.414862 +5 4.175961 3.862056 0.428568 +6 4.089641 3.769989 0.443156 +7 3.996232 3.692802 0.455285 +8 3.911177 3.609659 0.469151 +9 3.834589 3.517031 0.485899 +10 3.753677 3.444421 0.499121 +11 3.693849 3.378458 0.512520 +12 3.656308 3.304272 0.527849 +13 3.619973 3.243898 0.539743 +14 3.538301 3.187478 0.551900 +15 3.542358 3.146652 0.561410 +16 3.433141 3.108872 0.569471 +17 3.397583 3.086333 0.574742 +18 3.420117 3.073466 0.577351 +19 3.364507 3.063923 0.579112 +20 3.371653 3.063201 0.579386 +Total time: 28:17 +/home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-dvd/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-dvd/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.566780 0.456278 0.895000 +2 0.502839 0.438660 0.910000 +3 0.451509 0.439451 0.915000 +4 0.409567 0.458298 0.905000 +5 0.375980 0.451819 0.875000 +6 0.365615 0.429869 0.895000 +7 0.354849 0.430782 0.900000 +8 0.351484 0.429099 0.900000 +Total time: 01:58 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-dvd/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.29015526, tensor(0.8955)] +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-music/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-music/models/sp15k/qrnn_nl4.m +Training +Validation set not found using 10% of trn +Running tokenization lm... +Data lm, trn: 17946, val: 1994 +Running tokenization cls... +Data cls, trn: 1800, val: 200 +Running tokenization tst... +Data tst, trn: 200, val: 2000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 4.902670 4.297194 0.366058 +Total time: 01:43 +epoch train_loss valid_loss accuracy +1 4.611322 4.230604 0.375839 +2 4.375579 4.099567 0.396295 +3 4.221557 3.955517 0.419109 +4 4.053433 3.821369 0.439689 +5 3.936422 3.676852 0.463543 +6 3.822320 3.571660 0.481323 +7 3.738963 3.448354 0.505107 +8 3.660149 3.338475 0.528736 +9 3.517678 3.231433 0.552705 +10 3.506994 3.129620 0.575138 +11 3.405409 3.064906 0.591049 +12 3.381207 2.970587 0.613629 +13 3.252796 2.915546 0.627047 +14 3.214425 2.860644 0.640598 +15 3.154248 2.808179 0.653110 +16 3.101139 2.774935 0.661620 +17 3.052113 2.748979 0.667787 +18 2.992890 2.738183 0.670336 +19 3.059078 2.730828 0.671936 +20 2.990193 2.729673 0.672222 +Total time: 48:59 +/home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-music/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-music/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.506996 0.478671 0.880000 +2 0.468533 0.474114 0.900000 +3 0.426250 0.440435 0.910000 +4 0.382522 0.441184 0.920000 +5 0.368826 0.428872 0.925000 +6 0.359141 0.421278 0.925000 +7 0.345824 0.418836 0.925000 +8 0.348028 0.422893 0.920000 +Total time: 02:09 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-music/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.23497759, tensor(0.9340)] +Processing data/wiki/it-100/models/sp15k/qrnn_nl4.m +../cls/it-* +Processing data/wiki/ja-100/models/sp15k/qrnn_nl4.m +../cls/ja-* +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-books/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-books/models/sp15k/qrnn_nl4.m +Evaluating previously trained model +Validation set not found using 10% of trn +Data lm, trn: 156402, val: 17378 +Data cls, trn: 1800, val: 200 +Data tst, trn: 200, val: 2000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)'] +Loss and accuracy using (cls_best): [0.34613457, tensor(0.8630)] +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-dvd/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-dvd/models/sp15k/qrnn_nl4.m +Training +Validation set not found using 10% of trn +Running tokenization lm... +Data lm, trn: 65094, val: 7232 +Running tokenization cls... +Data cls, trn: 1800, val: 200 +Running tokenization tst... +Data tst, trn: 200, val: 2000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 5.160996 4.517923 0.367362 +Total time: 06:17 +epoch train_loss valid_loss accuracy +1 4.543629 4.324594 0.393028 +2 4.285625 4.133171 0.418336 +3 4.129646 3.980379 0.436407 +4 3.999502 3.868094 0.450510 +5 3.955512 3.801580 0.458919 +6 3.918293 3.744513 0.466177 +7 3.851580 3.696822 0.472531 +8 3.797014 3.660165 0.478426 +9 3.784533 3.629932 0.482919 +10 3.750223 3.600744 0.487628 +11 3.746155 3.572618 0.492204 +12 3.731790 3.549776 0.495799 +13 3.721592 3.523684 0.500092 +14 3.664623 3.501803 0.504120 +15 3.653851 3.482831 0.507431 +16 3.655076 3.466049 0.510687 +17 3.621635 3.454597 0.512781 +18 3.619130 3.447068 0.514104 +19 3.598743 3.443191 0.514797 +20 3.585334 3.441960 0.515080 +Total time: 2:55:48 +/home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-dvd/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-dvd/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.572977 0.500271 0.845000 +2 0.530233 0.530061 0.815000 +3 0.478677 0.534202 0.845000 +4 0.422542 0.536495 0.855000 +5 0.382862 0.508948 0.820000 +6 0.369442 0.500411 0.860000 +7 0.364325 0.500543 0.840000 +8 0.353970 0.498926 0.840000 +Total time: 02:12 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-dvd/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.351684, tensor(0.8575)] +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-music/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-music/models/sp15k/qrnn_nl4.m +Training +Validation set not found using 10% of trn +Running tokenization lm... +Data lm, trn: 53903, val: 5989 +Running tokenization cls... +Data cls, trn: 1800, val: 200 +Running tokenization tst... +Data tst, trn: 200, val: 2000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 5.209339 4.578931 0.361889 +Total time: 04:32 +epoch train_loss valid_loss accuracy +1 4.667555 4.387390 0.386283 +2 4.378979 4.193202 0.412063 +3 4.182499 4.027168 0.434250 +4 4.075960 3.904261 0.449052 +5 3.978663 3.817029 0.459996 +6 3.933847 3.751421 0.468568 +7 3.878065 3.694970 0.476144 +8 3.796052 3.649740 0.483267 +9 3.771566 3.604177 0.490818 +10 3.737682 3.573293 0.495396 +11 3.690553 3.530821 0.503269 +12 3.654486 3.498471 0.508781 +13 3.628883 3.467230 0.514346 +14 3.606242 3.443541 0.519053 +15 3.586881 3.420868 0.523608 +16 3.581954 3.402043 0.527174 +17 3.492296 3.383712 0.530720 +18 3.522726 3.376451 0.532236 +19 3.512796 3.371758 0.533116 +20 3.461179 3.370992 0.533302 +Total time: 2:06:47 +/home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-music/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-music/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.572634 0.558102 0.795000 +2 0.538641 0.521115 0.830000 +3 0.480845 0.544550 0.830000 +4 0.420194 0.540962 0.830000 +5 0.394283 0.502248 0.860000 +6 0.371833 0.520424 0.840000 +7 0.360100 0.511556 0.855000 +8 0.353953 0.511548 0.850000 +Total time: 01:51 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-music/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.34172097, tensor(0.8660)] +Processing data/wiki/ru-100/models/sp15k/qrnn_nl4.m +../cls/ru-* +Processing data/wiki/zh-100/models/sp15k/qrnn_nl4.m +../cls/zh-* +OrderedDict([('data/cls/de-books/models/sp15k/qrnn_nl4.m', 0.9319999814033508), + ('data/cls/de-dvd/models/sp15k/qrnn_nl4.m', 0.9054999947547913), + ('data/cls/de-music/models/sp15k/qrnn_nl4.m', 0.9300000071525574), + ('data/cls/en-books/models/sp15k/qrnn_nl4.m', 0.9075000286102295), + ('data/cls/en-dvd/models/sp15k/qrnn_nl4.m', 0.8930000066757202), + ('data/cls/en-music/models/sp15k/qrnn_nl4.m', 0.8945000171661377), + ('data/cls/fr-books/models/sp15k/qrnn_nl4.m', 0.9125000238418579), + ('data/cls/fr-dvd/models/sp15k/qrnn_nl4.m', 0.8955000042915344), + ('data/cls/fr-music/models/sp15k/qrnn_nl4.m', 0.9340000152587891), + ('data/cls/ja-books/models/sp15k/qrnn_nl4.m', 0.8629999756813049), + ('data/cls/ja-dvd/models/sp15k/qrnn_nl4.m', 0.8575000166893005), + ('data/cls/ja-music/models/sp15k/qrnn_nl4.m', 0.8659999966621399)]) +data/cls/de-books/models/sp15k/qrnn_nl4.m: 0.9319999814033508 +data/cls/de-dvd/models/sp15k/qrnn_nl4.m: 0.9054999947547913 +data/cls/de-music/models/sp15k/qrnn_nl4.m: 0.9300000071525574 +data/cls/en-books/models/sp15k/qrnn_nl4.m: 0.9075000286102295 +data/cls/en-dvd/models/sp15k/qrnn_nl4.m: 0.8930000066757202 +data/cls/en-music/models/sp15k/qrnn_nl4.m: 0.8945000171661377 +data/cls/fr-books/models/sp15k/qrnn_nl4.m: 0.9125000238418579 +data/cls/fr-dvd/models/sp15k/qrnn_nl4.m: 0.8955000042915344 +data/cls/fr-music/models/sp15k/qrnn_nl4.m: 0.9340000152587891 +data/cls/ja-books/models/sp15k/qrnn_nl4.m: 0.8629999756813049 +data/cls/ja-dvd/models/sp15k/qrnn_nl4.m: 0.8575000166893005 +data/cls/ja-music/models/sp15k/qrnn_nl4.m: 0.8659999966621399 +``` +# Debugging issues +## Without col merge +```` +python -m ulmfit cls --dataset-path data/cls/${LANG}-books --base-lm-path data/wiki-m/${LANG}-100/models/sp30k/lstm_nl4.m --lang=${LANG} --name 'nl4' - train 20 --bs 20 --num-cls-epochs=8 --lr-sched=single +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k/lstm_nl4.m +Validation set not found using 10% of trn +Data lm, trn: 33183, val: 3687 +Data cls, trn: 1800, val: 200 +Running tokenization tst... +Data tst, trn: 200, val: 2000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁.', "'", 's', '▁la', '▁le', '▁et', '▁l', '▁à'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki-m/fr-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki-m/fr-100/models/sp30k/lstm_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 3.356221 2.821012 0.518492 +Total time: 00:21 +epoch train_loss valid_loss accuracy +1 3.041214 2.734799 0.524577 +2 2.919576 2.648412 0.535661 +3 2.822292 2.542236 0.549206 +4 2.721790 2.414110 0.561852 +5 2.596515 2.276732 0.579841 +6 2.453715 2.140479 0.600370 +7 2.333764 2.000186 0.621349 +8 2.231092 1.873927 0.644259 +9 2.101130 1.765473 0.660529 +10 2.006949 1.666797 0.682196 +11 1.905025 1.584023 0.696058 +12 1.820798 1.513958 0.709841 +13 1.751217 1.456632 0.720846 +14 1.689076 1.410359 0.729947 +15 1.646113 1.371438 0.739868 +16 1.594153 1.346142 0.744577 +17 1.564375 1.332298 0.746693 +18 1.536557 1.322925 0.748995 +19 1.532926 1.319159 0.749444 +20 1.525449 1.318028 0.749815 +Total time: 08:51 +/home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k/lstm_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.588118 0.581087 0.700000 +2 0.504373 0.583527 0.720000 +3 0.412651 0.538866 0.750000 +4 0.295401 0.658459 0.750000 +5 0.212442 1.054068 0.720000 +6 0.126090 1.302099 0.745000 +7 0.078312 1.307932 0.760000 +8 0.050346 1.339740 0.745000 +Total time: 00:35 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k/lstm_nl4.m +Loss and accuracy using (cls_best): [1.3513571, tensor(0.7700)] +1.351357102394104 +0.7699999809265137 +```` + +### FR books +#### CLS second run 91.00% +````bash +python -m ulmfit cls --dataset-path data/cls/${LANG}-books --base-lm-path data/wiki-m/${LANG}-100/models/sp30k/lstm_nl4.m --lang=${LANG} --name 'nl4' - train 20 --bs 20 --num-cls-epochs=8 --lr-sched=single +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k/lstm_nl4.m +Validation set not found using 10% of trn +Running tokenization lm... +Data lm, trn: 33183, val: 3687 +Running tokenization cls... +Data cls, trn: 1800, val: 200 +Running tokenization tst... +Data tst, trn: 200, val: 2000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁.', "'", 's', '▁la', '▁le', '▁et', '▁l', '▁à'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki-m/fr-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki-m/fr-100/models/sp30k/lstm_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 3.790325 3.409294 0.367234 +Total time: 06:02 +epoch train_loss valid_loss accuracy +1 3.526303 3.326439 0.378936 +2 3.466923 3.226977 0.392378 +3 3.342312 3.111874 0.406997 +4 3.244619 2.992510 0.422330 +5 3.156150 2.877498 0.437467 +6 3.070326 2.762509 0.453874 +7 2.956969 2.651613 0.471552 +8 2.878008 2.535935 0.491058 +9 2.790110 2.438724 0.508560 +10 2.684145 2.323467 0.528415 +11 2.633781 2.231418 0.547093 +12 2.535126 2.143523 0.564889 +13 2.464436 2.055402 0.582077 +14 2.330094 1.989257 0.596582 +15 2.372371 1.924338 0.610048 +16 2.190224 1.866912 0.621738 +17 2.176868 1.834098 0.629221 +18 2.168293 1.809196 0.633879 +19 2.151132 1.797144 0.636382 +20 2.130476 1.793351 0.637044 +Total time: 2:30:05 +/home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k/lstm_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.314315 0.530879 0.865000 +2 0.336746 0.468635 0.865000 +3 0.255810 0.324242 0.870000 +4 0.149121 0.480570 0.885000 +5 0.093909 0.613743 0.890000 +6 0.091678 0.660452 0.885000 +7 0.049993 0.649642 0.910000 +8 0.034218 0.640008 0.910000 +Total time: 04:19 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k/lstm_nl4.m +Loss and accuracy using (cls_best): [0.5418505, tensor(0.9100)] +0.5418505072593689 +0.9100000262260437 +```` +#### CLS second run 89.70% +``` +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 4.037580 3.312217 0.364410 +Total time: 01:44 +epoch train_loss valid_loss accuracy +1 3.709982 3.256320 0.371825 +2 3.459413 3.150574 0.386972 +3 3.296628 3.037327 0.402039 +4 3.186458 2.914899 0.418413 +5 3.092632 2.817097 0.431216 +6 2.966957 2.726081 0.442906 +7 2.924824 2.647339 0.453871 +8 2.818279 2.561596 0.466795 +9 2.773893 2.501994 0.475877 +10 2.736084 2.438490 0.485978 +11 2.688937 2.370927 0.496899 +12 2.615245 2.314875 0.506508 +13 2.583292 2.260717 0.515725 +14 2.535631 2.220295 0.522666 +15 2.466035 2.179093 0.530148 +16 2.461427 2.151952 0.535315 +17 2.390641 2.131065 0.538749 +18 2.376235 2.116927 0.541430 +19 2.407630 2.115370 0.542039 +20 2.391378 2.112687 0.542522 +Total time: 46:33 +/home/n-waves/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k +Saving info /home/n-waves/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.466671 0.534682 0.745000 +2 0.358965 0.372612 0.875000 +3 0.251557 0.311034 0.900000 +4 0.166484 0.585425 0.865000 +5 0.101803 0.726341 0.900000 +6 0.072025 0.587875 0.885000 +7 0.045328 0.760989 0.890000 +8 0.027765 0.727203 0.890000 +Total time: 01:17 +Saving models at /home/n-waves/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.55982095, tensor(0.8970)] +0.5598209500312805 +0.8970000147819519 +``` + +````bash + python -m ulmfit eval --glob="wiki/*-100/models/sp15k/qrnn_nl4.m" --name nl4 --dataset-template='../cls/${lang}-books' --num-lm-epochs=20 --num-cls-epochs=8 --bs=18 --lr_sched=1cycle --label-smoothing-eps=0.1 +Processing data/wiki/de-100/models/sp15k/qrnn_nl4.m +../cls/de-books +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books/models/sp15k/qrnn_nl4.m +Training +Validation set not found using 10% of trn +Running tokenization lm... +Data lm, trn: 152523, val: 16947 +Running tokenization cls... +Data cls, trn: 1800, val: 200 +Running tokenization tst... +Data tst, trn: 200, val: 2000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) may be ABI-incompatible with PyTorch! +Please use a compiler that is ABI-compatible with GCC 4.9 and above. +See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html. + +See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6 +for instructions on how to install GCC 4.9 or higher. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler)) +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/de-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/de-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 4.659840 4.155068 0.438915 +Total time: 19:49 +epoch train_loss valid_loss accuracy +1 4.047657 3.906174 0.473263 +2 3.838604 3.750033 0.495044 +3 3.759745 3.645392 0.508066 +4 3.687292 3.590345 0.515256 +5 3.627978 3.558625 0.519270 +6 3.645294 3.535127 0.522458 +7 3.598592 3.514614 0.525545 +8 3.624991 3.498831 0.527508 +9 3.579310 3.484266 0.529842 +10 3.583765 3.466230 0.532676 +11 3.582043 3.449772 0.535180 +12 3.555684 3.431577 0.537932 +13 3.547026 3.412131 0.541149 +14 3.494887 3.394598 0.543857 +15 3.508622 3.377836 0.546771 +16 3.509469 3.362931 0.549247 +17 3.491808 3.351186 0.551273 +18 3.493448 3.342843 0.552928 +19 3.430579 3.338350 0.553565 +20 3.464007 3.337320 0.553756 +Total time: 9:30:17 +/home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.483316 0.432666 0.930000 +2 0.439075 0.419435 0.900000 +3 0.413648 0.415576 0.930000 +4 0.378170 0.421523 0.925000 +5 0.362544 0.410525 0.935000 +6 0.348756 0.407976 0.935000 +7 0.345272 0.403396 0.935000 +8 0.347165 0.408822 0.935000 +Total time: 02:45 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.23490253, tensor(0.9315)] +Processing data/wiki/en-100/models/sp15k/qrnn_nl4.m +../cls/en-books +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/en-books/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/en-books/models/sp15k/qrnn_nl4.m +Evaluating previously trained model +Validation set not found using 10% of trn +Data lm, trn: 48600, val: 5400 +Data cls, trn: 1800, val: 200 +Data tst, trn: 200, val: 2000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁the', '▁,', 's', '▁.', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed'] +Loss and accuracy using (cls_best): [0.2731999, tensor(0.9075)] +Processing data/wiki/es-100/models/sp15k/qrnn_nl4.m +../cls/es-books +Processing data/wiki/fr-100/models/sp15k/qrnn_nl4.m +../cls/fr-books +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k/qrnn_nl4.m +Evaluating previously trained model +Validation set not found using 10% of trn +Data lm, trn: 33183, val: 3687 +Data cls, trn: 1800, val: 200 +Data tst, trn: 200, val: 2000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à'] +Loss and accuracy using (cls_best): [0.27542278, tensor(0.9125)] +Processing data/wiki/it-100/models/sp15k/qrnn_nl4.m +../cls/it-books +Processing data/wiki/ja-100/models/sp15k/qrnn_nl4.m +../cls/ja-books +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-books/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-books/models/sp15k/qrnn_nl4.m +Training +Validation set not found using 10% of trn +Running tokenization lm... +Data lm, trn: 156402, val: 17378 +Running tokenization cls... +Data cls, trn: 1800, val: 200 +Running tokenization tst... +Data tst, trn: 200, val: 2000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 4.851995 4.372256 0.377945 +Total time: 15:17 +epoch train_loss valid_loss accuracy +1 4.281115 4.121970 0.414018 +2 4.072245 3.940537 0.438534 +3 3.903796 3.822031 0.453624 +4 3.863336 3.748684 0.462770 +5 3.818518 3.706158 0.468450 +6 3.755432 3.675938 0.472679 +7 3.738127 3.648453 0.476536 +8 3.717736 3.625716 0.479841 +9 3.720993 3.605317 0.483234 +10 3.695235 3.585903 0.486179 +11 3.696312 3.567543 0.489165 +12 3.688561 3.550402 0.492248 +13 3.654837 3.533271 0.495245 +14 3.667123 3.515244 0.498298 +15 3.610541 3.499110 0.501204 +16 3.589734 3.484431 0.503880 +17 3.586404 3.474879 0.505702 +18 3.590800 3.466892 0.507203 +19 3.570113 3.462836 0.508037 +20 3.544942 3.461981 0.508177 +Total time: 7:11:56 +/home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-books/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-books/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.536773 0.625124 0.775000 +2 0.509836 0.827109 0.735000 +3 0.455667 0.636377 0.805000 +4 0.411549 0.650928 0.725000 +5 0.371732 0.594132 0.795000 +6 0.363084 0.580896 0.810000 +7 0.351636 0.569889 0.790000 +8 0.343057 0.575629 0.790000 +Total time: 02:15 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/ja-books/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.3464668, tensor(0.8630)] +Processing data/wiki/ru-100/models/sp15k/qrnn_nl4.m +../cls/ru-books +Processing data/wiki/zh-100/models/sp15k/qrnn_nl4.m +../cls/zh-books +OrderedDict([('data/cls/de-books/models/sp15k/qrnn_nl4.m', 0.9315000176429749), + ('data/cls/en-books/models/sp15k/qrnn_nl4.m', 0.9075000286102295), + ('data/cls/fr-books/models/sp15k/qrnn_nl4.m', 0.9125000238418579), + ('data/cls/ja-books/models/sp15k/qrnn_nl4.m', 0.8629999756813049)]) +```` diff --git a/results/logs/cls/zeroshoot.md b/results/logs/cls/zeroshoot.md new file mode 100644 index 0000000..d6b7c71 --- /dev/null +++ b/results/logs/cls/zeroshoot.md @@ -0,0 +1,118 @@ +# Results on books dataset + +| | de | fr | +|-------------------------|-------|-------| +| laser zero shot from en | 84.15 | 83.90 | +| with ULMFIT QRNN sp15k | 89.60 | 87.84 | + +## Laser results + +| | en | de | fr | +|------|--------|-------|------| +| en: | 84.55 | 84.15 | 83.90| +| de: | 82.60 | 85.20 | 83.05| +| fr: | 77.20 | 82.95 | 84.85| + +## ULMFiT improvment +``` +data/cls/de-books-laser-en1/models/sp15k/qrnn_nl4.m: 0.8960000276565552 +data/cls/fr-books-laser-en1/models/sp15k/qrnn_nl4.m: 0.8784999847412109 +``` + +### Execution log +``` +python -m ulmfit eval --glob="cls/*-books/models/sp15k/qrnn_nl4.m" --name nl4 --dataset-template='${lang}-books-laser-en1' --num-lm-epochs=0 --num-cls-epochs=8 --bs=18 --lr_sched=1cycle --label-smoothing-eps=0.1 +Processing data/cls/de-books/models/sp15k/qrnn_nl4.m +de-books-laser-en1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books-laser-en1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books-laser-en1/models/sp15k/qrnn_nl4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books-laser-en1/de.dev.csv +Running tokenization lm... +Data lm, trn: 152523, val: 16947 +Running tokenization cls... +Data cls, trn: 1800, val: 200 +Running tokenization tst... +Data tst, trn: 200, val: 2000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) may be ABI-incompatible with PyTorch! +Please use a compiler that is ABI-compatible with GCC 4.9 and above. +See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html. + +See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6 +for instructions on how to install GCC 4.9 or higher. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler)) +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books-laser-en1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books-laser-en1/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.541837 0.487596 0.870000 +2 0.498016 0.490300 0.885000 +3 0.442417 0.479205 0.875000 +4 0.395640 0.528897 0.855000 +5 0.369408 0.521830 0.855000 +6 0.361129 0.481892 0.880000 +7 0.351095 0.481634 0.885000 +8 0.343147 0.481654 0.880000 +Total time: 02:35 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books-laser-en1/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.29498395, tensor(0.8960)] +Processing data/cls/en-books/models/sp15k/qrnn_nl4.m +en-books-laser-en1 +Processing data/cls/fr-books/models/sp15k/qrnn_nl4.m +fr-books-laser-en1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books-laser-en1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books-laser-en1/models/sp15k/qrnn_nl4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books-laser-en1/fr.dev.csv +Running tokenization lm... +Data lm, trn: 33183, val: 3687 +Running tokenization cls... +Data cls, trn: 1800, val: 200 +Running tokenization tst... +Data tst, trn: 200, val: 2000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books-laser-en1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books-laser-en1/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.551931 0.532421 0.835000 +2 0.509913 0.524893 0.880000 +3 0.433385 0.502657 0.860000 +4 0.397314 0.487201 0.880000 +5 0.365447 0.467523 0.885000 +6 0.356587 0.520736 0.855000 +7 0.353801 0.487093 0.875000 +8 0.343812 0.484453 0.880000 +Total time: 01:45 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books-laser-en1/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.32666296, tensor(0.8785)] +Processing data/cls/ja-books/models/sp15k/qrnn_nl4.m +ja-books-laser-en1 +OrderedDict([('data/cls/de-books-laser-en1/models/sp15k/qrnn_nl4.m', + 0.8960000276565552), + ('data/cls/fr-books-laser-en1/models/sp15k/qrnn_nl4.m', + 0.8784999847412109)]) +data/cls/de-books-laser-en1/models/sp15k/qrnn_nl4.m: 0.8960000276565552 +data/cls/fr-books-laser-en1/models/sp15k/qrnn_nl4.m: 0.8784999847412109 +``` + diff --git a/results/logs/de.md b/results/logs/de.md index 39a6e58..8590f5c 100644 --- a/results/logs/de.md +++ b/results/logs/de.md @@ -22,6 +22,72 @@ Total time: 19:18:33 data/wiki/de-100/models/sp15k Saving info data/wiki/de-100/models/sp15k/lstm_nl4.m/info.jso ``` +### MLDoc +```bash +LANG=de +python -m ulmfit cls --dataset-path data/mldoc-m/${LANG}-1 --base-lm-path data/wiki-m/${LANG}-100/models/sp15k/lstm_nl4.m --lang=${LANG} --name 'nl4' - train 20 --bs 20 --num-cls- +epochs=8 --lr_sched=1cycle +Max vocab: 15000 +Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc-m/de-1/models/sp15k +Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc-m/de-1/models/sp15k/lstm_nl4.m +Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc-m/de-1/de.dev.csv +Running tokenization lm... +Data lm, trn: 13500, val: 1500 +Running tokenization cls... +Data cls, trn: 1000, val: 1000 +Running tokenization tst... +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki-m/de-100/models/sp15k/lstm_nl4.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki-m/de-100/models/sp15k/lstm_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 2.333600 2.005051 0.596875 +Total time: 07:40 +epoch train_loss valid_loss accuracy +1 2.120653 1.886799 0.615784 +2 1.980713 1.763139 0.636041 +3 1.805195 1.655620 0.654068 +4 1.729641 1.564017 0.668772 +5 1.681813 1.491185 0.680613 +6 1.682965 1.422562 0.692458 +7 1.580731 1.357177 0.703143 +8 1.506753 1.297219 0.714487 +9 1.515824 1.235473 0.725413 +10 1.427750 1.178680 0.737216 +11 1.371839 1.118909 0.749590 +12 1.342978 1.068754 0.760473 +13 1.286842 1.011940 0.772384 +14 1.254822 0.960727 0.784244 +15 1.195136 0.919377 0.793910 +16 1.118260 0.881799 0.802814 +17 1.071546 0.855769 0.809040 +18 1.079081 0.839280 0.812895 +19 1.052724 0.831323 0.814723 +20 1.024207 0.829737 0.815070 +Total time: 3:08:58 +/home/test/workspace/ulmfit-multilingual/data/mldoc-m/de-1/models/sp15k +Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc-m/de-1/models/sp15k/lstm_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.539181 0.239851 0.938000 +2 0.326801 0.374512 0.917000 +3 0.225103 0.330872 0.945000 +4 0.121660 0.444890 0.938000 +5 0.078411 0.422513 0.948000 +6 0.061354 0.509489 0.949000 +7 0.029890 0.438118 0.949000 +8 0.014213 0.441808 0.949000 +Total time: 09:00 +Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc-m/de-1/models/sp15k/lstm_nl4.m +Loss and accuracy using (cls_best): [0.3710725, tensor(0.9553)] + +0.3710725009441376 +0.9552500247955322 +``` + ## VF60k LSTM nl 3 ### LM diff --git a/results/logs/label_smoothing.md b/results/logs/label_smoothing.md index c52df1e..4d17cee 100644 --- a/results/logs/label_smoothing.md +++ b/results/logs/label_smoothing.md @@ -2,6 +2,957 @@ ## QRNN 15k +## LM+CLS training +``` +python -m ulmfit eval --glob="wiki/*-100/models/sp15k/qrnn_nl4.m" --name nl4-sl --dataset-template='../mldoc/${lang}-1' --num-lm-epochs=20 --num-cls-epochs=8 --bs=18 --lr_sched=1cycle --label-smoothing-eps=0.1 +Processing data/wiki/de-100/models/sp15k/qrnn_nl4.m +../mldoc/de-1 +Max vocab: 15000 +Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k +Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-sl.m +Training +Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/de-1/de.dev.csv +Data lm, trn: 13500, val: 1500 +Data cls, trn: 1000, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/de-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/de-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 4.336932 3.600409 0.524499 + +Total time: 01:55 +epoch train_loss valid_loss accuracy +1 3.867303 3.446689 0.547847 + +2 3.475966 3.257031 0.578702 + +3 3.233705 3.085521 0.607024 + +4 3.170250 2.964533 0.627028 + +5 3.059212 2.876878 0.641094 + +6 2.965926 2.797152 0.654905 + +7 2.974514 2.743403 0.663470 + +8 2.858759 2.690824 0.672891 + +9 2.866673 2.646101 0.680956 + +10 2.814579 2.610239 0.687777 + +11 2.806775 2.577145 0.694683 + +12 2.741160 2.540292 0.702336 + +13 2.753407 2.506782 0.709361 + +14 2.720171 2.480167 0.715709 + +15 2.631490 2.452236 0.721706 + +16 2.587928 2.431256 0.726922 + +17 2.608380 2.417473 0.729975 + +18 2.564811 2.407112 0.732487 + +19 2.599782 2.403481 0.733463 + +20 2.603479 2.402438 0.733552 + +Total time: 1:18:42 +/home/test/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k +Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-sl.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.790572 0.610132 0.925000 + +2 0.655777 0.619519 0.945000 + +3 0.591051 0.654355 0.930000 + +4 0.541992 0.582572 0.944000 + +5 0.514052 0.574799 0.939000 + +6 0.490411 0.550166 0.949000 + +7 0.476344 0.553866 0.947000 + +8 0.469109 0.548871 0.947000 + +Total time: 02:43 +Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-sl.m +Loss and accuracy using (cls_best): [0.2010318, tensor(0.9610)] + +Processing data/wiki/en-100/models/sp15k/qrnn_nl4.m +../mldoc/en-1 +Max vocab: 15000 +Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k +Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-sl.m +Training +Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/en-1/en.dev.csv +Data lm, trn: 13500, val: 1500 +Data cls, trn: 1000, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁the', '▁,', 's', '▁.', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/en-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/en-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 5.361436 4.703674 0.343924 + +Total time: 02:55 +epoch train_loss valid_loss accuracy +1 4.757253 4.493189 0.372715 + +2 4.515738 4.291261 0.404511 + +3 4.301139 4.110973 0.430612 + +4 4.162306 3.964101 0.450053 + +5 4.062487 3.841125 0.466488 + +6 3.898028 3.740108 0.480628 + +7 3.876914 3.660982 0.493164 + +8 3.793781 3.593925 0.502977 + +9 3.736873 3.528259 0.513241 + +10 3.695738 3.477659 0.521709 + +11 3.668749 3.431972 0.529821 + +12 3.642119 3.385145 0.537860 + +13 3.556678 3.343567 0.545521 + +14 3.548823 3.305735 0.552502 + +15 3.520068 3.272878 0.558736 + +16 3.439619 3.247021 0.563504 + +17 3.391731 3.228240 0.567151 + +18 3.398134 3.217466 0.569319 + +19 3.402400 3.212110 0.570352 + +20 3.375334 3.210727 0.570589 + +Total time: 1:19:32 +/home/test/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k +Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-sl.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.855421 0.644234 0.903000 + +2 0.722607 0.665451 0.954000 + +3 0.629362 0.590857 0.945000 + +4 0.555874 0.562738 0.950000 + +5 0.522701 0.549048 0.953000 + +6 0.506758 0.536445 0.961000 + +7 0.489250 0.527361 0.963000 + +8 0.482953 0.528452 0.961000 + +Total time: 02:54 +Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-sl.m +Loss and accuracy using (cls_best): [0.20434816, tensor(0.9555)] + +Processing data/wiki/es-100/models/sp15k/qrnn_nl4.m +../mldoc/es-1 +Max vocab: 15000 +Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k +Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-sl.m +Training +Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 1000, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁.', '▁la', 's', '▁el', '▁en', '▁y', '▁a', "▁&'"] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/es-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/es-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 3.925853 3.293504 0.535753 + +Total time: 01:42 +epoch train_loss valid_loss accuracy +1 3.584036 3.131868 0.563887 + +2 3.341769 2.924815 0.605171 + +3 3.064540 2.760458 0.637288 + +4 2.979203 2.651128 0.657270 + +5 2.913760 2.569732 0.670629 + +6 2.901400 2.507033 0.682515 + +7 2.884516 2.454021 0.692617 + +8 2.759039 2.404587 0.703479 + +9 2.730353 2.367218 0.711349 + +10 2.657660 2.325339 0.720632 + +11 2.638513 2.292851 0.728599 + +12 2.629284 2.258947 0.737086 + +13 2.542013 2.226581 0.744815 + +14 2.464086 2.202000 0.750827 + +15 2.489060 2.177043 0.757989 + +16 2.446775 2.158471 0.762447 + +17 2.388175 2.144888 0.766083 + +18 2.415777 2.136921 0.768290 + +19 2.454445 2.132960 0.769424 + +20 2.346935 2.132173 0.769667 + +Total time: 47:14 +/home/test/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k +Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-sl.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.842615 0.671844 0.919000 + +2 0.705828 0.669006 0.948000 + +3 0.627629 0.572335 0.944000 + +4 0.576609 0.600053 0.953000 + +5 0.532899 0.542761 0.962000 + +6 0.503425 0.548742 0.961000 + +7 0.495654 0.535397 0.959000 + +8 0.487190 0.545798 0.958000 + +Total time: 02:16 +Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-sl.m +Loss and accuracy using (cls_best): [0.18526463, tensor(0.9582)] + +Processing data/wiki/fr-100/models/sp15k/qrnn_nl4.m +../mldoc/fr-1 +Max vocab: 15000 +Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k +Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-sl.m +Training +Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/fr-1/fr.dev.csv +Data lm, trn: 13500, val: 1500 +Running tokenization cls... +Data cls, trn: 1000, val: 1000 + +Running tokenization tst... +Data tst, trn: 1000, val: 4000 + +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 4.408209 3.734384 0.451908 + +Total time: 02:09 +epoch train_loss valid_loss accuracy +1 3.911323 3.613580 0.472285 + +2 3.678445 3.445187 0.503284 + +3 3.489969 3.296292 0.528945 + +4 3.381153 3.180339 0.549296 + +5 3.291956 3.100773 0.562257 + +6 3.184217 3.027092 0.575632 + +7 3.215341 2.965142 0.586544 + +8 3.119935 2.915341 0.596039 + +9 3.081539 2.870155 0.605426 + +10 3.096917 2.826453 0.614306 + +11 3.024909 2.786344 0.622573 + +12 2.940282 2.743246 0.632480 + +13 2.939400 2.713417 0.639012 + +14 2.920471 2.682074 0.646323 + +15 2.836955 2.652518 0.653338 + +16 2.873827 2.631899 0.658241 + +17 2.847641 2.615588 0.662295 + +18 2.856253 2.605571 0.664535 + +19 2.817845 2.601670 0.665505 + +20 2.827199 2.600590 0.665696 + +Total time: 1:13:04 +/home/test/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k +Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-sl.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.830361 0.645785 0.913000 + +2 0.718334 0.713889 0.903000 + +3 0.625460 0.608466 0.936000 + +4 0.549874 0.573567 0.938000 + +5 0.513573 0.563112 0.938000 + +6 0.497791 0.559489 0.948000 + +7 0.482823 0.547815 0.944000 + +8 0.473484 0.543763 0.946000 + +Total time: 02:36 +Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-sl.m +Loss and accuracy using (cls_best): [0.21287616, tensor(0.9480)] + +Processing data/wiki/it-100/models/sp15k/qrnn_nl4.m +../mldoc/it-1 +Max vocab: 15000 +Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k +Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-sl.m +Training +Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/it-1/it.dev.csv +Running tokenization lm... +Data lm, trn: 13500, val: 1500 + +Running tokenization cls... +Data cls, trn: 1000, val: 1000 + +Running tokenization tst... +Data tst, trn: 1000, val: 4000 + +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁di', '▁e', "▁&'", "'", '▁il', '▁la', '▁in', 'e'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/it-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/it-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 4.802729 3.930606 0.435333 + +Total time: 01:03 +epoch train_loss valid_loss accuracy +1 4.208684 3.763728 0.460475 + +2 3.899941 3.558932 0.495172 + +3 3.627373 3.365080 0.528145 + +4 3.479348 3.217953 0.552994 + +5 3.345057 3.106383 0.571176 + +6 3.203966 3.013706 0.587515 + +7 3.159807 2.928730 0.601662 + +8 3.123686 2.863181 0.613407 + +9 3.091257 2.802755 0.625887 + +10 2.993412 2.747775 0.636441 + +11 2.923112 2.694130 0.647843 + +12 2.910893 2.644428 0.658610 + +13 2.912500 2.606808 0.667447 + +14 2.809009 2.564682 0.676342 + +15 2.813561 2.530405 0.684753 + +16 2.768533 2.505601 0.691209 + +17 2.714267 2.487358 0.695374 + +18 2.704343 2.474279 0.698375 + +19 2.721650 2.469191 0.699875 + +20 2.692483 2.468237 0.700245 + +Total time: 43:31 +/home/test/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k +Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-sl.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.926403 0.718438 0.853000 + +2 0.791397 0.850076 0.824000 + +3 0.710437 0.707848 0.902000 + +4 0.626952 0.700860 0.882000 + +5 0.551851 0.648725 0.900000 + +6 0.527778 0.632797 0.906000 + +7 0.502474 0.621409 0.911000 + +8 0.489953 0.621797 0.910000 + +Total time: 01:32 +Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-sl.m +Loss and accuracy using (cls_best): [0.3240368, tensor(0.9005)] + +Processing data/wiki/ja-100/models/sp15k/qrnn_nl4.m +../mldoc/ja-1 +Max vocab: 15000 +Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k +Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-sl.m +Training +Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv +Running tokenization lm... +Data lm, trn: 13500, val: 1500 + +Running tokenization cls... +Data cls, trn: 1000, val: 1000 + +Running tokenization tst... +Data tst, trn: 1000, val: 4000 + +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 4.467716 3.639215 0.510628 + +Total time: 02:43 +epoch train_loss valid_loss accuracy +1 3.640961 3.393765 0.547888 + +2 3.245662 3.111493 0.597234 + +3 3.091152 2.893684 0.636629 + +4 2.874470 2.753393 0.660614 + +5 2.774781 2.660047 0.677299 + +6 2.818495 2.584401 0.690161 + +7 2.763403 2.525782 0.699487 + +8 2.689764 2.481472 0.708918 + +9 2.471829 2.443742 0.715523 + +10 2.558768 2.411052 0.722205 + +11 2.583986 2.380159 0.728743 + +12 2.416061 2.352447 0.734377 + +13 2.422695 2.327425 0.739690 + +14 2.447176 2.302086 0.745426 + +15 2.409782 2.280813 0.749981 + +16 2.431124 2.265426 0.753981 + +17 2.409947 2.255584 0.756481 + +18 2.426040 2.246758 0.758470 + +19 2.363041 2.244397 0.759102 + +20 2.397845 2.243302 0.759366 + +Total time: 1:20:37 +/home/test/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k +Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-sl.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.962460 0.721570 0.861000 + +2 0.850875 0.732539 0.873000 + +3 0.733097 0.733598 0.880000 + +4 0.639531 0.743423 0.882000 + +5 0.570058 0.702896 0.870000 + +6 0.525673 0.663320 0.892000 + +7 0.514369 0.668241 0.887000 + +8 0.500725 0.663006 0.886000 + +Total time: 03:16 +Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-sl.m +Loss and accuracy using (cls_best): [0.32758784, tensor(0.8988)] + +Processing data/wiki/ru-100/models/sp15k/qrnn_nl4.m +../mldoc/ru-1 +Max vocab: 15000 +Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k +Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-sl.m +Training +Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv +Running tokenization lm... +Data lm, trn: 9195, val: 1021 + +Running tokenization cls... +Data cls, trn: 1000, val: 1000 + +Running tokenization tst... +Data tst, trn: 1000, val: 4000 + +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 4.599347 3.756284 0.476954 + +Total time: 02:08 +epoch train_loss valid_loss accuracy +1 3.814071 3.501726 0.524080 + +2 3.543633 3.248534 0.571844 + +3 3.245142 3.051544 0.607123 + +4 3.074194 2.917155 0.630383 + +5 3.015727 2.814585 0.648273 + +6 2.936803 2.731585 0.663555 + +7 2.801445 2.659986 0.676864 + +8 2.829947 2.602137 0.688181 + +9 2.784838 2.547982 0.699379 + +10 2.705119 2.501527 0.709368 + +11 2.763923 2.456791 0.719173 + +12 2.597343 2.411362 0.730358 + +13 2.647648 2.374054 0.738579 + +14 2.527706 2.336248 0.747234 + +15 2.543835 2.306177 0.755421 + +16 2.478718 2.282218 0.761426 + +17 2.563173 2.261219 0.766674 + +18 2.480387 2.251179 0.769268 + +19 2.415822 2.244734 0.770911 + +20 2.459630 2.243680 0.771312 + +Total time: 1:04:34 +/home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k +Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-sl.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.010382 0.760864 0.848000 + +2 0.885819 0.800701 0.835000 + +3 0.768587 0.828160 0.844000 + +4 0.698592 0.751787 0.857000 + +5 0.620826 0.767858 0.856000 + +6 0.563309 0.727524 0.859000 + +7 0.524067 0.700363 0.878000 + +8 0.499860 0.707688 0.871000 + +Total time: 03:40 +Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-sl.m +Loss and accuracy using (cls_best): [0.40361115, tensor(0.8717)] + +Processing data/wiki/zh-100/models/sp15k/qrnn_nl4.m +../mldoc/zh-1 +Max vocab: 15000 +Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k +Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-sl.m +Training +Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv +Running tokenization lm... +Data lm, trn: 13500, val: 1500 + +Running tokenization cls... +Data cls, trn: 1000, val: 1000 + +Running tokenization tst... +Data tst, trn: 1000, val: 4000 + +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/zh-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/zh-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 3.804344 3.253225 0.571249 + +Total time: 02:19 +epoch train_loss valid_loss accuracy +1 3.309380 3.061855 0.600547 + +2 3.055568 2.865146 0.636541 + +3 2.877304 2.710331 0.663194 + +4 2.749444 2.604171 0.681028 + +5 2.715787 2.528694 0.693581 + +6 2.664271 2.477576 0.702474 + +7 2.598713 2.412279 0.715434 + +8 2.540510 2.367390 0.724008 + +9 2.499321 2.330683 0.731755 + +10 2.513472 2.290408 0.740227 + +11 2.397077 2.248312 0.749950 + +12 2.425433 2.212132 0.757908 + +13 2.364556 2.176752 0.767242 + +14 2.349984 2.142507 0.775855 + +15 2.321824 2.119729 0.781726 + +16 2.313458 2.095738 0.788297 + +17 2.239505 2.078650 0.792735 + +18 2.240292 2.069656 0.795083 + +19 2.250233 2.064083 0.796754 + +20 2.251804 2.063307 0.797006 + +Total time: 1:10:47 +/home/test/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k +Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-sl.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.920672 0.685620 0.870000 + +2 0.773749 0.700495 0.907000 + +3 0.663505 0.669162 0.908000 + +4 0.591225 0.621341 0.915000 + +5 0.542783 0.622516 0.919000 + +6 0.517919 0.608709 0.911000 + +7 0.489793 0.605009 0.918000 + +8 0.476783 0.597071 0.916000 + +Total time: 02:37 +Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-sl.m +Loss and accuracy using (cls_best): [0.29685277, tensor(0.9190)] + +OrderedDict([('data/mldoc/de-1/models/sp15k/qrnn_nl4-sl.m', 0.9610000252723694), + ('data/mldoc/en-1/models/sp15k/qrnn_nl4-sl.m', 0.9555000066757202), + ('data/mldoc/es-1/models/sp15k/qrnn_nl4-sl.m', 0.9582499861717224), + ('data/mldoc/fr-1/models/sp15k/qrnn_nl4-sl.m', 0.9480000138282776), + ('data/mldoc/it-1/models/sp15k/qrnn_nl4-sl.m', 0.9004999995231628), + ('data/mldoc/ja-1/models/sp15k/qrnn_nl4-sl.m', 0.8987500071525574), + ('data/mldoc/ru-1/models/sp15k/qrnn_nl4-sl.m', 0.871749997138977), + ('data/mldoc/zh-1/models/sp15k/qrnn_nl4-sl.m', + 0.9190000295639038)]) +data/mldoc/de-1/models/sp15k/qrnn_nl4-sl.m: 0.9610000252723694 +data/mldoc/en-1/models/sp15k/qrnn_nl4-sl.m: 0.9555000066757202 +data/mldoc/es-1/models/sp15k/qrnn_nl4-sl.m: 0.9582499861717224 +data/mldoc/fr-1/models/sp15k/qrnn_nl4-sl.m: 0.9480000138282776 +data/mldoc/it-1/models/sp15k/qrnn_nl4-sl.m: 0.9004999995231628 +data/mldoc/ja-1/models/sp15k/qrnn_nl4-sl.m: 0.8987500071525574 +data/mldoc/ru-1/models/sp15k/qrnn_nl4-sl.m: 0.871749997138977 +data/mldoc/zh-1/models/sp15k/qrnn_nl4-sl.m: 0.9190000295639038 +``` + + +## CLS training +### all +``` +python -m ulmfit eval --glob="mldoc/*-1/models/sp15k/qrnn_nl4.m" --name nl4-1cyc-sl-e4 --num-cls-epochs=4 --bs=18 --lr_sched=1cycle --label-smoothing-eps=0.1 +Processing data/mldoc/de-1/models/sp15k/qrnn_nl4.m +de-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/de.dev.csv +Data lm, trn: 13500, val: 1500 +Data cls, trn: 1000, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) may be ABI-incompatible with PyTorch! +Please use a compiler that is ABI-compatible with GCC 4.9 and above. +See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html. + +See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6 +for instructions on how to install GCC 4.9 or higher. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler)) +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.806209 0.620243 0.938000 +2 0.643088 0.608909 0.944000 +3 0.552762 0.577317 0.943000 +4 0.506282 0.566124 0.944000 +Total time: 01:09 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m +Loss and accuracy using (cls_best): [0.18408646, tensor(0.9597)] +Processing data/mldoc/en-1/models/sp15k/qrnn_nl4.m +en-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/en.dev.csv +Data lm, trn: 13500, val: 1500 +Data cls, trn: 1000, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁the', '▁,', 's', '▁.', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.856259 0.637753 0.935000 +2 0.713970 0.627611 0.918000 +3 0.594603 0.550310 0.947000 +4 0.526848 0.549133 0.954000 +Total time: 01:15 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m +Loss and accuracy using (cls_best): [0.20320596, tensor(0.9500)] +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 1000, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.804829 0.636135 0.906000 +2 0.712160 0.579012 0.952000 +3 0.605291 0.543731 0.965000 +4 0.531676 0.546145 0.966000 +Total time: 01:01 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m +Loss and accuracy using (cls_best): [0.18462537, tensor(0.9565)] +Processing data/mldoc/fr-1/models/sp15k/qrnn_nl4.m +fr-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/fr.dev.csv +Data lm, trn: 13500, val: 1500 +Data cls, trn: 1000, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.835704 0.657239 0.902000 +2 0.688311 0.679450 0.924000 +3 0.575970 0.579612 0.938000 +4 0.515154 0.565664 0.939000 +Total time: 01:11 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m +Loss and accuracy using (cls_best): [0.20844615, tensor(0.9435)] +Processing data/mldoc/it-1/models/sp15k/qrnn_nl4.m +it-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/it.dev.csv +Data lm, trn: 13500, val: 1500 +Data cls, trn: 1000, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁di', '▁e', "▁&'", "'", '▁il', '▁la', '▁in', 'e'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.914871 0.758207 0.848000 +2 0.799002 0.692626 0.877000 +3 0.658110 0.646415 0.888000 +4 0.567358 0.629301 0.912000 +Total time: 00:42 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m +Loss and accuracy using (cls_best): [0.30882642, tensor(0.9032)] +Processing data/mldoc/ja-1/models/sp15k/qrnn_nl4.m +ja-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv +Data lm, trn: 13500, val: 1500 +Data cls, trn: 1000, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.959891 0.771793 0.834000 +2 0.813159 0.684481 0.889000 +3 0.675869 0.698423 0.878000 +4 0.580597 0.689197 0.881000 +Total time: 01:24 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m +Loss and accuracy using (cls_best): [0.3306819, tensor(0.8967)] +Processing data/mldoc/zh-1/models/sp15k/qrnn_nl4.m +zh-1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv +Data lm, trn: 13500, val: 1500 +Data cls, trn: 1000, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有'] +Loss and accuracy using (cls_best): [0.28541276, tensor(0.9237)] +OrderedDict([('data/mldoc/de-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m', + 0.9597499966621399), + ('data/mldoc/en-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m', + 0.949999988079071), + ('data/mldoc/es-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m', + 0.9564999938011169), + ('data/mldoc/fr-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m', + 0.9434999823570251), + ('data/mldoc/it-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m', + 0.903249979019165), + ('data/mldoc/ja-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m', + 0.8967499732971191), + ('data/mldoc/zh-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m', + 0.9237499833106995)]) +data/mldoc/de-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m: 0.9597499966621399 +data/mldoc/en-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m: 0.949999988079071 +data/mldoc/es-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m: 0.9564999938011169 +data/mldoc/fr-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m: 0.9434999823570251 +data/mldoc/it-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m: 0.903249979019165 +data/mldoc/ja-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m: 0.8967499732971191 +data/mldoc/zh-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m: 0.9237499833106995 +``` +### ZH Exec 1 ``` python -m ulmfit eval --glob="mldoc/zh-1/models/sp15k/qrnn_nl4.m" --name nl4-1cyc-sl --num-cls-epochs=8 --bs=18 --lr_sched=1cycle --label-smoothing-eps=0.1 diff --git a/results/logs/noise/de10k-noise.md b/results/logs/mldoc/noise/de10k-noise.md similarity index 100% rename from results/logs/noise/de10k-noise.md rename to results/logs/mldoc/noise/de10k-noise.md diff --git a/results/logs/mldoc/noise/es10k-noise-lstm.md b/results/logs/mldoc/noise/es10k-noise-lstm.md new file mode 100644 index 0000000..b40dbe1 --- /dev/null +++ b/results/logs/mldoc/noise/es10k-noise-lstm.md @@ -0,0 +1,370 @@ +``` + python -m ulmfit eval_noise_resistance --lang=es --size=10 --prefix-name="val_" --model="sp30k/lstm_nl4.m" +Noise: 0 +Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m +es-10 +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_0.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que'] +Loss and accuracy using (cls_best): [0.32779965, tensor(0.9515)] +OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_0.m', + 0.9514999985694885)]) +Noise: 5 +Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m +es-10 +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_5.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que'] +Loss and accuracy using (cls_best): [0.33051395, tensor(0.9488)] +OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_5.m', + 0.9487500190734863)]) +Noise: 10 +Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m +es-10 +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_10.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que'] +Loss and accuracy using (cls_best): [0.22158922, tensor(0.9433)] +OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_10.m', + 0.9432500004768372)]) +Noise: 15 +Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m +es-10 +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_15.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que'] +Loss and accuracy using (cls_best): [0.25426567, tensor(0.9358)] +OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_15.m', + 0.9357500076293945)]) +Noise: 20 +Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m +es-10 +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_20.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que'] +Loss and accuracy using (cls_best): [0.32246214, tensor(0.9210)] +OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_20.m', + 0.9210000038146973)]) +Noise: 25 +Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m +es-10 +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_25.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que'] +Loss and accuracy using (cls_best): [0.823559, tensor(0.9095)] +OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_25.m', + 0.909500002861023)]) +Noise: 30 +Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m +es-10 +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_30.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que'] +Loss and accuracy using (cls_best): [0.5010365, tensor(0.8942)] +OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_30.m', + 0.8942499756813049)]) +Noise: 35 +Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m +es-10 +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_35.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que'] +Loss and accuracy using (cls_best): [0.95638776, tensor(0.5853)] +OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_35.m', + 0.5852500200271606)]) +Noise: 40 +Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m +es-10 +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_40.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que'] +Loss and accuracy using (cls_best): [1.1012905, tensor(0.5642)] +OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_40.m', + 0.5642499923706055)]) +Noise: 45 +Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m +es-10 +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_45.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que'] +Loss and accuracy using (cls_best): [1.6009017, tensor(0.3072)] +OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_45.m', + 0.3072499930858612)]) +Noise: 50 +Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m +es-10 +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_50.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que'] +Loss and accuracy using (cls_best): [1.5735056, tensor(0.3072)] +OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_50.m', + 0.3072499930858612)]) +Noise: 55 +Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m +es-10 +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_55.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 5201 examples, only 0.4500951575385917 have correct labels +Added noise to 550 examples, only 0.45 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.55tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que'] +Single training schedule +epoch train_loss valid_loss accuracy +1 1.150436 1.391014 0.321000 +2 1.190502 5.388964 0.313000 +3 1.216090 1.697217 0.221000 +4 1.221863 1.676644 0.221000 +5 1.213776 1.734900 0.221000 +6 1.195663 1.713853 0.221000 +7 1.211159 1.710040 0.221000 +8 1.197578 1.674693 0.221000 +Total time: 29:10 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_55.m +Loss and accuracy using (cls_best): [1.5282942, tensor(0.3072)] +OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_55.m', + 0.3072499930858612)]) +Noise: 60 +Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m +es-10 +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_60.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 5674 examples, only 0.4000845844787482 have correct labels +Added noise to 600 examples, only 0.4 have correct labels +Data lm, trn: 13013, val: 1445 +Running tokenization clsnoise0.6tv... +Data clsnoise0.6tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_60.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.176071 1.396755 0.321000 +2 1.211001 1.619019 0.289000 +3 1.229442 1.733743 0.261000 +4 1.190156 1.545205 0.312000 +5 1.182274 1.369377 0.308000 +6 1.169403 1.352204 0.304000 +7 1.166997 1.332295 0.316000 +8 1.165893 1.370641 0.313000 +Total time: 30:23 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_60.m +Loss and accuracy using (cls_best): [1.2368572, tensor(0.6102)] +OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_60.m', + 0.6102499961853027)]) +Noise: 65 +Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m +es-10 +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_65.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 6147 examples, only 0.35007401141890465 have correct labels +Added noise to 650 examples, only 0.35 have correct labels +Data lm, trn: 13013, val: 1445 +Running tokenization clsnoise0.65tv... +Data clsnoise0.65tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_65.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.179257 1.460323 0.295000 +2 1.220349 1.516707 0.222000 +3 1.211396 1.870125 0.242000 +4 1.187261 1.922184 0.308000 +5 1.201833 1.429372 0.300000 +6 1.187137 1.580070 0.264000 +7 1.162549 1.845004 0.294000 +8 1.162919 1.514930 0.313000 +Total time: 29:23 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_65.m +Loss and accuracy using (cls_best): [1.1729655, tensor(0.6385)] +OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_65.m', + 0.6384999752044678)]) +Noise: 70 +Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m +es-10 +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_70.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 6620 examples, only 0.30006343835906113 have correct labels +Added noise to 700 examples, only 0.3 have correct labels +Data lm, trn: 13013, val: 1445 +Running tokenization clsnoise0.7tv... +Data clsnoise0.7tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_70.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.155135 1.479137 0.312000 +2 1.190364 1.649113 0.288000 +3 1.220965 3.919039 0.280000 +4 1.222588 1.696949 0.258000 +5 1.220919 1.669896 0.264000 +6 1.217906 2.003806 0.257000 +7 1.216235 1.654473 0.258000 +8 1.217084 1.675933 0.258000 +Total time: 29:04 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_70.m +Loss and accuracy using (cls_best): [1.5526773, tensor(0.1828)] +OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_70.m', + 0.18275000154972076)]) +Noise: 75 +Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m +es-10 +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_75.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 7093 examples, only 0.2500528652992176 have correct labels +Added noise to 750 examples, only 0.25 have correct labels +Data lm, trn: 13013, val: 1445 +Running tokenization clsnoise0.75tv... +Data clsnoise0.75tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_75.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.170507 1.421675 0.344000 +2 1.221764 1.700004 0.246000 +3 1.215101 2.358311 0.263000 +4 1.243265 1.551931 0.257000 +5 1.222902 1.756996 0.271000 +6 1.215993 1.677014 0.266000 +7 1.225945 4.560951 0.263000 +8 1.219151 2.939914 0.245000 +Total time: 29:52 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_75.m +Loss and accuracy using (cls_best): [1.7072973, tensor(0.2465)] +OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_75.m', + 0.24650000035762787)]) + noise accuracy +0 0.00 0.95150 +1 0.05 0.94875 +2 0.10 0.94325 +3 0.15 0.93575 +4 0.20 0.92100 +5 0.25 0.90950 +6 0.30 0.89425 +7 0.35 0.58525 +8 0.40 0.56425 +9 0.45 0.30725 +10 0.50 0.30725 +11 0.55 0.30725 +12 0.60 0.61025 +13 0.65 0.63850 +14 0.70 0.18275 +15 0.75 0.24650 +``` \ No newline at end of file diff --git a/results/logs/noise/es10k-noise.md b/results/logs/mldoc/noise/es10k-noise.md similarity index 55% rename from results/logs/noise/es10k-noise.md rename to results/logs/mldoc/noise/es10k-noise.md index e39e197..c33efb1 100644 --- a/results/logs/noise/es10k-noise.md +++ b/results/logs/mldoc/noise/es10k-noise.md @@ -1,3 +1,981 @@ + +# Label Smoothing +## Epochs 4 +``` +python -m ulmfit eval_noise_resistance --lang=es --size=10 --prefix-name="sl-e4" --model="sp15k/qrnn_nl4.m" --num-cls-epochs=4 --label-smoothing-eps=0.1 +python -m ulmfit eval_noise_resistance --lang=es --size=10 --prefix-name="sl-e4" --model="sp15k/qrnn_nl4.m" --num-cls-epochs=4 --label-smoothing-eps=0.1 +Noise: 0 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e40.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) may be ABI-incompatible with PyTorch! +Please use a compiler that is ABI-compatible with GCC 4.9 and above. +See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html. + +See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6 +for instructions on how to install GCC 4.9 or higher. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler)) +Loss and accuracy using (cls_best): [0.2204297, tensor(0.9553)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e40.m', + 0.9552500247955322)]) +Noise: 5 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e45.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Loss and accuracy using (cls_best): [4.971248, tensor(0.8798)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e45.m', + 0.8797500133514404)]) +Noise: 10 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e410.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Loss and accuracy using (cls_best): [0.5768092, tensor(0.9420)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e410.m', + 0.9419999718666077)]) +Noise: 15 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e415.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Loss and accuracy using (cls_best): [1.2631954, tensor(0.9197)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e415.m', + 0.9197499752044678)]) +Noise: 20 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e420.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Loss and accuracy using (cls_best): [0.42572692, tensor(0.9237)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e420.m', + 0.9237499833106995)]) +Noise: 25 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e425.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Loss and accuracy using (cls_best): [29.74483, tensor(0.8648)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e425.m', + 0.8647500276565552)]) +Noise: 30 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e430.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Loss and accuracy using (cls_best): [0.51494944, tensor(0.9185)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e430.m', + 0.9185000061988831)]) +Noise: 35 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e435.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Loss and accuracy using (cls_best): [0.66567194, tensor(0.8848)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e435.m', + 0.8847500085830688)]) +Noise: 40 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e440.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Loss and accuracy using (cls_best): [2.3167746, tensor(0.8217)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e440.m', + 0.8217499852180481)]) +Noise: 45 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e445.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Loss and accuracy using (cls_best): [1.6398115, tensor(0.8192)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e445.m', + 0.8192499876022339)]) +Noise: 50 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e450.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 4729 examples, only 0.5 have correct labels +Added noise to 500 examples, only 0.5 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.5tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Single training schedule +epoch train_loss valid_loss accuracy +1 1.222034 1.372962 0.382000 +2 1.222928 9.350571 0.374000 +3 1.200542 1.636571 0.413000 +4 1.175799 7.368647 0.417000 +Total time: 05:31 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e450.m +Loss and accuracy using (cls_best): [5.1662917, tensor(0.7197)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e450.m', + 0.7197499871253967)]) +Noise: 55 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e455.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 5201 examples, only 0.4500951575385917 have correct labels +Added noise to 550 examples, only 0.45 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.55tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e455.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.241146 1.574603 0.338000 +2 1.220505 3.343982 0.314000 +3 1.215406 4.805650 0.381000 +4 1.192340 3.459251 0.368000 +Total time: 05:16 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e455.m +Loss and accuracy using (cls_best): [4.879584, tensor(0.6900)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e455.m', + 0.6899999976158142)]) +Noise: 60 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e460.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 5674 examples, only 0.4000845844787482 have correct labels +Added noise to 600 examples, only 0.4 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.6tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e460.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.258633 1.374315 0.319000 +2 1.249057 1.935162 0.279000 +3 1.235318 8.574917 0.325000 +4 1.224898 31.664907 0.356000 +Total time: 05:25 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e460.m +Loss and accuracy using (cls_best): [33.578053, tensor(0.5670)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e460.m', + 0.5669999718666077)]) +Noise: 65 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e465.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 6147 examples, only 0.35007401141890465 have correct labels +Added noise to 650 examples, only 0.35 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.65tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e465.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.259673 1.630593 0.307000 +2 1.256428 1.900148 0.237000 +3 1.241903 3.709883 0.307000 +4 1.218096 1.747542 0.292000 +Total time: 05:31 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e465.m +Loss and accuracy using (cls_best): [1.3016428, tensor(0.5238)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e465.m', + 0.5237500071525574)]) +Noise: 70 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e470.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 6620 examples, only 0.30006343835906113 have correct labels +Added noise to 700 examples, only 0.3 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.7tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e470.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.261316 1.444706 0.331000 +2 1.251065 6.526892 0.285000 +3 1.232021 19.925491 0.335000 +4 1.207373 1.669503 0.329000 +Total time: 05:17 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e470.m +Loss and accuracy using (cls_best): [1.6000191, tensor(0.1885)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e470.m', + 0.18850000202655792)]) +Noise: 75 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e475.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 7093 examples, only 0.2500528652992176 have correct labels +Added noise to 750 examples, only 0.25 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.75tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e475.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.256266 1.862535 0.346000 +2 1.245836 3.535186 0.307000 +3 1.222711 5.987287 0.335000 +4 1.206070 4.709743 0.322000 +Total time: 06:12 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e475.m +Loss and accuracy using (cls_best): [1.847491, tensor(0.1700)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e475.m', + 0.17000000178813934)]) + noise accuracy +0 0.00 0.95525 +1 0.05 0.87975 +2 0.10 0.94200 +3 0.15 0.91975 +4 0.20 0.92375 +5 0.25 0.86475 +6 0.30 0.91850 +7 0.35 0.88475 +8 0.40 0.82175 +9 0.45 0.81925 +10 0.50 0.71975 +11 0.55 0.69000 +12 0.60 0.56700 +13 0.65 0.52375 +14 0.70 0.18850 + +``` + +## Epochs 8 +``` +python -m ulmfit eval_noise_resistance --lang=es --size=10 --prefix-name="sl-e8" --model="sp15k/qrnn_nl4.m" --num-cls-epochs=8 --label-smoothing-eps=0.1 +Noise: 0 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e80.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Data lm, trn: 13013, val: 1445 +Data cls, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) may be ABI-incompatible with PyTorch! +Please use a compiler that is ABI-compatible with GCC 4.9 and above. +See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html. + +See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6 +for instructions on how to install GCC 4.9 or higher. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler)) +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e80.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.520412 0.696753 0.934000 +2 0.509825 0.735629 0.934000 +3 0.486594 0.684108 0.931000 +4 0.488381 0.579037 0.953000 +5 0.471751 0.588278 0.945000 +6 0.459802 0.567890 0.943000 +7 0.455027 0.545390 0.954000 +8 0.462613 0.575517 0.943000 +Total time: 11:12 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e80.m +Loss and accuracy using (cls_best): [0.27576917, tensor(0.9417)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e80.m', + 0.9417499899864197)]) +Noise: 5 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e85.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 472 examples, only 0.9500951575385916 have correct labels +Added noise to 50 examples, only 0.95 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.05tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e85.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.666560 0.852668 0.874000 +2 0.657355 5.546601 0.824000 +3 0.656841 7.308374 0.853000 +4 0.612917 24.591734 0.790000 +5 0.596588 12.358717 0.834000 +6 0.568417 4.996921 0.913000 +7 0.557468 1.554828 0.831000 +8 0.536724 1.115134 0.858000 +Total time: 10:51 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e85.m +Loss and accuracy using (cls_best): [0.6382615, tensor(0.9078)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e85.m', + 0.9077500104904175)]) +Noise: 10 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e810.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 945 examples, only 0.9000845844787482 have correct labels +Added noise to 100 examples, only 0.9 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.1tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e810.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.771364 0.982115 0.801000 +2 0.744242 0.947076 0.840000 +3 0.736343 1.086157 0.842000 +4 0.730165 0.987014 0.827000 +5 0.708371 5.287072 0.763000 +6 0.697188 0.855899 0.833000 +7 0.640460 0.931902 0.835000 +8 0.593337 0.915233 0.837000 +Total time: 10:48 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e810.m +Loss and accuracy using (cls_best): [0.45135674, tensor(0.9170)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e810.m', + 0.9169999957084656)]) +Noise: 15 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e815.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 1418 examples, only 0.8500740114189046 have correct labels +Added noise to 150 examples, only 0.85 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.15tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e815.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.848943 1.361678 0.751000 +2 0.857650 1.120813 0.662000 +3 0.861447 1.036685 0.809000 +4 0.824053 1.000713 0.786000 +5 0.821309 1.111803 0.799000 +6 0.754514 1.129427 0.782000 +7 0.695773 1.394005 0.765000 +8 0.638984 1.081989 0.780000 +Total time: 11:00 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e815.m +Loss and accuracy using (cls_best): [0.36475056, tensor(0.9005)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e815.m', + 0.9004999995231628)]) +Noise: 20 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e820.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 1891 examples, only 0.8000634383590611 have correct labels +Added noise to 200 examples, only 0.8 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.2tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e820.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.931807 1.271276 0.709000 +2 0.923600 1.298985 0.741000 +3 0.925085 5.663558 0.658000 +4 0.929284 1.188406 0.739000 +5 0.888162 3.878520 0.658000 +6 0.819503 1.259068 0.731000 +7 0.752936 1.150221 0.710000 +8 0.735175 1.149028 0.722000 +Total time: 11:07 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e820.m +Loss and accuracy using (cls_best): [0.4438091, tensor(0.8813)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e820.m', + 0.8812500238418579)]) +Noise: 25 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e825.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 2364 examples, only 0.7500528652992176 have correct labels +Added noise to 250 examples, only 0.75 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.25tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e825.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.999227 1.194783 0.646000 +2 0.990589 2.633189 0.626000 +3 1.018390 1.542320 0.659000 +4 0.987891 7.653442 0.605000 +5 0.972299 2.651095 0.646000 +6 0.922192 9.360953 0.637000 +7 0.878368 2.497859 0.640000 +8 0.847166 1.993811 0.639000 +Total time: 10:48 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e825.m +Loss and accuracy using (cls_best): [0.88353807, tensor(0.8367)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e825.m', + 0.8367499709129333)]) +Noise: 30 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e830.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 2837 examples, only 0.7000422922393741 have correct labels +Added noise to 300 examples, only 0.7 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.3tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e830.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.052917 1.325655 0.581000 +2 1.071630 1.998020 0.526000 +3 1.063711 31.339291 0.510000 +4 1.029802 2.206242 0.607000 +5 1.027847 1.763125 0.607000 +6 0.961119 1.473439 0.657000 +7 0.908768 1.629924 0.617000 +8 0.873362 1.462983 0.626000 +Total time: 10:44 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e830.m +Loss and accuracy using (cls_best): [0.61362606, tensor(0.8410)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e830.m', + 0.8410000205039978)]) +Noise: 35 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e835.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 3310 examples, only 0.6500317191795305 have correct labels +Added noise to 350 examples, only 0.65 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.35tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e835.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.124931 1.301690 0.549000 +2 1.098526 1.527998 0.599000 +3 1.109146 3.372168 0.557000 +4 1.085633 9.049232 0.536000 +5 1.040149 2.878901 0.552000 +6 0.999970 1.699484 0.548000 +7 0.924742 2.458920 0.519000 +8 0.916262 5.709455 0.517000 +Total time: 10:39 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e835.m +Loss and accuracy using (cls_best): [7.591171, tensor(0.6967)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e835.m', + 0.6967499852180481)]) +Noise: 40 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e840.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 3783 examples, only 0.600021146119687 have correct labels +Added noise to 400 examples, only 0.6 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.4tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e840.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.165653 1.350154 0.508000 +2 1.172212 1.732053 0.421000 +3 1.159003 9.825891 0.476000 +4 1.147642 2.909990 0.485000 +5 1.085988 4.217392 0.523000 +6 1.073084 3.288731 0.488000 +7 0.998839 1.664031 0.482000 +8 0.932477 1.921165 0.471000 +Total time: 11:06 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e840.m +Loss and accuracy using (cls_best): [0.8643208, tensor(0.7275)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e840.m', + 0.7275000214576721)]) +Noise: 45 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e845.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 4256 examples, only 0.5500105730598435 have correct labels +Added noise to 450 examples, only 0.55 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.45tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e845.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.195873 1.296093 0.433000 +2 1.181843 13.203069 0.375000 +3 1.202427 3.895014 0.350000 +4 1.180791 2.918823 0.442000 +5 1.168890 4.746016 0.477000 +6 1.138908 5.763408 0.452000 +7 1.085301 1.734959 0.487000 +8 1.019007 1.669769 0.467000 +Total time: 10:40 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e845.m +Loss and accuracy using (cls_best): [0.99686193, tensor(0.7107)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e845.m', + 0.7107499837875366)]) +Noise: 50 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e850.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 4729 examples, only 0.5 have correct labels +Added noise to 500 examples, only 0.5 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.5tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e850.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.206193 1.506727 0.400000 +2 1.219487 1.916529 0.383000 +3 1.201234 12.136375 0.383000 +4 1.193026 2.111413 0.403000 +5 1.184301 2.633834 0.438000 +6 1.145973 2.558009 0.437000 +7 1.099467 2.177719 0.410000 +8 1.061634 2.724488 0.401000 +Total time: 11:06 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e850.m +Loss and accuracy using (cls_best): [1.8331982, tensor(0.5920)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e850.m', + 0.5920000076293945)]) +Noise: 55 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e855.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 5201 examples, only 0.4500951575385917 have correct labels +Added noise to 550 examples, only 0.45 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.55tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e855.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.231457 1.337840 0.356000 +2 1.245993 12.007490 0.306000 +3 1.229736 1.784564 0.331000 +4 1.227334 4.005848 0.339000 +5 1.207836 12.369584 0.363000 +6 1.186945 15.869857 0.365000 +7 1.143967 22.024086 0.386000 +8 1.097784 4.130466 0.361000 +Total time: 10:52 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e855.m +Loss and accuracy using (cls_best): [1.1213393, tensor(0.6237)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e855.m', + 0.6237499713897705)]) +Noise: 60 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e860.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 5674 examples, only 0.4000845844787482 have correct labels +Added noise to 600 examples, only 0.4 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.6tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e860.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.253129 1.360811 0.344000 +2 1.259076 1.422704 0.292000 +3 1.249924 2.302250 0.258000 +4 1.238938 7.596085 0.318000 +5 1.226801 19.490450 0.361000 +6 1.220376 45.920719 0.358000 +7 1.186902 92.042252 0.361000 +8 1.169160 47.323799 0.352000 +Total time: 10:51 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e860.m +Loss and accuracy using (cls_best): [54.127697, tensor(0.5253)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e860.m', + 0.5252500176429749)]) +Noise: 65 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e865.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 6147 examples, only 0.35007401141890465 have correct labels +Added noise to 650 examples, only 0.35 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.65tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e865.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.258576 1.400468 0.285000 +2 1.252457 1.485225 0.303000 +3 1.264578 15.317787 0.257000 +4 1.245918 8.976856 0.300000 +5 1.239147 5.338088 0.296000 +6 1.226425 8.540084 0.314000 +7 1.206139 8.959650 0.300000 +8 1.185957 7.868751 0.322000 +Total time: 10:47 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e865.m +Loss and accuracy using (cls_best): [3.7213144, tensor(0.5070)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e865.m', + 0.5070000290870667)]) +Noise: 70 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e870.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 6620 examples, only 0.30006343835906113 have correct labels +Added noise to 700 examples, only 0.3 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.7tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e870.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.257724 1.391102 0.311000 +2 1.259833 1.376604 0.346000 +3 1.256289 4.146193 0.272000 +4 1.248032 21.711473 0.308000 +5 1.237731 104.512573 0.287000 +6 1.226114 14.212803 0.318000 +7 1.190652 3.960902 0.333000 +8 1.186640 2.436945 0.339000 +Total time: 10:38 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e870.m +Loss and accuracy using (cls_best): [2.232332, tensor(0.2713)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e870.m', + 0.27125000953674316)]) +Noise: 75 +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-10 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e875.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv +Added noise to 7093 examples, only 0.2500528652992176 have correct labels +Added noise to 750 examples, only 0.25 have correct labels +Data lm, trn: 13013, val: 1445 +Data clsnoise0.75tv, trn: 9458, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e875.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.246185 1.331556 0.343000 +2 1.247202 1.593753 0.334000 +3 1.248334 4.676108 0.320000 +4 1.235685 27.420618 0.289000 +5 1.221268 10.798445 0.330000 +6 1.201879 4.895516 0.335000 +7 1.164162 3.015471 0.353000 +8 1.146750 2.903884 0.353000 +Total time: 11:04 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e875.m +Loss and accuracy using (cls_best): [3.2696714, tensor(0.1305)] +OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e875.m', + 0.13050000369548798)]) + noise accuracy +0 0.00 0.94175 +1 0.05 0.90775 +2 0.10 0.91700 +3 0.15 0.90050 +4 0.20 0.88125 +5 0.25 0.83675 +6 0.30 0.84100 +7 0.35 0.69675 +8 0.40 0.72750 +9 0.45 0.71075 +10 0.50 0.59200 +11 0.55 0.62375 +12 0.60 0.52525 +13 0.65 0.50700 +14 0.70 0.27125 +``` + + # Correct VAL ``` python -m ulmfit eval_noise_resistance --lang=es --size=10 --prefix-name="val_" diff --git a/results/logs/qrnn-ru.md b/results/logs/qrnn-ru.md index a213dd6..6b6b121 100644 --- a/results/logs/qrnn-ru.md +++ b/results/logs/qrnn-ru.md @@ -1,15 +1,183 @@ # QRNN RU -## SP15k nl8 -data/wiki/ru-100/models/sp15k/qrnn_nl8.m +## SP15k nl4 +## LM +export CUDA_VISIBLE_DEVICES=3 +LANG=ru +python -m ulmfit lm --dataset-path data/wiki/ru-100 --bidir=False --qrnn=True --nl 4 --tokenizer='sp' --max-vocab 15000 --lang ru --name 'nl4' - train 10 --drop-mult=0 --bs=50 --label-smoothing-eps=0.1 +## SP15k nl8 +### LM +``` +5 2.869308 2.905951 0.466976 +6 2.768955 2.782804 0.481852 +7 2.654484 2.676304 0.495593 +8 2.585963 2.591748 0.508447 +9 2.512042 2.526819 0.518860 +10 2.520543 2.509287 0.521890 +Total time: 18:46:01 +data/wiki/ru-100/models/sp15k +Saving info data/wiki/ru-100/models/sp15k/qrnn_nl8.m/info.json +``` +### MLDoc +``` +python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_nl8.m --lang=${LANG} --name 'nl8' - train 20 --bs 20 --num-cls-epochs=8 --lr_sched=1cycle --label-smoothing-eps=0.1 +Max vocab: 15000 +Cache dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k +Model dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl8.m +Loading validation /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv +Running tokenization lm... +Data lm, trn: 9195, val: 1021 + +Running tokenization cls... +Data cls, trn: 1000, val: 1000 + +Running tokenization tst... +Data tst, trn: 1000, val: 4000 + +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl8.m/lm_best'), PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl8.m/../itos')] +epoch train_loss valid_loss accuracy +1 2.923423 2.334532 0.529978 + +Total time: 02:46 +epoch train_loss valid_loss accuracy +1 2.462077 2.150593 0.563281 + +2 2.230013 1.972095 0.596198 + +3 2.118523 1.812012 0.623204 + +4 1.916368 1.690016 0.644060 + +5 1.842718 1.585770 0.661704 + +6 1.748630 1.513972 0.674130 + +7 1.675032 1.447667 0.686207 + +8 1.628485 1.393949 0.695972 + +9 1.564814 1.330838 0.707272 + +10 1.553933 1.283114 0.715716 + +11 1.441891 1.234810 0.726201 + +12 1.496388 1.185676 0.735977 + +13 1.383019 1.141014 0.745528 + +14 1.256620 1.094201 0.755120 + +15 1.306187 1.052457 0.764280 + +16 1.297933 1.028387 0.769747 + +17 1.319773 1.004256 0.775285 + +18 1.178073 0.989788 0.778480 + +19 1.252248 0.982740 0.780057 + +20 1.177640 0.981201 0.780267 + +Total time: 1:24:58 +/home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k +Saving info /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl8.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.882775 0.510930 0.826000 + +2 0.683476 0.513669 0.847000 + +3 0.556661 0.590375 0.839000 + +4 0.454019 0.757216 0.828000 + +5 0.344460 0.549675 0.870000 + +6 0.246039 0.630242 0.861000 + +7 0.173423 0.649066 0.858000 + +8 0.098640 0.638015 0.867000 + +Total time: 05:11 +Saving models at /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl8.m +Loss and accuracy using (cls_best): [0.64393336, tensor(0.8683)] +``` + +### MLDoc nl8 -2nd +``` + python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME}-2 - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0 +.1 +Max vocab: 15000 +Cache dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k +Model dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl8-2.m +Loading validation /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv +Data lm, trn: 9195, val: 1021 +Data cls, trn: 1000, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl8.m/lm_best'), PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl8.m/../itos')] +epoch train_loss valid_loss accuracy +1 3.966292 3.450758 0.527065 +Total time: 02:58 +epoch train_loss valid_loss accuracy +1 3.495761 3.276329 0.560047 +2 3.319947 3.102911 0.593742 +3 3.137904 2.955317 0.620171 +4 3.040286 2.839161 0.642270 +5 2.869962 2.753622 0.658331 +6 2.905739 2.680881 0.672860 +7 2.836454 2.620925 0.685026 +8 2.857271 2.569716 0.695722 +9 2.702872 2.520050 0.705589 +10 2.701559 2.473591 0.715346 +11 2.740815 2.429558 0.725597 +12 2.646513 2.389550 0.735010 +13 2.587685 2.349614 0.744885 +14 2.546527 2.311087 0.754463 +15 2.568136 2.278581 0.762980 +16 2.492115 2.252367 0.769275 +17 2.338561 2.230529 0.775072 +18 2.447506 2.218215 0.778437 +19 2.364424 2.212115 0.780085 +20 2.367132 2.210520 0.780424 +Total time: 1:30:47 +/home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k +Saving info /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl8-2.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.969255 0.769736 0.843000 +2 0.846340 0.813483 0.839000 +3 0.718175 0.705339 0.867000 +4 0.609513 0.726442 0.875000 +Total time: 02:54 +Saving models at /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl8-2.m +Loss and accuracy using (cls_best): [0.4056449, tensor(0.8698)] +0.40564489364624023 +``` + + + + + + +## cls +``` export CUDA_VISIBLE_DEVICES=0 LANG=ru -python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_nl8.m --lang=${LANG} --name 'nl8' - train 20 --bs 20 --num-cls-epochs=8 --lr_sched=1cycle - -export CUDA_VISIBLE_DEVICES=0 -LANG=de -python -m ulmfit cls --dataset-path data/mldoc-m/${LANG}-1 --base-lm-path data/wiki-m/${LANG}-100/models/sp15k/lstm_nl4.m --lang=${LANG} --name 'nl4' - train 20 --bs 20 --num-cls-epochs=8 --lr_sched=1cycle - +python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_nl8.m --lang=${LANG} --name 'nl8' - train 20 --bs 20 --num-cls-epochs=8 --lr_sched=1cycle --label-smoothing-eps=0.1 +``` ## SP30k nl4 ### LM diff --git a/results/logs/ru.md b/results/logs/ru.md index f309aa0..dc469aa 100644 --- a/results/logs/ru.md +++ b/results/logs/ru.md @@ -17,7 +17,10 @@ Total time: 10:43:03 data/wiki/ru-100/models/sp15k Saving info data/wiki/ru-100/models/sp15k/qrnn_nl4.m/info.json ``` +```bash +python -m ulmfit cls --dataset-path data/mldoc/ru-1 --base-lm-path data/wiki/ru-100/models/sp30k/lstm_nl4.m --lang=ru --name 'nl4-100' --cuda-id=1 - train 20 --bs 40 --num-cls-epochs=2 +``` ## SP30k nl4 ### LM diff --git a/results/logs/ru/bs_vs_lr.md b/results/logs/ru/bs_vs_lr.md new file mode 100644 index 0000000..9d13a63 --- /dev/null +++ b/results/logs/ru/bs_vs_lr.md @@ -0,0 +1,123 @@ + +## BS=18, lr_mult=1.0 + +epoch train_loss valid_loss accuracy +1 4.427713 3.693394 0.484268 +Total time: 01:32 +epoch train_loss valid_loss accuracy +1 3.758918 3.446661 0.529820 +2 3.394254 3.199054 0.577411 +3 3.235364 3.014517 0.610520 +4 3.125459 2.871101 0.637153 +5 2.994313 2.773862 0.654470 +6 2.915075 2.693080 0.669942 +7 2.855732 2.622858 0.683629 +8 2.755074 2.572147 0.694145 +9 2.697898 2.517524 0.704816 +10 2.689881 2.468190 0.715927 +11 2.579573 2.432807 0.723324 +12 2.659464 2.387878 0.733931 +13 2.520637 2.344804 0.744233 +14 2.482952 2.315014 0.751855 +15 2.564730 2.279045 0.761163 +16 2.552707 2.255916 0.766971 +17 2.511244 2.240169 0.770991 +18 2.461429 2.228213 0.774309 +19 2.426440 2.222140 0.775745 +20 2.425955 2.221128 0.775836 +Total time: 1:14:17 + +## BS=500, lr_mult=1.0 +epoch train_loss valid_loss accuracy +1 5.536769 3.850831 0.444662 +Total time: 01:10 +epoch train_loss valid_loss accuracy +1 4.845898 3.781763 0.461471 +2 4.388605 3.643141 0.491225 +3 4.038255 3.464554 0.526143 + +## BS=500, lr_mult=27 +epoch train_loss valid_loss accuracy +1 7.234749 5.868155 0.312675 +Total time: 01:41 + +## BS=500, lr_mult=10 + BS=50 lr_mult=10 for cls +/data/wiki/ru-100/models/sp15k/qrnn_nl4sl.m/../itos')] +epoch train_loss valid_loss accuracy +1 5.052441 4.082105 0.439539 +Total time: 02:27 +epoch train_loss valid_loss accuracy +1 4.120197 3.712686 0.498216 +2 3.727043 3.373258 0.557896 +3 3.383009 3.109635 0.598970 +4 3.180799 2.938478 0.626816 +5 3.048913 2.812639 0.647257 +6 2.943903 2.727179 0.661784 +7 2.864300 2.650275 0.674248 +8 2.773810 2.583594 0.687063 +9 2.724850 2.529445 0.697573 +10 2.673996 2.473824 0.708698 +11 2.657637 2.431461 0.716904 +12 2.591277 2.372668 0.730318 +13 2.537707 2.323294 0.741157 +14 2.486507 2.280270 0.751768 +15 2.435933 2.238660 0.762545 +16 2.401303 2.208848 0.769561 +17 2.374117 2.184253 0.776400 +18 2.341421 2.169156 0.780388 +19 2.328202 2.163922 0.781700 +20 2.315462 2.161784 0.782105 +Total time: 1:07:09 +------------------- Checking the influence of number of epochs on the accuracy +(multifit) test@test:~/workspace/ulmfit-multilingual$ rm /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4sl-bs500.m/cls* +(multifit) test@test:~/workspace/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME}-bs500 - train 0 --bs 50 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1 --lr_mult=1 +Max vocab: 15000 +Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k +Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4sl-bs500.m +Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv +Data lm, trn: 9195, val: 1021 +Data cls, trn: 1000, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х'] +Single training schedule +epoch train_loss valid_loss accuracy +1 1.063845 1.111960 0.601000 +2 0.902245 0.766871 0.817000 +3 0.766261 0.707502 0.861000 +4 0.680053 0.694492 0.866000 +Total time: 01:22 +Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4sl-bs500.m +Loss and accuracy using (cls_best): [0.41532615, tensor(0.8630)] +0.41532614827156067 +0.8629999756813049 +(multifit) test@test:~/workspace/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME}-bs500 - train 0 --bs 50 --num-cls-epochs=16 --lr_sched=1cycle --label-smoothing-eps=0.1 --lr_mult=1 +Max vocab: 15000 +Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k +Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4sl-bs500.m +Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv +Data lm, trn: 9195, val: 1021 +Data cls, trn: 1000, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х'] +Loading last classifier +Single training schedule +epoch train_loss valid_loss accuracy +1 0.556688 0.706000 0.873000 +2 0.537578 0.717411 0.865000 +3 0.532326 0.775549 0.854000 +4 0.529178 0.767506 0.861000 +5 0.521306 0.797604 0.860000 +6 0.527344 0.736225 0.868000 +7 0.516393 0.724941 0.878000 +8 0.510422 0.716110 0.873000 +9 0.504320 0.701886 0.869000 +10 0.500323 0.676577 0.878000 +11 0.493490 0.682657 0.873000 +12 0.484450 0.682047 0.878000 +13 0.479248 0.682782 0.880000 +14 0.474778 0.688019 0.873000 +15 0.472664 0.685304 0.874000 +16 0.470747 0.677925 0.878000 +Total time: 07:57 diff --git a/results/logs/ru/lm-opti.md b/results/logs/ru/lm-opti.md new file mode 100644 index 0000000..122a3b6 --- /dev/null +++ b/results/logs/ru/lm-opti.md @@ -0,0 +1,260 @@ +CUDA_VISIBLE_DEVICES=0 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 5 --name 'nl5-merity' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 2500 - train 14 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1 +CUDA_VISIBLE_DEVICES=1 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-wide2' --max-vocab 15000 --lang ${LANG} --qrnn=True --nh 3100 - train 10 --bs=100 --drop_mult=0 --label-smoothing-eps=0.1 +CUDA_VISIBLE_DEVICES=2 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-merity' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 2500 - train 14 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1 +CUDA_VISIBLE_DEVICES=3 python -m ulmfit lm --dataset-path data/wiki/ru-100 --bidir=False --qrnn=True --nl 4 --tokenizer=sp --max-vocab 15000 --lang ru --name nl4sl - train 10 --drop-mult=0 --bs=50 --label-smoothing-eps=0.1 + +## 25vocab +CUDA_VISIBLE_DEVICES=3 python -m ulmfit lm --dataset-path data/wiki/ru-100 --bidir=False --qrnn=True --nl 4 --tokenizer=sp --max-vocab 25000 --lang ru --name nl4 - train 10 --drop-mult=0 --bs=50 --label-smoothing-eps=0.1 + +LANG=ru +CUDA_VISIBLE_DEVICES=0 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-merity-wide2' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 3100 - train 10 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1 + +##### CLS +export CUDA_VISIBLE_DEVICES=0 +LANG=ru +NAME=nl5-merity +python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1 + +export CUDA_VISIBLE_DEVICES=1 +LANG=ru +NAME=nl4-wide2 +python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1 + +export CUDA_VISIBLE_DEVICES=2 +LANG=ru +NAME=nl4-merity +python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1 + +export CUDA_VISIBLE_DEVICES=3 +LANG=ru +NAME=nl4sl +python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1 + + +-----------------------CLS1 +export CUDA_VISIBLE_DEVICES=0 +LANG=ru +NAME=nl4 +python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1 + + +export CUDA_VISIBLE_DEVICES=0 +LANG=ru +NAME=nl8 +python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1 + + +python -m ulmfit cls --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 5 --name 'nl5-merity' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 2500 - train 14 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1 + +CUDA_VISIBLE_DEVICES=1 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-wide2' --max-vocab 15000 --lang ${LANG} --qrnn=True --nh 3100 - train 10 --bs=100 --drop_mult=0 --label-smoothing-eps=0.1 +CUDA_VISIBLE_DEVICES=2 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-merity' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 2500 - train 14 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1 +CUDA_VISIBLE_DEVICES=3 python -m ulmfit lm --dataset-path data/wiki/ru-100 --bidir=False --qrnn=True --nl 4 --tokenizer=sp --max-vocab 15000 --lang ru --name nl4sl - train 10 --drop-mult=0 --bs=50 --label-smoothing-eps=0.1 +## + +------------------------ + +7 3.680504 3.678406 0.498396 +8 3.556062 3.596037 0.512345 +9 3.553716 3.535783 0.523509 +10 3.523366 3.515352 0.527935 +Total time: 20:03:59 +data/wiki/ru-100/models/sp15k +Saving info data/wiki/ru-100/models/sp15k/qrnn_ nl4sl.m/info.json + +### Ru +``` +export CUDA_VISIBLE_DEVICES=3 +LANG=ru +NAME=nl4sl +python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1 +Max vocab: 15000 +Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k +Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4sl.m +Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv +Data lm, trn: 9195, val: 1021 +Data cls, trn: 1000, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4sl.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4sl.m/../itos')] +epoch train_loss valid_loss accuracy +1 4.427713 3.693394 0.484268 +Total time: 01:32 +epoch train_loss valid_loss accuracy +1 3.758918 3.446661 0.529820 +2 3.394254 3.199054 0.577411 +3 3.235364 3.014517 0.610520 +4 3.125459 2.871101 0.637153 +5 2.994313 2.773862 0.654470 +6 2.915075 2.693080 0.669942 +7 2.855732 2.622858 0.683629 +8 2.755074 2.572147 0.694145 +9 2.697898 2.517524 0.704816 +10 2.689881 2.468190 0.715927 +11 2.579573 2.432807 0.723324 +12 2.659464 2.387878 0.733931 +13 2.520637 2.344804 0.744233 +14 2.482952 2.315014 0.751855 +15 2.564730 2.279045 0.761163 +16 2.552707 2.255916 0.766971 +17 2.511244 2.240169 0.770991 +18 2.461429 2.228213 0.774309 +19 2.426440 2.222140 0.775745 +20 2.425955 2.221128 0.775836 +Total time: 1:14:17 +/home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k +Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4sl.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.022382 0.779370 0.822000 +2 0.866379 0.792353 0.832000 +3 0.715650 0.698579 0.865000 +4 0.603621 0.693501 0.884000 +Total time: 02:05 +Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4sl.m +Loss and accuracy using (cls_best): [0.3978519, tensor(0.8723)] +0.3978519141674042 +0.8722500205039978 +``` + +---- + +```bash +$ export CUDA_VISIBLE_DEVICES=0 +$ LANG=ru +$ NAME=nl4 +$ python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1 +Max vocab: 15000 +Cache dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k +Model dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4.m +Loading validation /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv +Data lm, trn: 9195, val: 1021 +Data cls, trn: 1000, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х'] +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4.m/../itos')] +epoch train_loss valid_loss accuracy +1 4.531968 3.764185 0.474252 +Total time: 01:44 +epoch train_loss valid_loss accuracy +1 3.770046 3.506013 0.522443 +2 3.546580 3.251341 0.571620 +3 3.320569 3.055680 0.606364 +4 3.130226 2.912925 0.631395 +5 3.072772 2.809725 0.649728 +6 2.765424 2.731825 0.662963 +7 2.959237 2.662104 0.676203 +8 2.807999 2.600417 0.688423 +9 2.771271 2.548279 0.699473 +10 2.809488 2.501688 0.709020 +11 2.707221 2.454946 0.719196 +12 2.597226 2.417315 0.728432 +13 2.609972 2.376176 0.737923 +14 2.590427 2.341666 0.746216 +15 2.572995 2.306599 0.754747 +16 2.496636 2.285632 0.760806 +17 2.508584 2.266456 0.765147 +18 2.441373 2.253839 0.768449 +19 2.430915 2.249204 0.769536 +20 2.426130 2.247966 0.769886 +Total time: 47:33 +/home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k +Saving info /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.060299 0.890710 0.716000 +2 0.884965 0.769866 0.853000 +3 0.722994 0.723213 0.875000 +4 0.609488 0.730594 0.865000 +Total time: 01:14 +Saving models at /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.39589784, tensor(0.8692)] +0.39589783549308777 +0.8692499995231628 +``` + + +## wide 2 +```bash +$ CUDA_VISIBLE_DEVICES=1 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-wide2' --max-vocab 15000 --lang ${LANG} --qrnn=True --nh 3100 - train 10 --bs=100 --drop_mult=0 --label-smoothing-eps=0.1 +Max vocab: 15000 +Cache dir: data/wiki/ru-100/models/sp15k +Model dir: data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m +Wiki text was split to 193047 articles +Wiki text was split to 460 articles +Data lm, trn: 193047, val: 460 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Training lm from random weights +epoch train_loss valid_loss accuracy +1 3.908233 3.950865 0.463469 +2 3.738863 3.815026 0.477703 +3 3.696502 3.779513 0.483625 +4 3.692592 3.720908 0.490143 +5 3.600519 3.652444 0.501671 +6 3.564568 3.582584 0.511550 +7 3.472859 3.493226 0.525943 +8 3.390483 3.407970 0.541749 +9 3.351620 3.344207 0.552758 +10 3.329683 3.330087 0.556380 +Total time: 51:05:43 +data/wiki/ru-100/models/sp15k +Saving info data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/info.json +``` +### MLDoc +export CUDA_VISIBLE_DEVICES=1 +LANG=ru +NAME=nl4-wide2 +python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1 + + + +## Merity nl4 +```bash + CUDA_VISIBLE_DEVICES=2 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-merity' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 2500 - train 14 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1 +Max vocab: 15000 +Cache dir: data/wiki/ru-100/models/sp15k +Model dir: data/wiki/ru-100/models/sp15k/qrnn_nl4-merity.m +Wiki text was split to 193047 articles +Wiki text was split to 460 articles +Data lm, trn: 193047, val: 460 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Training lm from random weights +epoch train_loss valid_loss accuracy +1 3.965899 3.977734 0.460046 +2 3.806082 3.858176 0.472396 +3 3.839230 3.874757 0.469224 +4 3.762105 3.868653 0.469943 +5 3.800827 3.833991 0.474116 +6 3.755466 3.796329 0.479868 +7 3.691958 3.747888 0.487367 +8 3.660529 3.702986 0.493545 +9 3.593282 3.635035 0.504086 +10 3.585948 3.579200 0.513631 +11 3.473865 3.512114 0.525391 +12 3.451973 3.455807 0.535520 +13 3.418731 3.417129 0.542943 +14 3.385637 3.407541 0.545545 +Total time: 51:32:09 +data/wiki/ru-100/models/sp15k +Saving info data/wiki/ru-100/models/sp15k/qrnn_nl4-merity.m/info.json +``` + +#### MLDoc + +export CUDA_VISIBLE_DEVICES=2 +LANG=ru +NAME=nl4-merity +python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1 diff --git a/results/logs/ru/merity4.md b/results/logs/ru/merity4.md new file mode 100644 index 0000000..d84b258 --- /dev/null +++ b/results/logs/ru/merity4.md @@ -0,0 +1,101 @@ +## LM + + +### MLDoc 1 +``` +python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1 +Max vocab: 15000 +Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k +Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-merity.m +Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv +Data lm, trn: 9195, val: 1021 +Data cls, trn: 1000, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4-merity.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4-merity.m/../itos')] +epoch train_loss valid_loss accuracy +1 4.411345 3.660489 0.486965 +Total time: 02:43 +epoch train_loss valid_loss accuracy +1 3.613334 3.372079 0.544188 +2 3.325245 3.100234 0.596406 +3 3.181919 2.906442 0.631586 +4 3.010830 2.767429 0.656378 +5 2.880418 2.663865 0.676339 +6 2.825526 2.571074 0.694140 +7 2.766901 2.483362 0.711652 +8 2.601965 2.417213 0.726853 +9 2.569160 2.341699 0.744193 +10 2.588142 2.272457 0.760294 +11 2.494011 2.198197 0.779175 +12 2.421921 2.135517 0.795854 +13 2.396429 2.075012 0.812815 +14 2.306572 2.019140 0.828851 +15 2.281730 1.966554 0.843595 +16 2.206670 1.927567 0.854515 +17 2.143836 1.901352 0.862114 +18 2.141715 1.884954 0.867003 +19 2.070353 1.876935 0.869214 +20 2.066195 1.874844 0.869665 +Total time: 2:12:21 +/home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k +Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-merity.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.994393 0.755689 0.844000 +2 0.859871 0.822650 0.856000 +3 0.678185 0.721333 0.859000 +4 0.586906 0.693618 0.878000 +Total time: 04:17 +Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-merity.m +Loss and accuracy using (cls_best): [0.3872361, tensor(0.8777)] +0.387236088514328 +0.8777499794960022 +``` + +### MLDoc 2 +``` +(multifit) test@test:~/workspace/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/mldoc/${LANG}-1/models/sp15k/qrnn_${NAME}. +m --lang=${LANG} --name ${NAME}-16 - train 0 --bs 18 --num-cls-epochs=16 --lr_sched=1cycle --label-smoothing-eps=0.1 +Max vocab: 15000 +Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k +Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-merity-16.m +Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv +Data lm, trn: 9195, val: 1021 +Data cls, trn: 1000, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k +Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-merity-16.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.081481 0.837775 0.781000 +2 0.901621 0.798574 0.858000 +3 0.778870 0.826576 0.859000 +4 0.693465 0.787875 0.833000 +5 0.639763 0.841092 0.861000 +6 0.595044 0.731504 0.853000 +7 0.576115 0.796013 0.819000 +8 0.544098 0.744034 0.875000 +9 0.531359 0.699035 0.879000 +10 0.513886 0.698310 0.879000 +11 0.495473 0.686897 0.864000 +12 0.489863 0.688584 0.881000 +13 0.481086 0.675660 0.881000 +14 0.479960 0.684917 0.883000 +15 0.490157 0.687865 0.882000 +16 0.486081 0.679104 0.882000 +Total time: 15:26 +Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-merity-16.m +Loss and accuracy using (cls_best): [0.4047818, tensor(0.8737)] +0.4047817885875702 +0.8737499713897705 +``` \ No newline at end of file diff --git a/results/logs/ru/merity5.md b/results/logs/ru/merity5.md new file mode 100644 index 0000000..fe8bddf --- /dev/null +++ b/results/logs/ru/merity5.md @@ -0,0 +1,38 @@ +## LM +```bash +CUDA_VISIBLE_DEVICES=0 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 5 --name 'nl5-mer +ity' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 2500 - train 14 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1 +Max vocab: 15000 +Cache dir: data/wiki/ru-100/models/sp15k +Model dir: data/wiki/ru-100/models/sp15k/qrnn_nl5-merity.m +Wiki text was split to 193047 articles +Wiki text was split to 460 articles +Data lm, trn: 193047, val: 460 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Training lm from random weights +epoch train_loss valid_loss accuracy +1 3.969639 4.024787 0.452887 +2 3.814622 3.834142 0.476612 +3 3.798372 3.846118 0.473666 +4 3.742609 3.835311 0.474612 +5 3.715114 3.790690 0.480469 +6 3.652987 3.748408 0.486146 +7 3.573350 3.697325 0.493774 +8 3.589853 3.637134 0.504189 +9 3.558110 3.583030 0.512137 +10 3.501382 3.510491 0.524148 +11 3.408982 3.437177 0.536634 +12 3.402717 3.373548 0.548113 +13 3.293624 3.331311 0.556288 +14 3.309859 3.322777 0.558426 +Total time: 68:05:15 +data/wiki/ru-100/models/sp15k +Saving info data/wiki/ru-100/models/sp15k/qrnn_nl5-merity.m/info.json +``` + +### MLDoc 1 +``` + +``` \ No newline at end of file diff --git a/results/logs/ru/wide2.md b/results/logs/ru/wide2.md new file mode 100644 index 0000000..1637791 --- /dev/null +++ b/results/logs/ru/wide2.md @@ -0,0 +1,183 @@ + 3 2.812496 2.877055 0.468569 + +4 2.705551 2.792535 0.479420 + +5 2.649598 2.726415 0.487439 + +6 2.599835 2.635610 0.499679 + +7 2.574639 2.554657 0.512358 + +8 2.489573 2.475936 0.523280 + +9 2.396540 2.415555 0.534089 + +10 2.374290 2.401968 0.536601 + +Total time: 15:49:20 +data/wiki/ru-100/models/sp15k +Saving info data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/info.json +Fire trace: +1. Initial component +2. Accessed property "lm" (/home/test/workspace/ulmfit-multilingual/ulmfit/__main__.py:32) +3. Called routine "LMHyperParams" (/home/test/workspace/ulmfit-multilingual/ulmfit/__main__.py:32) +4. Accessed property "train" (/home/test/workspace/ulmfit-multilingual/ulmfit/pretrain_lm.py:174) +5. Called routine "train_lm" (/home/test/workspace/ulmfit-multilingual/ulmfit/pretrain_lm.py:174) +6. ('Could not consume arg:', '--nh') + +Type: NoneType +String form: None + +Usage: __main__.py lm --dataset-path data/wiki/ru-100 --tokenizer=sp --nl 4 --name nl4-wide2 --max-vocab 15000 --lang ru --qrnn=True - train 10 --bs=100 --drop_mult=0 - +(multifit) test@test:~/workspace/ulmfit-multilingual$ less data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/info.json +(multifit) test@test:~/workspace/ulmfit-multilingual$ CUDA_VISIBLE_DEVICES=1 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-wide2' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=100 --drop_mult=0 ^C100 -- +(multifit) test@test:~/workspace/ulmfit-multilingual$ mv data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/ data/wiki/ru-100/models/sp15k/qrnn_nl4-2.m/ +(multifit) test@test:~/workspace/ulmfit-multilingual$ less data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/info.json^C +(multifit) test@test:~/workspace/ulmfit-multilingual$ CUDA_VISIBLE_DEVICES=1 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-wid +e2' --max-vocab 15000 --lang ${LANG} --qrnn=True --nh 3100 - train 10 --bs=100 --drop_mult=0 +Max vocab: 15000 +Cache dir: data/wiki/ru-100/models/sp15k +Model dir: data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m +^CTraceback (most recent call last): + File "/home/test/anaconda3/envs/multifit/lib/python3.7/runpy.py", line 193, in _run_module_as_main + "__main__", mod_spec) + File "/home/test/anaconda3/envs/multifit/lib/python3.7/runpy.py", line 85, in _run_code + exec(code, run_globals) + File "/home/test/workspace/ulmfit-multilingual/ulmfit/__main__.py", line 119, in + fire.Fire(ULMFiT()) + File "/home/test/anaconda3/envs/multifit/lib/python3.7/site-packages/fire/core.py", line 127, in Fire + component_trace = _Fire(component, args, context, name) + File "/home/test/anaconda3/envs/multifit/lib/python3.7/site-packages/fire/core.py", line 366, in _Fire + component, remaining_args) + File "/home/test/anaconda3/envs/multifit/lib/python3.7/site-packages/fire/core.py", line 542, in _CallCallable + result = fn(*varargs, **kwargs) + File "/home/test/workspace/ulmfit-multilingual/ulmfit/pretrain_lm.py", line 176, in train_lm + data_lm = self.load_wiki_data(bs=bs) if data_lm is None else data_lm + File "/home/test/workspace/ulmfit-multilingual/ulmfit/pretrain_lm.py", line 253, in load_wiki_data + train_df=read_wiki_articles(trn_path), + File "/home/test/workspace/ulmfit-multilingual/ulmfit/pretrain_lm.py", line 48, in read_wiki_articles + if i < len(lines)-2 and lines[i+1].strip() == "" and istitle(lines[i+2]): + File "/home/test/workspace/ulmfit-multilingual/ulmfit/pretrain_lm.py", line 39, in istitle + return len(re.findall(r'^ ?= [^=]* = ?$', line)) != 0 + File "/home/test/anaconda3/envs/multifit/lib/python3.7/re.py", line 223, in findall + return _compile(pattern, flags).findall(string) +KeyboardInterrupt +^C +(multifit) test@test:~/workspace/ulmfit-multilingual$ CUDA_VISIBLE_DEVICES=1 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-wide2' --max-vocab 15000 --lang ${LANG} --qrnn=True --nh 3100 - train 10 --bs=100 --drop_mult=0 --label-smoothing-eps=0.1 +Max vocab: 15000 +Cache dir: data/wiki/ru-100/models/sp15k +Model dir: data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m +Wiki text was split to 193047 articles +Wiki text was split to 460 articles +Data lm, trn: 193047, val: 460 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Training lm from random weights +epoch train_loss valid_loss accuracy +1 3.908233 3.950865 0.463469 +2 3.738863 3.815026 0.477703 +3 3.696502 3.779513 0.483625 +4 3.692592 3.720908 0.490143 +5 3.600519 3.652444 0.501671 +6 3.564568 3.582584 0.511550 +7 3.472859 3.493226 0.525943 +8 3.390483 3.407970 0.541749 +9 3.351620 3.344207 0.552758 +10 3.329683 3.330087 0.556380 +Total time: 51:05:43 +data/wiki/ru-100/models/sp15k +Saving info data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/info.json +(multifit) test@test:~/workspace/ulmfit-multilingual$ export CUDA_VISIBLE_DEVICES=1 +(multifit) test@test:~/workspace/ulmfit-multilingual$ LANG=ru +(multifit) test@test:~/workspace/ulmfit-multilingual$ NAME=nl4-wide2 +(multifit) test@test:~/workspace/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1 +Max vocab: 15000 +Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k +Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-wide2.m +Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv +Data lm, trn: 9195, val: 1021 +Data cls, trn: 1000, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/../itos')] +epoch train_loss valid_loss accuracy +1 3.777423 3.222261 0.564503 +Total time: 04:35 +epoch train_loss valid_loss accuracy +1 3.292465 3.029143 0.602257 +2 3.034045 2.858176 0.634576 +3 2.943366 2.710314 0.665116 +4 2.722069 2.596702 0.687515 +5 2.819853 2.508158 0.705020 +6 2.734984 2.417240 0.724748 +7 2.674353 2.332395 0.743694 +8 2.527344 2.251373 0.762892 +9 2.473972 2.168185 0.784043 +10 2.359504 2.093983 0.803255 +11 2.287590 2.019540 0.823566 +12 2.254421 1.943832 0.845138 +13 2.203321 1.884380 0.863381 +14 2.142532 1.824186 0.881509 +15 2.121573 1.777664 0.894901 +16 2.013238 1.740772 0.905824 +17 2.026189 1.715271 0.913569 +18 1.904322 1.700163 0.917917 +19 1.889113 1.692539 0.919811 +20 1.903118 1.691033 0.920319 +Total time: 3:10:09 +/home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k +Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-wide2.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.006922 0.880313 0.788000 +2 0.823572 0.782953 0.860000 +3 0.679078 0.749164 0.872000 +4 0.579215 0.707200 0.872000 +Total time: 06:30 +Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-wide2.m +Loss and accuracy using (cls_best): [0.3935929, tensor(0.8708)] +0.393592894077301 +0.8707500100135803 +(multifit) test@test:~/workspace/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/mldoc/${LANG}-1/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME}-16 - train 0 --bs 18 --num-cls-epochs=16 --lr_sched=1cycle --label-smoothing-eps=0.1 +Max vocab: 15000 +Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k +Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-wide2-16.m +Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv +Data lm, trn: 9195, val: 1021 +Data cls, trn: 1000, val: 1000 +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k +Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-wide2-16.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 1.067446 0.822965 0.824000 +2 0.897088 0.845636 0.826000 +3 0.778055 0.828693 0.847000 +4 0.685080 0.893327 0.823000 +5 0.620457 0.929057 0.800000 +6 0.587644 0.802154 0.859000 +7 0.570255 0.713434 0.872000 +8 0.543071 0.705259 0.871000 +9 0.517465 0.715090 0.867000 +10 0.498291 0.695459 0.876000 +11 0.497857 0.698052 0.862000 +12 0.486924 0.681911 0.878000 +13 0.479041 0.676714 0.874000 +14 0.475131 0.677843 0.878000 +15 0.467238 0.672065 0.876000 +16 0.476889 0.680850 0.875000 +Total time: 23:47 +Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-wide2-16.m +Loss and accuracy using (cls_best): [0.41155785, tensor(0.8700)] +0.4115578532218933 +0.8700000047683716 \ No newline at end of file diff --git a/results/logs/zeroshot.md b/results/logs/zeroshot.md index 074f859..fbb6199 100644 --- a/results/logs/zeroshot.md +++ b/results/logs/zeroshot.md @@ -90,9 +90,405 @@ zh from fr | Test: 79.40% | classes: 33.60 31.12 9.07 26.20 ``` +#### ULMFit zershot on laser-en1k 4 epochs +```bash +python -m ulmfit eval --glob="mldoc/*-1/models/sp15k/qrnn_nl4.m" --dataset_template="{lang}-1*-laser-en1" --name nl4 --num-cls-epochs=4 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=18 + python -m ulmfit eval --glob="mldoc/*-1/models/sp15k/qrnn_nl4.m" --dataset_template='${lang}-1*-laser-en1' --name nl4 --num-cls-epochs=4 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=18 +Processing data/mldoc/de-1/models/sp15k/qrnn_nl4.m +de-1*-laser-en1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1-laser-en1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1-laser-en1/models/sp15k/qrnn_nl4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1-laser-en1/de.dev.csv +Running tokenization lm... +Data lm, trn: 13500, val: 1500 +Running tokenization cls... +Data cls, trn: 1000, val: 1000 +Running tokenization tst... +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) may be ABI-incompatible with PyTorch! +Please use a compiler that is ABI-compatible with GCC 4.9 and above. +See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html. + +See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6 +for instructions on how to install GCC 4.9 or higher. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler)) +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1-laser-en1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1-laser-en1/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.858069 0.756738 0.853000 +2 0.737577 0.687798 0.909000 +3 0.611922 0.615140 0.919000 +4 0.544274 0.608824 0.909000 +Total time: 01:08 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1-laser-en1/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.31768194, tensor(0.9133)] +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10-laser-en1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10-laser-en1/models/sp15k/qrnn_nl4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10-laser-en1/de.dev.csv +Running tokenization lm... +Data lm, trn: 13500, val: 1500 +Running tokenization cls... +Data cls, trn: 10000, val: 1000 +Running tokenization tst... +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10-laser-en1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10-laser-en1/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.665453 0.600660 0.924000 +2 0.624768 0.595788 0.922000 +3 0.576251 0.580166 0.930000 +4 0.520507 0.569867 0.930000 +Total time: 09:38 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10-laser-en1/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.3278069, tensor(0.9190)] +Processing data/mldoc/en-1/models/sp15k/qrnn_nl4.m +en-1*-laser-en1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1-laser-en1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1-laser-en1/models/sp15k/qrnn_nl4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1-laser-en1/en.dev.csv +Running tokenization lm... +Data lm, trn: 13500, val: 1500 +Running tokenization cls... +Data cls, trn: 1000, val: 1000 +Running tokenization tst... +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁the', '▁,', 's', '▁.', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1-laser-en1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1-laser-en1/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.836131 0.760151 0.887000 +2 0.691355 0.666257 0.905000 +3 0.587016 0.603976 0.932000 +4 0.529849 0.584768 0.943000 +Total time: 01:09 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1-laser-en1/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.20341124, tensor(0.9503)] +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-10-laser-en1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-10-laser-en1/models/sp15k/qrnn_nl4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-10-laser-en1/en.dev.csv +Running tokenization lm... +Data lm, trn: 13500, val: 1500 +Running tokenization cls... +Data cls, trn: 10000, val: 1000 +Running tokenization tst... +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁the', '▁,', 's', '▁.', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-10-laser-en1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-10-laser-en1/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.676697 0.611570 0.931000 +2 0.647868 0.629156 0.916000 +3 0.578947 0.546653 0.943000 +4 0.539242 0.550864 0.949000 +Total time: 10:25 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-10-laser-en1/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.2546631, tensor(0.9490)] +Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m +es-1*-laser-en1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1-laser-en1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1-laser-en1/models/sp15k/qrnn_nl4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1-laser-en1/es.dev.csv +Running tokenization lm... +Data lm, trn: 13013, val: 1445 +Running tokenization cls... +Data cls, trn: 1000, val: 1000 +Running tokenization tst... +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1-laser-en1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1-laser-en1/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.947175 0.774158 0.854000 +2 0.827687 0.747528 0.859000 +3 0.698278 0.726713 0.881000 +4 0.603821 0.729449 0.878000 +Total time: 00:58 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1-laser-en1/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.77420205, tensor(0.7893)] +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10-laser-en1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10-laser-en1/models/sp15k/qrnn_nl4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10-laser-en1/es.dev.csv +Running tokenization lm... +Data lm, trn: 13013, val: 1445 +Running tokenization cls... +Data cls, trn: 9458, val: 1000 +Running tokenization tst... +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10-laser-en1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10-laser-en1/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.654948 0.968341 0.863000 +2 0.634143 0.844979 0.873000 +3 0.559031 0.719894 0.890000 +4 0.524448 0.675569 0.887000 +Total time: 05:36 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10-laser-en1/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.70765454, tensor(0.7880)] +Processing data/mldoc/fr-1/models/sp15k/qrnn_nl4.m +fr-1*-laser-en1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1-laser-en1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1-laser-en1/models/sp15k/qrnn_nl4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1-laser-en1/fr.dev.csv +Running tokenization lm... +Data lm, trn: 13500, val: 1500 +Running tokenization cls... +Data cls, trn: 1000, val: 1000 +Running tokenization tst... +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1-laser-en1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1-laser-en1/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.882636 0.720057 0.851000 +2 0.766720 0.790358 0.847000 +3 0.661309 0.669355 0.877000 +4 0.584607 0.676029 0.889000 +Total time: 01:05 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1-laser-en1/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.32164142, tensor(0.8945)] +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-10-laser-en1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-10-laser-en1/models/sp15k/qrnn_nl4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-10-laser-en1/fr.dev.csv +Running tokenization lm... +Data lm, trn: 13500, val: 1500 +Running tokenization cls... +Data cls, trn: 10000, val: 1000 +Running tokenization tst... +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-10-laser-en1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-10-laser-en1/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.721444 0.714190 0.865000 +2 0.684221 0.688866 0.879000 +3 0.616377 0.624161 0.904000 +4 0.576405 0.630186 0.904000 +Total time: 09:37 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-10-laser-en1/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.39518934, tensor(0.8848)] +Processing data/mldoc/it-1/models/sp15k/qrnn_nl4.m +it-1*-laser-en1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1-laser-en1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1-laser-en1/models/sp15k/qrnn_nl4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1-laser-en1/it.dev.csv +Running tokenization lm... +Data lm, trn: 13500, val: 1500 +Running tokenization cls... +Data cls, trn: 1000, val: 1000 +Running tokenization tst... +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁di', '▁e', "▁&'", "'", '▁il', '▁la', '▁in', 'e'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1-laser-en1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1-laser-en1/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.963395 0.843734 0.777000 +2 0.892776 0.890245 0.785000 +3 0.755211 0.799284 0.815000 +4 0.629971 0.796769 0.820000 +Total time: 00:40 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1-laser-en1/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.76029295, tensor(0.7600)] +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-10-laser-en1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-10-laser-en1/models/sp15k/qrnn_nl4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-10-laser-en1/it.dev.csv +Running tokenization lm... +Data lm, trn: 13500, val: 1500 +Running tokenization cls... +Data cls, trn: 10000, val: 1000 +Running tokenization tst... +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁di', '▁e', "▁&'", "'", '▁il', '▁la', '▁in', 'e'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-10-laser-en1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-10-laser-en1/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.828429 0.801399 0.820000 +2 0.769261 0.786845 0.816000 +3 0.717665 0.734139 0.845000 +4 0.597664 0.741355 0.843000 +Total time: 05:29 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-10-laser-en1/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.7915614, tensor(0.7605)] +Processing data/mldoc/ja-1/models/sp15k/qrnn_nl4.m +ja-1*-laser-en1 +Processing data/mldoc/zh-1/models/sp15k/qrnn_nl4.m +zh-1*-laser-en1 +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-en1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-en1/models/sp15k/qrnn_nl4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-en1/zh.dev.csv +Running tokenization lm... +Data lm, trn: 13500, val: 1500 +Running tokenization cls... +Data cls, trn: 1000, val: 1000 +Running tokenization tst... +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-en1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-en1/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.964628 1.016147 0.710000 +2 0.836267 0.885741 0.789000 +3 0.694938 0.789230 0.811000 +4 0.594315 0.809480 0.811000 +Total time: 01:08 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-en1/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.5295076, tensor(0.8245)] +Max vocab: 15000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-10-laser-en1/models/sp15k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-10-laser-en1/models/sp15k/qrnn_nl4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-10-laser-en1/zh.dev.csv +Running tokenization lm... +Data lm, trn: 13500, val: 1500 +Running tokenization cls... +Data cls, trn: 10000, val: 1000 +Running tokenization tst... +Data tst, trn: 1000, val: 4000 +Size of vocabulary: 15000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有'] +Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15} +Loading pretrained model +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-10-laser-en1/models/sp15k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-10-laser-en1/models/sp15k/qrnn_nl4.m/info.json +Single training schedule +epoch train_loss valid_loss accuracy +1 0.823208 0.774504 0.821000 +2 0.771960 0.769799 0.822000 +3 0.682731 0.724021 0.847000 +4 0.595054 0.744821 0.836000 +Total time: 09:42 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-10-laser-en1/models/sp15k/qrnn_nl4.m +Loss and accuracy using (cls_best): [0.59163076, tensor(0.8127)] +OrderedDict([('data/mldoc/de-1-laser-en1/models/sp15k/qrnn_nl4.m', + 0.9132500290870667), + ('data/mldoc/de-10-laser-en1/models/sp15k/qrnn_nl4.m', + 0.9190000295639038), + ('data/mldoc/en-1-laser-en1/models/sp15k/qrnn_nl4.m', + 0.9502500295639038), + ('data/mldoc/en-10-laser-en1/models/sp15k/qrnn_nl4.m', + 0.9490000009536743), + ('data/mldoc/es-1-laser-en1/models/sp15k/qrnn_nl4.m', + 0.7892500162124634), + ('data/mldoc/es-10-laser-en1/models/sp15k/qrnn_nl4.m', + 0.7879999876022339), + ('data/mldoc/fr-1-laser-en1/models/sp15k/qrnn_nl4.m', + 0.8945000171661377), + ('data/mldoc/fr-10-laser-en1/models/sp15k/qrnn_nl4.m', + 0.8847500085830688), + ('data/mldoc/it-1-laser-en1/models/sp15k/qrnn_nl4.m', + 0.7599999904632568), + ('data/mldoc/it-10-laser-en1/models/sp15k/qrnn_nl4.m', + 0.7605000138282776), + ('data/mldoc/zh-1-laser-en1/models/sp15k/qrnn_nl4.m', + 0.8245000243186951), + ('data/mldoc/zh-10-laser-en1/models/sp15k/qrnn_nl4.m', + 0.812749981880188)]) +data/mldoc/de-1-laser-en1/models/sp15k/qrnn_nl4.m: 0.9132500290870667 +data/mldoc/de-10-laser-en1/models/sp15k/qrnn_nl4.m: 0.9190000295639038 +data/mldoc/en-1-laser-en1/models/sp15k/qrnn_nl4.m: 0.9502500295639038 +data/mldoc/en-10-laser-en1/models/sp15k/qrnn_nl4.m: 0.9490000009536743 +data/mldoc/es-1-laser-en1/models/sp15k/qrnn_nl4.m: 0.7892500162124634 +data/mldoc/es-10-laser-en1/models/sp15k/qrnn_nl4.m: 0.7879999876022339 +data/mldoc/fr-1-laser-en1/models/sp15k/qrnn_nl4.m: 0.8945000171661377 +data/mldoc/fr-10-laser-en1/models/sp15k/qrnn_nl4.m: 0.8847500085830688 +data/mldoc/it-1-laser-en1/models/sp15k/qrnn_nl4.m: 0.7599999904632568 +data/mldoc/it-10-laser-en1/models/sp15k/qrnn_nl4.m: 0.7605000138282776 +data/mldoc/zh-1-laser-en1/models/sp15k/qrnn_nl4.m: 0.8245000243186951 +``` + + #### Evaluation of Laser 1k Performance ``` -python -m ulmfit eval --glob="mldoc/*-1/models/sp60k/lstm_nl4.m" --dataset_template="{}-laser-*" --name nl4 --cuda-id=0 ✘ 130 +python -m ulmfit eval --glob="mldoc/*-1/models/sp60k/lstm_nl4.m" --dataset_template="{}-laser-*" --name nl4 --cuda-id=0 Max vocab: 60000 Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-fr/models/sp60k Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-fr/models/sp60k/lstm_nl4.m