wassname/multifit
1
0
Fork 0
mirror of https://github.com/wassname/multifit.git synced 2026-08-31 12:10:51 +08:00
Code Issues Packages Projects Releases Wiki Activity

Some logs from previous experiments

Browse Source
This commit is contained in:
Piotr Czapla
2019-02-27 20:31:38 +01:00
parent f750586114
commit 4742a68d28
19 changed files with 5881 additions and 243 deletions
Download Patch File Download Diff File Expand all files Collapse all files
results/MLDoc.md
+8 -5
View File
@@ -9,8 +9,9 @@
|ULMFiT L30k-100 | | 91.35 | 83.32 | 88.77 | 77.99 | 71.12 | 72.20 | |
|ULMFiT L30k | | 95.4 | 95.15 | 93.67 | 88.42 | 89.20 | **87.27** | 90.20 |
|ULMFiT sp-fixed | | **95.6** | 94.80 | 94.20 | 88.52 | 88.72 | 86.85 | 90.47 |
|ULMFIT Q15k 1cyc| **94.62** | **95.65** | 95.15 | **94.42** | **89.92** | 89.60 | | 90.78/89.82 |
|ULMFIT Q15k 1c l| | | | | | | | **92.22** |
|ULMFIT Q15k 1cyc| 94.62 | **95.65** | 95.15 | **94.42** | 89.92 | 89.60 | | 90.78/89.82 |
|ULMFIT Q15k 1c l| **94.99** | | 95.64 | 94.34 | **90.32** | 89.67 | 87.67^ | **92.22** |
|ULMFIT Q15k 1cfl| **95.55** | **96.10** | 95.82 | 94.80 | **90.04** | 89.87 | 87.17 | **91.90** |
|ULMFIT L30k 1cyc| | **95.85** | **96.32** | **94.82** | 89.87 | **90.45** | **87.94** | 92.02/91.64 |
- L30k - LSTM sp30k trained using gradual unfreezing
@@ -18,6 +19,8 @@
- ULMFiT sp-fixed - --||-- with fixed tokenization
- Q15k 1cyc - QRNN sp15k trained using 1cycle learning rate schedule
- L30k 1cyc - LSTM sp30k trained using 1cycle learning rate schedule
- We checked LSTM on sp15k on DE and got 95.53% accuracy which is comparable to QRNN sp15k
- ^ - 16 epochs qrnn_nl4sl-bs500
## Zero shot approaches - LSTM
@@ -77,9 +80,9 @@
| Impr 10k over 10k | 37% | 12% | 39% | 15% | 9% | 23% |
| Impr 10k over 1k | 34% | 19% | 29% | 21% | 11% | 26% |
| Impr 1k over 1k | 43% | 20% | 30% | 17% | 10% | 16% |
All ULMFiT examples above were trained on 1k training data generated by a LASER classification model
| ULMFiT qrnn on 1k LSRen1k | 91.32 | 78.92 | 89.45 | 75.99 | | 82.45 |
| ULMFiT qrnn on 10k LSRen1k| 91.90 | 78.79 | 88.47 | 76.05 | | |
## Noise resistance
results/logs/100examples.md
+1017
View File
File diff suppressed because it is too large Load Diff
results/logs/_experiment-log.md
-57
View File
@@ -1,57 +0,0 @@
## QRNN sp15k
```
cd fastai # go to fast ai
git checkout ulfit_multilingual
git pull
cd ../ulmfit-multilingual # go to ulmfit
git checkout master
git pull
export CUDA_VISIBLE_DEVICES=1
LANG=fr
python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0
## Jeremy
export CUDA_VISIBLE_DEVICES=2
LANG=it
python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0
export CUDA_VISIBLE_DEVICES=3
LANG=ru
python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0
```
```
#
export CUDA_VISIBLE_DEVICES=0
LANG=de
python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0
#
## Piotr
export CUDA_VISIBLE_DEVICES=1
LANG=es
python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0
export CUDA_VISIBLE_DEVICES=0
LANG=zh
python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0
```
# trained
export CUDA_VISIBLE_DEVICES=0
LANG=ja
python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0
done V100
```
export CUDA_VISIBLE_DEVICES=0
LANG=en
python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0
```
results/logs/cls.md
-172
View File
@@ -1,172 +0,0 @@
# without col merge
````
python -m ulmfit cls --dataset-path data/cls/${LANG}-books --base-lm-path data/wiki-m/${LANG}-100/models/sp30k/lstm_nl4.m --lang=${LANG} --name 'nl4' - train 20 --bs 20 --num-cls-epochs=8 --lr-sched=single
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k/lstm_nl4.m
Validation set not found using 10% of trn
Data lm, trn: 33183, val: 3687
Data cls, trn: 1800, val: 200
Running tokenization tst...
Data tst, trn: 200, val: 2000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁.', "'", 's', '▁la', '▁le', '▁et', '▁l', '▁à']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki-m/fr-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki-m/fr-100/models/sp30k/lstm_nl4.m/../itos')]
epoch train_loss valid_loss accuracy
1 3.356221 2.821012 0.518492
Total time: 00:21
epoch train_loss valid_loss accuracy
1 3.041214 2.734799 0.524577
2 2.919576 2.648412 0.535661
3 2.822292 2.542236 0.549206
4 2.721790 2.414110 0.561852
5 2.596515 2.276732 0.579841
6 2.453715 2.140479 0.600370
7 2.333764 2.000186 0.621349
8 2.231092 1.873927 0.644259
9 2.101130 1.765473 0.660529
10 2.006949 1.666797 0.682196
11 1.905025 1.584023 0.696058
12 1.820798 1.513958 0.709841
13 1.751217 1.456632 0.720846
14 1.689076 1.410359 0.729947
15 1.646113 1.371438 0.739868
16 1.594153 1.346142 0.744577
17 1.564375 1.332298 0.746693
18 1.536557 1.322925 0.748995
19 1.532926 1.319159 0.749444
20 1.525449 1.318028 0.749815
Total time: 08:51
/home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k/lstm_nl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.588118 0.581087 0.700000
2 0.504373 0.583527 0.720000
3 0.412651 0.538866 0.750000
4 0.295401 0.658459 0.750000
5 0.212442 1.054068 0.720000
6 0.126090 1.302099 0.745000
7 0.078312 1.307932 0.760000
8 0.050346 1.339740 0.745000
Total time: 00:35
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k/lstm_nl4.m
Loss and accuracy using (cls_best): [1.3513571, tensor(0.7700)]
1.351357102394104
0.7699999809265137
````
### FR books
````bash
python -m ulmfit cls --dataset-path data/cls/${LANG}-books --base-lm-path data/wiki-m/${LANG}-100/models/sp30k/lstm_nl4.m --lang=${LANG} --name 'nl4' - train 20 --bs 20 --num-cls-epochs=8 --lr-sched=single
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k/lstm_nl4.m
Validation set not found using 10% of trn
Running tokenization lm...
Data lm, trn: 33183, val: 3687
Running tokenization cls...
Data cls, trn: 1800, val: 200
Running tokenization tst...
Data tst, trn: 200, val: 2000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁.', "'", 's', '▁la', '▁le', '▁et', '▁l', '▁à']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki-m/fr-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki-m/fr-100/models/sp30k/lstm_nl4.m/../itos')]
epoch train_loss valid_loss accuracy
1 3.790325 3.409294 0.367234
Total time: 06:02
epoch train_loss valid_loss accuracy
1 3.526303 3.326439 0.378936
2 3.466923 3.226977 0.392378
3 3.342312 3.111874 0.406997
4 3.244619 2.992510 0.422330
5 3.156150 2.877498 0.437467
6 3.070326 2.762509 0.453874
7 2.956969 2.651613 0.471552
8 2.878008 2.535935 0.491058
9 2.790110 2.438724 0.508560
10 2.684145 2.323467 0.528415
11 2.633781 2.231418 0.547093
12 2.535126 2.143523 0.564889
13 2.464436 2.055402 0.582077
14 2.330094 1.989257 0.596582
15 2.372371 1.924338 0.610048
16 2.190224 1.866912 0.621738
17 2.176868 1.834098 0.629221
18 2.168293 1.809196 0.633879
19 2.151132 1.797144 0.636382
20 2.130476 1.793351 0.637044
Total time: 2:30:05
/home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k/lstm_nl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.314315 0.530879 0.865000
2 0.336746 0.468635 0.865000
3 0.255810 0.324242 0.870000
4 0.149121 0.480570 0.885000
5 0.093909 0.613743 0.890000
6 0.091678 0.660452 0.885000
7 0.049993 0.649642 0.910000
8 0.034218 0.640008 0.910000
Total time: 04:19
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k/lstm_nl4.m
Loss and accuracy using (cls_best): [0.5418505, tensor(0.9100)]
0.5418505072593689
0.9100000262260437
````
```
Loading pretrained model
Unknown tokens 0, first 100: []
Training lm from: [PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/../itos')]
epoch train_loss valid_loss accuracy
1 4.037580 3.312217 0.364410
Total time: 01:44
epoch train_loss valid_loss accuracy
1 3.709982 3.256320 0.371825
2 3.459413 3.150574 0.386972
3 3.296628 3.037327 0.402039
4 3.186458 2.914899 0.418413
5 3.092632 2.817097 0.431216
6 2.966957 2.726081 0.442906
7 2.924824 2.647339 0.453871
8 2.818279 2.561596 0.466795
9 2.773893 2.501994 0.475877
10 2.736084 2.438490 0.485978
11 2.688937 2.370927 0.496899
12 2.615245 2.314875 0.506508
13 2.583292 2.260717 0.515725
14 2.535631 2.220295 0.522666
15 2.466035 2.179093 0.530148
16 2.461427 2.151952 0.535315
17 2.390641 2.131065 0.538749
18 2.376235 2.116927 0.541430
19 2.407630 2.115370 0.542039
20 2.391378 2.112687 0.542522
Total time: 46:33
/home/n-waves/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k
Saving info /home/n-waves/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k/qrnn_nl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.466671 0.534682 0.745000
2 0.358965 0.372612 0.875000
3 0.251557 0.311034 0.900000
4 0.166484 0.585425 0.865000
5 0.101803 0.726341 0.900000
6 0.072025 0.587875 0.885000
7 0.045328 0.760989 0.890000
8 0.027765 0.727203 0.890000
Total time: 01:17
Saving models at /home/n-waves/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k/qrnn_nl4.m
Loss and accuracy using (cls_best): [0.55982095, tensor(0.8970)]
0.5598209500312805
0.8970000147819519
```
results/logs/cls/ulmfit on cls.md
+1092
View File
File diff suppressed because it is too large Load Diff
results/logs/cls/zeroshoot.md
+118
View File
@@ -0,0 +1,118 @@
# Results on books dataset
| | de | fr |
|-------------------------|-------|-------|
| laser zero shot from en | 84.15 | 83.90 |
| with ULMFIT QRNN sp15k | 89.60 | 87.84 |
## Laser results
| | en | de | fr |
|------|--------|-------|------|
| en: | 84.55 | 84.15 | 83.90|
| de: | 82.60 | 85.20 | 83.05|
| fr: | 77.20 | 82.95 | 84.85|
## ULMFiT improvment
```
data/cls/de-books-laser-en1/models/sp15k/qrnn_nl4.m: 0.8960000276565552
data/cls/fr-books-laser-en1/models/sp15k/qrnn_nl4.m: 0.8784999847412109
```
### Execution log
```
python -m ulmfit eval --glob="cls/*-books/models/sp15k/qrnn_nl4.m" --name nl4 --dataset-template='${lang}-books-laser-en1' --num-lm-epochs=0 --num-cls-epochs=8 --bs=18 --lr_sched=1cycle --label-smoothing-eps=0.1
Processing data/cls/de-books/models/sp15k/qrnn_nl4.m
de-books-laser-en1
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books-laser-en1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books-laser-en1/models/sp15k/qrnn_nl4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books-laser-en1/de.dev.csv
Running tokenization lm...
Data lm, trn: 152523, val: 16947
Running tokenization cls...
Data cls, trn: 1800, val: 200
Running tokenization tst...
Data tst, trn: 200, val: 2000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
!! WARNING !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Your compiler (c++) may be ABI-incompatible with PyTorch!
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
for instructions on how to install GCC 4.9 or higher.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! WARNING !!
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books-laser-en1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books-laser-en1/models/sp15k/qrnn_nl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.541837 0.487596 0.870000
2 0.498016 0.490300 0.885000
3 0.442417 0.479205 0.875000
4 0.395640 0.528897 0.855000
5 0.369408 0.521830 0.855000
6 0.361129 0.481892 0.880000
7 0.351095 0.481634 0.885000
8 0.343147 0.481654 0.880000
Total time: 02:35
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books-laser-en1/models/sp15k/qrnn_nl4.m
Loss and accuracy using (cls_best): [0.29498395, tensor(0.8960)]
Processing data/cls/en-books/models/sp15k/qrnn_nl4.m
en-books-laser-en1
Processing data/cls/fr-books/models/sp15k/qrnn_nl4.m
fr-books-laser-en1
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books-laser-en1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books-laser-en1/models/sp15k/qrnn_nl4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books-laser-en1/fr.dev.csv
Running tokenization lm...
Data lm, trn: 33183, val: 3687
Running tokenization cls...
Data cls, trn: 1800, val: 200
Running tokenization tst...
Data tst, trn: 200, val: 2000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books-laser-en1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books-laser-en1/models/sp15k/qrnn_nl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.551931 0.532421 0.835000
2 0.509913 0.524893 0.880000
3 0.433385 0.502657 0.860000
4 0.397314 0.487201 0.880000
5 0.365447 0.467523 0.885000
6 0.356587 0.520736 0.855000
7 0.353801 0.487093 0.875000
8 0.343812 0.484453 0.880000
Total time: 01:45
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books-laser-en1/models/sp15k/qrnn_nl4.m
Loss and accuracy using (cls_best): [0.32666296, tensor(0.8785)]
Processing data/cls/ja-books/models/sp15k/qrnn_nl4.m
ja-books-laser-en1
OrderedDict([('data/cls/de-books-laser-en1/models/sp15k/qrnn_nl4.m',
0.8960000276565552),
('data/cls/fr-books-laser-en1/models/sp15k/qrnn_nl4.m',
0.8784999847412109)])
data/cls/de-books-laser-en1/models/sp15k/qrnn_nl4.m: 0.8960000276565552
data/cls/fr-books-laser-en1/models/sp15k/qrnn_nl4.m: 0.8784999847412109
```
results/logs/de.md
+66
View File
@@ -22,6 +22,72 @@ Total time: 19:18:33
data/wiki/de-100/models/sp15k
Saving info data/wiki/de-100/models/sp15k/lstm_nl4.m/info.jso
```
### MLDoc
```bash
LANG=de
python -m ulmfit cls --dataset-path data/mldoc-m/${LANG}-1 --base-lm-path data/wiki-m/${LANG}-100/models/sp15k/lstm_nl4.m --lang=${LANG} --name 'nl4' - train 20 --bs 20 --num-cls-
epochs=8 --lr_sched=1cycle
Max vocab: 15000
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc-m/de-1/models/sp15k
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc-m/de-1/models/sp15k/lstm_nl4.m
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc-m/de-1/de.dev.csv
Running tokenization lm...
Data lm, trn: 13500, val: 1500
Running tokenization cls...
Data cls, trn: 1000, val: 1000
Running tokenization tst...
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki-m/de-100/models/sp15k/lstm_nl4.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki-m/de-100/models/sp15k/lstm_nl4.m/../itos')]
epoch train_loss valid_loss accuracy
1 2.333600 2.005051 0.596875
Total time: 07:40
epoch train_loss valid_loss accuracy
1 2.120653 1.886799 0.615784
2 1.980713 1.763139 0.636041
3 1.805195 1.655620 0.654068
4 1.729641 1.564017 0.668772
5 1.681813 1.491185 0.680613
6 1.682965 1.422562 0.692458
7 1.580731 1.357177 0.703143
8 1.506753 1.297219 0.714487
9 1.515824 1.235473 0.725413
10 1.427750 1.178680 0.737216
11 1.371839 1.118909 0.749590
12 1.342978 1.068754 0.760473
13 1.286842 1.011940 0.772384
14 1.254822 0.960727 0.784244
15 1.195136 0.919377 0.793910
16 1.118260 0.881799 0.802814
17 1.071546 0.855769 0.809040
18 1.079081 0.839280 0.812895
19 1.052724 0.831323 0.814723
20 1.024207 0.829737 0.815070
Total time: 3:08:58
/home/test/workspace/ulmfit-multilingual/data/mldoc-m/de-1/models/sp15k
Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc-m/de-1/models/sp15k/lstm_nl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.539181 0.239851 0.938000
2 0.326801 0.374512 0.917000
3 0.225103 0.330872 0.945000
4 0.121660 0.444890 0.938000
5 0.078411 0.422513 0.948000
6 0.061354 0.509489 0.949000
7 0.029890 0.438118 0.949000
8 0.014213 0.441808 0.949000
Total time: 09:00
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc-m/de-1/models/sp15k/lstm_nl4.m
Loss and accuracy using (cls_best): [0.3710725, tensor(0.9553)]
0.3710725009441376
0.9552500247955322
```
## VF60k LSTM nl 3
### LM
results/logs/label_smoothing.md
+951
View File
@@ -2,6 +2,957 @@
## QRNN 15k
## LM+CLS training
```
python -m ulmfit eval --glob="wiki/*-100/models/sp15k/qrnn_nl4.m" --name nl4-sl --dataset-template='../mldoc/${lang}-1' --num-lm-epochs=20 --num-cls-epochs=8 --bs=18 --lr_sched=1cycle --label-smoothing-eps=0.1
Processing data/wiki/de-100/models/sp15k/qrnn_nl4.m
../mldoc/de-1
Max vocab: 15000
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-sl.m
Training
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/de-1/de.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/de-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/de-100/models/sp15k/qrnn_nl4.m/../itos')]
epoch train_loss valid_loss accuracy
1 4.336932 3.600409 0.524499
Total time: 01:55
epoch train_loss valid_loss accuracy
1 3.867303 3.446689 0.547847
2 3.475966 3.257031 0.578702
3 3.233705 3.085521 0.607024
4 3.170250 2.964533 0.627028
5 3.059212 2.876878 0.641094
6 2.965926 2.797152 0.654905
7 2.974514 2.743403 0.663470
8 2.858759 2.690824 0.672891
9 2.866673 2.646101 0.680956
10 2.814579 2.610239 0.687777
11 2.806775 2.577145 0.694683
12 2.741160 2.540292 0.702336
13 2.753407 2.506782 0.709361
14 2.720171 2.480167 0.715709
15 2.631490 2.452236 0.721706
16 2.587928 2.431256 0.726922
17 2.608380 2.417473 0.729975
18 2.564811 2.407112 0.732487
19 2.599782 2.403481 0.733463
20 2.603479 2.402438 0.733552
Total time: 1:18:42
/home/test/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k
Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-sl.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.790572 0.610132 0.925000
2 0.655777 0.619519 0.945000
3 0.591051 0.654355 0.930000
4 0.541992 0.582572 0.944000
5 0.514052 0.574799 0.939000
6 0.490411 0.550166 0.949000
7 0.476344 0.553866 0.947000
8 0.469109 0.548871 0.947000
Total time: 02:43
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-sl.m
Loss and accuracy using (cls_best): [0.2010318, tensor(0.9610)]
Processing data/wiki/en-100/models/sp15k/qrnn_nl4.m
../mldoc/en-1
Max vocab: 15000
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-sl.m
Training
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/en-1/en.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁the', '▁,', 's', '▁.', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/en-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/en-100/models/sp15k/qrnn_nl4.m/../itos')]
epoch train_loss valid_loss accuracy
1 5.361436 4.703674 0.343924
Total time: 02:55
epoch train_loss valid_loss accuracy
1 4.757253 4.493189 0.372715
2 4.515738 4.291261 0.404511
3 4.301139 4.110973 0.430612
4 4.162306 3.964101 0.450053
5 4.062487 3.841125 0.466488
6 3.898028 3.740108 0.480628
7 3.876914 3.660982 0.493164
8 3.793781 3.593925 0.502977
9 3.736873 3.528259 0.513241
10 3.695738 3.477659 0.521709
11 3.668749 3.431972 0.529821
12 3.642119 3.385145 0.537860
13 3.556678 3.343567 0.545521
14 3.548823 3.305735 0.552502
15 3.520068 3.272878 0.558736
16 3.439619 3.247021 0.563504
17 3.391731 3.228240 0.567151
18 3.398134 3.217466 0.569319
19 3.402400 3.212110 0.570352
20 3.375334 3.210727 0.570589
Total time: 1:19:32
/home/test/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k
Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-sl.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.855421 0.644234 0.903000
2 0.722607 0.665451 0.954000
3 0.629362 0.590857 0.945000
4 0.555874 0.562738 0.950000
5 0.522701 0.549048 0.953000
6 0.506758 0.536445 0.961000
7 0.489250 0.527361 0.963000
8 0.482953 0.528452 0.961000
Total time: 02:54
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-sl.m
Loss and accuracy using (cls_best): [0.20434816, tensor(0.9555)]
Processing data/wiki/es-100/models/sp15k/qrnn_nl4.m
../mldoc/es-1
Max vocab: 15000
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-sl.m
Training
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁.', '▁la', 's', '▁el', '▁en', '▁y', '▁a', "▁&'"]
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/es-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/es-100/models/sp15k/qrnn_nl4.m/../itos')]
epoch train_loss valid_loss accuracy
1 3.925853 3.293504 0.535753
Total time: 01:42
epoch train_loss valid_loss accuracy
1 3.584036 3.131868 0.563887
2 3.341769 2.924815 0.605171
3 3.064540 2.760458 0.637288
4 2.979203 2.651128 0.657270
5 2.913760 2.569732 0.670629
6 2.901400 2.507033 0.682515
7 2.884516 2.454021 0.692617
8 2.759039 2.404587 0.703479
9 2.730353 2.367218 0.711349
10 2.657660 2.325339 0.720632
11 2.638513 2.292851 0.728599
12 2.629284 2.258947 0.737086
13 2.542013 2.226581 0.744815
14 2.464086 2.202000 0.750827
15 2.489060 2.177043 0.757989
16 2.446775 2.158471 0.762447
17 2.388175 2.144888 0.766083
18 2.415777 2.136921 0.768290
19 2.454445 2.132960 0.769424
20 2.346935 2.132173 0.769667
Total time: 47:14
/home/test/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k
Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-sl.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.842615 0.671844 0.919000
2 0.705828 0.669006 0.948000
3 0.627629 0.572335 0.944000
4 0.576609 0.600053 0.953000
5 0.532899 0.542761 0.962000
6 0.503425 0.548742 0.961000
7 0.495654 0.535397 0.959000
8 0.487190 0.545798 0.958000
Total time: 02:16
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-sl.m
Loss and accuracy using (cls_best): [0.18526463, tensor(0.9582)]
Processing data/wiki/fr-100/models/sp15k/qrnn_nl4.m
../mldoc/fr-1
Max vocab: 15000
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-sl.m
Training
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/fr-1/fr.dev.csv
Data lm, trn: 13500, val: 1500
Running tokenization cls...
Data cls, trn: 1000, val: 1000
Running tokenization tst...
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/../itos')]
epoch train_loss valid_loss accuracy
1 4.408209 3.734384 0.451908
Total time: 02:09
epoch train_loss valid_loss accuracy
1 3.911323 3.613580 0.472285
2 3.678445 3.445187 0.503284
3 3.489969 3.296292 0.528945
4 3.381153 3.180339 0.549296
5 3.291956 3.100773 0.562257
6 3.184217 3.027092 0.575632
7 3.215341 2.965142 0.586544
8 3.119935 2.915341 0.596039
9 3.081539 2.870155 0.605426
10 3.096917 2.826453 0.614306
11 3.024909 2.786344 0.622573
12 2.940282 2.743246 0.632480
13 2.939400 2.713417 0.639012
14 2.920471 2.682074 0.646323
15 2.836955 2.652518 0.653338
16 2.873827 2.631899 0.658241
17 2.847641 2.615588 0.662295
18 2.856253 2.605571 0.664535
19 2.817845 2.601670 0.665505
20 2.827199 2.600590 0.665696
Total time: 1:13:04
/home/test/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k
Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-sl.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.830361 0.645785 0.913000
2 0.718334 0.713889 0.903000
3 0.625460 0.608466 0.936000
4 0.549874 0.573567 0.938000
5 0.513573 0.563112 0.938000
6 0.497791 0.559489 0.948000
7 0.482823 0.547815 0.944000
8 0.473484 0.543763 0.946000
Total time: 02:36
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-sl.m
Loss and accuracy using (cls_best): [0.21287616, tensor(0.9480)]
Processing data/wiki/it-100/models/sp15k/qrnn_nl4.m
../mldoc/it-1
Max vocab: 15000
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-sl.m
Training
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/it-1/it.dev.csv
Running tokenization lm...
Data lm, trn: 13500, val: 1500
Running tokenization cls...
Data cls, trn: 1000, val: 1000
Running tokenization tst...
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁di', '▁e', "▁&'", "'", '▁il', '▁la', '▁in', 'e']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/it-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/it-100/models/sp15k/qrnn_nl4.m/../itos')]
epoch train_loss valid_loss accuracy
1 4.802729 3.930606 0.435333
Total time: 01:03
epoch train_loss valid_loss accuracy
1 4.208684 3.763728 0.460475
2 3.899941 3.558932 0.495172
3 3.627373 3.365080 0.528145
4 3.479348 3.217953 0.552994
5 3.345057 3.106383 0.571176
6 3.203966 3.013706 0.587515
7 3.159807 2.928730 0.601662
8 3.123686 2.863181 0.613407
9 3.091257 2.802755 0.625887
10 2.993412 2.747775 0.636441
11 2.923112 2.694130 0.647843
12 2.910893 2.644428 0.658610
13 2.912500 2.606808 0.667447
14 2.809009 2.564682 0.676342
15 2.813561 2.530405 0.684753
16 2.768533 2.505601 0.691209
17 2.714267 2.487358 0.695374
18 2.704343 2.474279 0.698375
19 2.721650 2.469191 0.699875
20 2.692483 2.468237 0.700245
Total time: 43:31
/home/test/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k
Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-sl.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.926403 0.718438 0.853000
2 0.791397 0.850076 0.824000
3 0.710437 0.707848 0.902000
4 0.626952 0.700860 0.882000
5 0.551851 0.648725 0.900000
6 0.527778 0.632797 0.906000
7 0.502474 0.621409 0.911000
8 0.489953 0.621797 0.910000
Total time: 01:32
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-sl.m
Loss and accuracy using (cls_best): [0.3240368, tensor(0.9005)]
Processing data/wiki/ja-100/models/sp15k/qrnn_nl4.m
../mldoc/ja-1
Max vocab: 15000
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-sl.m
Training
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv
Running tokenization lm...
Data lm, trn: 13500, val: 1500
Running tokenization cls...
Data cls, trn: 1000, val: 1000
Running tokenization tst...
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp15k/qrnn_nl4.m/../itos')]
epoch train_loss valid_loss accuracy
1 4.467716 3.639215 0.510628
Total time: 02:43
epoch train_loss valid_loss accuracy
1 3.640961 3.393765 0.547888
2 3.245662 3.111493 0.597234
3 3.091152 2.893684 0.636629
4 2.874470 2.753393 0.660614
5 2.774781 2.660047 0.677299
6 2.818495 2.584401 0.690161
7 2.763403 2.525782 0.699487
8 2.689764 2.481472 0.708918
9 2.471829 2.443742 0.715523
10 2.558768 2.411052 0.722205
11 2.583986 2.380159 0.728743
12 2.416061 2.352447 0.734377
13 2.422695 2.327425 0.739690
14 2.447176 2.302086 0.745426
15 2.409782 2.280813 0.749981
16 2.431124 2.265426 0.753981
17 2.409947 2.255584 0.756481
18 2.426040 2.246758 0.758470
19 2.363041 2.244397 0.759102
20 2.397845 2.243302 0.759366
Total time: 1:20:37
/home/test/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k
Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-sl.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.962460 0.721570 0.861000
2 0.850875 0.732539 0.873000
3 0.733097 0.733598 0.880000
4 0.639531 0.743423 0.882000
5 0.570058 0.702896 0.870000
6 0.525673 0.663320 0.892000
7 0.514369 0.668241 0.887000
8 0.500725 0.663006 0.886000
Total time: 03:16
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-sl.m
Loss and accuracy using (cls_best): [0.32758784, tensor(0.8988)]
Processing data/wiki/ru-100/models/sp15k/qrnn_nl4.m
../mldoc/ru-1
Max vocab: 15000
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-sl.m
Training
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
Running tokenization lm...
Data lm, trn: 9195, val: 1021
Running tokenization cls...
Data cls, trn: 1000, val: 1000
Running tokenization tst...
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4.m/../itos')]
epoch train_loss valid_loss accuracy
1 4.599347 3.756284 0.476954
Total time: 02:08
epoch train_loss valid_loss accuracy
1 3.814071 3.501726 0.524080
2 3.543633 3.248534 0.571844
3 3.245142 3.051544 0.607123
4 3.074194 2.917155 0.630383
5 3.015727 2.814585 0.648273
6 2.936803 2.731585 0.663555
7 2.801445 2.659986 0.676864
8 2.829947 2.602137 0.688181
9 2.784838 2.547982 0.699379
10 2.705119 2.501527 0.709368
11 2.763923 2.456791 0.719173
12 2.597343 2.411362 0.730358
13 2.647648 2.374054 0.738579
14 2.527706 2.336248 0.747234
15 2.543835 2.306177 0.755421
16 2.478718 2.282218 0.761426
17 2.563173 2.261219 0.766674
18 2.480387 2.251179 0.769268
19 2.415822 2.244734 0.770911
20 2.459630 2.243680 0.771312
Total time: 1:04:34
/home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-sl.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.010382 0.760864 0.848000
2 0.885819 0.800701 0.835000
3 0.768587 0.828160 0.844000
4 0.698592 0.751787 0.857000
5 0.620826 0.767858 0.856000
6 0.563309 0.727524 0.859000
7 0.524067 0.700363 0.878000
8 0.499860 0.707688 0.871000
Total time: 03:40
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-sl.m
Loss and accuracy using (cls_best): [0.40361115, tensor(0.8717)]
Processing data/wiki/zh-100/models/sp15k/qrnn_nl4.m
../mldoc/zh-1
Max vocab: 15000
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-sl.m
Training
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv
Running tokenization lm...
Data lm, trn: 13500, val: 1500
Running tokenization cls...
Data cls, trn: 1000, val: 1000
Running tokenization tst...
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/zh-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/zh-100/models/sp15k/qrnn_nl4.m/../itos')]
epoch train_loss valid_loss accuracy
1 3.804344 3.253225 0.571249
Total time: 02:19
epoch train_loss valid_loss accuracy
1 3.309380 3.061855 0.600547
2 3.055568 2.865146 0.636541
3 2.877304 2.710331 0.663194
4 2.749444 2.604171 0.681028
5 2.715787 2.528694 0.693581
6 2.664271 2.477576 0.702474
7 2.598713 2.412279 0.715434
8 2.540510 2.367390 0.724008
9 2.499321 2.330683 0.731755
10 2.513472 2.290408 0.740227
11 2.397077 2.248312 0.749950
12 2.425433 2.212132 0.757908
13 2.364556 2.176752 0.767242
14 2.349984 2.142507 0.775855
15 2.321824 2.119729 0.781726
16 2.313458 2.095738 0.788297
17 2.239505 2.078650 0.792735
18 2.240292 2.069656 0.795083
19 2.250233 2.064083 0.796754
20 2.251804 2.063307 0.797006
Total time: 1:10:47
/home/test/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k
Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-sl.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.920672 0.685620 0.870000
2 0.773749 0.700495 0.907000
3 0.663505 0.669162 0.908000
4 0.591225 0.621341 0.915000
5 0.542783 0.622516 0.919000
6 0.517919 0.608709 0.911000
7 0.489793 0.605009 0.918000
8 0.476783 0.597071 0.916000
Total time: 02:37
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-sl.m
Loss and accuracy using (cls_best): [0.29685277, tensor(0.9190)]
OrderedDict([('data/mldoc/de-1/models/sp15k/qrnn_nl4-sl.m', 0.9610000252723694),
('data/mldoc/en-1/models/sp15k/qrnn_nl4-sl.m', 0.9555000066757202),
('data/mldoc/es-1/models/sp15k/qrnn_nl4-sl.m', 0.9582499861717224),
('data/mldoc/fr-1/models/sp15k/qrnn_nl4-sl.m', 0.9480000138282776),
('data/mldoc/it-1/models/sp15k/qrnn_nl4-sl.m', 0.9004999995231628),
('data/mldoc/ja-1/models/sp15k/qrnn_nl4-sl.m', 0.8987500071525574),
('data/mldoc/ru-1/models/sp15k/qrnn_nl4-sl.m', 0.871749997138977),
('data/mldoc/zh-1/models/sp15k/qrnn_nl4-sl.m',
0.9190000295639038)])
data/mldoc/de-1/models/sp15k/qrnn_nl4-sl.m: 0.9610000252723694
data/mldoc/en-1/models/sp15k/qrnn_nl4-sl.m: 0.9555000066757202
data/mldoc/es-1/models/sp15k/qrnn_nl4-sl.m: 0.9582499861717224
data/mldoc/fr-1/models/sp15k/qrnn_nl4-sl.m: 0.9480000138282776
data/mldoc/it-1/models/sp15k/qrnn_nl4-sl.m: 0.9004999995231628
data/mldoc/ja-1/models/sp15k/qrnn_nl4-sl.m: 0.8987500071525574
data/mldoc/ru-1/models/sp15k/qrnn_nl4-sl.m: 0.871749997138977
data/mldoc/zh-1/models/sp15k/qrnn_nl4-sl.m: 0.9190000295639038
```
## CLS training
### all
```
python -m ulmfit eval --glob="mldoc/*-1/models/sp15k/qrnn_nl4.m" --name nl4-1cyc-sl-e4 --num-cls-epochs=4 --bs=18 --lr_sched=1cycle --label-smoothing-eps=0.1
Processing data/mldoc/de-1/models/sp15k/qrnn_nl4.m
de-1
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/de.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
!! WARNING !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Your compiler (c++) may be ABI-incompatible with PyTorch!
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
for instructions on how to install GCC 4.9 or higher.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! WARNING !!
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.806209 0.620243 0.938000
2 0.643088 0.608909 0.944000
3 0.552762 0.577317 0.943000
4 0.506282 0.566124 0.944000
Total time: 01:09
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m
Loss and accuracy using (cls_best): [0.18408646, tensor(0.9597)]
Processing data/mldoc/en-1/models/sp15k/qrnn_nl4.m
en-1
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/en.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁the', '▁,', 's', '▁.', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.856259 0.637753 0.935000
2 0.713970 0.627611 0.918000
3 0.594603 0.550310 0.947000
4 0.526848 0.549133 0.954000
Total time: 01:15
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m
Loss and accuracy using (cls_best): [0.20320596, tensor(0.9500)]
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-1
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.804829 0.636135 0.906000
2 0.712160 0.579012 0.952000
3 0.605291 0.543731 0.965000
4 0.531676 0.546145 0.966000
Total time: 01:01
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m
Loss and accuracy using (cls_best): [0.18462537, tensor(0.9565)]
Processing data/mldoc/fr-1/models/sp15k/qrnn_nl4.m
fr-1
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/fr.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.835704 0.657239 0.902000
2 0.688311 0.679450 0.924000
3 0.575970 0.579612 0.938000
4 0.515154 0.565664 0.939000
Total time: 01:11
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m
Loss and accuracy using (cls_best): [0.20844615, tensor(0.9435)]
Processing data/mldoc/it-1/models/sp15k/qrnn_nl4.m
it-1
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/it.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁di', '▁e', "▁&'", "'", '▁il', '▁la', '▁in', 'e']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.914871 0.758207 0.848000
2 0.799002 0.692626 0.877000
3 0.658110 0.646415 0.888000
4 0.567358 0.629301 0.912000
Total time: 00:42
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m
Loss and accuracy using (cls_best): [0.30882642, tensor(0.9032)]
Processing data/mldoc/ja-1/models/sp15k/qrnn_nl4.m
ja-1
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.959891 0.771793 0.834000
2 0.813159 0.684481 0.889000
3 0.675869 0.698423 0.878000
4 0.580597 0.689197 0.881000
Total time: 01:24
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m
Loss and accuracy using (cls_best): [0.3306819, tensor(0.8967)]
Processing data/mldoc/zh-1/models/sp15k/qrnn_nl4.m
zh-1
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有']
Loss and accuracy using (cls_best): [0.28541276, tensor(0.9237)]
OrderedDict([('data/mldoc/de-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m',
0.9597499966621399),
('data/mldoc/en-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m',
0.949999988079071),
('data/mldoc/es-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m',
0.9564999938011169),
('data/mldoc/fr-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m',
0.9434999823570251),
('data/mldoc/it-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m',
0.903249979019165),
('data/mldoc/ja-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m',
0.8967499732971191),
('data/mldoc/zh-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m',
0.9237499833106995)])
data/mldoc/de-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m: 0.9597499966621399
data/mldoc/en-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m: 0.949999988079071
data/mldoc/es-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m: 0.9564999938011169
data/mldoc/fr-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m: 0.9434999823570251
data/mldoc/it-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m: 0.903249979019165
data/mldoc/ja-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m: 0.8967499732971191
data/mldoc/zh-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m: 0.9237499833106995
```
### ZH
Exec 1
```
python -m ulmfit eval --glob="mldoc/zh-1/models/sp15k/qrnn_nl4.m" --name nl4-1cyc-sl --num-cls-epochs=8 --bs=18 --lr_sched=1cycle --label-smoothing-eps=0.1
results/logs/noise/de10k-noise.md → results/logs/mldoc/noise/de10k-noise.md
View File
results/logs/mldoc/noise/es10k-noise-lstm.md
+370
View File
@@ -0,0 +1,370 @@
```
python -m ulmfit eval_noise_resistance --lang=es --size=10 --prefix-name="val_" --model="sp30k/lstm_nl4.m"
Noise: 0
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
es-10
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_0.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
Loss and accuracy using (cls_best): [0.32779965, tensor(0.9515)]
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_0.m',
0.9514999985694885)])
Noise: 5
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
es-10
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_5.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
Loss and accuracy using (cls_best): [0.33051395, tensor(0.9488)]
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_5.m',
0.9487500190734863)])
Noise: 10
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
es-10
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_10.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
Loss and accuracy using (cls_best): [0.22158922, tensor(0.9433)]
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_10.m',
0.9432500004768372)])
Noise: 15
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
es-10
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_15.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
Loss and accuracy using (cls_best): [0.25426567, tensor(0.9358)]
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_15.m',
0.9357500076293945)])
Noise: 20
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
es-10
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_20.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
Loss and accuracy using (cls_best): [0.32246214, tensor(0.9210)]
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_20.m',
0.9210000038146973)])
Noise: 25
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
es-10
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_25.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
Loss and accuracy using (cls_best): [0.823559, tensor(0.9095)]
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_25.m',
0.909500002861023)])
Noise: 30
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
es-10
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_30.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
Loss and accuracy using (cls_best): [0.5010365, tensor(0.8942)]
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_30.m',
0.8942499756813049)])
Noise: 35
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
es-10
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_35.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
Loss and accuracy using (cls_best): [0.95638776, tensor(0.5853)]
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_35.m',
0.5852500200271606)])
Noise: 40
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
es-10
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_40.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
Loss and accuracy using (cls_best): [1.1012905, tensor(0.5642)]
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_40.m',
0.5642499923706055)])
Noise: 45
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
es-10
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_45.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
Loss and accuracy using (cls_best): [1.6009017, tensor(0.3072)]
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_45.m',
0.3072499930858612)])
Noise: 50
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
es-10
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_50.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
Loss and accuracy using (cls_best): [1.5735056, tensor(0.3072)]
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_50.m',
0.3072499930858612)])
Noise: 55
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
es-10
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_55.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 5201 examples, only 0.4500951575385917 have correct labels
Added noise to 550 examples, only 0.45 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.55tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
Single training schedule
epoch train_loss valid_loss accuracy
1 1.150436 1.391014 0.321000
2 1.190502 5.388964 0.313000
3 1.216090 1.697217 0.221000
4 1.221863 1.676644 0.221000
5 1.213776 1.734900 0.221000
6 1.195663 1.713853 0.221000
7 1.211159 1.710040 0.221000
8 1.197578 1.674693 0.221000
Total time: 29:10
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_55.m
Loss and accuracy using (cls_best): [1.5282942, tensor(0.3072)]
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_55.m',
0.3072499930858612)])
Noise: 60
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
es-10
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_60.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 5674 examples, only 0.4000845844787482 have correct labels
Added noise to 600 examples, only 0.4 have correct labels
Data lm, trn: 13013, val: 1445
Running tokenization clsnoise0.6tv...
Data clsnoise0.6tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_60.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.176071 1.396755 0.321000
2 1.211001 1.619019 0.289000
3 1.229442 1.733743 0.261000
4 1.190156 1.545205 0.312000
5 1.182274 1.369377 0.308000
6 1.169403 1.352204 0.304000
7 1.166997 1.332295 0.316000
8 1.165893 1.370641 0.313000
Total time: 30:23
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_60.m
Loss and accuracy using (cls_best): [1.2368572, tensor(0.6102)]
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_60.m',
0.6102499961853027)])
Noise: 65
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
es-10
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_65.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 6147 examples, only 0.35007401141890465 have correct labels
Added noise to 650 examples, only 0.35 have correct labels
Data lm, trn: 13013, val: 1445
Running tokenization clsnoise0.65tv...
Data clsnoise0.65tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_65.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.179257 1.460323 0.295000
2 1.220349 1.516707 0.222000
3 1.211396 1.870125 0.242000
4 1.187261 1.922184 0.308000
5 1.201833 1.429372 0.300000
6 1.187137 1.580070 0.264000
7 1.162549 1.845004 0.294000
8 1.162919 1.514930 0.313000
Total time: 29:23
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_65.m
Loss and accuracy using (cls_best): [1.1729655, tensor(0.6385)]
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_65.m',
0.6384999752044678)])
Noise: 70
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
es-10
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_70.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 6620 examples, only 0.30006343835906113 have correct labels
Added noise to 700 examples, only 0.3 have correct labels
Data lm, trn: 13013, val: 1445
Running tokenization clsnoise0.7tv...
Data clsnoise0.7tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_70.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.155135 1.479137 0.312000
2 1.190364 1.649113 0.288000
3 1.220965 3.919039 0.280000
4 1.222588 1.696949 0.258000
5 1.220919 1.669896 0.264000
6 1.217906 2.003806 0.257000
7 1.216235 1.654473 0.258000
8 1.217084 1.675933 0.258000
Total time: 29:04
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_70.m
Loss and accuracy using (cls_best): [1.5526773, tensor(0.1828)]
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_70.m',
0.18275000154972076)])
Noise: 75
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
es-10
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_75.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 7093 examples, only 0.2500528652992176 have correct labels
Added noise to 750 examples, only 0.25 have correct labels
Data lm, trn: 13013, val: 1445
Running tokenization clsnoise0.75tv...
Data clsnoise0.75tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_75.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.170507 1.421675 0.344000
2 1.221764 1.700004 0.246000
3 1.215101 2.358311 0.263000
4 1.243265 1.551931 0.257000
5 1.222902 1.756996 0.271000
6 1.215993 1.677014 0.266000
7 1.225945 4.560951 0.263000
8 1.219151 2.939914 0.245000
Total time: 29:52
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_75.m
Loss and accuracy using (cls_best): [1.7072973, tensor(0.2465)]
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_75.m',
0.24650000035762787)])
noise accuracy
0 0.00 0.95150
1 0.05 0.94875
2 0.10 0.94325
3 0.15 0.93575
4 0.20 0.92100
5 0.25 0.90950
6 0.30 0.89425
7 0.35 0.58525
8 0.40 0.56425
9 0.45 0.30725
10 0.50 0.30725
11 0.55 0.30725
12 0.60 0.61025
13 0.65 0.63850
14 0.70 0.18275
15 0.75 0.24650
```
results/logs/noise/es10k-noise.md → results/logs/mldoc/noise/es10k-noise.md
+978
View File
@@ -1,3 +1,981 @@
# Label Smoothing
## Epochs 4
```
python -m ulmfit eval_noise_resistance --lang=es --size=10 --prefix-name="sl-e4" --model="sp15k/qrnn_nl4.m" --num-cls-epochs=4 --label-smoothing-eps=0.1
python -m ulmfit eval_noise_resistance --lang=es --size=10 --prefix-name="sl-e4" --model="sp15k/qrnn_nl4.m" --num-cls-epochs=4 --label-smoothing-eps=0.1
Noise: 0
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e40.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
!! WARNING !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Your compiler (c++) may be ABI-incompatible with PyTorch!
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
for instructions on how to install GCC 4.9 or higher.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! WARNING !!
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
Loss and accuracy using (cls_best): [0.2204297, tensor(0.9553)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e40.m',
0.9552500247955322)])
Noise: 5
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e45.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Loss and accuracy using (cls_best): [4.971248, tensor(0.8798)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e45.m',
0.8797500133514404)])
Noise: 10
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e410.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Loss and accuracy using (cls_best): [0.5768092, tensor(0.9420)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e410.m',
0.9419999718666077)])
Noise: 15
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e415.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Loss and accuracy using (cls_best): [1.2631954, tensor(0.9197)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e415.m',
0.9197499752044678)])
Noise: 20
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e420.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Loss and accuracy using (cls_best): [0.42572692, tensor(0.9237)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e420.m',
0.9237499833106995)])
Noise: 25
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e425.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Loss and accuracy using (cls_best): [29.74483, tensor(0.8648)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e425.m',
0.8647500276565552)])
Noise: 30
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e430.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Loss and accuracy using (cls_best): [0.51494944, tensor(0.9185)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e430.m',
0.9185000061988831)])
Noise: 35
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e435.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Loss and accuracy using (cls_best): [0.66567194, tensor(0.8848)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e435.m',
0.8847500085830688)])
Noise: 40
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e440.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Loss and accuracy using (cls_best): [2.3167746, tensor(0.8217)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e440.m',
0.8217499852180481)])
Noise: 45
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e445.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Loss and accuracy using (cls_best): [1.6398115, tensor(0.8192)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e445.m',
0.8192499876022339)])
Noise: 50
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e450.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 4729 examples, only 0.5 have correct labels
Added noise to 500 examples, only 0.5 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.5tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Single training schedule
epoch train_loss valid_loss accuracy
1 1.222034 1.372962 0.382000
2 1.222928 9.350571 0.374000
3 1.200542 1.636571 0.413000
4 1.175799 7.368647 0.417000
Total time: 05:31
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e450.m
Loss and accuracy using (cls_best): [5.1662917, tensor(0.7197)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e450.m',
0.7197499871253967)])
Noise: 55
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e455.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 5201 examples, only 0.4500951575385917 have correct labels
Added noise to 550 examples, only 0.45 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.55tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e455.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.241146 1.574603 0.338000
2 1.220505 3.343982 0.314000
3 1.215406 4.805650 0.381000
4 1.192340 3.459251 0.368000
Total time: 05:16
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e455.m
Loss and accuracy using (cls_best): [4.879584, tensor(0.6900)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e455.m',
0.6899999976158142)])
Noise: 60
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e460.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 5674 examples, only 0.4000845844787482 have correct labels
Added noise to 600 examples, only 0.4 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.6tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e460.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.258633 1.374315 0.319000
2 1.249057 1.935162 0.279000
3 1.235318 8.574917 0.325000
4 1.224898 31.664907 0.356000
Total time: 05:25
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e460.m
Loss and accuracy using (cls_best): [33.578053, tensor(0.5670)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e460.m',
0.5669999718666077)])
Noise: 65
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e465.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 6147 examples, only 0.35007401141890465 have correct labels
Added noise to 650 examples, only 0.35 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.65tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e465.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.259673 1.630593 0.307000
2 1.256428 1.900148 0.237000
3 1.241903 3.709883 0.307000
4 1.218096 1.747542 0.292000
Total time: 05:31
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e465.m
Loss and accuracy using (cls_best): [1.3016428, tensor(0.5238)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e465.m',
0.5237500071525574)])
Noise: 70
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e470.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 6620 examples, only 0.30006343835906113 have correct labels
Added noise to 700 examples, only 0.3 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.7tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e470.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.261316 1.444706 0.331000
2 1.251065 6.526892 0.285000
3 1.232021 19.925491 0.335000
4 1.207373 1.669503 0.329000
Total time: 05:17
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e470.m
Loss and accuracy using (cls_best): [1.6000191, tensor(0.1885)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e470.m',
0.18850000202655792)])
Noise: 75
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e475.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 7093 examples, only 0.2500528652992176 have correct labels
Added noise to 750 examples, only 0.25 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.75tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e475.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.256266 1.862535 0.346000
2 1.245836 3.535186 0.307000
3 1.222711 5.987287 0.335000
4 1.206070 4.709743 0.322000
Total time: 06:12
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e475.m
Loss and accuracy using (cls_best): [1.847491, tensor(0.1700)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e475.m',
0.17000000178813934)])
noise accuracy
0 0.00 0.95525
1 0.05 0.87975
2 0.10 0.94200
3 0.15 0.91975
4 0.20 0.92375
5 0.25 0.86475
6 0.30 0.91850
7 0.35 0.88475
8 0.40 0.82175
9 0.45 0.81925
10 0.50 0.71975
11 0.55 0.69000
12 0.60 0.56700
13 0.65 0.52375
14 0.70 0.18850
```
## Epochs 8
```
python -m ulmfit eval_noise_resistance --lang=es --size=10 --prefix-name="sl-e8" --model="sp15k/qrnn_nl4.m" --num-cls-epochs=8 --label-smoothing-eps=0.1
Noise: 0
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e80.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
!! WARNING !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Your compiler (c++) may be ABI-incompatible with PyTorch!
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
for instructions on how to install GCC 4.9 or higher.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! WARNING !!
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e80.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.520412 0.696753 0.934000
2 0.509825 0.735629 0.934000
3 0.486594 0.684108 0.931000
4 0.488381 0.579037 0.953000
5 0.471751 0.588278 0.945000
6 0.459802 0.567890 0.943000
7 0.455027 0.545390 0.954000
8 0.462613 0.575517 0.943000
Total time: 11:12
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e80.m
Loss and accuracy using (cls_best): [0.27576917, tensor(0.9417)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e80.m',
0.9417499899864197)])
Noise: 5
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e85.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 472 examples, only 0.9500951575385916 have correct labels
Added noise to 50 examples, only 0.95 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.05tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e85.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.666560 0.852668 0.874000
2 0.657355 5.546601 0.824000
3 0.656841 7.308374 0.853000
4 0.612917 24.591734 0.790000
5 0.596588 12.358717 0.834000
6 0.568417 4.996921 0.913000
7 0.557468 1.554828 0.831000
8 0.536724 1.115134 0.858000
Total time: 10:51
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e85.m
Loss and accuracy using (cls_best): [0.6382615, tensor(0.9078)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e85.m',
0.9077500104904175)])
Noise: 10
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e810.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 945 examples, only 0.9000845844787482 have correct labels
Added noise to 100 examples, only 0.9 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.1tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e810.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.771364 0.982115 0.801000
2 0.744242 0.947076 0.840000
3 0.736343 1.086157 0.842000
4 0.730165 0.987014 0.827000
5 0.708371 5.287072 0.763000
6 0.697188 0.855899 0.833000
7 0.640460 0.931902 0.835000
8 0.593337 0.915233 0.837000
Total time: 10:48
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e810.m
Loss and accuracy using (cls_best): [0.45135674, tensor(0.9170)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e810.m',
0.9169999957084656)])
Noise: 15
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e815.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 1418 examples, only 0.8500740114189046 have correct labels
Added noise to 150 examples, only 0.85 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.15tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e815.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.848943 1.361678 0.751000
2 0.857650 1.120813 0.662000
3 0.861447 1.036685 0.809000
4 0.824053 1.000713 0.786000
5 0.821309 1.111803 0.799000
6 0.754514 1.129427 0.782000
7 0.695773 1.394005 0.765000
8 0.638984 1.081989 0.780000
Total time: 11:00
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e815.m
Loss and accuracy using (cls_best): [0.36475056, tensor(0.9005)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e815.m',
0.9004999995231628)])
Noise: 20
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e820.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 1891 examples, only 0.8000634383590611 have correct labels
Added noise to 200 examples, only 0.8 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.2tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e820.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.931807 1.271276 0.709000
2 0.923600 1.298985 0.741000
3 0.925085 5.663558 0.658000
4 0.929284 1.188406 0.739000
5 0.888162 3.878520 0.658000
6 0.819503 1.259068 0.731000
7 0.752936 1.150221 0.710000
8 0.735175 1.149028 0.722000
Total time: 11:07
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e820.m
Loss and accuracy using (cls_best): [0.4438091, tensor(0.8813)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e820.m',
0.8812500238418579)])
Noise: 25
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e825.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 2364 examples, only 0.7500528652992176 have correct labels
Added noise to 250 examples, only 0.75 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.25tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e825.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.999227 1.194783 0.646000
2 0.990589 2.633189 0.626000
3 1.018390 1.542320 0.659000
4 0.987891 7.653442 0.605000
5 0.972299 2.651095 0.646000
6 0.922192 9.360953 0.637000
7 0.878368 2.497859 0.640000
8 0.847166 1.993811 0.639000
Total time: 10:48
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e825.m
Loss and accuracy using (cls_best): [0.88353807, tensor(0.8367)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e825.m',
0.8367499709129333)])
Noise: 30
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e830.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 2837 examples, only 0.7000422922393741 have correct labels
Added noise to 300 examples, only 0.7 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.3tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e830.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.052917 1.325655 0.581000
2 1.071630 1.998020 0.526000
3 1.063711 31.339291 0.510000
4 1.029802 2.206242 0.607000
5 1.027847 1.763125 0.607000
6 0.961119 1.473439 0.657000
7 0.908768 1.629924 0.617000
8 0.873362 1.462983 0.626000
Total time: 10:44
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e830.m
Loss and accuracy using (cls_best): [0.61362606, tensor(0.8410)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e830.m',
0.8410000205039978)])
Noise: 35
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e835.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 3310 examples, only 0.6500317191795305 have correct labels
Added noise to 350 examples, only 0.65 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.35tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e835.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.124931 1.301690 0.549000
2 1.098526 1.527998 0.599000
3 1.109146 3.372168 0.557000
4 1.085633 9.049232 0.536000
5 1.040149 2.878901 0.552000
6 0.999970 1.699484 0.548000
7 0.924742 2.458920 0.519000
8 0.916262 5.709455 0.517000
Total time: 10:39
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e835.m
Loss and accuracy using (cls_best): [7.591171, tensor(0.6967)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e835.m',
0.6967499852180481)])
Noise: 40
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e840.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 3783 examples, only 0.600021146119687 have correct labels
Added noise to 400 examples, only 0.6 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.4tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e840.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.165653 1.350154 0.508000
2 1.172212 1.732053 0.421000
3 1.159003 9.825891 0.476000
4 1.147642 2.909990 0.485000
5 1.085988 4.217392 0.523000
6 1.073084 3.288731 0.488000
7 0.998839 1.664031 0.482000
8 0.932477 1.921165 0.471000
Total time: 11:06
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e840.m
Loss and accuracy using (cls_best): [0.8643208, tensor(0.7275)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e840.m',
0.7275000214576721)])
Noise: 45
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e845.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 4256 examples, only 0.5500105730598435 have correct labels
Added noise to 450 examples, only 0.55 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.45tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e845.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.195873 1.296093 0.433000
2 1.181843 13.203069 0.375000
3 1.202427 3.895014 0.350000
4 1.180791 2.918823 0.442000
5 1.168890 4.746016 0.477000
6 1.138908 5.763408 0.452000
7 1.085301 1.734959 0.487000
8 1.019007 1.669769 0.467000
Total time: 10:40
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e845.m
Loss and accuracy using (cls_best): [0.99686193, tensor(0.7107)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e845.m',
0.7107499837875366)])
Noise: 50
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e850.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 4729 examples, only 0.5 have correct labels
Added noise to 500 examples, only 0.5 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.5tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e850.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.206193 1.506727 0.400000
2 1.219487 1.916529 0.383000
3 1.201234 12.136375 0.383000
4 1.193026 2.111413 0.403000
5 1.184301 2.633834 0.438000
6 1.145973 2.558009 0.437000
7 1.099467 2.177719 0.410000
8 1.061634 2.724488 0.401000
Total time: 11:06
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e850.m
Loss and accuracy using (cls_best): [1.8331982, tensor(0.5920)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e850.m',
0.5920000076293945)])
Noise: 55
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e855.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 5201 examples, only 0.4500951575385917 have correct labels
Added noise to 550 examples, only 0.45 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.55tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e855.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.231457 1.337840 0.356000
2 1.245993 12.007490 0.306000
3 1.229736 1.784564 0.331000
4 1.227334 4.005848 0.339000
5 1.207836 12.369584 0.363000
6 1.186945 15.869857 0.365000
7 1.143967 22.024086 0.386000
8 1.097784 4.130466 0.361000
Total time: 10:52
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e855.m
Loss and accuracy using (cls_best): [1.1213393, tensor(0.6237)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e855.m',
0.6237499713897705)])
Noise: 60
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e860.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 5674 examples, only 0.4000845844787482 have correct labels
Added noise to 600 examples, only 0.4 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.6tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e860.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.253129 1.360811 0.344000
2 1.259076 1.422704 0.292000
3 1.249924 2.302250 0.258000
4 1.238938 7.596085 0.318000
5 1.226801 19.490450 0.361000
6 1.220376 45.920719 0.358000
7 1.186902 92.042252 0.361000
8 1.169160 47.323799 0.352000
Total time: 10:51
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e860.m
Loss and accuracy using (cls_best): [54.127697, tensor(0.5253)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e860.m',
0.5252500176429749)])
Noise: 65
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e865.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 6147 examples, only 0.35007401141890465 have correct labels
Added noise to 650 examples, only 0.35 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.65tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e865.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.258576 1.400468 0.285000
2 1.252457 1.485225 0.303000
3 1.264578 15.317787 0.257000
4 1.245918 8.976856 0.300000
5 1.239147 5.338088 0.296000
6 1.226425 8.540084 0.314000
7 1.206139 8.959650 0.300000
8 1.185957 7.868751 0.322000
Total time: 10:47
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e865.m
Loss and accuracy using (cls_best): [3.7213144, tensor(0.5070)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e865.m',
0.5070000290870667)])
Noise: 70
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e870.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 6620 examples, only 0.30006343835906113 have correct labels
Added noise to 700 examples, only 0.3 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.7tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e870.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.257724 1.391102 0.311000
2 1.259833 1.376604 0.346000
3 1.256289 4.146193 0.272000
4 1.248032 21.711473 0.308000
5 1.237731 104.512573 0.287000
6 1.226114 14.212803 0.318000
7 1.190652 3.960902 0.333000
8 1.186640 2.436945 0.339000
Total time: 10:38
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e870.m
Loss and accuracy using (cls_best): [2.232332, tensor(0.2713)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e870.m',
0.27125000953674316)])
Noise: 75
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-10
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e875.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
Added noise to 7093 examples, only 0.2500528652992176 have correct labels
Added noise to 750 examples, only 0.25 have correct labels
Data lm, trn: 13013, val: 1445
Data clsnoise0.75tv, trn: 9458, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e875.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.246185 1.331556 0.343000
2 1.247202 1.593753 0.334000
3 1.248334 4.676108 0.320000
4 1.235685 27.420618 0.289000
5 1.221268 10.798445 0.330000
6 1.201879 4.895516 0.335000
7 1.164162 3.015471 0.353000
8 1.146750 2.903884 0.353000
Total time: 11:04
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e875.m
Loss and accuracy using (cls_best): [3.2696714, tensor(0.1305)]
OrderedDict([('data/mldoc/es-10/models/sp15k/qrnn_nl4_sl-e875.m',
0.13050000369548798)])
noise accuracy
0 0.00 0.94175
1 0.05 0.90775
2 0.10 0.91700
3 0.15 0.90050
4 0.20 0.88125
5 0.25 0.83675
6 0.30 0.84100
7 0.35 0.69675
8 0.40 0.72750
9 0.45 0.71075
10 0.50 0.59200
11 0.55 0.62375
12 0.60 0.52525
13 0.65 0.50700
14 0.70 0.27125
```
# Correct VAL
```
python -m ulmfit eval_noise_resistance --lang=es --size=10 --prefix-name="val_"
results/logs/qrnn-ru.md
+176 -8
View File
@@ -1,15 +1,183 @@
# QRNN RU
## SP15k nl8
data/wiki/ru-100/models/sp15k/qrnn_nl8.m
## SP15k nl4
## LM
export CUDA_VISIBLE_DEVICES=3
LANG=ru
python -m ulmfit lm --dataset-path data/wiki/ru-100 --bidir=False --qrnn=True --nl 4 --tokenizer='sp' --max-vocab 15000 --lang ru --name 'nl4' - train 10 --drop-mult=0 --bs=50 --label-smoothing-eps=0.1
## SP15k nl8
### LM
```
5 2.869308 2.905951 0.466976
6 2.768955 2.782804 0.481852
7 2.654484 2.676304 0.495593
8 2.585963 2.591748 0.508447
9 2.512042 2.526819 0.518860
10 2.520543 2.509287 0.521890
Total time: 18:46:01
data/wiki/ru-100/models/sp15k
Saving info data/wiki/ru-100/models/sp15k/qrnn_nl8.m/info.json
```
### MLDoc
```
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_nl8.m --lang=${LANG} --name 'nl8' - train 20 --bs 20 --num-cls-epochs=8 --lr_sched=1cycle --label-smoothing-eps=0.1
Max vocab: 15000
Cache dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
Model dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl8.m
Loading validation /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
Running tokenization lm...
Data lm, trn: 9195, val: 1021
Running tokenization cls...
Data cls, trn: 1000, val: 1000
Running tokenization tst...
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
Training lm from: [PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl8.m/lm_best'), PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl8.m/../itos')]
epoch train_loss valid_loss accuracy
1 2.923423 2.334532 0.529978
Total time: 02:46
epoch train_loss valid_loss accuracy
1 2.462077 2.150593 0.563281
2 2.230013 1.972095 0.596198
3 2.118523 1.812012 0.623204
4 1.916368 1.690016 0.644060
5 1.842718 1.585770 0.661704
6 1.748630 1.513972 0.674130
7 1.675032 1.447667 0.686207
8 1.628485 1.393949 0.695972
9 1.564814 1.330838 0.707272
10 1.553933 1.283114 0.715716
11 1.441891 1.234810 0.726201
12 1.496388 1.185676 0.735977
13 1.383019 1.141014 0.745528
14 1.256620 1.094201 0.755120
15 1.306187 1.052457 0.764280
16 1.297933 1.028387 0.769747
17 1.319773 1.004256 0.775285
18 1.178073 0.989788 0.778480
19 1.252248 0.982740 0.780057
20 1.177640 0.981201 0.780267
Total time: 1:24:58
/home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
Saving info /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl8.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.882775 0.510930 0.826000
2 0.683476 0.513669 0.847000
3 0.556661 0.590375 0.839000
4 0.454019 0.757216 0.828000
5 0.344460 0.549675 0.870000
6 0.246039 0.630242 0.861000
7 0.173423 0.649066 0.858000
8 0.098640 0.638015 0.867000
Total time: 05:11
Saving models at /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl8.m
Loss and accuracy using (cls_best): [0.64393336, tensor(0.8683)]
```
### MLDoc nl8 -2nd
```
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME}-2 - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0
.1
Max vocab: 15000
Cache dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
Model dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl8-2.m
Loading validation /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
Data lm, trn: 9195, val: 1021
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
Training lm from: [PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl8.m/lm_best'), PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl8.m/../itos')]
epoch train_loss valid_loss accuracy
1 3.966292 3.450758 0.527065
Total time: 02:58
epoch train_loss valid_loss accuracy
1 3.495761 3.276329 0.560047
2 3.319947 3.102911 0.593742
3 3.137904 2.955317 0.620171
4 3.040286 2.839161 0.642270
5 2.869962 2.753622 0.658331
6 2.905739 2.680881 0.672860
7 2.836454 2.620925 0.685026
8 2.857271 2.569716 0.695722
9 2.702872 2.520050 0.705589
10 2.701559 2.473591 0.715346
11 2.740815 2.429558 0.725597
12 2.646513 2.389550 0.735010
13 2.587685 2.349614 0.744885
14 2.546527 2.311087 0.754463
15 2.568136 2.278581 0.762980
16 2.492115 2.252367 0.769275
17 2.338561 2.230529 0.775072
18 2.447506 2.218215 0.778437
19 2.364424 2.212115 0.780085
20 2.367132 2.210520 0.780424
Total time: 1:30:47
/home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
Saving info /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl8-2.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.969255 0.769736 0.843000
2 0.846340 0.813483 0.839000
3 0.718175 0.705339 0.867000
4 0.609513 0.726442 0.875000
Total time: 02:54
Saving models at /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl8-2.m
Loss and accuracy using (cls_best): [0.4056449, tensor(0.8698)]
0.40564489364624023
```
## cls
```
export CUDA_VISIBLE_DEVICES=0
LANG=ru
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_nl8.m --lang=${LANG} --name 'nl8' - train 20 --bs 20 --num-cls-epochs=8 --lr_sched=1cycle
export CUDA_VISIBLE_DEVICES=0
LANG=de
python -m ulmfit cls --dataset-path data/mldoc-m/${LANG}-1 --base-lm-path data/wiki-m/${LANG}-100/models/sp15k/lstm_nl4.m --lang=${LANG} --name 'nl4' - train 20 --bs 20 --num-cls-epochs=8 --lr_sched=1cycle
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_nl8.m --lang=${LANG} --name 'nl8' - train 20 --bs 20 --num-cls-epochs=8 --lr_sched=1cycle --label-smoothing-eps=0.1
```
## SP30k nl4
### LM
results/logs/ru.md
+3
View File
@@ -17,7 +17,10 @@ Total time: 10:43:03
data/wiki/ru-100/models/sp15k
Saving info data/wiki/ru-100/models/sp15k/qrnn_nl4.m/info.json
```
```bash
python -m ulmfit cls --dataset-path data/mldoc/ru-1 --base-lm-path data/wiki/ru-100/models/sp30k/lstm_nl4.m --lang=ru --name 'nl4-100' --cuda-id=1 - train 20 --bs 40 --num-cls-epochs=2
```
## SP30k nl4
### LM
results/logs/ru/bs_vs_lr.md
+123
View File
@@ -0,0 +1,123 @@
## BS=18, lr_mult=1.0
epoch train_loss valid_loss accuracy
1 4.427713 3.693394 0.484268
Total time: 01:32
epoch train_loss valid_loss accuracy
1 3.758918 3.446661 0.529820
2 3.394254 3.199054 0.577411
3 3.235364 3.014517 0.610520
4 3.125459 2.871101 0.637153
5 2.994313 2.773862 0.654470
6 2.915075 2.693080 0.669942
7 2.855732 2.622858 0.683629
8 2.755074 2.572147 0.694145
9 2.697898 2.517524 0.704816
10 2.689881 2.468190 0.715927
11 2.579573 2.432807 0.723324
12 2.659464 2.387878 0.733931
13 2.520637 2.344804 0.744233
14 2.482952 2.315014 0.751855
15 2.564730 2.279045 0.761163
16 2.552707 2.255916 0.766971
17 2.511244 2.240169 0.770991
18 2.461429 2.228213 0.774309
19 2.426440 2.222140 0.775745
20 2.425955 2.221128 0.775836
Total time: 1:14:17
## BS=500, lr_mult=1.0
epoch train_loss valid_loss accuracy
1 5.536769 3.850831 0.444662
Total time: 01:10
epoch train_loss valid_loss accuracy
1 4.845898 3.781763 0.461471
2 4.388605 3.643141 0.491225
3 4.038255 3.464554 0.526143
## BS=500, lr_mult=27
epoch train_loss valid_loss accuracy
1 7.234749 5.868155 0.312675
Total time: 01:41
## BS=500, lr_mult=10 + BS=50 lr_mult=10 for cls
/data/wiki/ru-100/models/sp15k/qrnn_nl4sl.m/../itos')]
epoch train_loss valid_loss accuracy
1 5.052441 4.082105 0.439539
Total time: 02:27
epoch train_loss valid_loss accuracy
1 4.120197 3.712686 0.498216
2 3.727043 3.373258 0.557896
3 3.383009 3.109635 0.598970
4 3.180799 2.938478 0.626816
5 3.048913 2.812639 0.647257
6 2.943903 2.727179 0.661784
7 2.864300 2.650275 0.674248
8 2.773810 2.583594 0.687063
9 2.724850 2.529445 0.697573
10 2.673996 2.473824 0.708698
11 2.657637 2.431461 0.716904
12 2.591277 2.372668 0.730318
13 2.537707 2.323294 0.741157
14 2.486507 2.280270 0.751768
15 2.435933 2.238660 0.762545
16 2.401303 2.208848 0.769561
17 2.374117 2.184253 0.776400
18 2.341421 2.169156 0.780388
19 2.328202 2.163922 0.781700
20 2.315462 2.161784 0.782105
Total time: 1:07:09
------------------- Checking the influence of number of epochs on the accuracy
(multifit) test@test:~/workspace/ulmfit-multilingual$ rm /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4sl-bs500.m/cls*
(multifit) test@test:~/workspace/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME}-bs500 - train 0 --bs 50 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1 --lr_mult=1
Max vocab: 15000
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4sl-bs500.m
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
Data lm, trn: 9195, val: 1021
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
Single training schedule
epoch train_loss valid_loss accuracy
1 1.063845 1.111960 0.601000
2 0.902245 0.766871 0.817000
3 0.766261 0.707502 0.861000
4 0.680053 0.694492 0.866000
Total time: 01:22
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4sl-bs500.m
Loss and accuracy using (cls_best): [0.41532615, tensor(0.8630)]
0.41532614827156067
0.8629999756813049
(multifit) test@test:~/workspace/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME}-bs500 - train 0 --bs 50 --num-cls-epochs=16 --lr_sched=1cycle --label-smoothing-eps=0.1 --lr_mult=1
Max vocab: 15000
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4sl-bs500.m
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
Data lm, trn: 9195, val: 1021
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
Loading last classifier
Single training schedule
epoch train_loss valid_loss accuracy
1 0.556688 0.706000 0.873000
2 0.537578 0.717411 0.865000
3 0.532326 0.775549 0.854000
4 0.529178 0.767506 0.861000
5 0.521306 0.797604 0.860000
6 0.527344 0.736225 0.868000
7 0.516393 0.724941 0.878000
8 0.510422 0.716110 0.873000
9 0.504320 0.701886 0.869000
10 0.500323 0.676577 0.878000
11 0.493490 0.682657 0.873000
12 0.484450 0.682047 0.878000
13 0.479248 0.682782 0.880000
14 0.474778 0.688019 0.873000
15 0.472664 0.685304 0.874000
16 0.470747 0.677925 0.878000
Total time: 07:57
results/logs/ru/lm-opti.md
+260
View File
@@ -0,0 +1,260 @@
CUDA_VISIBLE_DEVICES=0 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 5 --name 'nl5-merity' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 2500 - train 14 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1
CUDA_VISIBLE_DEVICES=1 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-wide2' --max-vocab 15000 --lang ${LANG} --qrnn=True --nh 3100 - train 10 --bs=100 --drop_mult=0 --label-smoothing-eps=0.1
CUDA_VISIBLE_DEVICES=2 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-merity' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 2500 - train 14 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1
CUDA_VISIBLE_DEVICES=3 python -m ulmfit lm --dataset-path data/wiki/ru-100 --bidir=False --qrnn=True --nl 4 --tokenizer=sp --max-vocab 15000 --lang ru --name nl4sl - train 10 --drop-mult=0 --bs=50 --label-smoothing-eps=0.1
## 25vocab
CUDA_VISIBLE_DEVICES=3 python -m ulmfit lm --dataset-path data/wiki/ru-100 --bidir=False --qrnn=True --nl 4 --tokenizer=sp --max-vocab 25000 --lang ru --name nl4 - train 10 --drop-mult=0 --bs=50 --label-smoothing-eps=0.1
LANG=ru
CUDA_VISIBLE_DEVICES=0 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-merity-wide2' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 3100 - train 10 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1
##### CLS
export CUDA_VISIBLE_DEVICES=0
LANG=ru
NAME=nl5-merity
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
export CUDA_VISIBLE_DEVICES=1
LANG=ru
NAME=nl4-wide2
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
export CUDA_VISIBLE_DEVICES=2
LANG=ru
NAME=nl4-merity
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
export CUDA_VISIBLE_DEVICES=3
LANG=ru
NAME=nl4sl
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
-----------------------CLS1
export CUDA_VISIBLE_DEVICES=0
LANG=ru
NAME=nl4
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
export CUDA_VISIBLE_DEVICES=0
LANG=ru
NAME=nl8
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
python -m ulmfit cls --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 5 --name 'nl5-merity' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 2500 - train 14 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1
CUDA_VISIBLE_DEVICES=1 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-wide2' --max-vocab 15000 --lang ${LANG} --qrnn=True --nh 3100 - train 10 --bs=100 --drop_mult=0 --label-smoothing-eps=0.1
CUDA_VISIBLE_DEVICES=2 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-merity' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 2500 - train 14 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1
CUDA_VISIBLE_DEVICES=3 python -m ulmfit lm --dataset-path data/wiki/ru-100 --bidir=False --qrnn=True --nl 4 --tokenizer=sp --max-vocab 15000 --lang ru --name nl4sl - train 10 --drop-mult=0 --bs=50 --label-smoothing-eps=0.1
##
------------------------
7 3.680504 3.678406 0.498396
8 3.556062 3.596037 0.512345
9 3.553716 3.535783 0.523509
10 3.523366 3.515352 0.527935
Total time: 20:03:59
data/wiki/ru-100/models/sp15k
Saving info data/wiki/ru-100/models/sp15k/qrnn_ nl4sl.m/info.json
### Ru
```
export CUDA_VISIBLE_DEVICES=3
LANG=ru
NAME=nl4sl
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
Max vocab: 15000
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4sl.m
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
Data lm, trn: 9195, val: 1021
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4sl.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4sl.m/../itos')]
epoch train_loss valid_loss accuracy
1 4.427713 3.693394 0.484268
Total time: 01:32
epoch train_loss valid_loss accuracy
1 3.758918 3.446661 0.529820
2 3.394254 3.199054 0.577411
3 3.235364 3.014517 0.610520
4 3.125459 2.871101 0.637153
5 2.994313 2.773862 0.654470
6 2.915075 2.693080 0.669942
7 2.855732 2.622858 0.683629
8 2.755074 2.572147 0.694145
9 2.697898 2.517524 0.704816
10 2.689881 2.468190 0.715927
11 2.579573 2.432807 0.723324
12 2.659464 2.387878 0.733931
13 2.520637 2.344804 0.744233
14 2.482952 2.315014 0.751855
15 2.564730 2.279045 0.761163
16 2.552707 2.255916 0.766971
17 2.511244 2.240169 0.770991
18 2.461429 2.228213 0.774309
19 2.426440 2.222140 0.775745
20 2.425955 2.221128 0.775836
Total time: 1:14:17
/home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4sl.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.022382 0.779370 0.822000
2 0.866379 0.792353 0.832000
3 0.715650 0.698579 0.865000
4 0.603621 0.693501 0.884000
Total time: 02:05
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4sl.m
Loss and accuracy using (cls_best): [0.3978519, tensor(0.8723)]
0.3978519141674042
0.8722500205039978
```
----
```bash
$ export CUDA_VISIBLE_DEVICES=0
$ LANG=ru
$ NAME=nl4
$ python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
Max vocab: 15000
Cache dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
Model dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4.m
Loading validation /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
Data lm, trn: 9195, val: 1021
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
Training lm from: [PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4.m/../itos')]
epoch train_loss valid_loss accuracy
1 4.531968 3.764185 0.474252
Total time: 01:44
epoch train_loss valid_loss accuracy
1 3.770046 3.506013 0.522443
2 3.546580 3.251341 0.571620
3 3.320569 3.055680 0.606364
4 3.130226 2.912925 0.631395
5 3.072772 2.809725 0.649728
6 2.765424 2.731825 0.662963
7 2.959237 2.662104 0.676203
8 2.807999 2.600417 0.688423
9 2.771271 2.548279 0.699473
10 2.809488 2.501688 0.709020
11 2.707221 2.454946 0.719196
12 2.597226 2.417315 0.728432
13 2.609972 2.376176 0.737923
14 2.590427 2.341666 0.746216
15 2.572995 2.306599 0.754747
16 2.496636 2.285632 0.760806
17 2.508584 2.266456 0.765147
18 2.441373 2.253839 0.768449
19 2.430915 2.249204 0.769536
20 2.426130 2.247966 0.769886
Total time: 47:33
/home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
Saving info /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.060299 0.890710 0.716000
2 0.884965 0.769866 0.853000
3 0.722994 0.723213 0.875000
4 0.609488 0.730594 0.865000
Total time: 01:14
Saving models at /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4.m
Loss and accuracy using (cls_best): [0.39589784, tensor(0.8692)]
0.39589783549308777
0.8692499995231628
```
## wide 2
```bash
$ CUDA_VISIBLE_DEVICES=1 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-wide2' --max-vocab 15000 --lang ${LANG} --qrnn=True --nh 3100 - train 10 --bs=100 --drop_mult=0 --label-smoothing-eps=0.1
Max vocab: 15000
Cache dir: data/wiki/ru-100/models/sp15k
Model dir: data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m
Wiki text was split to 193047 articles
Wiki text was split to 460 articles
Data lm, trn: 193047, val: 460
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Training lm from random weights
epoch train_loss valid_loss accuracy
1 3.908233 3.950865 0.463469
2 3.738863 3.815026 0.477703
3 3.696502 3.779513 0.483625
4 3.692592 3.720908 0.490143
5 3.600519 3.652444 0.501671
6 3.564568 3.582584 0.511550
7 3.472859 3.493226 0.525943
8 3.390483 3.407970 0.541749
9 3.351620 3.344207 0.552758
10 3.329683 3.330087 0.556380
Total time: 51:05:43
data/wiki/ru-100/models/sp15k
Saving info data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/info.json
```
### MLDoc
export CUDA_VISIBLE_DEVICES=1
LANG=ru
NAME=nl4-wide2
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
## Merity nl4
```bash
CUDA_VISIBLE_DEVICES=2 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-merity' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 2500 - train 14 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1
Max vocab: 15000
Cache dir: data/wiki/ru-100/models/sp15k
Model dir: data/wiki/ru-100/models/sp15k/qrnn_nl4-merity.m
Wiki text was split to 193047 articles
Wiki text was split to 460 articles
Data lm, trn: 193047, val: 460
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Training lm from random weights
epoch train_loss valid_loss accuracy
1 3.965899 3.977734 0.460046
2 3.806082 3.858176 0.472396
3 3.839230 3.874757 0.469224
4 3.762105 3.868653 0.469943
5 3.800827 3.833991 0.474116
6 3.755466 3.796329 0.479868
7 3.691958 3.747888 0.487367
8 3.660529 3.702986 0.493545
9 3.593282 3.635035 0.504086
10 3.585948 3.579200 0.513631
11 3.473865 3.512114 0.525391
12 3.451973 3.455807 0.535520
13 3.418731 3.417129 0.542943
14 3.385637 3.407541 0.545545
Total time: 51:32:09
data/wiki/ru-100/models/sp15k
Saving info data/wiki/ru-100/models/sp15k/qrnn_nl4-merity.m/info.json
```
#### MLDoc
export CUDA_VISIBLE_DEVICES=2
LANG=ru
NAME=nl4-merity
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
results/logs/ru/merity4.md
+101
View File
@@ -0,0 +1,101 @@
## LM
### MLDoc 1
```
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
Max vocab: 15000
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-merity.m
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
Data lm, trn: 9195, val: 1021
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4-merity.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4-merity.m/../itos')]
epoch train_loss valid_loss accuracy
1 4.411345 3.660489 0.486965
Total time: 02:43
epoch train_loss valid_loss accuracy
1 3.613334 3.372079 0.544188
2 3.325245 3.100234 0.596406
3 3.181919 2.906442 0.631586
4 3.010830 2.767429 0.656378
5 2.880418 2.663865 0.676339
6 2.825526 2.571074 0.694140
7 2.766901 2.483362 0.711652
8 2.601965 2.417213 0.726853
9 2.569160 2.341699 0.744193
10 2.588142 2.272457 0.760294
11 2.494011 2.198197 0.779175
12 2.421921 2.135517 0.795854
13 2.396429 2.075012 0.812815
14 2.306572 2.019140 0.828851
15 2.281730 1.966554 0.843595
16 2.206670 1.927567 0.854515
17 2.143836 1.901352 0.862114
18 2.141715 1.884954 0.867003
19 2.070353 1.876935 0.869214
20 2.066195 1.874844 0.869665
Total time: 2:12:21
/home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-merity.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.994393 0.755689 0.844000
2 0.859871 0.822650 0.856000
3 0.678185 0.721333 0.859000
4 0.586906 0.693618 0.878000
Total time: 04:17
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-merity.m
Loss and accuracy using (cls_best): [0.3872361, tensor(0.8777)]
0.387236088514328
0.8777499794960022
```
### MLDoc 2
```
(multifit) test@test:~/workspace/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/mldoc/${LANG}-1/models/sp15k/qrnn_${NAME}.
m --lang=${LANG} --name ${NAME}-16 - train 0 --bs 18 --num-cls-epochs=16 --lr_sched=1cycle --label-smoothing-eps=0.1
Max vocab: 15000
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-merity-16.m
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
Data lm, trn: 9195, val: 1021
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-merity-16.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.081481 0.837775 0.781000
2 0.901621 0.798574 0.858000
3 0.778870 0.826576 0.859000
4 0.693465 0.787875 0.833000
5 0.639763 0.841092 0.861000
6 0.595044 0.731504 0.853000
7 0.576115 0.796013 0.819000
8 0.544098 0.744034 0.875000
9 0.531359 0.699035 0.879000
10 0.513886 0.698310 0.879000
11 0.495473 0.686897 0.864000
12 0.489863 0.688584 0.881000
13 0.481086 0.675660 0.881000
14 0.479960 0.684917 0.883000
15 0.490157 0.687865 0.882000
16 0.486081 0.679104 0.882000
Total time: 15:26
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-merity-16.m
Loss and accuracy using (cls_best): [0.4047818, tensor(0.8737)]
0.4047817885875702
0.8737499713897705
```
results/logs/ru/merity5.md
+38
View File
@@ -0,0 +1,38 @@
## LM
```bash
CUDA_VISIBLE_DEVICES=0 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 5 --name 'nl5-mer
ity' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 2500 - train 14 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1
Max vocab: 15000
Cache dir: data/wiki/ru-100/models/sp15k
Model dir: data/wiki/ru-100/models/sp15k/qrnn_nl5-merity.m
Wiki text was split to 193047 articles
Wiki text was split to 460 articles
Data lm, trn: 193047, val: 460
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Training lm from random weights
epoch train_loss valid_loss accuracy
1 3.969639 4.024787 0.452887
2 3.814622 3.834142 0.476612
3 3.798372 3.846118 0.473666
4 3.742609 3.835311 0.474612
5 3.715114 3.790690 0.480469
6 3.652987 3.748408 0.486146
7 3.573350 3.697325 0.493774
8 3.589853 3.637134 0.504189
9 3.558110 3.583030 0.512137
10 3.501382 3.510491 0.524148
11 3.408982 3.437177 0.536634
12 3.402717 3.373548 0.548113
13 3.293624 3.331311 0.556288
14 3.309859 3.322777 0.558426
Total time: 68:05:15
data/wiki/ru-100/models/sp15k
Saving info data/wiki/ru-100/models/sp15k/qrnn_nl5-merity.m/info.json
```
### MLDoc 1
```
```
results/logs/ru/wide2.md
+183
View File
@@ -0,0 +1,183 @@
3 2.812496 2.877055 0.468569
4 2.705551 2.792535 0.479420
5 2.649598 2.726415 0.487439
6 2.599835 2.635610 0.499679
7 2.574639 2.554657 0.512358
8 2.489573 2.475936 0.523280
9 2.396540 2.415555 0.534089
10 2.374290 2.401968 0.536601
Total time: 15:49:20
data/wiki/ru-100/models/sp15k
Saving info data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/info.json
Fire trace:
1. Initial component
2. Accessed property "lm" (/home/test/workspace/ulmfit-multilingual/ulmfit/__main__.py:32)
3. Called routine "LMHyperParams" (/home/test/workspace/ulmfit-multilingual/ulmfit/__main__.py:32)
4. Accessed property "train" (/home/test/workspace/ulmfit-multilingual/ulmfit/pretrain_lm.py:174)
5. Called routine "train_lm" (/home/test/workspace/ulmfit-multilingual/ulmfit/pretrain_lm.py:174)
6. ('Could not consume arg:', '--nh')
Type: NoneType
String form: None
Usage: __main__.py lm --dataset-path data/wiki/ru-100 --tokenizer=sp --nl 4 --name nl4-wide2 --max-vocab 15000 --lang ru --qrnn=True - train 10 --bs=100 --drop_mult=0 -
(multifit) test@test:~/workspace/ulmfit-multilingual$ less data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/info.json
(multifit) test@test:~/workspace/ulmfit-multilingual$ CUDA_VISIBLE_DEVICES=1 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-wide2' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=100 --drop_mult=0 ^C100 --
(multifit) test@test:~/workspace/ulmfit-multilingual$ mv data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/ data/wiki/ru-100/models/sp15k/qrnn_nl4-2.m/
(multifit) test@test:~/workspace/ulmfit-multilingual$ less data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/info.json^C
(multifit) test@test:~/workspace/ulmfit-multilingual$ CUDA_VISIBLE_DEVICES=1 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-wid
e2' --max-vocab 15000 --lang ${LANG} --qrnn=True --nh 3100 - train 10 --bs=100 --drop_mult=0
Max vocab: 15000
Cache dir: data/wiki/ru-100/models/sp15k
Model dir: data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m
^CTraceback (most recent call last):
File "/home/test/anaconda3/envs/multifit/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/home/test/anaconda3/envs/multifit/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/test/workspace/ulmfit-multilingual/ulmfit/__main__.py", line 119, in <module>
fire.Fire(ULMFiT())
File "/home/test/anaconda3/envs/multifit/lib/python3.7/site-packages/fire/core.py", line 127, in Fire
component_trace = _Fire(component, args, context, name)
File "/home/test/anaconda3/envs/multifit/lib/python3.7/site-packages/fire/core.py", line 366, in _Fire
component, remaining_args)
File "/home/test/anaconda3/envs/multifit/lib/python3.7/site-packages/fire/core.py", line 542, in _CallCallable
result = fn(*varargs, **kwargs)
File "/home/test/workspace/ulmfit-multilingual/ulmfit/pretrain_lm.py", line 176, in train_lm
data_lm = self.load_wiki_data(bs=bs) if data_lm is None else data_lm
File "/home/test/workspace/ulmfit-multilingual/ulmfit/pretrain_lm.py", line 253, in load_wiki_data
train_df=read_wiki_articles(trn_path),
File "/home/test/workspace/ulmfit-multilingual/ulmfit/pretrain_lm.py", line 48, in read_wiki_articles
if i < len(lines)-2 and lines[i+1].strip() == "" and istitle(lines[i+2]):
File "/home/test/workspace/ulmfit-multilingual/ulmfit/pretrain_lm.py", line 39, in istitle
return len(re.findall(r'^ ?= [^=]* = ?$', line)) != 0
File "/home/test/anaconda3/envs/multifit/lib/python3.7/re.py", line 223, in findall
return _compile(pattern, flags).findall(string)
KeyboardInterrupt
^C
(multifit) test@test:~/workspace/ulmfit-multilingual$ CUDA_VISIBLE_DEVICES=1 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-wide2' --max-vocab 15000 --lang ${LANG} --qrnn=True --nh 3100 - train 10 --bs=100 --drop_mult=0 --label-smoothing-eps=0.1
Max vocab: 15000
Cache dir: data/wiki/ru-100/models/sp15k
Model dir: data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m
Wiki text was split to 193047 articles
Wiki text was split to 460 articles
Data lm, trn: 193047, val: 460
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Training lm from random weights
epoch train_loss valid_loss accuracy
1 3.908233 3.950865 0.463469
2 3.738863 3.815026 0.477703
3 3.696502 3.779513 0.483625
4 3.692592 3.720908 0.490143
5 3.600519 3.652444 0.501671
6 3.564568 3.582584 0.511550
7 3.472859 3.493226 0.525943
8 3.390483 3.407970 0.541749
9 3.351620 3.344207 0.552758
10 3.329683 3.330087 0.556380
Total time: 51:05:43
data/wiki/ru-100/models/sp15k
Saving info data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/info.json
(multifit) test@test:~/workspace/ulmfit-multilingual$ export CUDA_VISIBLE_DEVICES=1
(multifit) test@test:~/workspace/ulmfit-multilingual$ LANG=ru
(multifit) test@test:~/workspace/ulmfit-multilingual$ NAME=nl4-wide2
(multifit) test@test:~/workspace/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
Max vocab: 15000
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-wide2.m
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
Data lm, trn: 9195, val: 1021
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/../itos')]
epoch train_loss valid_loss accuracy
1 3.777423 3.222261 0.564503
Total time: 04:35
epoch train_loss valid_loss accuracy
1 3.292465 3.029143 0.602257
2 3.034045 2.858176 0.634576
3 2.943366 2.710314 0.665116
4 2.722069 2.596702 0.687515
5 2.819853 2.508158 0.705020
6 2.734984 2.417240 0.724748
7 2.674353 2.332395 0.743694
8 2.527344 2.251373 0.762892
9 2.473972 2.168185 0.784043
10 2.359504 2.093983 0.803255
11 2.287590 2.019540 0.823566
12 2.254421 1.943832 0.845138
13 2.203321 1.884380 0.863381
14 2.142532 1.824186 0.881509
15 2.121573 1.777664 0.894901
16 2.013238 1.740772 0.905824
17 2.026189 1.715271 0.913569
18 1.904322 1.700163 0.917917
19 1.889113 1.692539 0.919811
20 1.903118 1.691033 0.920319
Total time: 3:10:09
/home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-wide2.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.006922 0.880313 0.788000
2 0.823572 0.782953 0.860000
3 0.679078 0.749164 0.872000
4 0.579215 0.707200 0.872000
Total time: 06:30
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-wide2.m
Loss and accuracy using (cls_best): [0.3935929, tensor(0.8708)]
0.393592894077301
0.8707500100135803
(multifit) test@test:~/workspace/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/mldoc/${LANG}-1/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME}-16 - train 0 --bs 18 --num-cls-epochs=16 --lr_sched=1cycle --label-smoothing-eps=0.1
Max vocab: 15000
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-wide2-16.m
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
Data lm, trn: 9195, val: 1021
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-wide2-16.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.067446 0.822965 0.824000
2 0.897088 0.845636 0.826000
3 0.778055 0.828693 0.847000
4 0.685080 0.893327 0.823000
5 0.620457 0.929057 0.800000
6 0.587644 0.802154 0.859000
7 0.570255 0.713434 0.872000
8 0.543071 0.705259 0.871000
9 0.517465 0.715090 0.867000
10 0.498291 0.695459 0.876000
11 0.497857 0.698052 0.862000
12 0.486924 0.681911 0.878000
13 0.479041 0.676714 0.874000
14 0.475131 0.677843 0.878000
15 0.467238 0.672065 0.876000
16 0.476889 0.680850 0.875000
Total time: 23:47
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-wide2-16.m
Loss and accuracy using (cls_best): [0.41155785, tensor(0.8700)]
0.4115578532218933
0.8700000047683716
results/logs/zeroshot.md
+397 -1
View File
@@ -90,9 +90,405 @@ zh from fr
| Test: 79.40% | classes: 33.60 31.12 9.07 26.20
```
#### ULMFit zershot on laser-en1k 4 epochs
```bash
python -m ulmfit eval --glob="mldoc/*-1/models/sp15k/qrnn_nl4.m" --dataset_template="{lang}-1*-laser-en1" --name nl4 --num-cls-epochs=4 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=18
python -m ulmfit eval --glob="mldoc/*-1/models/sp15k/qrnn_nl4.m" --dataset_template='${lang}-1*-laser-en1' --name nl4 --num-cls-epochs=4 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=18
Processing data/mldoc/de-1/models/sp15k/qrnn_nl4.m
de-1*-laser-en1
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1-laser-en1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1-laser-en1/models/sp15k/qrnn_nl4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1-laser-en1/de.dev.csv
Running tokenization lm...
Data lm, trn: 13500, val: 1500
Running tokenization cls...
Data cls, trn: 1000, val: 1000
Running tokenization tst...
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
!! WARNING !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Your compiler (c++) may be ABI-incompatible with PyTorch!
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
for instructions on how to install GCC 4.9 or higher.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! WARNING !!
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1-laser-en1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1-laser-en1/models/sp15k/qrnn_nl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.858069 0.756738 0.853000
2 0.737577 0.687798 0.909000
3 0.611922 0.615140 0.919000
4 0.544274 0.608824 0.909000
Total time: 01:08
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1-laser-en1/models/sp15k/qrnn_nl4.m
Loss and accuracy using (cls_best): [0.31768194, tensor(0.9133)]
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10-laser-en1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10-laser-en1/models/sp15k/qrnn_nl4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10-laser-en1/de.dev.csv
Running tokenization lm...
Data lm, trn: 13500, val: 1500
Running tokenization cls...
Data cls, trn: 10000, val: 1000
Running tokenization tst...
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10-laser-en1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10-laser-en1/models/sp15k/qrnn_nl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.665453 0.600660 0.924000
2 0.624768 0.595788 0.922000
3 0.576251 0.580166 0.930000
4 0.520507 0.569867 0.930000
Total time: 09:38
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10-laser-en1/models/sp15k/qrnn_nl4.m
Loss and accuracy using (cls_best): [0.3278069, tensor(0.9190)]
Processing data/mldoc/en-1/models/sp15k/qrnn_nl4.m
en-1*-laser-en1
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1-laser-en1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1-laser-en1/models/sp15k/qrnn_nl4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1-laser-en1/en.dev.csv
Running tokenization lm...
Data lm, trn: 13500, val: 1500
Running tokenization cls...
Data cls, trn: 1000, val: 1000
Running tokenization tst...
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁the', '▁,', 's', '▁.', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1-laser-en1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1-laser-en1/models/sp15k/qrnn_nl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.836131 0.760151 0.887000
2 0.691355 0.666257 0.905000
3 0.587016 0.603976 0.932000
4 0.529849 0.584768 0.943000
Total time: 01:09
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1-laser-en1/models/sp15k/qrnn_nl4.m
Loss and accuracy using (cls_best): [0.20341124, tensor(0.9503)]
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-10-laser-en1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-10-laser-en1/models/sp15k/qrnn_nl4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-10-laser-en1/en.dev.csv
Running tokenization lm...
Data lm, trn: 13500, val: 1500
Running tokenization cls...
Data cls, trn: 10000, val: 1000
Running tokenization tst...
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁the', '▁,', 's', '▁.', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-10-laser-en1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-10-laser-en1/models/sp15k/qrnn_nl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.676697 0.611570 0.931000
2 0.647868 0.629156 0.916000
3 0.578947 0.546653 0.943000
4 0.539242 0.550864 0.949000
Total time: 10:25
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-10-laser-en1/models/sp15k/qrnn_nl4.m
Loss and accuracy using (cls_best): [0.2546631, tensor(0.9490)]
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
es-1*-laser-en1
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1-laser-en1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1-laser-en1/models/sp15k/qrnn_nl4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1-laser-en1/es.dev.csv
Running tokenization lm...
Data lm, trn: 13013, val: 1445
Running tokenization cls...
Data cls, trn: 1000, val: 1000
Running tokenization tst...
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1-laser-en1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1-laser-en1/models/sp15k/qrnn_nl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.947175 0.774158 0.854000
2 0.827687 0.747528 0.859000
3 0.698278 0.726713 0.881000
4 0.603821 0.729449 0.878000
Total time: 00:58
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1-laser-en1/models/sp15k/qrnn_nl4.m
Loss and accuracy using (cls_best): [0.77420205, tensor(0.7893)]
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10-laser-en1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10-laser-en1/models/sp15k/qrnn_nl4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10-laser-en1/es.dev.csv
Running tokenization lm...
Data lm, trn: 13013, val: 1445
Running tokenization cls...
Data cls, trn: 9458, val: 1000
Running tokenization tst...
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10-laser-en1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10-laser-en1/models/sp15k/qrnn_nl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.654948 0.968341 0.863000
2 0.634143 0.844979 0.873000
3 0.559031 0.719894 0.890000
4 0.524448 0.675569 0.887000
Total time: 05:36
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10-laser-en1/models/sp15k/qrnn_nl4.m
Loss and accuracy using (cls_best): [0.70765454, tensor(0.7880)]
Processing data/mldoc/fr-1/models/sp15k/qrnn_nl4.m
fr-1*-laser-en1
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1-laser-en1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1-laser-en1/models/sp15k/qrnn_nl4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1-laser-en1/fr.dev.csv
Running tokenization lm...
Data lm, trn: 13500, val: 1500
Running tokenization cls...
Data cls, trn: 1000, val: 1000
Running tokenization tst...
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1-laser-en1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1-laser-en1/models/sp15k/qrnn_nl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.882636 0.720057 0.851000
2 0.766720 0.790358 0.847000
3 0.661309 0.669355 0.877000
4 0.584607 0.676029 0.889000
Total time: 01:05
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1-laser-en1/models/sp15k/qrnn_nl4.m
Loss and accuracy using (cls_best): [0.32164142, tensor(0.8945)]
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-10-laser-en1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-10-laser-en1/models/sp15k/qrnn_nl4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-10-laser-en1/fr.dev.csv
Running tokenization lm...
Data lm, trn: 13500, val: 1500
Running tokenization cls...
Data cls, trn: 10000, val: 1000
Running tokenization tst...
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-10-laser-en1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-10-laser-en1/models/sp15k/qrnn_nl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.721444 0.714190 0.865000
2 0.684221 0.688866 0.879000
3 0.616377 0.624161 0.904000
4 0.576405 0.630186 0.904000
Total time: 09:37
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-10-laser-en1/models/sp15k/qrnn_nl4.m
Loss and accuracy using (cls_best): [0.39518934, tensor(0.8848)]
Processing data/mldoc/it-1/models/sp15k/qrnn_nl4.m
it-1*-laser-en1
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1-laser-en1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1-laser-en1/models/sp15k/qrnn_nl4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1-laser-en1/it.dev.csv
Running tokenization lm...
Data lm, trn: 13500, val: 1500
Running tokenization cls...
Data cls, trn: 1000, val: 1000
Running tokenization tst...
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁di', '▁e', "▁&'", "'", '▁il', '▁la', '▁in', 'e']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1-laser-en1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1-laser-en1/models/sp15k/qrnn_nl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.963395 0.843734 0.777000
2 0.892776 0.890245 0.785000
3 0.755211 0.799284 0.815000
4 0.629971 0.796769 0.820000
Total time: 00:40
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1-laser-en1/models/sp15k/qrnn_nl4.m
Loss and accuracy using (cls_best): [0.76029295, tensor(0.7600)]
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-10-laser-en1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-10-laser-en1/models/sp15k/qrnn_nl4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-10-laser-en1/it.dev.csv
Running tokenization lm...
Data lm, trn: 13500, val: 1500
Running tokenization cls...
Data cls, trn: 10000, val: 1000
Running tokenization tst...
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁di', '▁e', "▁&'", "'", '▁il', '▁la', '▁in', 'e']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-10-laser-en1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-10-laser-en1/models/sp15k/qrnn_nl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.828429 0.801399 0.820000
2 0.769261 0.786845 0.816000
3 0.717665 0.734139 0.845000
4 0.597664 0.741355 0.843000
Total time: 05:29
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-10-laser-en1/models/sp15k/qrnn_nl4.m
Loss and accuracy using (cls_best): [0.7915614, tensor(0.7605)]
Processing data/mldoc/ja-1/models/sp15k/qrnn_nl4.m
ja-1*-laser-en1
Processing data/mldoc/zh-1/models/sp15k/qrnn_nl4.m
zh-1*-laser-en1
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-en1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-en1/models/sp15k/qrnn_nl4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-en1/zh.dev.csv
Running tokenization lm...
Data lm, trn: 13500, val: 1500
Running tokenization cls...
Data cls, trn: 1000, val: 1000
Running tokenization tst...
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-en1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-en1/models/sp15k/qrnn_nl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.964628 1.016147 0.710000
2 0.836267 0.885741 0.789000
3 0.694938 0.789230 0.811000
4 0.594315 0.809480 0.811000
Total time: 01:08
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-en1/models/sp15k/qrnn_nl4.m
Loss and accuracy using (cls_best): [0.5295076, tensor(0.8245)]
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-10-laser-en1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-10-laser-en1/models/sp15k/qrnn_nl4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-10-laser-en1/zh.dev.csv
Running tokenization lm...
Data lm, trn: 13500, val: 1500
Running tokenization cls...
Data cls, trn: 10000, val: 1000
Running tokenization tst...
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-10-laser-en1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-10-laser-en1/models/sp15k/qrnn_nl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.823208 0.774504 0.821000
2 0.771960 0.769799 0.822000
3 0.682731 0.724021 0.847000
4 0.595054 0.744821 0.836000
Total time: 09:42
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-10-laser-en1/models/sp15k/qrnn_nl4.m
Loss and accuracy using (cls_best): [0.59163076, tensor(0.8127)]
OrderedDict([('data/mldoc/de-1-laser-en1/models/sp15k/qrnn_nl4.m',
0.9132500290870667),
('data/mldoc/de-10-laser-en1/models/sp15k/qrnn_nl4.m',
0.9190000295639038),
('data/mldoc/en-1-laser-en1/models/sp15k/qrnn_nl4.m',
0.9502500295639038),
('data/mldoc/en-10-laser-en1/models/sp15k/qrnn_nl4.m',
0.9490000009536743),
('data/mldoc/es-1-laser-en1/models/sp15k/qrnn_nl4.m',
0.7892500162124634),
('data/mldoc/es-10-laser-en1/models/sp15k/qrnn_nl4.m',
0.7879999876022339),
('data/mldoc/fr-1-laser-en1/models/sp15k/qrnn_nl4.m',
0.8945000171661377),
('data/mldoc/fr-10-laser-en1/models/sp15k/qrnn_nl4.m',
0.8847500085830688),
('data/mldoc/it-1-laser-en1/models/sp15k/qrnn_nl4.m',
0.7599999904632568),
('data/mldoc/it-10-laser-en1/models/sp15k/qrnn_nl4.m',
0.7605000138282776),
('data/mldoc/zh-1-laser-en1/models/sp15k/qrnn_nl4.m',
0.8245000243186951),
('data/mldoc/zh-10-laser-en1/models/sp15k/qrnn_nl4.m',
0.812749981880188)])
data/mldoc/de-1-laser-en1/models/sp15k/qrnn_nl4.m: 0.9132500290870667
data/mldoc/de-10-laser-en1/models/sp15k/qrnn_nl4.m: 0.9190000295639038
data/mldoc/en-1-laser-en1/models/sp15k/qrnn_nl4.m: 0.9502500295639038
data/mldoc/en-10-laser-en1/models/sp15k/qrnn_nl4.m: 0.9490000009536743
data/mldoc/es-1-laser-en1/models/sp15k/qrnn_nl4.m: 0.7892500162124634
data/mldoc/es-10-laser-en1/models/sp15k/qrnn_nl4.m: 0.7879999876022339
data/mldoc/fr-1-laser-en1/models/sp15k/qrnn_nl4.m: 0.8945000171661377
data/mldoc/fr-10-laser-en1/models/sp15k/qrnn_nl4.m: 0.8847500085830688
data/mldoc/it-1-laser-en1/models/sp15k/qrnn_nl4.m: 0.7599999904632568
data/mldoc/it-10-laser-en1/models/sp15k/qrnn_nl4.m: 0.7605000138282776
data/mldoc/zh-1-laser-en1/models/sp15k/qrnn_nl4.m: 0.8245000243186951
```
#### Evaluation of Laser 1k Performance
```
python -m ulmfit eval --glob="mldoc/*-1/models/sp60k/lstm_nl4.m" --dataset_template="{}-laser-*" --name nl4 --cuda-id=0 ✘ 130
python -m ulmfit eval --glob="mldoc/*-1/models/sp60k/lstm_nl4.m" --dataset_template="{}-laser-*" --name nl4 --cuda-id=0
Max vocab: 60000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-fr/models/sp60k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-fr/models/sp60k/lstm_nl4.m