wassname/multifit
1
0
Fork 0
mirror of https://github.com/wassname/multifit.git synced 2026-09-09 11:27:26 +08:00
Code Issues Packages Projects Releases Wiki Activity

New results lstm 30k 1cyc

Browse Source
This commit is contained in:
Piotr Czapla
2019-02-22 12:01:25 +01:00
parent 99d6b22447
commit c69d31c420
6 changed files with 2628 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
results/MLDoc.md
+2 -2
View File
@@ -9,8 +9,8 @@
|ULMFiT L30k-100 | | 91.35 | 83.32 | 88.77 | 77.99 | 71.12 | 72.20 | |
|ULMFiT L30k | | 95.4 | 95.15 | 93.67 | 88.42 | 89.20 | **87.27** | 90.20 |
|ULMFiT sp-fixed | | **95.6** | 94.80 | 94.20 | 88.52 | 88.72 | 86.85 | 90.47 |
|ULMFIT Q15k 1cyc| **94.62** | **95.65** | 95.15 | **94.42** | **89.92** | **89.60** | | 90.78/89.82 |
|ULMFIT L30k 1cyc| | | **95.95** | | | | | **92.02** |
|ULMFIT Q15k 1cyc| **94.62** | **95.65** | 95.15 | **94.42** | **89.92** | 89.60 | | 90.78/89.82 |
|ULMFIT L30k 1cyc| | **95.85** | **96.32** | **94.82** | 89.87 | **90.45** | **87.94** | **92.02/91.64** |
- L30k - LSTM sp30k trained using gradual unfreezing
- L30k-100 - --||-- **on 100 samples**
- ULMFiT sp-fixed - --||-- with fixed tokenization
results/logs/cls.md
+49 -1
View File
@@ -121,4 +121,52 @@ Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/m
Loss and accuracy using (cls_best): [0.5418505, tensor(0.9100)]
0.5418505072593689
0.9100000262260437
````
````
```
Loading pretrained model
Unknown tokens 0, first 100: []
Training lm from: [PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/../itos')]
epoch train_loss valid_loss accuracy
1 4.037580 3.312217 0.364410
Total time: 01:44
epoch train_loss valid_loss accuracy
1 3.709982 3.256320 0.371825
2 3.459413 3.150574 0.386972
3 3.296628 3.037327 0.402039
4 3.186458 2.914899 0.418413
5 3.092632 2.817097 0.431216
6 2.966957 2.726081 0.442906
7 2.924824 2.647339 0.453871
8 2.818279 2.561596 0.466795
9 2.773893 2.501994 0.475877
10 2.736084 2.438490 0.485978
11 2.688937 2.370927 0.496899
12 2.615245 2.314875 0.506508
13 2.583292 2.260717 0.515725
14 2.535631 2.220295 0.522666
15 2.466035 2.179093 0.530148
16 2.461427 2.151952 0.535315
17 2.390641 2.131065 0.538749
18 2.376235 2.116927 0.541430
19 2.407630 2.115370 0.542039
20 2.391378 2.112687 0.542522
Total time: 46:33
/home/n-waves/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k
Saving info /home/n-waves/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k/qrnn_nl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.466671 0.534682 0.745000
2 0.358965 0.372612 0.875000
3 0.251557 0.311034 0.900000
4 0.166484 0.585425 0.865000
5 0.101803 0.726341 0.900000
6 0.072025 0.587875 0.885000
7 0.045328 0.760989 0.890000
8 0.027765 0.727203 0.890000
Total time: 01:17
Saving models at /home/n-waves/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k/qrnn_nl4.m
Loss and accuracy using (cls_best): [0.55982095, tensor(0.8970)]
0.5598209500312805
0.8970000147819519
```
results/logs/common.md
+236
View File
@@ -1,6 +1,242 @@
# MLDoc
### Different training schedules
### 1cycle -lstm
```
(fastaiv1) pczapla@galatea ~/w/ulmfit-multilingual ❯❯❯ python -m ulmfit eval --glob="mldoc/*-1/models/sp30k/lstm_nl4.m" --name nl4-1cyc --num-cls-epochs=8 --bs=18 --lr_sched=1cycle ✘ 1
Processing data/mldoc/de-1/models/sp30k/lstm_nl4.m
de-1
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4-1cyc.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/de.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', '▁und', '▁in', 'en', "▁&'", 's', '-']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4-1cyc.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.610423 0.287707 0.920000
2 0.390499 0.266688 0.948000
3 0.366716 0.302463 0.933000
4 0.248321 0.305547 0.937000
5 0.166564 0.411075 0.948000
6 0.083940 0.406182 0.950000
7 0.033326 0.388105 0.949000
8 0.014658 0.397507 0.948000
Total time: 06:42
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4-1cyc.m
Loss and accuracy using (cls_best): [0.3040595, tensor(0.9585)]
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
es-1
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4-1cyc.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4-1cyc.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.610724 0.278892 0.925000
2 0.372022 0.348428 0.937000
3 0.310411 0.386958 0.927000
4 0.215536 0.273834 0.958000
5 0.163195 0.319600 0.958000
6 0.085268 0.313287 0.961000
7 0.037369 0.347500 0.961000
8 0.016851 0.338436 0.963000
Total time: 05:36
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4-1cyc.m
Loss and accuracy using (cls_best): [0.31034237, tensor(0.9632)]
Processing data/mldoc/fr-1/models/sp30k/lstm_nl4.m
fr-1
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k/lstm_nl4-1cyc.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/fr.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁.', "'", 's', '▁la', '▁le', '▁et', '▁l', '▁à']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k/lstm_nl4-1cyc.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.642809 0.240702 0.928000
2 0.420564 0.658542 0.852000
3 0.443345 0.244053 0.927000
4 0.338779 0.335634 0.914000
5 0.224778 0.263748 0.928000
6 0.116705 0.280655 0.944000
7 0.072063 0.287557 0.945000
8 0.048084 0.289200 0.946000
Total time: 06:33
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k/lstm_nl4-1cyc.m
Loss and accuracy using (cls_best): [0.29398218, tensor(0.9482)]
Processing data/mldoc/it-1/models/sp30k/lstm_nl4.m
it-1
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k/lstm_nl4-1cyc.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/it.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁di', "▁&'", "'", '▁e', '▁il', '▁la', 'e', '▁in']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k/lstm_nl4-1cyc.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.736070 0.391008 0.859000
2 0.512531 0.614638 0.860000
3 0.343422 0.594530 0.862000
4 0.370786 0.540225 0.884000
5 0.234727 0.591903 0.892000
6 0.141539 0.589971 0.906000
7 0.073315 0.544248 0.906000
8 0.038477 0.580940 0.904000
Total time: 03:48
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k/lstm_nl4-1cyc.m
Loss and accuracy using (cls_best): [0.6642357, tensor(0.8988)]
Processing data/mldoc/ja-1/models/sp30k/lstm_nl4.m
ja-1
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4-1cyc.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁の', '▁。', '▁に', '▁を', '▁は', '▁年', '▁が', '▁)', '▁(']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4-1cyc.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.804430 0.435780 0.837000
2 0.576932 0.452903 0.839000
3 0.499791 0.640789 0.806000
4 0.418024 0.610898 0.839000
5 0.259578 0.582953 0.868000
6 0.188161 0.719131 0.888000
7 0.101508 0.766175 0.877000
8 0.072221 0.795415 0.883000
Total time: 08:19
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4-1cyc.m
Loss and accuracy using (cls_best): [0.636261, tensor(0.9045)]
Processing data/mldoc/ru-1/models/sp30k/lstm_nl4.m
ru-1
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k/lstm_nl4-1cyc.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
Data lm, trn: 9195, val: 1021
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', '▁и', 'е', 'и', 'й', '▁на', 'х']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k/lstm_nl4-1cyc.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.821141 0.532432 0.814000
2 0.611059 0.457023 0.866000
3 0.464408 0.484035 0.870000
4 0.446560 0.477454 0.858000
5 0.299095 0.906959 0.858000
6 0.176480 0.709579 0.875000
7 0.089683 0.781081 0.876000
8 0.047031 0.772935 0.877000
Total time: 08:58
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k/lstm_nl4-1cyc.m
Loss and accuracy using (cls_best): [0.8528109, tensor(0.8795)]
Processing data/mldoc/zh-1/models/sp30k/lstm_nl4.m
zh-1
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁中', '▁人', '▁是']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.696346 0.335083 0.895000
2 0.505075 0.360600 0.906000
3 0.427076 0.462661 0.883000
4 0.351177 0.489026 0.919000
5 0.244958 0.415151 0.918000
6 0.156452 0.494367 0.926000
7 0.085807 0.471611 0.927000
8 0.046887 0.484232 0.929000
Total time: 06:41
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc.m
Loss and accuracy using (cls_best): [0.50999963, tensor(0.9165)]
OrderedDict([('data/mldoc/de-1/models/sp30k/lstm_nl4-1cyc.m',
0.9585000276565552),
('data/mldoc/es-1/models/sp30k/lstm_nl4-1cyc.m',
0.9632499814033508),
('data/mldoc/fr-1/models/sp30k/lstm_nl4-1cyc.m',
0.9482499957084656),
('data/mldoc/it-1/models/sp30k/lstm_nl4-1cyc.m',
0.8987500071525574),
('data/mldoc/ja-1/models/sp30k/lstm_nl4-1cyc.m',
0.9045000076293945),
('data/mldoc/ru-1/models/sp30k/lstm_nl4-1cyc.m',
0.8794999718666077),
('data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc.m',
0.9164999723434448)])
data/mldoc/de-1/models/sp30k/lstm_nl4-1cyc.m: 0.9585000276565552
data/mldoc/es-1/models/sp30k/lstm_nl4-1cyc.m: 0.9632499814033508
data/mldoc/fr-1/models/sp30k/lstm_nl4-1cyc.m: 0.9482499957084656
data/mldoc/it-1/models/sp30k/lstm_nl4-1cyc.m: 0.8987500071525574
data/mldoc/ja-1/models/sp30k/lstm_nl4-1cyc.m: 0.9045000076293945
data/mldoc/ru-1/models/sp30k/lstm_nl4-1cyc.m: 0.8794999718666077
data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc.m: 0.9164999723434448
```
### 2cycle
```bash
results/logs/de.md
+24
View File
@@ -1,4 +1,28 @@
# DE
## SP15k LSTM nl4
```
$ python -m ulmfit lm --dataset-path data/wiki/de-100 --cuda-id=0 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang de --qrnn=False - train 10 --bs=100 --drop_mult=0
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der',
'▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Training lm from random weights
epoch train_loss valid_loss accuracy
1 2.519809 2.600072 0.529963
2 2.436580 2.538897 0.534651
3 2.402220 2.510569 0.537314
4 2.305741 2.439347 0.546574
5 2.265683 2.376482 0.553794
6 2.210663 2.305362 0.562672
7 2.134196 2.230041 0.572958
8 2.085375 2.150917 0.584621
9 2.037781 2.097170 0.593747
10 1.986773 2.081469 0.595799
Total time: 19:18:33
data/wiki/de-100/models/sp15k
Saving info data/wiki/de-100/models/sp15k/lstm_nl4.m/info.jso
```
## VF60k LSTM nl 3
### LM
```
results/logs/noise/de10k-noise.md
+1158
View File
File diff suppressed because it is too large Load Diff
results/logs/noise/es10k-noise.md
+1159
View File
File diff suppressed because it is too large Load Diff