wassname/multifit
1
0
Fork 0
mirror of https://github.com/wassname/multifit.git synced 2026-09-09 11:27:26 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #38 from n-waves/lbl_smoothing

Browse Source 
Label smoothing
This commit is contained in:
Piotr Czapla
2019-02-22 16:57:47 +01:00
committed by GitHub
parent 99d6b22447 260faa703c
commit 852336c29b
12 changed files with 2961 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
results/MLDoc.md
+4 -2
View File
@@ -9,8 +9,10 @@
|ULMFiT L30k-100 | | 91.35 | 83.32 | 88.77 | 77.99 | 71.12 | 72.20 | |
|ULMFiT L30k | | 95.4 | 95.15 | 93.67 | 88.42 | 89.20 | **87.27** | 90.20 |
|ULMFiT sp-fixed | | **95.6** | 94.80 | 94.20 | 88.52 | 88.72 | 86.85 | 90.47 |
|ULMFIT Q15k 1cyc| **94.62** | **95.65** | 95.15 | **94.42** | **89.92** | **89.60** | | 90.78/89.82 |
|ULMFIT L30k 1cyc| | | **95.95** | | | | | **92.02** |
|ULMFIT Q15k 1cyc| **94.62** | **95.65** | 95.15 | **94.42** | **89.92** | 89.60 | | 90.78/89.82 |
|ULMFIT Q15k 1c l| | | | | | | | **92.22** |
|ULMFIT L30k 1cyc| | **95.85** | **96.32** | **94.82** | 89.87 | **90.45** | **87.94** | 92.02/91.64 |
- L30k - LSTM sp30k trained using gradual unfreezing
- L30k-100 - --||-- **on 100 samples**
- ULMFiT sp-fixed - --||-- with fixed tokenization
results/logs/cls.md
+49 -1
View File
@@ -121,4 +121,52 @@ Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/m
Loss and accuracy using (cls_best): [0.5418505, tensor(0.9100)]
0.5418505072593689
0.9100000262260437
````
````
```
Loading pretrained model
Unknown tokens 0, first 100: []
Training lm from: [PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/../itos')]
epoch train_loss valid_loss accuracy
1 4.037580 3.312217 0.364410
Total time: 01:44
epoch train_loss valid_loss accuracy
1 3.709982 3.256320 0.371825
2 3.459413 3.150574 0.386972
3 3.296628 3.037327 0.402039
4 3.186458 2.914899 0.418413
5 3.092632 2.817097 0.431216
6 2.966957 2.726081 0.442906
7 2.924824 2.647339 0.453871
8 2.818279 2.561596 0.466795
9 2.773893 2.501994 0.475877
10 2.736084 2.438490 0.485978
11 2.688937 2.370927 0.496899
12 2.615245 2.314875 0.506508
13 2.583292 2.260717 0.515725
14 2.535631 2.220295 0.522666
15 2.466035 2.179093 0.530148
16 2.461427 2.151952 0.535315
17 2.390641 2.131065 0.538749
18 2.376235 2.116927 0.541430
19 2.407630 2.115370 0.542039
20 2.391378 2.112687 0.542522
Total time: 46:33
/home/n-waves/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k
Saving info /home/n-waves/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k/qrnn_nl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.466671 0.534682 0.745000
2 0.358965 0.372612 0.875000
3 0.251557 0.311034 0.900000
4 0.166484 0.585425 0.865000
5 0.101803 0.726341 0.900000
6 0.072025 0.587875 0.885000
7 0.045328 0.760989 0.890000
8 0.027765 0.727203 0.890000
Total time: 01:17
Saving models at /home/n-waves/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp15k/qrnn_nl4.m
Loss and accuracy using (cls_best): [0.55982095, tensor(0.8970)]
0.5598209500312805
0.8970000147819519
```
results/logs/common.md
+236
View File
@@ -1,6 +1,242 @@
# MLDoc
### Different training schedules
### 1cycle -lstm
```
(fastaiv1) pczapla@galatea ~/w/ulmfit-multilingual ❯❯❯ python -m ulmfit eval --glob="mldoc/*-1/models/sp30k/lstm_nl4.m" --name nl4-1cyc --num-cls-epochs=8 --bs=18 --lr_sched=1cycle ✘ 1
Processing data/mldoc/de-1/models/sp30k/lstm_nl4.m
de-1
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4-1cyc.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/de.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', '▁und', '▁in', 'en', "▁&'", 's', '-']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4-1cyc.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.610423 0.287707 0.920000
2 0.390499 0.266688 0.948000
3 0.366716 0.302463 0.933000
4 0.248321 0.305547 0.937000
5 0.166564 0.411075 0.948000
6 0.083940 0.406182 0.950000
7 0.033326 0.388105 0.949000
8 0.014658 0.397507 0.948000
Total time: 06:42
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4-1cyc.m
Loss and accuracy using (cls_best): [0.3040595, tensor(0.9585)]
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
es-1
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4-1cyc.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv
Data lm, trn: 13013, val: 1445
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4-1cyc.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.610724 0.278892 0.925000
2 0.372022 0.348428 0.937000
3 0.310411 0.386958 0.927000
4 0.215536 0.273834 0.958000
5 0.163195 0.319600 0.958000
6 0.085268 0.313287 0.961000
7 0.037369 0.347500 0.961000
8 0.016851 0.338436 0.963000
Total time: 05:36
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4-1cyc.m
Loss and accuracy using (cls_best): [0.31034237, tensor(0.9632)]
Processing data/mldoc/fr-1/models/sp30k/lstm_nl4.m
fr-1
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k/lstm_nl4-1cyc.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/fr.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁.', "'", 's', '▁la', '▁le', '▁et', '▁l', '▁à']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k/lstm_nl4-1cyc.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.642809 0.240702 0.928000
2 0.420564 0.658542 0.852000
3 0.443345 0.244053 0.927000
4 0.338779 0.335634 0.914000
5 0.224778 0.263748 0.928000
6 0.116705 0.280655 0.944000
7 0.072063 0.287557 0.945000
8 0.048084 0.289200 0.946000
Total time: 06:33
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k/lstm_nl4-1cyc.m
Loss and accuracy using (cls_best): [0.29398218, tensor(0.9482)]
Processing data/mldoc/it-1/models/sp30k/lstm_nl4.m
it-1
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k/lstm_nl4-1cyc.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/it.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁di', "▁&'", "'", '▁e', '▁il', '▁la', 'e', '▁in']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k/lstm_nl4-1cyc.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.736070 0.391008 0.859000
2 0.512531 0.614638 0.860000
3 0.343422 0.594530 0.862000
4 0.370786 0.540225 0.884000
5 0.234727 0.591903 0.892000
6 0.141539 0.589971 0.906000
7 0.073315 0.544248 0.906000
8 0.038477 0.580940 0.904000
Total time: 03:48
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k/lstm_nl4-1cyc.m
Loss and accuracy using (cls_best): [0.6642357, tensor(0.8988)]
Processing data/mldoc/ja-1/models/sp30k/lstm_nl4.m
ja-1
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4-1cyc.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁の', '▁。', '▁に', '▁を', '▁は', '▁年', '▁が', '▁)', '▁(']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4-1cyc.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.804430 0.435780 0.837000
2 0.576932 0.452903 0.839000
3 0.499791 0.640789 0.806000
4 0.418024 0.610898 0.839000
5 0.259578 0.582953 0.868000
6 0.188161 0.719131 0.888000
7 0.101508 0.766175 0.877000
8 0.072221 0.795415 0.883000
Total time: 08:19
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4-1cyc.m
Loss and accuracy using (cls_best): [0.636261, tensor(0.9045)]
Processing data/mldoc/ru-1/models/sp30k/lstm_nl4.m
ru-1
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k/lstm_nl4-1cyc.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
Data lm, trn: 9195, val: 1021
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', '▁и', 'е', 'и', 'й', '▁на', 'х']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k/lstm_nl4-1cyc.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.821141 0.532432 0.814000
2 0.611059 0.457023 0.866000
3 0.464408 0.484035 0.870000
4 0.446560 0.477454 0.858000
5 0.299095 0.906959 0.858000
6 0.176480 0.709579 0.875000
7 0.089683 0.781081 0.876000
8 0.047031 0.772935 0.877000
Total time: 08:58
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k/lstm_nl4-1cyc.m
Loss and accuracy using (cls_best): [0.8528109, tensor(0.8795)]
Processing data/mldoc/zh-1/models/sp30k/lstm_nl4.m
zh-1
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁中', '▁人', '▁是']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.696346 0.335083 0.895000
2 0.505075 0.360600 0.906000
3 0.427076 0.462661 0.883000
4 0.351177 0.489026 0.919000
5 0.244958 0.415151 0.918000
6 0.156452 0.494367 0.926000
7 0.085807 0.471611 0.927000
8 0.046887 0.484232 0.929000
Total time: 06:41
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc.m
Loss and accuracy using (cls_best): [0.50999963, tensor(0.9165)]
OrderedDict([('data/mldoc/de-1/models/sp30k/lstm_nl4-1cyc.m',
0.9585000276565552),
('data/mldoc/es-1/models/sp30k/lstm_nl4-1cyc.m',
0.9632499814033508),
('data/mldoc/fr-1/models/sp30k/lstm_nl4-1cyc.m',
0.9482499957084656),
('data/mldoc/it-1/models/sp30k/lstm_nl4-1cyc.m',
0.8987500071525574),
('data/mldoc/ja-1/models/sp30k/lstm_nl4-1cyc.m',
0.9045000076293945),
('data/mldoc/ru-1/models/sp30k/lstm_nl4-1cyc.m',
0.8794999718666077),
('data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc.m',
0.9164999723434448)])
data/mldoc/de-1/models/sp30k/lstm_nl4-1cyc.m: 0.9585000276565552
data/mldoc/es-1/models/sp30k/lstm_nl4-1cyc.m: 0.9632499814033508
data/mldoc/fr-1/models/sp30k/lstm_nl4-1cyc.m: 0.9482499957084656
data/mldoc/it-1/models/sp30k/lstm_nl4-1cyc.m: 0.8987500071525574
data/mldoc/ja-1/models/sp30k/lstm_nl4-1cyc.m: 0.9045000076293945
data/mldoc/ru-1/models/sp30k/lstm_nl4-1cyc.m: 0.8794999718666077
data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc.m: 0.9164999723434448
```
### 2cycle
```bash
results/logs/de.md
+24
View File
@@ -1,4 +1,28 @@
# DE
## SP15k LSTM nl4
```
$ python -m ulmfit lm --dataset-path data/wiki/de-100 --cuda-id=0 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang de --qrnn=False - train 10 --bs=100 --drop_mult=0
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der',
'▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Training lm from random weights
epoch train_loss valid_loss accuracy
1 2.519809 2.600072 0.529963
2 2.436580 2.538897 0.534651
3 2.402220 2.510569 0.537314
4 2.305741 2.439347 0.546574
5 2.265683 2.376482 0.553794
6 2.210663 2.305362 0.562672
7 2.134196 2.230041 0.572958
8 2.085375 2.150917 0.584621
9 2.037781 2.097170 0.593747
10 1.986773 2.081469 0.595799
Total time: 19:18:33
data/wiki/de-100/models/sp15k
Saving info data/wiki/de-100/models/sp15k/lstm_nl4.m/info.jso
```
## VF60k LSTM nl 3
### LM
```
results/logs/label_smoothing.md
+223
View File
@@ -0,0 +1,223 @@
# MLDoc
## QRNN 15k
Exec 1
```
python -m ulmfit eval --glob="mldoc/zh-1/models/sp15k/qrnn_nl4.m" --name nl4-1cyc-sl --num-cls-epochs=8 --bs=18 --lr_sched=1cycle --label-smoothing-eps=0.1
Processing data/mldoc/zh-1/models/sp15k/qrnn_nl4.m
zh-1
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-1cyc-sl.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-1cyc-sl.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.859153 0.767961 0.864000
2 0.768888 0.775161 0.904000
3 0.658956 0.685653 0.902000
4 0.589073 0.618438 0.923000
5 0.540008 0.622157 0.915000
6 0.508080 0.606979 0.914000
7 0.487228 0.599491 0.918000
8 0.477516 0.602196 0.923000
Total time: 02:18
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-1cyc-sl.m
Loss and accuracy using (cls_best): [0.2829206, tensor(0.9205)]
OrderedDict([('data/mldoc/zh-1/models/sp15k/qrnn_nl4-1cyc-sl.m',
0.9204999804496765)])
data/mldoc/zh-1/models/sp15k/qrnn_nl4-1cyc-sl.m: 0.9204999804496765
```
Exec 2
````python -m ulmfit eval --glob="mldoc/zh-1/models/sp15k/qrnn_nl4.m" --name nl4-1cyc-sl1 --num-cls-epochs=8 --bs=18 --lr_sched=1cycle --label-smoothing-eps=0.1
Processing data/mldoc/zh-1/models/sp15k/qrnn_nl4.m
zh-1
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-1cyc-sl1.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-1cyc-sl1.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.881513 0.712176 0.865000
2 0.743687 0.665091 0.906000
3 0.677436 0.687689 0.873000
4 0.595139 0.626483 0.920000
5 0.542732 0.600652 0.914000
6 0.512080 0.597546 0.916000
7 0.487021 0.597065 0.912000
8 0.476598 0.596792 0.914000
Total time: 02:20
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-1cyc-sl1.m
Loss and accuracy using (cls_best): [0.29172945, tensor(0.9178)]
OrderedDict([('data/mldoc/zh-1/models/sp15k/qrnn_nl4-1cyc-sl1.m',
0.9177500009536743)])
data/mldoc/zh-1/models/sp15k/qrnn_nl4-1cyc-sl1.m: 0.9177500009536743
````
Exec 4
```bash
python -m ulmfit eval --glob="mldoc/zh-1/models/sp15k/qrnn_nl4.m" --name nl4-1cyc-sl-e4 --num-cls-epochs=4 --bs=18 --lr_sched=1cycle --label-smoothing-eps=0.1 ✘ 130
Processing data/mldoc/zh-1/models/sp15k/qrnn_nl4.m
zh-1
Max vocab: 15000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 15000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.880415 0.677291 0.901000
2 0.729670 0.659975 0.911000
3 0.624817 0.603056 0.921000
4 0.542027 0.601961 0.921000
Total time: 01:08
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m
Loss and accuracy using (cls_best): [0.28558904, tensor(0.9222)]
OrderedDict([('data/mldoc/zh-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m',
0.922249972820282)])
data/mldoc/zh-1/models/sp15k/qrnn_nl4-1cyc-sl-e4.m: 0.922249972820282
```
## LSTM sp30k
### 0.1
```bash
python -m ulmfit eval --glob="mldoc/zh-1/models/sp30k/lstm_nl4.m" --name nl4-1cyc-sl --num-cls-epochs=8 --bs=18 --lr_sched=1cycle --label-smoothing-eps=0.1
Processing data/mldoc/zh-1/models/sp30k/lstm_nl4.m
zh-1
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc-sl.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁中', '▁人', '▁是']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc-sl.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 0.870432 0.670671 0.882000
2 0.754248 0.824157 0.895000
3 0.654601 0.727428 0.885000
4 0.602772 0.668668 0.901000
5 0.542110 0.625137 0.903000
6 0.506150 0.617842 0.913000
7 0.480944 0.616885 0.912000
8 0.472876 0.614381 0.911000
Total time: 06:38
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc-sl.m
Loss and accuracy using (cls_best): [0.2977172, tensor(0.9233)]
OrderedDict([('data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc-sl.m',
0.9232500195503235)])
data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc-sl.m: 0.9232500195503235
```
### 0.2
```bash
python -m ulmfit eval --glob="mldoc/zh-1/models/sp30k/lstm_nl4.m" --name nl4-1cyc-sl2 --num-cls-epochs=8 --bs=18 --lr_sched=1cycle --label-smoothing-eps=0.2
Processing data/mldoc/zh-1/models/sp30k/lstm_nl4.m
zh-1
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc-sl2.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁中', '▁人', '▁是']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc-sl2.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.045619 0.908213 0.874000
2 0.957379 0.857977 0.921000
3 0.891791 0.852157 0.905000
4 0.845289 0.849923 0.914000
5 0.818228 0.848613 0.921000
6 0.787021 0.840483 0.920000
7 0.776123 0.844006 0.919000
8 0.762384 0.857240 0.916000
Total time: 06:33
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc-sl2.m
Loss and accuracy using (cls_best): [0.40299156, tensor(0.9170)]
OrderedDict([('data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc-sl2.m',
0.9169999957084656)])
data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc-sl2.m: 0.9169999957084656
```
### 0.4
```bash
python -m ulmfit eval --glob="mldoc/zh-1/models/sp30k/lstm_nl4.m" --name nl4-1cyc-sl4 --num-cls-epochs=8 --bs=18 --lr_sched=1cycle --label-smoothing-eps=0.4
Processing data/mldoc/zh-1/models/sp30k/lstm_nl4.m
zh-1
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc-sl4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv
Data lm, trn: 13500, val: 1500
Data cls, trn: 1000, val: 1000
Data tst, trn: 1000, val: 4000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁中', '▁人', '▁是']
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
Loading pretrained model
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc-sl4.m/info.json
Single training schedule
epoch train_loss valid_loss accuracy
1 1.251581 1.183341 0.898000
2 1.214358 1.201266 0.834000
3 1.190343 1.165525 0.919000
4 1.168018 1.172510 0.903000
5 1.149965 1.161660 0.914000
6 1.140140 1.161689 0.915000
7 1.135877 1.159853 0.912000
8 1.134425 1.160039 0.911000
Total time: 06:34
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc-sl4.m
Loss and accuracy using (cls_best): [0.64041936, tensor(0.9195)]
OrderedDict([('data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc-sl4.m',
0.9194999933242798)])
data/mldoc/zh-1/models/sp30k/lstm_nl4-1cyc-sl4.m: 0.9194999933242798
```
results/logs/noise/de10k-noise.md
+1158
View File
File diff suppressed because it is too large Load Diff
results/logs/noise/es10k-noise.md
+1159
View File
File diff suppressed because it is too large Load Diff
results/logs/qrnn-ru.md
+12
View File
@@ -1,4 +1,16 @@
# QRNN RU
## SP15k nl8
data/wiki/ru-100/models/sp15k/qrnn_nl8.m
export CUDA_VISIBLE_DEVICES=0
LANG=ru
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_nl8.m --lang=${LANG} --name 'nl8' - train 20 --bs 20 --num-cls-epochs=8 --lr_sched=1cycle
export CUDA_VISIBLE_DEVICES=0
LANG=de
python -m ulmfit cls --dataset-path data/mldoc-m/${LANG}-1 --base-lm-path data/wiki-m/${LANG}-100/models/sp15k/lstm_nl4.m --lang=${LANG} --name 'nl4' - train 20 --bs 20 --num-cls-epochs=8 --lr_sched=1cycle
## SP30k nl4
### LM
```
tests/test_end_to_end.py
+23 -1
View File
@@ -39,7 +39,8 @@ def get_test_data():
copy_head(wt / 'en.wiki.train.tokens', test_wt / 'en.wiki.valid.tokens', n=600*sz)
copy_head(wt / 'en.wiki.train.tokens', test_wt / 'en.wiki.test.tokens', n=600*sz)
copy_head(imdb / 'train.csv', test_imdb / 'train.csv', n=10*sz)
copy_head(imdb / 'train.csv', test_imdb / 'test.csv', n=6*sz)
copy_head(imdb / 'train.csv', test_imdb / 'test.csv', n=6 * sz)
copy_head(imdb / 'train.csv', test_imdb / 'dev.csv', n=6 * sz)
copy_head(imdb / 'train.csv', test_imdb / 'unsup.csv', n=1*sz)
return test_data, test_wt
@@ -109,12 +110,33 @@ def test_ulmfit_fastai_end_to_end():
qrnn=False,
tokenizer='f',
max_vocab=100,
nl=1,
name=lm_name,
)
exp.train_lm(num_epochs=1, bs=2)
exp2 = ulmfit.train_clas.CLSHyperParams.from_lm(test_data / 'imdb', exp.model_dir)
exp2.train_cls(num_lm_epochs=0, unfreeze=False, bs=4, )
def test_ulmfit_fastai_end_to_end_label_smoothing():
""" Test ulmfit with sentencepiece tokenizer on small wikipedia dataset.
"""
test_data, wt2 = get_test_data()
lm_name = 'end-to-end-test-fastai'
exp = ulmfit.pretrain_lm.LMHyperParams(
dataset_path=wt2,
lang='en',
cuda_id=cuda_id,
qrnn=False,
tokenizer='f',
max_vocab=100,
name=lm_name,
)
exp.train_lm(num_epochs=1, bs=2, label_smoothing_eps=0.1)
exp2 = ulmfit.train_clas.CLSHyperParams.from_lm(test_data / 'imdb', exp.model_dir)
exp2.train_cls(num_lm_epochs=0, unfreeze=False, bs=4, label_smoothing_eps=0.1 )
def test_ulmfit_fastai_bidir_end_to_end():
""" Test ulmfit with sentencepiece tokenizer on small wikipedia dataset.
"""
ulmfit/__main__.py
+29 -10
View File
@@ -1,12 +1,13 @@
import gc
import os
import pprint
import tarfile
import shutil
from collections import OrderedDict
from functools import wraps
import pandas as pd
import fire
from .pretrain_lm import LMHyperParams, np
from .pretrain_lm import LMHyperParams
from .train_clas import CLSHyperParams
from pathlib import Path
from string import Template
@@ -49,19 +50,37 @@ class ULMFiT:
return FireView(train=params.train_cls, validate_cls=params.validate_cls)
def eval_noise_resistance(self, lang="de"):
results = {}
def eval_noise_resistance(self, lang="de", size=1, prefix_name="", model="sp15k/qrnn_nl4.m"):
def first_or_default(l, default=None):
l = list(l)
if l:
return l[0]
return default
results= []
for noise in range(0, 80, 5):
print("Noise: ", noise)
d = self.eval(glob=f"mldoc/{lang}-1/models/sp15k/qrnn_nl4.m",
name=f"nl4_{noise}",
d = self.eval(glob=f"mldoc/{lang}-1/models/{model}",
name=f"nl4_{prefix_name}{noise}",
noise=noise/100,
dataset_template='${lang}-'+str(size),
num_cls_epochs=8,
bs=18,
lr_sched="1cycle")
results.update(d)
np.save('results.npy', results)
print(results)
val = first_or_default(d.values(), default=-1)
results.append((noise/100, val))
df = pd.DataFrame(results, columns=["noise", "accuracy"])
df.to_csv(f"noise_{lang}-{size}{prefix_name}.csv")
print(df)
def tar(self, model_path):
params = CLSHyperParams.from_json(model_path)
tar_name = f"models/{params.lang}-{params.tokenizer_prefix}-{params.model_name}.tar"
print("Storing model in", tar_name)
with tarfile.open(tar_name, mode="w") as tar:
for g in map(params.model_dir.glob, ['*_last.*', 'info.json', 'info.json', '../spm.*', '../itos.*',]):
for f in g:
print("Adding", f, f.relative_to("data"))
tar.add(f, f.relative_to("data"))
def eval(self, glob="mldoc/*-1/models/sp30k/lstm_nl4.m", dataset_template='${lang}-1', name="tmp-100", num_lm_epochs=0, cuda_id=0, **trn_params):
results = OrderedDict()
@@ -71,7 +90,7 @@ class ULMFiT:
try:
params = CLSHyperParams.from_lm(dataset_path, base_model, lang=lang, name=name, cuda_id=cuda_id)
key = str(params.model_dir.relative_to(Path.cwd()))
if (params.model_dir/"cls_last.pth").exists():
if (params.model_dir/"cls_best.pth").exists():
print("Evaluating previously trained model")
results[key] = params.validate_cls()[1]
else:
ulmfit/pretrain_lm.py
+19 -4
View File
@@ -171,10 +171,10 @@ class LMHyperParams:
with (self.model_dir / 'info.json').open("w") as fp: json.dump(vals, fp)
print("Saving info", self.model_dir / 'info.json')
def train_lm(self, num_epochs=20, data_lm=None, bs=70, true_wd=False, drop_mult=0.0, lr=5e-3):
def train_lm(self, num_epochs=20, data_lm=None, bs=70, true_wd=False, drop_mult=0.0, lr=5e-3, label_smoothing_eps=0.0):
self.model_dir.mkdir(exist_ok=True, parents=True)
data_lm = self.load_wiki_data(bs=bs) if data_lm is None else data_lm
learn = self.create_lm_learner(data_lm, drop_mult=drop_mult)
learn = self.create_lm_learner(data_lm, drop_mult=drop_mult, label_smoothing_eps=label_smoothing_eps)
learn.true_wd = true_wd
if num_epochs > 0:
@@ -204,7 +204,7 @@ class LMHyperParams:
# do we need to return `learn'? it adds noise to Fire output
#return learn
def create_lm_learner(self, data_lm, dps=None, **kwargs):
def create_lm_learner(self, data_lm, dps=None, label_smoothing_eps=0.0, **kwargs):
assert self.bidir == False, "bidirectional model is not yet supported"
config = dict(emb_sz=self.emb_sz, n_hid=self.nh, n_layers=self.nl, pad_token=PAD_TOKEN_ID, qrnn=self.qrnn,
tie_weights=True, out_bias=True)
@@ -230,6 +230,8 @@ class LMHyperParams:
learn.callback_fns += [partial(CSVLogger, filename=f"{learn.model_dir}/lm-history"),
# partial(SaveModelCallback, every='improvement', name='lm') disabled due to Memory issues
]
if label_smoothing_eps > 0.0:
learn.loss_func = FlattenedLoss(LabelSmoothingCrossEntropy, eps=label_smoothing_eps)
return learn
def load_train_text(self):
@@ -316,13 +318,26 @@ class LMHyperParams:
d.update(kwargs)
return cls(**d)
@classmethod
def from_json(cls, model_path, **kwargs):
def from_json(cls, model_path:Path, **kwargs):
model_path = Path(model_path).resolve()
name = re.search(r"[a-z]+_(.+).m", model_path.name).group(1)
with open(model_path / 'info.json', 'r') as f:
d = json.load(f)
d.update(kwargs)
d['name'] = name
dataset_path = path_strip(model_path, "data", "models").parent
d['dataset_path'] = str(dataset_path)
d['lang'] = infer_lang_from_dataset(dataset_path.name)
return cls(**d)
def infer_lang_from_dataset(name:str):
return name.split("-")[0]
def path_strip(path, from_folder, to_folder):
to_p = [p for p in path.parents if p.name == to_folder][0]
from_p = [p for p in path.parents if p.name == from_folder][0]
return to_p.relative_to(from_p.parent)
def validate_lm(self):
if not self.exp.subword and self.exp.max_vocab is None:
raise NotImplementedError("figure out how to validate and save results")
ulmfit/train_clas.py
+25 -14
View File
@@ -64,7 +64,8 @@ class CLSHyperParams(LMHyperParams):
learn.fit_one_cycle(num_cls_epochs-4, slice(1e-2 / (2.6 ** 4), 1e-2), moms=(0.8, 0.7), wd=1e-7)
def train_cls(self, num_lm_epochs, unfreeze=True, num_cls_frozen_epochs=1, bs=40, drop_mul_lm=0.3, drop_mul_cls=0.5,
use_test_for_validation=False, num_cls_epochs=2, limit=None, noise=0.0, cls_max_len=20*70, lr_sched='layered'):
use_test_for_validation=False, num_cls_epochs=2, limit=None, noise=0.0, cls_max_len=20*70, lr_sched='layered',
label_smoothing_eps=0.0):
assert use_test_for_validation == False, "use_test_for_validation=True is not supported"
self.model_dir.mkdir(exist_ok=True, parents=True)
@@ -73,10 +74,10 @@ class CLSHyperParams(LMHyperParams):
data_clas, data_lm, data_tst = self.load_cls_data(bs, limit=limit, noise=noise)
if self.need_fine_tune_lm: self.train_lm(num_lm_epochs, data_lm=data_lm, drop_mult=drop_mul_lm)
learn = self.create_cls_learner(data_clas, drop_mult=drop_mul_cls, max_len=cls_max_len)
if self.need_fine_tune_lm: self.train_lm(num_lm_epochs, data_lm=data_lm, drop_mult=drop_mul_lm, label_smoothing_eps=label_smoothing_eps)
learn = self.create_cls_learner(data_clas, drop_mult=drop_mul_cls, max_len=cls_max_len, label_smoothing_eps=label_smoothing_eps)
try:
learn.load('cls_last')
learn.load('cls_best')
print("Loading last classifier")
except FileNotFoundError:
learn.load_encoder(ENC_BEST)
@@ -84,6 +85,8 @@ class CLSHyperParams(LMHyperParams):
if hasattr(self, 'lr_schedule_'+lr_sched):
learn.true_wd = True
getattr(self, 'lr_schedule_'+lr_sched)(learn, num_cls_epochs)
else:
raise ValueError(f"Wrong lr_sched: {lr_sched}")
print(f"Saving models at {learn.path / learn.model_dir}")
learn.save('cls_last', with_opt=False)
@@ -91,7 +94,7 @@ class CLSHyperParams(LMHyperParams):
del learn
return self.validate_cls('cls_best', bs=bs, data_tst=data_tst, learn=None)
def validate_cls(self, save_name='cls_last', bs=40, data_tst=None, learn=None):
def validate_cls(self, save_name='cls_best', bs=40, data_tst=None, learn=None):
if data_tst is None:
_, _, data_tst = self.load_cls_data(bs)
if learn is None:
@@ -102,7 +105,7 @@ class CLSHyperParams(LMHyperParams):
print(f"Loss and accuracy using ({save_name}):", results)
return list(map(float, results))
def create_cls_learner(self, data_clas, dps=None, **kwargs):
def create_cls_learner(self, data_clas, dps=None, label_smoothing_eps=0.0, **kwargs):
assert self.bidir == False, "bidirectional model is not yet supported"
config = dict(emb_sz=self.emb_sz, n_hid=self.nh, n_layers=self.nl, pad_token=PAD_TOKEN_ID, qrnn=self.qrnn)
config.update(dps or self.dps)
@@ -121,6 +124,8 @@ class CLSHyperParams(LMHyperParams):
learn.callback_fns += [partial(CSVLogger, filename=f"{learn.model_dir}/cls-history"),
#partial(SaveModelCallback, every='improvement', name='cls_best') disabled due to memory issues
]
if label_smoothing_eps > 0.0:
learn.loss_func = FlattenedLoss(LabelSmoothingCrossEntropy, eps=label_smoothing_eps)
return learn
def load_cls_data(self, bs, **kwargs):
@@ -178,6 +183,17 @@ class CLSHyperParams(LMHyperParams):
kwargs.update(dict(trn_df=trn_df, val_df=val_df, tst_df=tst_df, unsup_df=unsup_df))
return kwargs
def add_noise(self, trn_df, noise):
count = len(trn_df)
labels = trn_df[0].unique()
assert np.issubdtype(labels.dtype, np.integer), "noise only works on numerical numbers"
modulo = labels.max() + 1
idx_to_distrub = np.random.permutation(count)[:int(count * noise)]
trn_df.loc[idx_to_distrub, [0]] = (np.random.randint(1, modulo - 1, size=len(idx_to_distrub)) +
trn_df.loc[idx_to_distrub][0]) % modulo
print(f"Added noise to {len(idx_to_distrub)} examples, only {(count - len(idx_to_distrub)) / count} have correct labels")
return trn_df
def databunches(self, bs, trn_df, val_df, tst_df, unsup_df, add_trn_to_lm=True, use_moses=False, force=False, limit=None, noise=0.0):
lm_trn_df = pd.concat([unsup_df, val_df, tst_df] + ([trn_df] if add_trn_to_lm else []))
val_len = max(int(len(lm_trn_df) * 0.1), 2)
@@ -192,14 +208,9 @@ class CLSHyperParams(LMHyperParams):
cls_name=f'{cls_name}limit{limit}'
if noise > 0.0:
count = len(trn_df)
labels = trn_df[0].unique()
assert np.issubdtype(labels.dtype, np.integer), "noise only works on numerical numbers"
modulo = labels.max()+1
idx_to_distrub = np.random.permutation(count)[:int(count * noise)]
trn_df.loc[idx_to_distrub, [0]] = (np.random.randint(1, modulo-1, size=len(idx_to_distrub)) + trn_df.loc[idx_to_distrub][0]) % modulo
print(f"Added noise to {len(idx_to_distrub)} examples, only {(count-len(idx_to_distrub))/count} have correct labels")
cls_name = f'{cls_name}noise{noise}'
trn_df = self.add_noise(trn_df, noise)
val_df = self.add_noise(val_df, noise)
cls_name = f'{cls_name}noise{noise}tv'
args = self.tokenizer_to_fastai_args(sp_data_func=lambda: trn_df[1], use_moses=use_moses)
data_lm = self.lm_databunch('lm', train_df=lm_trn_df, valid_df=lm_val_df, bs=bs, force=force, **args)