mirror of
https://github.com/wassname/multifit.git
synced 2026-09-20 13:00:41 +08:00
270 lines
14 KiB
Markdown
270 lines
14 KiB
Markdown
# ES
|
|
|
|
## SP30k LSTM nl 4
|
|
### LM
|
|
````
|
|
python -m ulmfit lm --dataset-path data/wiki/es-100 --cuda-id=0 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 30000 --lang es --qrnn=False - train 10 --bs=50 --drop_mult=0
|
|
Running tokenization
|
|
Wiki text was split to 96224 articles
|
|
Wiki text was split to 105 articles
|
|
Size of vocabulary: 30000
|
|
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
|
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': None, 'pretrained_model': None, 'drop_mult': 0} dps: [0.25 0.1 0.2 0.02 0.15]
|
|
Training lm from random weights
|
|
epoch train_loss valid_loss accuracy
|
|
1 3.269541 3.451855 0.387471
|
|
2 3.161740 3.423016 0.386158
|
|
3 3.187431 3.419638 0.388626
|
|
4 3.115763 3.357066 0.393877
|
|
5 2.996527 3.291787 0.402488
|
|
6 3.021759 3.202183 0.410873
|
|
7 2.998267 3.104373 0.422624
|
|
8 2.827225 3.006537 0.436010
|
|
9 2.784576 2.937735 0.446654
|
|
10 2.789913 2.918509 0.450055
|
|
data/wiki/es-100/models/sp30k
|
|
Saving info data/wiki/es-100/models/sp30k/lstm_nl4.m/info.json
|
|
````
|
|
|
|
### MLDoc
|
|
|
|
```
|
|
python -m ulmfit cls --dataset-path data/mldoc/es-1 --base-lm-path data/wiki/es-100/models/sp30k/lstm_nl4.m --lang=es --name 'nl4' --cuda-id=1 - train 20 --bs 40 --num-cls-epochs=2
|
|
Max vocab: 30000
|
|
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k
|
|
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4.m
|
|
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv
|
|
Running tokenization...
|
|
Saving tokenized: cls.trn 13013, cls.val 1445
|
|
Running tokenization...
|
|
Saving tokenized: cls.trn 1000, cls.val 1000
|
|
Size of vocabulary: 30000
|
|
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
|
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/es-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/es-100/models/sp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
|
Unknown tokens 0, first 100: []
|
|
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/es-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/es-100/models/sp30k/lstm_nl4.m/../itos')]
|
|
epoch train_loss valid_loss accuracy
|
|
1 2.805415 2.188974 0.537779
|
|
epoch train_loss valid_loss accuracy
|
|
1 2.429727 1.989691 0.569048
|
|
2 2.218828 1.794969 0.603721
|
|
3 2.015097 1.644815 0.629609
|
|
4 1.877210 1.537773 0.646898
|
|
5 1.775648 1.450283 0.660861
|
|
6 1.749334 1.377085 0.672146
|
|
7 1.601073 1.311101 0.684400
|
|
8 1.564420 1.251074 0.694900
|
|
9 1.532728 1.197607 0.704779
|
|
10 1.391921 1.145408 0.716044
|
|
11 1.379958 1.093550 0.726937
|
|
12 1.324111 1.048308 0.735890
|
|
13 1.344113 1.007926 0.745691
|
|
14 1.243085 0.969521 0.754591
|
|
15 1.230809 0.937330 0.762675
|
|
16 1.162501 0.913408 0.768044
|
|
17 1.170092 0.894892 0.773239
|
|
18 1.110860 0.884449 0.775603
|
|
19 1.115907 0.880448 0.776671
|
|
20 1.083033 0.878421 0.776931
|
|
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k
|
|
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4.m/info.json
|
|
Starting classifier training
|
|
epoch train_loss valid_loss accuracy
|
|
1 0.621574 0.391042 0.856000
|
|
epoch train_loss valid_loss accuracy
|
|
1 0.411668 0.215625 0.935000
|
|
epoch train_loss valid_loss accuracy
|
|
1 0.340519 0.222422 0.935000
|
|
epoch train_loss valid_loss accuracy
|
|
1 0.281729 0.192193 0.949000
|
|
2 0.262074 0.202975 0.945000
|
|
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4.m
|
|
Loss and accuracy using (cls_best): [0.1749019, tensor(0.9515)]
|
|
```
|
|
|
|
|
|
|
|
## ES optimization
|
|
|
|
|
|
|
|
### Smaler vocab 15k
|
|
#### LM
|
|
```
|
|
python -m ulmfit lm --dataset-path data/wiki-m/es-100 --cuda-id=0 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 \ ✘ 1
|
|
--lang es --qrnn=False - train 10 --bs=50 --drop_mult=0
|
|
Max vocab: 15000
|
|
Cache dir: data/wiki-m/es-100/models/sp15k
|
|
Model dir: data/wiki-m/es-100/models/sp15k/lstm_nl4.m
|
|
Tokenized data loaded
|
|
Size of vocabulary: 15000
|
|
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
|
|
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
|
Training lm from random weights
|
|
epoch train_loss valid_loss accuracy
|
|
1 2.961702 3.187043 0.403149
|
|
Better model found at epoch 1 with val_loss value: 3.1870434284210205.
|
|
2 2.928991 3.170802 0.402026
|
|
Better model found at epoch 2 with val_loss value: 3.170802354812622.
|
|
3 2.931906 3.128328 0.407816
|
|
Better model found at epoch 3 with val_loss value: 3.128328323364258.
|
|
4 2.869332 3.072160 0.414345
|
|
Better model found at epoch 4 with val_loss value: 3.072160243988037.
|
|
5 2.803377 2.997071 0.424847
|
|
Better model found at epoch 5 with val_loss value: 2.997070550918579.
|
|
6 2.758087 2.927369 0.432256
|
|
Better model found at epoch 6 with val_loss value: 2.927368640899658.
|
|
7 2.657733 2.825029 0.446440
|
|
Better model found at epoch 7 with val_loss value: 2.8250293731689453.
|
|
8 2.563273 2.728652 0.459271
|
|
Better model found at epoch 8 with val_loss value: 2.7286524772644043.
|
|
9 2.475741 2.654844 0.470864
|
|
Better model found at epoch 9 with val_loss value: 2.654844045639038.
|
|
10 2.428898 2.634355 0.474821
|
|
Better model found at epoch 10 with val_loss value: 2.634355306625366.
|
|
Total time: 17:53:59
|
|
data/wiki-m/es-100/models/sp15k
|
|
Saving info data/wiki-m/es-100/models/sp15k/lstm_nl4.m/info.json
|
|
```
|
|
|
|
#### MLDoc
|
|
```
|
|
python -m ulmfit cls --dataset-path data/mldoc/es-1 --base-lm-path data/wiki-m/es-100/models/sp15k/lstm_nl4.m --lang=es --name 'nl4' --cuda-id=0 - train 20 --bs 20 --num-cls-epochs=8
|
|
Max vocab: 15000
|
|
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k
|
|
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/lstm_nl4.m
|
|
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv
|
|
Running tokenization...
|
|
Saving tokenized: cls.trn 13013, cls.val 1445
|
|
Running tokenization...
|
|
Saving tokenized: cls.trn 1000, cls.val 1000
|
|
Size of vocabulary: 15000
|
|
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
|
|
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
|
Loading pretrained model
|
|
Unknown tokens 0, first 100: []
|
|
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki-m/es-100/models/sp15k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki-m/es-100/models/sp15k/lstm_nl4.m/../itos')]
|
|
epoch train_loss valid_loss accuracy
|
|
1 2.355042 1.850417 0.589128
|
|
Better model found at epoch 1 with val_loss value: 1.850416898727417.
|
|
Total time: 03:25
|
|
epoch train_loss valid_loss accuracy
|
|
1 2.087364 1.677666 0.619909
|
|
Better model found at epoch 1 with val_loss value: 1.6776657104492188.
|
|
2 1.880730 1.522996 0.648134
|
|
Better model found at epoch 2 with val_loss value: 1.5229955911636353.
|
|
3 1.767530 1.403644 0.668117
|
|
Better model found at epoch 3 with val_loss value: 1.4036436080932617.
|
|
4 1.659950 1.309353 0.684900
|
|
Better model found at epoch 4 with val_loss value: 1.3093526363372803.
|
|
5 1.546585 1.232220 0.699358
|
|
Better model found at epoch 5 with val_loss value: 1.2322196960449219.
|
|
6 1.592862 1.161846 0.713034
|
|
Better model found at epoch 6 with val_loss value: 1.1618456840515137.
|
|
7 1.444965 1.098108 0.726811
|
|
Better model found at epoch 7 with val_loss value: 1.0981075763702393.
|
|
8 1.340874 1.029193 0.741337
|
|
Better model found at epoch 8 with val_loss value: 1.0291931629180908.
|
|
9 1.351407 0.974317 0.753408
|
|
Better model found at epoch 9 with val_loss value: 0.9743167757987976.
|
|
10 1.231713 0.915328 0.767088
|
|
Better model found at epoch 10 with val_loss value: 0.9153280854225159.
|
|
11 1.151926 0.852391 0.782414
|
|
Better model found at epoch 11 with val_loss value: 0.852391242980957.
|
|
12 1.163565 0.794699 0.797228
|
|
Better model found at epoch 12 with val_loss value: 0.7946987152099609.
|
|
13 1.054929 0.743652 0.810518
|
|
Better model found at epoch 13 with val_loss value: 0.74365234375.
|
|
14 0.974651 0.695024 0.823344
|
|
Better model found at epoch 14 with val_loss value: 0.6950243711471558.
|
|
15 0.869718 0.651691 0.834510
|
|
Better model found at epoch 15 with val_loss value: 0.6516908407211304.
|
|
16 0.889763 0.615112 0.844947
|
|
Better model found at epoch 16 with val_loss value: 0.6151121258735657.
|
|
17 0.843503 0.590130 0.851694
|
|
Better model found at epoch 17 with val_loss value: 0.5901297926902771.
|
|
18 0.752870 0.575217 0.855496
|
|
Better model found at epoch 18 with val_loss value: 0.5752172470092773.
|
|
19 0.807087 0.567187 0.857605
|
|
Better model found at epoch 19 with val_loss value: 0.5671872496604919.
|
|
20 0.784531 0.566082 0.857827
|
|
Better model found at epoch 20 with val_loss value: 0.5660821199417114.
|
|
Total time: 1:26:48
|
|
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k
|
|
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/lstm_nl4.m/info.json
|
|
Starting classifier training
|
|
epoch train_loss valid_loss accuracy
|
|
1 0.616963 0.275447 0.923000
|
|
Better model found at epoch 1 with val_loss value: 0.27544698119163513.
|
|
Total time: 00:24
|
|
epoch train_loss valid_loss accuracy
|
|
1 0.425890 0.201535 0.932000
|
|
Better model found at epoch 1 with val_loss value: 0.2015346735715866.
|
|
Total time: 00:27
|
|
epoch train_loss valid_loss accuracy
|
|
1 0.265563 0.186434 0.951000
|
|
Better model found at epoch 1 with val_loss value: 0.18643426895141602.
|
|
Total time: 00:32
|
|
epoch train_loss valid_loss accuracy
|
|
1 0.162097 0.180026 0.955000
|
|
Better model found at epoch 1 with val_loss value: 0.1800260841846466.
|
|
2 0.161748 0.187014 0.957000
|
|
3 0.142789 0.166486 0.961000
|
|
Better model found at epoch 3 with val_loss value: 0.1664857715368271.
|
|
4 0.105920 0.173207 0.963000
|
|
5 0.078164 0.184849 0.962000
|
|
6 0.070540 0.189451 0.962000
|
|
7 0.051490 0.208019 0.959000
|
|
8 0.047614 0.193699 0.962000
|
|
Total time: 05:20
|
|
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/lstm_nl4.m
|
|
Loss and accuracy using (cls_best): [0.1623594, tensor(0.9538)]
|
|
0.16235940158367157
|
|
0.9537500143051147
|
|
```
|
|
|
|
### Larger dropout - no luck
|
|
```
|
|
python -m ulmfit cls --dataset-path data/mldoc/es-1 --base-lm-path data/wiki-m/es-100/models/sp30k/lstm_nl4.m --lang=es --name 'nl4-drop' --cuda-id=0 - train 0 --bs 20 --num-cls-epochs=8 --drop-mul-lm=0.5 --drop-mul-cls=0.8
|
|
Max vocab: 30000
|
|
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k
|
|
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4-drop.m
|
|
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv
|
|
Running tokenization...
|
|
Saving tokenized: cls.trn 13013, cls.val 1445
|
|
Running tokenization...
|
|
Saving tokenized: cls.trn 1000, cls.val 1000
|
|
Size of vocabulary: 30000
|
|
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
|
Starting classifier training
|
|
epoch train_loss valid_loss accuracy
|
|
1 1.310594 0.903076 0.724000
|
|
Better model found at epoch 1 with val_loss value: 0.9030755758285522.
|
|
Total time: 00:22
|
|
epoch train_loss valid_loss accuracy
|
|
1 1.209348 0.714140 0.756000
|
|
Better model found at epoch 1 with val_loss value: 0.714139997959137.
|
|
Total time: 00:23
|
|
epoch train_loss valid_loss accuracy
|
|
1 1.122636 0.627526 0.797000
|
|
Better model found at epoch 1 with val_loss value: 0.6275263428688049.
|
|
Total time: 00:29
|
|
epoch train_loss valid_loss accuracy
|
|
1 1.102433 0.593338 0.801000
|
|
Better model found at epoch 1 with val_loss value: 0.5933384895324707.
|
|
2 1.096480 0.543266 0.818000
|
|
Better model found at epoch 2 with val_loss value: 0.5432664155960083.
|
|
3 1.082919 0.501089 0.837000
|
|
Better model found at epoch 3 with val_loss value: 0.5010889172554016.
|
|
4 1.069694 0.518807 0.812000
|
|
5 1.040208 0.508399 0.825000
|
|
6 1.032841 0.512187 0.838000
|
|
7 1.031225 0.504557 0.825000
|
|
8 1.016486 0.502335 0.837000
|
|
Total time: 04:56
|
|
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4-drop.m
|
|
Loss and accuracy using (cls_best): [0.52473265, tensor(0.8160)]
|
|
0.5247326493263245
|
|
0.8159999847412109
|
|
```
|