wassname/multifit
1
0
Fork 0
mirror of https://github.com/wassname/multifit.git synced 2026-09-09 11:27:26 +08:00
Code Issues Packages Projects Releases Wiki Activity

Zeroshot MLDoc results for ulmfit trained on 10k examples

Browse Source
This commit is contained in:
Piotr Czapla
2019-02-17 19:03:02 +01:00
parent 1ee1dd950d
commit 0dda4b4c2c
3 changed files with 418 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
results/MLDoc.md
+47 -5
View File
@@ -2,14 +2,15 @@
## Supervised classification results on MLDoc
| Model | en | de | es | fr | it | ja | ru | zh |
|----------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|------------|
|LASER 0 shot | 80.75 | 87.03 | 82.60 | 82.83 | 73.25 | 60.95 | 68.83 | 72.90 |
|LASER | 90.73 | 92.70 | 88.75 | 90.80 | 85.93 | 85.15 | 84.65 | 88.98 |
|MultiCCA | 92.2 | 93.70 | 94.45 | 92.05 | 85.55 | 85.35 | 85.65 | 87.30 |
|ULMFiT | | **95.4** | **95.15** | **93.67** | **88.42** | **89.20** | **87.27** | 90.20 |
|ULMFiT sp-fixed | | 95.6 | 94.80 | 94.20 | 88.52 | 88.72 | 86.85 | 90.47 |
|ULMFiT 100 | | 91.35 | 83.32 | 88.77 | 77.99 | 71.12 | 72.20 | |
|Bert Multi | 93.23% | 94.0% | **95.15** | 93.20 | 85.82 | 87.48 | 86.85 | **90.72** |
|ULMFiT 100 | | 91.35 | 83.32 | 88.77 | 77.99 | 71.12 | 72.20 | |
|ULMFiT Zeroshot from Laser | | 94.48 | 86.93 | 88.78 | 79.35 | | 72.88 | 85.55 |
|ULMFiT | | 95.4 | **95.15** | 93.67 | 88.42 | **89.20** | **87.27** | 90.20 |
|ULMFiT sp-fixed | | **95.6** | 94.80 | **94.20** | **88.52** | 88.72 | 86.85 | 90.47 |
|Bert Multi | 93.23% | 94.0% | **95.15** | 93.20 | 85.82 | 87.48 | 86.85 | **90.72** |
^ - sp60k lstm nl 4
## Zero shot approaches
@@ -30,6 +31,47 @@
| | | | | | | |
| Bert Multilingual-EN | 74.50 | 61.85 | 69.77 | 57.73 | 51.10 | 64.08 |
### From Laser trained on French data
| Model Name | de | es | fr | it | ru | zh |
|---------------------------|-------|-------|----|-------|-------|-------|
| LASER fr 10k | 91.65 | 81.05 | | 75.08 | 70.73 | 76.33 |
| LASER fr 1k | 88.75 | 80.12 | | 72.58 | 67.35 | 79.4 |
| ULMFiT 10k on LASER-fr10k | 94.48 | 84.10 | | 77.93 | 72.87 | 84.53 |
| ULMFiT 10k on LASER-fr1k | 92.30 | 82.10 | | 75.52 | 69.52 | 85.55 |
| ULMFiT 1k on LASER-fr1k | 92.22 | 81.00 | | 76.88 | 68.33 | 84.65 |
| | | | | | | |
| Impr 10k over 10k | 34% | 16% | | 11% | 7% | 35% |
| Impr 10k over 1k | 32% | 10% | | 11% | 7% | 30% |
| Impr 1k over 1k | 31% | 4% | | 16% | 3% | 25% |
### From Laser trained on German data
| Model Name | de | es | fr | it | ru | zh |
|---------------------------|----|-------|-------|-------|-------|-------|
| LASER de 10k | | 83.5 | 82.85 | 76.6 | 68.8 | 73.12 |
| LASER de 1k | | 81.4 | 81.5 | 74.53 | 64.58 | 73.2 |
| ULMFiT 10k on LASER-de10k | | 86.92 | 87.17 | 79.35 | 70.15 | 78.15 |
| ULMFiT 10k on LASER-de1k | | 84.65 | 87.48 | 78.70 | 67.65 | 77.50 |
| ULMFiT 1k on LASER-de1k | | 85.5 | 87.37 | 78.75 | 66.95 | 72.32 |
| | | | | | | |
| Impr 10k over 10k | | 21% | 25% | 12% | 4% | 19% |
| Impr 10k over 1k | | 17% | 32% | 16% | 9% | 16% |
| Impr 1k over 1k | | 22% | 32% | 17% | 7% | -3% |
### From Laser trained on English data
| Model Name | de | es | fr | it | ru | zh |
|---------------------------|-------|-------|-------|-------|-------|-------|
| LASER en 10k | 87.43 | 77.38 | 78.7 | 72.53 | 67.7 | 75.18 |
| LASER en 1k | 87.65 | 75.48 | 84 | 71.18 | 66.58 | 76.65 |
| ULMFiT 10k on LASER-en10k | 92.05 | 80.05 | 86.95 | 76.65 | 70.57 | 80.85 |
| ULMFiT 10k on LASER-en1k | 91.80 | 80.10 | 88.67 | 77.32 | 70.25 | 82.73 |
| ULMFiT 1k on LASER-en1k | 92.95 | 80.50 | 88.78 | 76.20 | 70.05 | 80.45 |
| | | | | | | |
| Impr 10k over 10k | 37% | 12% | 39% | 15% | 9% | 23% |
| Impr 10k over 1k | 34% | 19% | 29% | 21% | 11% | 26% |
| Impr 1k over 1k | 43% | 20% | 30% | 17% | 10% | 16% |
All ULMFiT examples above were trained on 1k training data generated by a LASER classification model
## Noise resistance
results/logs/common.md
+228
View File
@@ -53,7 +53,235 @@ python -m ulmfit eval --glob="mldoc/${lang}-1/models/sp30k/lstm_nl4.m" --name nl
## Fix the sentence piece tokenizer
```
python -m ulmfit eval --glob="mldoc/*-1/models/bsp30k/lstm_nl4.m" --name nl4 --cuda-id=0
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/de.dev.csv
Running tokenization...
Saving tokenized: cls.trn 13500, cls.val 1500
Running tokenization...
Saving tokenized: cls.trn 1000, cls.val 1000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', '▁und', '▁in', 'en', "▁&'", 's', '-']
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/bsp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/bsp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4.m/info.json
Starting classifier training
epoch train_loss valid_loss accuracy
1 0.480367 0.263987 0.930000
epoch train_loss valid_loss accuracy
1 0.340463 0.209449 0.940000
epoch train_loss valid_loss accuracy
1 0.235290 0.214566 0.952000
epoch train_loss valid_loss accuracy
1 0.169588 0.217762 0.952000
2 0.175439 0.215570 0.946000
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4.m
Loss and accuracy using (cls_best): [0.15467079, tensor(0.9563)]
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv
Running tokenization...
Saving tokenized: cls.trn 13013, cls.val 1445
Running tokenization...
Saving tokenized: cls.trn 1000, cls.val 1000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/bsp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/bsp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4.m/info.json
Starting classifier training
epoch train_loss valid_loss accuracy
1 0.609305 0.343920 0.914000
epoch train_loss valid_loss accuracy
1 0.380833 0.204708 0.947000
epoch train_loss valid_loss accuracy
1 0.311563 0.210382 0.943000
epoch train_loss valid_loss accuracy
1 0.266655 0.192309 0.955000
2 0.235553 0.183249 0.953000
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4.m
Loss and accuracy using (cls_best): [0.19250762, tensor(0.9427)]
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k/lstm_nl4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/fr.dev.csv
Running tokenization...
Saving tokenized: cls.trn 13500, cls.val 1500
Running tokenization...
Saving tokenized: cls.trn 1000, cls.val 1000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁.', "'", 's', '▁la', '▁le', '▁et', '▁l', '▁à']
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/bsp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/bsp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k/lstm_nl4.m/info.json
Starting classifier training
epoch train_loss valid_loss accuracy
1 0.527443 0.342664 0.901000
epoch train_loss valid_loss accuracy
1 0.359752 0.203693 0.936000
epoch train_loss valid_loss accuracy
1 0.272514 0.188933 0.938000
epoch train_loss valid_loss accuracy
1 0.191520 0.183208 0.938000
2 0.201412 0.178796 0.942000
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k/lstm_nl4.m
Loss and accuracy using (cls_best): [0.18170285, tensor(0.9420)]
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k/lstm_nl4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/it.dev.csv
Running tokenization...
Saving tokenized: cls.trn 13500, cls.val 1500
Running tokenization...
Saving tokenized: cls.trn 1000, cls.val 1000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁di', "▁&'", "'", '▁e', '▁il', '▁la', 'e', '▁in']
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/bsp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/bsp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k/lstm_nl4.m/info.json
Starting classifier training
epoch train_loss valid_loss accuracy
1 0.754286 0.599928 0.783000
epoch train_loss valid_loss accuracy
1 0.501399 0.379078 0.851000
epoch train_loss valid_loss accuracy
1 0.408768 0.345188 0.867000
epoch train_loss valid_loss accuracy
1 0.324877 0.335110 0.872000
2 0.291118 0.336596 0.879000
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k/lstm_nl4.m
Loss and accuracy using (cls_best): [0.33244577, tensor(0.8852)]
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv
Tokenized data loaded, lm.trn 13500, lm.val 1500
Tokenized data loaded, cls.trn 1000, cls.val 1000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁の', '▁。', '▁に', '▁を', '▁は', '▁年', '▁が', '▁)', '▁(']
Loss and accuracy using (cls_last): [0.330675, tensor(0.8873)]
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k/lstm_nl4.m
Evaluating previously trained model
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
Tokenized data loaded, lm.trn 9195, lm.val 1021
Tokenized data loaded, cls.trn 1000, cls.val 1000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', '▁и', 'е', 'и', 'й', '▁на', 'х']
Loss and accuracy using (cls_last): [0.3987146, tensor(0.8680)]
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv
Running tokenization...
Saving tokenized: cls.trn 13500, cls.val 1500
Running tokenization...
Saving tokenized: cls.trn 1000, cls.val 1000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁中', '▁人', '▁是']
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/bsp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/bsp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4.m/info.json
Starting classifier training
epoch train_loss valid_loss accuracy
1 0.612067 0.487164 0.860000
epoch train_loss valid_loss accuracy
1 0.441519 0.343330 0.886000
epoch train_loss valid_loss accuracy
1 0.384575 0.318655 0.897000
epoch train_loss valid_loss accuracy
1 0.298987 0.305157 0.899000
2 0.293190 0.312572 0.901000
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4.m
Loss and accuracy using (cls_best): [0.28432375, tensor(0.9047)]
OrderedDict([('data/mldoc/de-1/models/sp30k/lstm_nl4.m', 0.956250011920929),
('data/mldoc/es-1/models/sp30k/lstm_nl4.m', 0.9427499771118164),
('data/mldoc/fr-1/models/sp30k/lstm_nl4.m', 0.9419999718666077),
('data/mldoc/it-1/models/sp30k/lstm_nl4.m', 0.8852499723434448),
('data/mldoc/ja-1/models/sp30k/lstm_nl4.m', 0.8872500061988831),
('data/mldoc/ru-1/models/sp30k/lstm_nl4.m', 0.8679999709129333),
('data/mldoc/zh-1/models/sp30k/lstm_nl4.m', 0.9047499895095825)])
--- Additonal run on ru
python -m ulmfit eval --glob="mldoc/ru-1/models/bsp30k/lstm_nl4.m" --name nl4 --cuda-id=0 ✘ 1
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k/lstm_nl4.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
Running tokenization...
Saving tokenized: cls.trn 9195, cls.val 1021
Running tokenization...
Saving tokenized: cls.trn 1000, cls.val 1000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', '▁и', 'е', 'и', 'й', '▁на', 'х']
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/bsp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/bsp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k/lstm_nl4.m/info.json
Starting classifier training
epoch train_loss valid_loss accuracy
1 0.814752 0.555627 0.808000
epoch train_loss valid_loss accuracy
1 0.659532 0.427135 0.855000
epoch train_loss valid_loss accuracy
1 0.508609 0.427321 0.851000
epoch train_loss valid_loss accuracy
1 0.440108 0.396991 0.872000
2 0.440024 0.388976 0.866000
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k/lstm_nl4.m
Loss and accuracy using (cls_best): [0.3959306, tensor(0.8685)]
----
----
python -m ulmfit eval --glob="mldoc/es-1/models/bsp30k/lstm_nl4.m" --name nl4-2nd --cuda-id=0
Max vocab: 30000
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4-2nd.m
Training
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv
Tokenized data loaded, lm.trn 13013, lm.val 1445
Tokenized data loaded, cls.trn 1000, cls.val 1000
Size of vocabulary: 30000
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/bsp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/bsp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
Unknown tokens 0, first 100: []
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4-2nd.m/info.json
Starting classifier training
epoch train_loss valid_loss accuracy
1 0.654984 0.453887 0.818000
epoch train_loss valid_loss accuracy
1 0.451552 0.220058 0.934000
epoch train_loss valid_loss accuracy
1 0.323974 0.193342 0.949000
epoch train_loss valid_loss accuracy
1 0.244755 0.201804 0.945000
2 0.237175 0.183736 0.953000
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4-2nd.m
Loss and accuracy using (cls_best): [0.18296617, tensor(0.9480)]
OrderedDict([('data/mldoc/es-1/models/sp30k/lstm_nl4-2nd.m',
0.9480000138282776)])
----
```
### LIMIT LOgs
```
results/logs/zeroshot.md
+143 -10
View File
@@ -1,6 +1,7 @@
## Laser Perforamnce
Accuracy matrix:
### 1k
| Train | en | de | es | fr | it | ru | zh |
|-------|-------|-------|-------|-------|-------|-------|-------|
@@ -12,8 +13,84 @@ Accuracy matrix:
| ru: | 72.38 | 81.65 | 65.73 | 71.30 | 63.33 | 85.45 | 59.58 |
| zh: | 74.98 | 81.35 | 72.20 | 73.28 | 70.08 | 66.23 | 88.30 |
### 10k
| Train | en | de | es | fr | it | ru | zh |
|-------|-------|-------|-------|-------|-------|-------|-------|
| en: | 92.70 | 87.43 | 77.38 | 78.70 | 72.53 | 67.70 | 75.18 |
| de: | 81.60 | 95.40 | 83.50 | 82.85 | 76.60 | 68.80 | 73.12 |
| es: | 73.48 | 87.13 | 94.40 | 81.63 | 76.70 | 58.65 | 72.98 |
| fr: | 85.08 | 91.65 | 81.05 | 93.65 | 75.08 | 70.73 | 76.33 |
| it: | 76.75 | 86.68 | 82.55 | 82.65 | 87.80 | 65.90 | 73.35 |
| ru: | 75.23 | 81.88 | 66.83 | 68.60 | 67.38 | 87.00 | 62.68 |
| zh: | 76.05 | 82.05 | 68.40 | 77.38 | 68.45 | 66.88 | 90.38 |
## Evaluation of Laser Performance
### laser 1k
#### Building dataset
```bash
for SRC_LANG in en de fr; do ✘ 130
for LANG in en de es fr it ru zh; do
echo $LANG from $SRC_LANG
python ../../source/classify.py embed10000/mldoc.${SRC_LANG}-${SRC_LANG}.h5 ~/workspace/ulmfit-multilingual/data/mldoc/${LANG}-10 --suffix=10 | grep Test:
done
done
```
```
for SRC_LANG in en de fr; do ✘ 130
for LANG in en de es fr it ru zh; do
echo $LANG from $SRC_LANG
python ../../source/classify.py embed1000/mldoc.${SRC_LANG}-${SRC_LANG}.h5 ~/workspace/ulmfit-multilingual/data/mldoc/${LANG}-1 | grep Test:
done
done
en from en
| Test: 91.48% | classes: 23.77 24.90 26.25 25.07
de from en
| Test: 87.65% | classes: 21.98 24.45 27.65 25.93
es from en
| Test: 75.48% | classes: 21.60 15.82 22.10 40.48
fr from en
| Test: 84.00% | classes: 23.18 29.12 27.90 19.80
it from en
| Test: 71.18% | classes: 23.65 22.88 25.68 27.80
ru from en
| Test: 66.58% | classes: 29.48 13.78 34.52 22.23
zh from en
| Test: 76.65% | classes: 30.25 31.30 13.93 24.52
en from de
| Test: 78.23% | classes: 31.80 17.73 30.15 20.32
de from de
| Test: 93.50% | classes: 24.45 25.45 26.00 24.10
es from de
| Test: 81.40% | classes: 24.15 25.77 20.12 29.95
fr from de
| Test: 81.50% | classes: 25.52 29.45 27.45 17.57
it from de
| Test: 74.53% | classes: 24.70 27.25 22.43 25.62
ru from de
| Test: 64.58% | classes: 45.62 9.12 26.73 18.52
zh from de
| Test: 73.20% | classes: 31.20 43.38 7.60 17.82
en from fr
| Test: 81.30% | classes: 28.95 18.02 24.98 28.05
de from fr
| Test: 88.75% | classes: 24.00 23.75 24.85 27.40
es from fr
| Test: 80.12% | classes: 24.50 14.82 18.40 42.27
fr from fr
| Test: 90.85% | classes: 24.50 24.75 24.68 26.07
it from fr
| Test: 72.58% | classes: 25.45 24.10 17.50 32.95
ru from fr
| Test: 67.35% | classes: 47.15 13.62 16.68 22.55
zh from fr
| Test: 79.40% | classes: 33.60 31.12 9.07 26.20
```
#### Evaluation of Laser 1k Performance
```
python -m ulmfit eval --glob="mldoc/*-1/models/sp60k/lstm_nl4.m" --dataset_template="{}-laser-*" --name nl4 --cuda-id=0 ✘ 130
Max vocab: 60000
@@ -703,17 +780,76 @@ OrderedDict([('data/mldoc/de-1-laser-de/models/sp30k/lstm_nl4.m',
```
### Building dataset
### Laser 10k
### Building dataset 10k
```
for SRC_LANG in en de fr; do ✘ 130
for SRC_LANG in en de fr; do ✘ 130
for LANG in en de es fr it ru zh; do
echo $LANG from $SRC_LANG
python ../../source/classify.py embed/mldoc.${SRC_LANG}-${SRC_LANG}.h5 ~/workspace/ulmfit-multilingual/data/mldoc/${LANG}-1 | grep Test:
python ../../source/classify.py embed10000/mldoc.${SRC_LANG}-${SRC_LANG}.h5 ~/workspace/ulmfit-multilingual/data/mldoc/${LANG}-10 --suffix=10 | grep Test:
done
done
zsh: command not found: ✘
en from en
| Test: 92.70% | classes: 24.65 25.52 26.20 23.62
de from en
| Test: 87.43% | classes: 20.05 25.55 28.88 25.52
es from en
| Test: 77.38% | classes: 24.70 15.40 22.27 37.62
fr from en
| Test: 78.70% | classes: 17.65 29.75 34.60 18.00
it from en
| Test: 72.53% | classes: 20.75 24.60 28.52 26.12
ru from en
| Test: 67.70% | classes: 33.40 17.12 29.35 20.12
zh from en
| Test: 75.18% | classes: 27.70 39.00 12.07 21.23
zsh: command not found: ✘
en from de
| Test: 81.60% | classes: 31.62 23.82 24.93 19.62
de from de
| Test: 95.40% | classes: 24.57 26.00 25.62 23.80
es from de
| Test: 83.50% | classes: 28.73 22.60 18.65 30.02
fr from de
| Test: 82.85% | classes: 27.52 29.23 25.23 18.02
it from de
| Test: 76.60% | classes: 26.52 25.62 21.43 26.43
ru from de
| Test: 68.80% | classes: 48.48 13.22 19.43 18.88
zh from de
| Test: 73.12% | classes: 33.12 40.10 10.53 16.25
zsh: command not found: ✘
en from fr
| Test: 85.08% | classes: 25.43 23.43 25.43 25.73
de from fr
| Test: 91.65% | classes: 23.15 27.10 25.00 24.75
es from fr
| Test: 81.05% | classes: 24.18 17.68 18.93 39.23
fr from fr
| Test: 93.65% | classes: 24.25 25.10 25.80 24.85
it from fr
| Test: 75.08% | classes: 24.07 26.85 18.25 30.82
ru from fr
| Test: 70.73% | classes: 43.17 19.15 16.57 21.10
zh from fr
| Test: 76.33% | classes: 34.23 34.05 10.95 20.77
```
1 10
### Building dataset 1k
```
for SRC_LANG in en de fr; do
for LANG in en de es fr it ru zh; do
echo $LANG from $SRC_LANG
python ../../source/classify.py embed1000/mldoc.${SRC_LANG}-${SRC_LANG}.h5 ~/workspace/ulmfit-multilingual/data/mldoc/${LANG}-10 --suffix=1 | grep Test:
done
done
for SRC_LANG in en de fr; do
for LANG in en de es fr it ru zh; do
echo $LANG from $SRC_LANG
python ../../source/classify.py embed1000/mldoc.${SRC_LANG}-${SRC_LANG}.h5 ~/workspace/ulmfit-multilingual/data/mldoc/${LANG}-10 --suffix=1 | grep Test:
done
done
en from en
| Test: 91.48% | classes: 23.77 24.90 26.25 25.07
de from en
@@ -757,9 +893,6 @@ ru from fr
zh from fr
| Test: 79.40% | classes: 33.60 31.12 9.07 26.20
```
### No Unfreeze
#### one epoch
```