From 0dda4b4c2c92f0d1248916db07fde669393bec1e Mon Sep 17 00:00:00 2001 From: Piotr Czapla Date: Sun, 17 Feb 2019 19:03:02 +0100 Subject: [PATCH] Zeroshot MLDoc results for ulmfit trained on 10k examples --- results/MLDoc.md | 52 ++++++++- results/logs/common.md | 228 +++++++++++++++++++++++++++++++++++++++ results/logs/zeroshot.md | 153 ++++++++++++++++++++++++-- 3 files changed, 418 insertions(+), 15 deletions(-) diff --git a/results/MLDoc.md b/results/MLDoc.md index 84bf1e8..c39f213 100644 --- a/results/MLDoc.md +++ b/results/MLDoc.md @@ -2,14 +2,15 @@ ## Supervised classification results on MLDoc | Model | en | de | es | fr | it | ja | ru | zh | |----------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|------------| +|LASER 0 shot | 80.75 | 87.03 | 82.60 | 82.83 | 73.25 | 60.95 | 68.83 | 72.90 | |LASER | 90.73 | 92.70 | 88.75 | 90.80 | 85.93 | 85.15 | 84.65 | 88.98 | |MultiCCA | 92.2 | 93.70 | 94.45 | 92.05 | 85.55 | 85.35 | 85.65 | 87.30 | -|ULMFiT | | **95.4** | **95.15** | **93.67** | **88.42** | **89.20** | **87.27** | 90.20 | -|ULMFiT sp-fixed | | 95.6 | 94.80 | 94.20 | 88.52 | 88.72 | 86.85 | 90.47 | -|ULMFiT 100 | | 91.35 | 83.32 | 88.77 | 77.99 | 71.12 | 72.20 | | -|Bert Multi | 93.23% | 94.0% | **95.15** | 93.20 | 85.82 | 87.48 | 86.85 | **90.72** | +|ULMFiT 100 | | 91.35 | 83.32 | 88.77 | 77.99 | 71.12 | 72.20 | | +|ULMFiT Zeroshot from Laser | | 94.48 | 86.93 | 88.78 | 79.35 | | 72.88 | 85.55 | +|ULMFiT | | 95.4 | **95.15** | 93.67 | 88.42 | **89.20** | **87.27** | 90.20 | +|ULMFiT sp-fixed | | **95.6** | 94.80 | **94.20** | **88.52** | 88.72 | 86.85 | 90.47 | +|Bert Multi | 93.23% | 94.0% | **95.15** | 93.20 | 85.82 | 87.48 | 86.85 | **90.72** | -^ - sp60k lstm nl 4 ## Zero shot approaches @@ -30,6 +31,47 @@ | | | | | | | | | Bert Multilingual-EN | 74.50 | 61.85 | 69.77 | 57.73 | 51.10 | 64.08 | + +### From Laser trained on French data +| Model Name | de | es | fr | it | ru | zh | +|---------------------------|-------|-------|----|-------|-------|-------| +| LASER fr 10k | 91.65 | 81.05 | | 75.08 | 70.73 | 76.33 | +| LASER fr 1k | 88.75 | 80.12 | | 72.58 | 67.35 | 79.4 | +| ULMFiT 10k on LASER-fr10k | 94.48 | 84.10 | | 77.93 | 72.87 | 84.53 | +| ULMFiT 10k on LASER-fr1k | 92.30 | 82.10 | | 75.52 | 69.52 | 85.55 | +| ULMFiT 1k on LASER-fr1k | 92.22 | 81.00 | | 76.88 | 68.33 | 84.65 | +| | | | | | | | +| Impr 10k over 10k | 34% | 16% | | 11% | 7% | 35% | +| Impr 10k over 1k | 32% | 10% | | 11% | 7% | 30% | +| Impr 1k over 1k | 31% | 4% | | 16% | 3% | 25% | + +### From Laser trained on German data +| Model Name | de | es | fr | it | ru | zh | +|---------------------------|----|-------|-------|-------|-------|-------| +| LASER de 10k | | 83.5 | 82.85 | 76.6 | 68.8 | 73.12 | +| LASER de 1k | | 81.4 | 81.5 | 74.53 | 64.58 | 73.2 | +| ULMFiT 10k on LASER-de10k | | 86.92 | 87.17 | 79.35 | 70.15 | 78.15 | +| ULMFiT 10k on LASER-de1k | | 84.65 | 87.48 | 78.70 | 67.65 | 77.50 | +| ULMFiT 1k on LASER-de1k | | 85.5 | 87.37 | 78.75 | 66.95 | 72.32 | +| | | | | | | | +| Impr 10k over 10k | | 21% | 25% | 12% | 4% | 19% | +| Impr 10k over 1k | | 17% | 32% | 16% | 9% | 16% | +| Impr 1k over 1k | | 22% | 32% | 17% | 7% | -3% | + +### From Laser trained on English data +| Model Name | de | es | fr | it | ru | zh | +|---------------------------|-------|-------|-------|-------|-------|-------| +| LASER en 10k | 87.43 | 77.38 | 78.7 | 72.53 | 67.7 | 75.18 | +| LASER en 1k | 87.65 | 75.48 | 84 | 71.18 | 66.58 | 76.65 | +| ULMFiT 10k on LASER-en10k | 92.05 | 80.05 | 86.95 | 76.65 | 70.57 | 80.85 | +| ULMFiT 10k on LASER-en1k | 91.80 | 80.10 | 88.67 | 77.32 | 70.25 | 82.73 | +| ULMFiT 1k on LASER-en1k | 92.95 | 80.50 | 88.78 | 76.20 | 70.05 | 80.45 | +| | | | | | | | +| Impr 10k over 10k | 37% | 12% | 39% | 15% | 9% | 23% | +| Impr 10k over 1k | 34% | 19% | 29% | 21% | 11% | 26% | +| Impr 1k over 1k | 43% | 20% | 30% | 17% | 10% | 16% | + + All ULMFiT examples above were trained on 1k training data generated by a LASER classification model ## Noise resistance diff --git a/results/logs/common.md b/results/logs/common.md index 9142b2b..2f84604 100644 --- a/results/logs/common.md +++ b/results/logs/common.md @@ -53,7 +53,235 @@ python -m ulmfit eval --glob="mldoc/${lang}-1/models/sp30k/lstm_nl4.m" --name nl +## Fix the sentence piece tokenizer +``` +python -m ulmfit eval --glob="mldoc/*-1/models/bsp30k/lstm_nl4.m" --name nl4 --cuda-id=0 +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/de.dev.csv +Running tokenization... +Saving tokenized: cls.trn 13500, cls.val 1500 +Running tokenization... +Saving tokenized: cls.trn 1000, cls.val 1000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁.', '▁,', '▁der', '▁die', '▁und', '▁in', 'en', "▁&'", 's', '-'] +Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/bsp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/bsp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15] +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4.m/info.json +Starting classifier training +epoch train_loss valid_loss accuracy +1 0.480367 0.263987 0.930000 +epoch train_loss valid_loss accuracy +1 0.340463 0.209449 0.940000 +epoch train_loss valid_loss accuracy +1 0.235290 0.214566 0.952000 +epoch train_loss valid_loss accuracy +1 0.169588 0.217762 0.952000 +2 0.175439 0.215570 0.946000 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4.m +Loss and accuracy using (cls_best): [0.15467079, tensor(0.9563)] +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv +Running tokenization... +Saving tokenized: cls.trn 13013, cls.val 1445 +Running tokenization... +Saving tokenized: cls.trn 1000, cls.val 1000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que'] +Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/bsp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/bsp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15] +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4.m/info.json +Starting classifier training +epoch train_loss valid_loss accuracy +1 0.609305 0.343920 0.914000 +epoch train_loss valid_loss accuracy +1 0.380833 0.204708 0.947000 +epoch train_loss valid_loss accuracy +1 0.311563 0.210382 0.943000 +epoch train_loss valid_loss accuracy +1 0.266655 0.192309 0.955000 +2 0.235553 0.183249 0.953000 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4.m +Loss and accuracy using (cls_best): [0.19250762, tensor(0.9427)] +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k/lstm_nl4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/fr.dev.csv +Running tokenization... +Saving tokenized: cls.trn 13500, cls.val 1500 +Running tokenization... +Saving tokenized: cls.trn 1000, cls.val 1000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁.', "'", 's', '▁la', '▁le', '▁et', '▁l', '▁à'] +Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/bsp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/bsp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15] +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k/lstm_nl4.m/info.json +Starting classifier training +epoch train_loss valid_loss accuracy +1 0.527443 0.342664 0.901000 +epoch train_loss valid_loss accuracy +1 0.359752 0.203693 0.936000 +epoch train_loss valid_loss accuracy +1 0.272514 0.188933 0.938000 +epoch train_loss valid_loss accuracy +1 0.191520 0.183208 0.938000 +2 0.201412 0.178796 0.942000 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k/lstm_nl4.m +Loss and accuracy using (cls_best): [0.18170285, tensor(0.9420)] +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k/lstm_nl4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/it.dev.csv +Running tokenization... +Saving tokenized: cls.trn 13500, cls.val 1500 +Running tokenization... +Saving tokenized: cls.trn 1000, cls.val 1000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁di', "▁&'", "'", '▁e', '▁il', '▁la', 'e', '▁in'] +Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/bsp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/bsp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15] +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k/lstm_nl4.m/info.json +Starting classifier training +epoch train_loss valid_loss accuracy +1 0.754286 0.599928 0.783000 +epoch train_loss valid_loss accuracy +1 0.501399 0.379078 0.851000 +epoch train_loss valid_loss accuracy +1 0.408768 0.345188 0.867000 +epoch train_loss valid_loss accuracy +1 0.324877 0.335110 0.872000 +2 0.291118 0.336596 0.879000 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k/lstm_nl4.m +Loss and accuracy using (cls_best): [0.33244577, tensor(0.8852)] +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv +Tokenized data loaded, lm.trn 13500, lm.val 1500 +Tokenized data loaded, cls.trn 1000, cls.val 1000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁、', '▁の', '▁。', '▁に', '▁を', '▁は', '▁年', '▁が', '▁)', '▁('] +Loss and accuracy using (cls_last): [0.330675, tensor(0.8873)] +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k/lstm_nl4.m +Evaluating previously trained model +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv +Tokenized data loaded, lm.trn 9195, lm.val 1021 +Tokenized data loaded, cls.trn 1000, cls.val 1000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁в', 'а', '▁и', 'е', 'и', 'й', '▁на', 'х'] +Loss and accuracy using (cls_last): [0.3987146, tensor(0.8680)] +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv +Running tokenization... +Saving tokenized: cls.trn 13500, cls.val 1500 +Running tokenization... +Saving tokenized: cls.trn 1000, cls.val 1000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁中', '▁人', '▁是'] +Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/bsp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/bsp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15] +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4.m/info.json +Starting classifier training +epoch train_loss valid_loss accuracy +1 0.612067 0.487164 0.860000 +epoch train_loss valid_loss accuracy +1 0.441519 0.343330 0.886000 +epoch train_loss valid_loss accuracy +1 0.384575 0.318655 0.897000 +epoch train_loss valid_loss accuracy +1 0.298987 0.305157 0.899000 +2 0.293190 0.312572 0.901000 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4.m +Loss and accuracy using (cls_best): [0.28432375, tensor(0.9047)] +OrderedDict([('data/mldoc/de-1/models/sp30k/lstm_nl4.m', 0.956250011920929), + ('data/mldoc/es-1/models/sp30k/lstm_nl4.m', 0.9427499771118164), + ('data/mldoc/fr-1/models/sp30k/lstm_nl4.m', 0.9419999718666077), + ('data/mldoc/it-1/models/sp30k/lstm_nl4.m', 0.8852499723434448), + ('data/mldoc/ja-1/models/sp30k/lstm_nl4.m', 0.8872500061988831), + ('data/mldoc/ru-1/models/sp30k/lstm_nl4.m', 0.8679999709129333), + ('data/mldoc/zh-1/models/sp30k/lstm_nl4.m', 0.9047499895095825)]) + +--- Additonal run on ru +python -m ulmfit eval --glob="mldoc/ru-1/models/bsp30k/lstm_nl4.m" --name nl4 --cuda-id=0 ✘ 1 +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k/lstm_nl4.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv +Running tokenization... +Saving tokenized: cls.trn 9195, cls.val 1021 +Running tokenization... +Saving tokenized: cls.trn 1000, cls.val 1000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁,', '▁.', '▁в', 'а', '▁и', 'е', 'и', 'й', '▁на', 'х'] +Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/bsp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/bsp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15] +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k/lstm_nl4.m/info.json +Starting classifier training +epoch train_loss valid_loss accuracy +1 0.814752 0.555627 0.808000 +epoch train_loss valid_loss accuracy +1 0.659532 0.427135 0.855000 +epoch train_loss valid_loss accuracy +1 0.508609 0.427321 0.851000 +epoch train_loss valid_loss accuracy +1 0.440108 0.396991 0.872000 +2 0.440024 0.388976 0.866000 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k/lstm_nl4.m +Loss and accuracy using (cls_best): [0.3959306, tensor(0.8685)] +---- +---- + python -m ulmfit eval --glob="mldoc/es-1/models/bsp30k/lstm_nl4.m" --name nl4-2nd --cuda-id=0 +Max vocab: 30000 +Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k +Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4-2nd.m +Training +Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv +Tokenized data loaded, lm.trn 13013, lm.val 1445 +Tokenized data loaded, cls.trn 1000, cls.val 1000 +Size of vocabulary: 30000 +First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que'] +Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/bsp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/bsp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15] +Unknown tokens 0, first 100: [] +/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k +Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4-2nd.m/info.json +Starting classifier training +epoch train_loss valid_loss accuracy +1 0.654984 0.453887 0.818000 +epoch train_loss valid_loss accuracy +1 0.451552 0.220058 0.934000 +epoch train_loss valid_loss accuracy +1 0.323974 0.193342 0.949000 +epoch train_loss valid_loss accuracy +1 0.244755 0.201804 0.945000 +2 0.237175 0.183736 0.953000 +Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4-2nd.m +Loss and accuracy using (cls_best): [0.18296617, tensor(0.9480)] +OrderedDict([('data/mldoc/es-1/models/sp30k/lstm_nl4-2nd.m', + 0.9480000138282776)]) +---- + +``` ### LIMIT LOgs ``` diff --git a/results/logs/zeroshot.md b/results/logs/zeroshot.md index 0eeb0f1..074f859 100644 --- a/results/logs/zeroshot.md +++ b/results/logs/zeroshot.md @@ -1,6 +1,7 @@ ## Laser Perforamnce Accuracy matrix: +### 1k | Train | en | de | es | fr | it | ru | zh | |-------|-------|-------|-------|-------|-------|-------|-------| @@ -12,8 +13,84 @@ Accuracy matrix: | ru: | 72.38 | 81.65 | 65.73 | 71.30 | 63.33 | 85.45 | 59.58 | | zh: | 74.98 | 81.35 | 72.20 | 73.28 | 70.08 | 66.23 | 88.30 | +### 10k + +| Train | en | de | es | fr | it | ru | zh | +|-------|-------|-------|-------|-------|-------|-------|-------| +| en: | 92.70 | 87.43 | 77.38 | 78.70 | 72.53 | 67.70 | 75.18 | +| de: | 81.60 | 95.40 | 83.50 | 82.85 | 76.60 | 68.80 | 73.12 | +| es: | 73.48 | 87.13 | 94.40 | 81.63 | 76.70 | 58.65 | 72.98 | +| fr: | 85.08 | 91.65 | 81.05 | 93.65 | 75.08 | 70.73 | 76.33 | +| it: | 76.75 | 86.68 | 82.55 | 82.65 | 87.80 | 65.90 | 73.35 | +| ru: | 75.23 | 81.88 | 66.83 | 68.60 | 67.38 | 87.00 | 62.68 | +| zh: | 76.05 | 82.05 | 68.40 | 77.38 | 68.45 | 66.88 | 90.38 | -## Evaluation of Laser Performance + + +### laser 1k +#### Building dataset +```bash +for SRC_LANG in en de fr; do ✘ 130 + for LANG in en de es fr it ru zh; do + echo $LANG from $SRC_LANG + python ../../source/classify.py embed10000/mldoc.${SRC_LANG}-${SRC_LANG}.h5 ~/workspace/ulmfit-multilingual/data/mldoc/${LANG}-10 --suffix=10 | grep Test: + done +done +``` + +``` +for SRC_LANG in en de fr; do ✘ 130 + for LANG in en de es fr it ru zh; do + echo $LANG from $SRC_LANG + python ../../source/classify.py embed1000/mldoc.${SRC_LANG}-${SRC_LANG}.h5 ~/workspace/ulmfit-multilingual/data/mldoc/${LANG}-1 | grep Test: + done +done + +en from en + | Test: 91.48% | classes: 23.77 24.90 26.25 25.07 +de from en + | Test: 87.65% | classes: 21.98 24.45 27.65 25.93 +es from en + | Test: 75.48% | classes: 21.60 15.82 22.10 40.48 +fr from en + | Test: 84.00% | classes: 23.18 29.12 27.90 19.80 +it from en + | Test: 71.18% | classes: 23.65 22.88 25.68 27.80 +ru from en + | Test: 66.58% | classes: 29.48 13.78 34.52 22.23 +zh from en + | Test: 76.65% | classes: 30.25 31.30 13.93 24.52 +en from de + | Test: 78.23% | classes: 31.80 17.73 30.15 20.32 +de from de + | Test: 93.50% | classes: 24.45 25.45 26.00 24.10 +es from de + | Test: 81.40% | classes: 24.15 25.77 20.12 29.95 +fr from de + | Test: 81.50% | classes: 25.52 29.45 27.45 17.57 +it from de + | Test: 74.53% | classes: 24.70 27.25 22.43 25.62 +ru from de + | Test: 64.58% | classes: 45.62 9.12 26.73 18.52 +zh from de + | Test: 73.20% | classes: 31.20 43.38 7.60 17.82 +en from fr + | Test: 81.30% | classes: 28.95 18.02 24.98 28.05 +de from fr + | Test: 88.75% | classes: 24.00 23.75 24.85 27.40 +es from fr + | Test: 80.12% | classes: 24.50 14.82 18.40 42.27 +fr from fr + | Test: 90.85% | classes: 24.50 24.75 24.68 26.07 +it from fr + | Test: 72.58% | classes: 25.45 24.10 17.50 32.95 +ru from fr + | Test: 67.35% | classes: 47.15 13.62 16.68 22.55 +zh from fr + | Test: 79.40% | classes: 33.60 31.12 9.07 26.20 +``` + +#### Evaluation of Laser 1k Performance ``` python -m ulmfit eval --glob="mldoc/*-1/models/sp60k/lstm_nl4.m" --dataset_template="{}-laser-*" --name nl4 --cuda-id=0 ✘ 130 Max vocab: 60000 @@ -703,17 +780,76 @@ OrderedDict([('data/mldoc/de-1-laser-de/models/sp30k/lstm_nl4.m', ``` - -### Building dataset - +### Laser 10k +### Building dataset 10k ``` -for SRC_LANG in en de fr; do ✘ 130 + for SRC_LANG in en de fr; do ✘ 130 for LANG in en de es fr it ru zh; do echo $LANG from $SRC_LANG - python ../../source/classify.py embed/mldoc.${SRC_LANG}-${SRC_LANG}.h5 ~/workspace/ulmfit-multilingual/data/mldoc/${LANG}-1 | grep Test: + python ../../source/classify.py embed10000/mldoc.${SRC_LANG}-${SRC_LANG}.h5 ~/workspace/ulmfit-multilingual/data/mldoc/${LANG}-10 --suffix=10 | grep Test: done done - +zsh: command not found: ✘ +en from en + | Test: 92.70% | classes: 24.65 25.52 26.20 23.62 +de from en + | Test: 87.43% | classes: 20.05 25.55 28.88 25.52 +es from en + | Test: 77.38% | classes: 24.70 15.40 22.27 37.62 +fr from en + | Test: 78.70% | classes: 17.65 29.75 34.60 18.00 +it from en + | Test: 72.53% | classes: 20.75 24.60 28.52 26.12 +ru from en + | Test: 67.70% | classes: 33.40 17.12 29.35 20.12 +zh from en + | Test: 75.18% | classes: 27.70 39.00 12.07 21.23 +zsh: command not found: ✘ +en from de + | Test: 81.60% | classes: 31.62 23.82 24.93 19.62 +de from de + | Test: 95.40% | classes: 24.57 26.00 25.62 23.80 +es from de + | Test: 83.50% | classes: 28.73 22.60 18.65 30.02 +fr from de + | Test: 82.85% | classes: 27.52 29.23 25.23 18.02 +it from de + | Test: 76.60% | classes: 26.52 25.62 21.43 26.43 +ru from de + | Test: 68.80% | classes: 48.48 13.22 19.43 18.88 +zh from de + | Test: 73.12% | classes: 33.12 40.10 10.53 16.25 +zsh: command not found: ✘ +en from fr + | Test: 85.08% | classes: 25.43 23.43 25.43 25.73 +de from fr + | Test: 91.65% | classes: 23.15 27.10 25.00 24.75 +es from fr + | Test: 81.05% | classes: 24.18 17.68 18.93 39.23 +fr from fr + | Test: 93.65% | classes: 24.25 25.10 25.80 24.85 +it from fr + | Test: 75.08% | classes: 24.07 26.85 18.25 30.82 +ru from fr + | Test: 70.73% | classes: 43.17 19.15 16.57 21.10 +zh from fr + | Test: 76.33% | classes: 34.23 34.05 10.95 20.77 + ``` + 1 10 +### Building dataset 1k +``` + for SRC_LANG in en de fr; do + for LANG in en de es fr it ru zh; do + echo $LANG from $SRC_LANG + python ../../source/classify.py embed1000/mldoc.${SRC_LANG}-${SRC_LANG}.h5 ~/workspace/ulmfit-multilingual/data/mldoc/${LANG}-10 --suffix=1 | grep Test: + done + done + for SRC_LANG in en de fr; do + for LANG in en de es fr it ru zh; do + echo $LANG from $SRC_LANG + python ../../source/classify.py embed1000/mldoc.${SRC_LANG}-${SRC_LANG}.h5 ~/workspace/ulmfit-multilingual/data/mldoc/${LANG}-10 --suffix=1 | grep Test: + done + done en from en | Test: 91.48% | classes: 23.77 24.90 26.25 25.07 de from en @@ -757,9 +893,6 @@ ru from fr zh from fr | Test: 79.40% | classes: 33.60 31.12 9.07 26.20 ``` - - - ### No Unfreeze #### one epoch ```