mirror of
https://github.com/wassname/multifit.git
synced 2026-09-09 11:27:26 +08:00
96 lines
7.0 KiB
Markdown
96 lines
7.0 KiB
Markdown
|
|
## Supervised classification results on MLDoc
|
|
| Model | en | de | es | fr | it | ja | ru | zh |
|
|
|----------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|------------|
|
|
|LASER 0 shot | 80.75 | 87.03 | 82.60 | 82.83 | 73.25 | 60.95 | 68.83 | 72.90 |
|
|
|LASER | 90.73 | 92.70 | 88.75 | 90.80 | 85.93 | 85.15 | 84.65 | 88.98 |
|
|
|MultiCCA | 92.2 | 93.70 | 94.45 | 92.05 | 85.55 | 85.35 | 85.65 | 87.30 |
|
|
|Bert Multi | 93.23 | 94.0 | 95.15 | 93.20 | 85.82 | 87.48 | 86.85 | 90.72 |
|
|
|ULMFiT L30k-100 | | 91.35 | 83.32 | 88.77 | 77.99 | 71.12 | 72.20 | |
|
|
|ULMFiT L30k | | 95.4 | 95.15 | 93.67 | 88.42 | 89.20 | **87.27** | 90.20 |
|
|
|ULMFiT sp-fixed | | **95.6** | 94.80 | 94.20 | 88.52 | 88.72 | 86.85 | 90.47 |
|
|
|ULMFIT Q15k 1cyc| 94.62 | **95.65** | 95.15 | **94.42** | 89.92 | 89.60 | | 90.78/89.82 |
|
|
|ULMFIT Q15k 1c l| **94.99** | | 95.64 | 94.34 | **90.32** | 89.67 | 87.67^ | **92.22** |
|
|
|ULMFIT Q15k 1cfl| **95.55** | **96.10** | 95.82 | 94.80 | **90.04** | 89.87 | 87.17 | **91.90** |
|
|
|ULMFIT L30k 1cyc| | **95.85** | **96.32** | **94.82** | 89.87 | **90.45** | **87.94** | 92.02/91.64 |
|
|
|
|
- L30k - LSTM sp30k trained using gradual unfreezing
|
|
- L30k-100 - --||-- **on 100 samples**
|
|
- ULMFiT sp-fixed - --||-- with fixed tokenization
|
|
- Q15k 1cyc - QRNN sp15k trained using 1cycle learning rate schedule
|
|
- L30k 1cyc - LSTM sp30k trained using 1cycle learning rate schedule
|
|
- We checked LSTM on sp15k on DE and got 95.53% accuracy which is comparable to QRNN sp15k
|
|
- ^ - 16 epochs qrnn_nl4sl-bs500
|
|
|
|
## Zero shot approaches - LSTM
|
|
|
|
| Model | de | es | fr | it | ru | zh |
|
|
|----------------------|------------|------------|-----------|-----------|-----------|-----------|
|
|
| LASER-de | | 81.40 | 81.50 | 74.53 | 64.58 | 73.20 |
|
|
| LASER-fr | 88.75 | 80.12 | | 72.58 | 67.35 | 79.40 |
|
|
| LASER-en | 87.65 | 75.48 | 84.00 | 71.18 | 66.58 | 76.65 |
|
|
| | | | | | | |
|
|
| ULMFiT on LASER-de | | **85.50** | 87.37 | **78.75** | 66.95 | 72.32 |
|
|
| ULMFiT on LASER-fr | 92.22 | 81.00 | | 76.88 | 68.33 | **84.65** |
|
|
| ULMFiT on LASER-en | **92.95** | 80.50 | **88.78** | 76.20 | **70.05** | 80.45 |
|
|
| | | | | | | |
|
|
| % impr over LASER-de | | 22% | 32% | 17% | 7% | *-3%* |
|
|
| % impr over LASER-fr | 31% | 4% | | 16% | 3% | 25% |
|
|
| % impr over LASER-en | 43% | 20% | 30% | 17% | 10% | 16% |
|
|
| ULMFiT 100 for comp. | 91.35 | 83.32 | 88.77 | 77.99 | 71.12 | |
|
|
| | | | | | | |
|
|
| Bert Multilingual-EN | 74.50 | 61.85 | 69.77 | 57.73 | 51.10 | 64.08 |
|
|
|
|
|
|
### From Laser trained on French data
|
|
| Model Name | de | es | fr | it | ru | zh |
|
|
|---------------------------|-------|-------|----|-------|-------|-------|
|
|
| LASER fr 10k | 91.65 | 81.05 | | 75.08 | 70.73 | 76.33 |
|
|
| LASER fr 1k | 88.75 | 80.12 | | 72.58 | 67.35 | 79.4 |
|
|
| ULMFiT 10k on LASER-fr10k | 94.48 | 84.10 | | 77.93 | 72.87 | 84.53 |
|
|
| ULMFiT 10k on LASER-fr1k | 92.30 | 82.10 | | 75.52 | 69.52 | 85.55 |
|
|
| ULMFiT 1k on LASER-fr1k | 92.22 | 81.00 | | 76.88 | 68.33 | 84.65 |
|
|
| | | | | | | |
|
|
| Impr 10k over 10k | 34% | 16% | | 11% | 7% | 35% |
|
|
| Impr 10k over 1k | 32% | 10% | | 11% | 7% | 30% |
|
|
| Impr 1k over 1k | 31% | 4% | | 16% | 3% | 25% |
|
|
|
|
### From Laser trained on German data
|
|
| Model Name | de | es | fr | it | ru | zh |
|
|
|---------------------------|----|-------|-------|-------|-------|-------|
|
|
| LASER de 10k | | 83.5 | 82.85 | 76.6 | 68.8 | 73.12 |
|
|
| LASER de 1k | | 81.4 | 81.5 | 74.53 | 64.58 | 73.2 |
|
|
| ULMFiT 10k on LASER-de10k | | 86.92 | 87.17 | 79.35 | 70.15 | 78.15 |
|
|
| ULMFiT 10k on LASER-de1k | | 84.65 | 87.48 | 78.70 | 67.65 | 77.50 |
|
|
| ULMFiT 1k on LASER-de1k | | 85.5 | 87.37 | 78.75 | 66.95 | 72.32 |
|
|
| | | | | | | |
|
|
| Impr 10k over 10k | | 21% | 25% | 12% | 4% | 19% |
|
|
| Impr 10k over 1k | | 17% | 32% | 16% | 9% | 16% |
|
|
| Impr 1k over 1k | | 22% | 32% | 17% | 7% | -3% |
|
|
|
|
### From Laser trained on English data
|
|
| Model Name | de | es | fr | it | ru | zh |
|
|
|---------------------------|-------|-------|-------|-------|-------|-------|
|
|
| LASER en 10k | 87.43 | 77.38 | 78.7 | 72.53 | 67.7 | 75.18 |
|
|
| LASER en 1k | 87.65 | 75.48 | 84 | 71.18 | 66.58 | 76.65 |
|
|
| ULMFiT 10k on LASER-en10k | 92.05 | 80.05 | 86.95 | 76.65 | 70.57 | 80.85 |
|
|
| ULMFiT 10k on LASER-en1k | 91.80 | 80.10 | 88.67 | 77.32 | 70.25 | 82.73 |
|
|
| ULMFiT 1k on LASER-en1k | 92.95 | 80.50 | 88.78 | 76.20 | 70.05 | 80.45 |
|
|
| | | | | | | |
|
|
| Impr 10k over 10k | 37% | 12% | 39% | 15% | 9% | 23% |
|
|
| Impr 10k over 1k | 34% | 19% | 29% | 21% | 11% | 26% |
|
|
| Impr 1k over 1k | 43% | 20% | 30% | 17% | 10% | 16% |
|
|
| ULMFiT qrnn on 1k LSRen1k | 91.32 | 78.92 | 89.45 | 75.99 | | 82.45 |
|
|
| ULMFiT qrnn on 10k LSRen1k| 91.90 | 78.79 | 88.47 | 76.05 | | |
|
|
|
|
|
|
## Noise resistance
|
|
|
|
| Model | en | de | es | fr | it | ja | ru | zh |
|
|
|---------------------------------|------------|-----------|-----------|-----------|-----------|-----------|-----------|------------|
|
|
|LASER 0 shot | 80.75 (en) | 87.03 (fr)| 82.60 (it)| 82.83 (de)| 73.25 (de)| 60.95 (en)| 68.83 (it)| 72.90 (de) |
|
|
|ULMFiT | | **95.4** | **95.15** | **93.67** | **88.42** | **89.20** | **87.27** | |
|
|
| % of noise | 20% | 13% | 18% | 18% | 27% | 40% | 32% | 28% |
|
|
|ULMFiT trained on 1k noisy exmp. | | 94.49 | 93.12 | 90.49 | 83.72 | 74.72 | 75.67 | |
|
|
|