mirror of
https://github.com/wassname/multifit.git
synced 2026-10-07 11:25:58 +08:00
Piotr Czapla
cd47b3b5dc
The issue was that Moses was executed after pre_rules when use_moses = True, But when data set was pre tokenized with Moses (use_moses=False) the pre_rules were executed after.
So our wikipedia had the following processing:
- raw text
- Moses
- pre_rules
- split(' ') # fastai BaseTokenizer
- post_rules
- sentence piece
While mldoc had the following tokenziation
- raw text
- pre_rules
- Moses
- post_rules
- sentence piece
After fix I've retrained the classfiers (without finetuning) and I haven't notice huge changes in the performance. 4 languages received slight improvment 4 got a slight decrease in performance.
3.1 KiB
3.1 KiB
Supervised classification results on MLDoc
| Model | en | de | es | fr | it | ja | ru | zh |
|---|---|---|---|---|---|---|---|---|
| LASER | 90.73 | 92.70 | 88.75 | 90.80 | 85.93 | 85.15 | 84.65 | 88.98 |
| MultiCCA | 92.2 | 93.70 | 94.45 | 92.05 | 85.55 | 85.35 | 85.65 | 87.30 |
| ULMFiT | 95.4 | 95.15 | 93.67 | 88.42 | 89.20 | 87.27 | 90.20 | |
| ULMFiT sp-fixed | 95.6 | 94.80 | 94.20 | 88.52 | 88.72 | 86.85 | 90.47 | |
| ULMFiT 100 | 91.35 | 83.32 | 88.77 | 77.99 | 71.12 | 72.20 |
^ - sp60k lstm nl 4
Zero shot approaches
| Model | de | es | fr | it | ru | zh |
|---|---|---|---|---|---|---|
| LASER-de | 81.40 | 81.50 | 74.53 | 64.58 | 73.20 | |
| LASER-fr | 88.75 | 80.12 | 72.58 | 67.35 | 79.40 | |
| LASER-en | 87.65 | 75.48 | 84.00 | 71.18 | 66.58 | 76.65 |
| ULMFiT on LASER-de | 85.50 | 87.37 | 78.75 | 66.95 | 72.32 | |
| ULMFiT on LASER-fr | 92.22 | 81.00 | 76.88 | 68.33 | 84.65 | |
| ULMFiT on LASER-en | 92.95 | 80.50 | 88.78 | 76.20 | 70.05 | 80.45 |
| % impr over LASER-de | 22% | 32% | 17% | 7% | -3% | |
| % impr over LASER-fr | 31% | 4% | 16% | 3% | 25% | |
| % impr over LASER-en | 43% | 20% | 30% | 17% | 10% | 16% |
| ULMFiT 100 for comp. | 91.35 | 83.32 | 88.77 | 77.99 | 71.12 |
All ULMFiT examples above were trained on 1k training data generated by a LASER classification model
Noise resistance
| Model | en | de | es | fr | it | ja | ru | zh |
|---|---|---|---|---|---|---|---|---|
| LASER 0 shot | 80.75 (en) | 87.03 (fr) | 82.60 (it) | 82.83 (de) | 73.25 (de) | 60.95 (en) | 68.83 (it) | 72.90 (de) |
| ULMFiT | 95.4 | 95.15 | 93.67 | 88.42 | 89.20 | 87.27 | ||
| % of noise | 20% | 13% | 18% | 18% | 27% | 40% | 32% | 28% |
| ULMFiT trained on 1k noisy exmp. | 94.49 | 93.12 | 90.49 | 83.72 | 74.72 | 75.67 |