mirror of
https://github.com/wassname/multifit.git
synced 2026-10-15 13:21:11 +08:00
Piotr Czapla
cd47b3b5dc
The issue was that Moses was executed after pre_rules when use_moses = True, But when data set was pre tokenized with Moses (use_moses=False) the pre_rules were executed after.
So our wikipedia had the following processing:
- raw text
- Moses
- pre_rules
- split(' ') # fastai BaseTokenizer
- post_rules
- sentence piece
While mldoc had the following tokenziation
- raw text
- pre_rules
- Moses
- post_rules
- sentence piece
After fix I've retrained the classfiers (without finetuning) and I haven't notice huge changes in the performance. 4 languages received slight improvment 4 got a slight decrease in performance.
41 lines
3.1 KiB
Markdown
41 lines
3.1 KiB
Markdown
|
|
## Supervised classification results on MLDoc
|
|
| Model | en | de | es | fr | it | ja | ru | zh |
|
|
|----------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|------------|
|
|
|LASER | 90.73 | 92.70 | 88.75 | 90.80 | 85.93 | 85.15 | 84.65 | 88.98 |
|
|
|MultiCCA | 92.2 | 93.70 | 94.45 | 92.05 | 85.55 | 85.35 | 85.65 | 87.30 |
|
|
|ULMFiT | | **95.4** | **95.15** | **93.67** | **88.42** | **89.20** | **87.27** | **90.20** |
|
|
|ULMFiT sp-fixed | | 95.6 | 94.80 | 94.20 | 88.52 | 88.72 | 86.85 | 90.47 |
|
|
|ULMFiT 100 | | 91.35 | 83.32 | 88.77 | 77.99 | 71.12 | 72.20 | |
|
|
|
|
^ - sp60k lstm nl 4
|
|
|
|
## Zero shot approaches
|
|
|
|
| Model | de | es | fr | it | ru | zh |
|
|
|----------------------|------------|------------|-----------|-----------|-----------|-----------|
|
|
| LASER-de | | 81.40 | 81.50 | 74.53 | 64.58 | 73.20 |
|
|
| LASER-fr | 88.75 | 80.12 | | 72.58 | 67.35 | 79.40 |
|
|
| LASER-en | 87.65 | 75.48 | 84.00 | 71.18 | 66.58 | 76.65 |
|
|
| | | | | | | |
|
|
| ULMFiT on LASER-de | | **85.50** | 87.37 | **78.75** | 66.95 | 72.32 |
|
|
| ULMFiT on LASER-fr | 92.22 | 81.00 | | 76.88 | 68.33 | **84.65** |
|
|
| ULMFiT on LASER-en | **92.95** | 80.50 | **88.78** | 76.20 | **70.05** | 80.45 |
|
|
| | | | | | | |
|
|
| % impr over LASER-de | | 22% | 32% | 17% | 7% | *-3%* |
|
|
| % impr over LASER-fr | 31% | 4% | | 16% | 3% | 25% |
|
|
| % impr over LASER-en | 43% | 20% | 30% | 17% | 10% | 16% |
|
|
| ULMFiT 100 for comp. | 91.35 | 83.32 | 88.77 | 77.99 | 71.12 | |
|
|
|
|
All ULMFiT examples above were trained on 1k training data generated by a LASER classification model
|
|
|
|
## Noise resistance
|
|
|
|
| Model | en | de | es | fr | it | ja | ru | zh |
|
|
|---------------------------------|------------|-----------|-----------|-----------|-----------|-----------|-----------|------------|
|
|
|LASER 0 shot | 80.75 (en) | 87.03 (fr)| 82.60 (it)| 82.83 (de)| 73.25 (de)| 60.95 (en)| 68.83 (it)| 72.90 (de) |
|
|
|ULMFiT | | **95.4** | **95.15** | **93.67** | **88.42** | **89.20** | **87.27** | |
|
|
| % of noise | 20% | 13% | 18% | 18% | 27% | 40% | 32% | 28% |
|
|
|ULMFiT trained on 1k noisy exmp. | | 94.49 | 93.12 | 90.49 | 83.72 | 74.72 | 75.67 | |
|
|
|