multifit/results/MLDoc.md

Supervised classification results on MLDoc

Model	en	de	es	fr	it	ja	ru	zh
LASER	90.73	92.70	88.75	90.80	85.93	85.15	84.65	88.98
MultiCCA	92.2	93.70	94.45	92.05	85.55	85.35	85.65	87.30
ULMFiT		95.4	95.15	93.67	88.42	89.20	87.27	90.20
ULMFiT sp-fixed		95.6	94.80	94.20	88.52	88.72	86.85	90.47
ULMFiT 100		91.35	83.32	88.77	77.99	71.12	72.20
Bert Multi	93.23%	94.0%	95.15	93.20	85.82	87.48	86.85	90.72

^ - sp60k lstm nl 4

Zero shot approaches

Model	de	es	fr	it	ru	zh
LASER-de		81.40	81.50	74.53	64.58	73.20
LASER-fr	88.75	80.12		72.58	67.35	79.40
LASER-en	87.65	75.48	84.00	71.18	66.58	76.65
ULMFiT on LASER-de		85.50	87.37	78.75	66.95	72.32
ULMFiT on LASER-fr	92.22	81.00		76.88	68.33	84.65
ULMFiT on LASER-en	92.95	80.50	88.78	76.20	70.05	80.45
% impr over LASER-de		22%	32%	17%	7%	-3%
% impr over LASER-fr	31%	4%		16%	3%	25%
% impr over LASER-en	43%	20%	30%	17%	10%	16%
ULMFiT 100 for comp.	91.35	83.32	88.77	77.99	71.12
Bert Multilingual-EN	74.50	61.85	69.77	57.73	51.10	64.08

All ULMFiT examples above were trained on 1k training data generated by a LASER classification model

Noise resistance

Model	en	de	es	fr	it	ja	ru	zh
LASER 0 shot	80.75 (en)	87.03 (fr)	82.60 (it)	82.83 (de)	73.25 (de)	60.95 (en)	68.83 (it)	72.90 (de)
ULMFiT		95.4	95.15	93.67	88.42	89.20	87.27
% of noise	20%	13%	18%	18%	27%	40%	32%	28%
ULMFiT trained on 1k noisy exmp.		94.49	93.12	90.49	83.72	74.72	75.67