Piotr Czapla cd47b3b5dc Fix use_moses=True for mldoc so that it is identical to wiki with uses_moses=False ...

The issue was that Moses was executed after pre_rules when use_moses = True, But when data set was pre tokenized with Moses (use_moses=False) the pre_rules were executed  after.
So our wikipedia had the following processing:
- raw text
- Moses
- pre_rules
- split(' ') # fastai BaseTokenizer
- post_rules
- sentence piece

While mldoc had the following tokenziation
- raw text
- pre_rules
- Moses
- post_rules
- sentence piece

After fix I've retrained the classfiers (without finetuning) and I haven't notice huge changes in the performance. 4 languages received slight improvment 4 got a slight decrease in performance.

2019-02-14 22:35:20 +01:00

..

__init__.py

Updated README.md

2018-11-08 21:05:48 +01:00

data.py

Fix BiLM training after update to newest fastai

2018-12-31 12:13:53 +01:00

learner.py

fix imports in learner (fastai adpatation)

2018-12-27 15:15:09 +01:00

models.py

Fix BiLM training after update to newest fastai

2018-12-31 12:13:53 +01:00

utils.py

Fix use_moses=True for mldoc so that it is identical to wiki with uses_moses=False

2019-02-14 22:35:20 +01:00