Refactor tokenization

So that we can instantiate tokenization before we know what dataset we want to use it on. Previously it was tidly copuled.
This commit is contained in:
Piotr Czapla
2019-10-15 04:17:54 +02:00
parent 77a2780a6b
commit 70c74a1cc5
5 changed files with 147 additions and 82 deletions
+6
View File
@@ -12,3 +12,9 @@ You can evaulate any model with the following command:
```bash
python -m ulmfit load data/mldoc/de-1/models/fsp15k/multfit_fp16 classifier validate data/mldoc/de-1
```
```bash
python -m ulmfit new multifit_fp16_nl3 pretrain-lm train- data/wiki/wikitext-103
```