Piotr Czapla 70c74a1cc5 Refactor tokenization ...

So that we can instantiate tokenization before we know what dataset we want to use it on. Previously it was tidly copuled.

2019-10-15 04:17:54 +02:00

..

__init__.py

Refactor tokenization

2019-10-15 04:17:54 +02:00

create_wikitext.py

New bolerplate code

2019-09-07 16:19:09 +02:00

dataset.py

Refactor tokenization

2019-10-15 04:17:54 +02:00

postprocess_wikitext.py

New bolerplate code

2019-09-07 16:19:09 +02:00

utils.py

Remove unused code

2019-09-07 21:17:32 +02:00