Piotr Czapla 70c74a1cc5 Refactor tokenization ...

So that we can instantiate tokenization before we know what dataset we want to use it on. Previously it was tidly copuled.

2019-10-15 04:17:54 +02:00

..

sotabench_setup.sh

Add sotabench scripts

2019-10-15 00:09:18 +02:00

sotabench.py

Refactor tokenization

2019-10-15 04:17:54 +02:00