Piotr Czapla
82d6a30b11
- train on whole articles when tokenizer is sentencepice - add moses tokenizer to get the same tokens on imdb as on wt103 - apply pre / post processing rules to moses tokenzier so that sentencepiece works on correctly preprocessed text (lowercased with html removed) - add alpha implementation on xnli (no tests) - remove vocab adaptation when swiching from wiki to imdb. As otherwise we get 50% of missing words and 93% accuracy on imdb
Todo
- Update these docs
Getting Started
Download and Extract the Wikipedia corpus
In Linux, you can do all the following steps automatically with prepare_wiki.sh
Manual Instructions
We use the WikiExtractor.py. It is a Python script that extracts and cleans text from a Wikipedia database dump.
At the end of this step, you should have the following directory structure inside ulmfit:
|- data
|- wiki
|- wiki_dumps
|- wiki_extr
|- wikiextractor
The extracted data should be in the folder wiki_extr -> language name e.g.en (english), fr (french) hi (hindi) and so on.
Create and Post Process WikiText
Create and Post-Process
If you used the automated shell script from previous step, this might look something like
python create_wikitext.py -i data/wiki_extr/hi -o data/wiki/hi -l hi
for hindi (unicode: 'hi')
This should create two splits of your Wikimedia Dumps: a small and large one.
Then, use the postprocess_wikitext.py script to finish post processing. This processes numbers, builds a vocab, and limits the vocabulary size. This might look following for Hindi (hi)
python postprocess_wikitext.py data/wiki/hi-2 hi
python postprocess_wikitext.py data/wiki/hi-100 hi