Piotr Czapla
0085c18ae0
The issue was that the code assumed that empty lines have space followed by a new line, which isn't the case for datasets generated by our scripts.
Todo
- Update these docs
Getting Started
Download and Extract the Wikipedia corpus
In Linux, you can do all the following steps automatically with prepare_wiki.sh
Manual Instructions
We use the WikiExtractor.py. It is a Python script that extracts and cleans text from a Wikipedia database dump.
At the end of this step, you should have the following directory structure inside ulmfit:
|- data
|- wiki
|- wiki_dumps
|- wiki_extr
|- wikiextractor
The extracted data should be in the folder wiki_extr -> language name e.g.en (english), fr (french) hi (hindi) and so on.
Create and Post Process WikiText
Create and Post-Process
If you used the automated shell script from previous step, this might look something like
python create_wikitext.py -i data/wiki_extr/hi -o data/wiki/hi -l hi
for hindi (unicode: 'hi')
This should create two splits of your Wikimedia Dumps: a small and large one.
Then, use the postprocess_wikitext.py script to finish post processing. This processes numbers, builds a vocab, and limits the vocabulary size. This might look following for Hindi (hi)
python postprocess_wikitext.py data/wiki/hi-2 hi
python postprocess_wikitext.py data/wiki/hi-100 hi