Todo

Update these docs

Getting Started

Download and Extract the Wikipedia corpus

In Linux, you can do all the following steps automatically with prepare_wiki.sh

Manual Instructions

We use the WikiExtractor.py. It is a Python script that extracts and cleans text from a Wikipedia database dump.

At the end of this step, you should have the following directory structure inside ulmfit:


|- data
  |- wiki
  |- wiki_dumps
  |- wiki_extr
|- wikiextractor

The extracted data should be in the folder wiki_extr -> language name e.g.en (english), fr (french) hi (hindi) and so on.

Create and Post Process WikiText

Create and Post-Process

If you used the automated shell script from previous step, this might look something like

python create_wikitext.py -i data/wiki_extr/hi -o data/wiki/hi -l hi

for hindi (unicode: 'hi')

This should create two splits of your Wikimedia Dumps: a small and large one.

Then, use the postprocess_wikitext.py script to finish post processing. This processes numbers, builds a vocab, and limits the vocabulary size. This might look following for Hindi (hi)

python postprocess_wikitext.py data/wiki/hi-2 hi
python postprocess_wikitext.py data/wiki/hi-100 hi