diff --git a/ulmfit/README.md b/ulmfit/README.md index c7d60f8..d0ceb69 100644 --- a/ulmfit/README.md +++ b/ulmfit/README.md @@ -1,2 +1,50 @@ -# todo -- [] create new docs \ No newline at end of file +# Todo +- [ ] Update these docs + +Getting Started +--- + +## Download and Extract the Wikipedia corpus + +In Linux, you can do all the following steps automatically with [prepare_wiki.sh](./prepare_wiki.sh) + +**Manual Instructions** + +We use the [WikiExtractor.py](http://medialab.di.unipi.it/wiki/Wikipedia_Extractor). It is a Python script that extracts and cleans text from a [Wikipedia database dump](http://download.wikimedia.org/). + +At the end of this step, you should have the following directory structure inside ulmfit: +```bash + +|- data + |- wiki + |- wiki_dumps + |- wiki_extr +|- wikiextractor +``` +The extracted data should be in the folder `wiki_extr` -> language name e.g.`en` (english), `fr` (french) `hi` (hindi) and so on. + +## Create and Post Process WikiText + +### Get the Dependencies + +**Python Fire**: To install Python Fire with pip, run: `pip install fire` + +To install Python Fire with conda, run: `conda install fire -c conda-forge` + +**Moses Tokenizer**: To install Moses Tokenizer: `pip install -U sacremoses` + +Use the Python script [create_wikitext.py](./create_wikitext.py) to process the extracted Wikipedia documents. + +### Create and Post-Process +If you used the automated shell script from previous step, this might look something like +```bash +python create_wikitext.py -i data/wiki_extr/hi -o data/hindi -l hi +``` +for hindi (unicode: 'hi') + +This should create two splits of your Wikimedia Dumps: a small and large one. + +_**Then**_, use the [postprocess_wikitext.py](./postprocess_wikitext.py) script to finish post processing. This processes numbers, builds a vocab, and limits the vocabulary size. This might look following for Hindi (`hi`) +```bash +python postprocess_wikitext.py -i data/hindi -l hi +```