diff --git a/ulmfit/README.md b/ulmfit/README.md index 9b8be42..d0ceb69 100644 --- a/ulmfit/README.md +++ b/ulmfit/README.md @@ -24,8 +24,18 @@ At the end of this step, you should have the following directory structure insid The extracted data should be in the folder `wiki_extr` -> language name e.g.`en` (english), `fr` (french) `hi` (hindi) and so on. ## Create and Post Process WikiText + +### Get the Dependencies + +**Python Fire**: To install Python Fire with pip, run: `pip install fire` + +To install Python Fire with conda, run: `conda install fire -c conda-forge` + +**Moses Tokenizer**: To install Moses Tokenizer: `pip install -U sacremoses` + Use the Python script [create_wikitext.py](./create_wikitext.py) to process the extracted Wikipedia documents. +### Create and Post-Process If you used the automated shell script from previous step, this might look something like ```bash python create_wikitext.py -i data/wiki_extr/hi -o data/hindi -l hi