From 820ff231bd202776d105fc7da0fbfacf6529c0fe Mon Sep 17 00:00:00 2001 From: Nirant Date: Wed, 14 Nov 2018 17:04:47 +0530 Subject: [PATCH 1/2] Add Download and Process Docs --- ulmfit/README.md | 42 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 40 insertions(+), 2 deletions(-) diff --git a/ulmfit/README.md b/ulmfit/README.md index c7d60f8..9b8be42 100644 --- a/ulmfit/README.md +++ b/ulmfit/README.md @@ -1,2 +1,40 @@ -# todo -- [] create new docs \ No newline at end of file +# Todo +- [ ] Update these docs + +Getting Started +--- + +## Download and Extract the Wikipedia corpus + +In Linux, you can do all the following steps automatically with [prepare_wiki.sh](./prepare_wiki.sh) + +**Manual Instructions** + +We use the [WikiExtractor.py](http://medialab.di.unipi.it/wiki/Wikipedia_Extractor). It is a Python script that extracts and cleans text from a [Wikipedia database dump](http://download.wikimedia.org/). + +At the end of this step, you should have the following directory structure inside ulmfit: +```bash + +|- data + |- wiki + |- wiki_dumps + |- wiki_extr +|- wikiextractor +``` +The extracted data should be in the folder `wiki_extr` -> language name e.g.`en` (english), `fr` (french) `hi` (hindi) and so on. + +## Create and Post Process WikiText +Use the Python script [create_wikitext.py](./create_wikitext.py) to process the extracted Wikipedia documents. + +If you used the automated shell script from previous step, this might look something like +```bash +python create_wikitext.py -i data/wiki_extr/hi -o data/hindi -l hi +``` +for hindi (unicode: 'hi') + +This should create two splits of your Wikimedia Dumps: a small and large one. + +_**Then**_, use the [postprocess_wikitext.py](./postprocess_wikitext.py) script to finish post processing. This processes numbers, builds a vocab, and limits the vocabulary size. This might look following for Hindi (`hi`) +```bash +python postprocess_wikitext.py -i data/hindi -l hi +``` From 5f845724d0d25dffd7d7e1f95464216fd7912701 Mon Sep 17 00:00:00 2001 From: Nirant Date: Wed, 14 Nov 2018 17:13:40 +0530 Subject: [PATCH 2/2] Add dependencies --- ulmfit/README.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/ulmfit/README.md b/ulmfit/README.md index 9b8be42..d0ceb69 100644 --- a/ulmfit/README.md +++ b/ulmfit/README.md @@ -24,8 +24,18 @@ At the end of this step, you should have the following directory structure insid The extracted data should be in the folder `wiki_extr` -> language name e.g.`en` (english), `fr` (french) `hi` (hindi) and so on. ## Create and Post Process WikiText + +### Get the Dependencies + +**Python Fire**: To install Python Fire with pip, run: `pip install fire` + +To install Python Fire with conda, run: `conda install fire -c conda-forge` + +**Moses Tokenizer**: To install Moses Tokenizer: `pip install -U sacremoses` + Use the Python script [create_wikitext.py](./create_wikitext.py) to process the extracted Wikipedia documents. +### Create and Post-Process If you used the automated shell script from previous step, this might look something like ```bash python create_wikitext.py -i data/wiki_extr/hi -o data/hindi -l hi