wassname/multifit
1
0
Fork 0
mirror of https://github.com/wassname/multifit.git synced 2026-09-10 12:12:50 +08:00
Code Issues Packages Projects Releases Wiki Activity

Add dependencies

Browse Source
This commit is contained in:
Nirant
2018-11-14 17:13:40 +05:30
committed by GitHub
parent 820ff231bd
commit 5f845724d0
1 changed files with 10 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ulmfit/README.md
+10
View File
@@ -24,8 +24,18 @@ At the end of this step, you should have the following directory structure insid
The extracted data should be in the folder `wiki_extr` -> language name e.g.`en` (english), `fr` (french) `hi` (hindi) and so on.
## Create and Post Process WikiText
### Get the Dependencies
**Python Fire**: To install Python Fire with pip, run: `pip install fire`
To install Python Fire with conda, run: `conda install fire -c conda-forge`
**Moses Tokenizer**: To install Moses Tokenizer: `pip install -U sacremoses`
Use the Python script [create_wikitext.py](./create_wikitext.py) to process the extracted Wikipedia documents.
### Create and Post-Process
If you used the automated shell script from previous step, this might look something like
```bash
python create_wikitext.py -i data/wiki_extr/hi -o data/hindi -l hi