wassname/multifit
1
0
Fork 0
mirror of https://github.com/wassname/multifit.git synced 2026-08-28 12:52:26 +08:00
Code Issues Packages Projects Releases Wiki Activity

Update to the docs

Browse Source
This commit is contained in:
Piotr Czapla
2018-11-16 00:17:41 +01:00
committed by GitHub
parent 87afa54f63
commit 363130bcff
1 changed files with 3 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ulmfit/README.md
+3 -2
View File
@@ -38,7 +38,7 @@ Use the Python script [create_wikitext.py](./create_wikitext.py) to process the
### Create and Post-Process
If you used the automated shell script from previous step, this might look something like
```bash
python create_wikitext.py -i data/wiki_extr/hi -o data/hindi -l hi
python create_wikitext.py -i data/wiki_extr/hi -o data/wiki/hi -l hi
```
for hindi (unicode: 'hi')
@@ -46,5 +46,6 @@ This should create two splits of your Wikimedia Dumps: a small and large one.
_**Then**_, use the [postprocess_wikitext.py](./postprocess_wikitext.py) script to finish post processing. This processes numbers, builds a vocab, and limits the vocabulary size. This might look following for Hindi (`hi`)
```bash
python postprocess_wikitext.py -i data/hindi -l hi
python postprocess_wikitext.py data/wiki/hi-2 hi
python postprocess_wikitext.py data/wiki/hi-100 hi
```