mirror of
https://github.com/wassname/multifit.git
synced 2026-09-09 11:27:26 +08:00
42 lines
1.4 KiB
Markdown
42 lines
1.4 KiB
Markdown
# Todo
|
|
- [ ] Update these docs
|
|
|
|
Getting Started
|
|
---
|
|
|
|
## Download and Extract the Wikipedia corpus
|
|
|
|
In Linux, you can do all the following steps automatically with [prepare_wiki.sh](./prepare_wiki.sh)
|
|
|
|
**Manual Instructions**
|
|
|
|
We use the [WikiExtractor.py](http://medialab.di.unipi.it/wiki/Wikipedia_Extractor). It is a Python script that extracts and cleans text from a [Wikipedia database dump](http://download.wikimedia.org/).
|
|
|
|
At the end of this step, you should have the following directory structure inside ulmfit:
|
|
```bash
|
|
|
|
|- data
|
|
|- wiki
|
|
|- wiki_dumps
|
|
|- wiki_extr
|
|
|- wikiextractor
|
|
```
|
|
The extracted data should be in the folder `wiki_extr` -> language name e.g.`en` (english), `fr` (french) `hi` (hindi) and so on.
|
|
|
|
## Create and Post Process WikiText
|
|
|
|
### Create and Post-Process
|
|
If you used the automated shell script from previous step, this might look something like
|
|
```bash
|
|
python create_wikitext.py -i data/wiki_extr/hi -o data/wiki/hi -l hi
|
|
```
|
|
for hindi (unicode: 'hi')
|
|
|
|
This should create two splits of your Wikimedia Dumps: a small and large one.
|
|
|
|
_**Then**_, use the [postprocess_wikitext.py](./postprocess_wikitext.py) script to finish post processing. This processes numbers, builds a vocab, and limits the vocabulary size. This might look following for Hindi (`hi`)
|
|
```bash
|
|
python postprocess_wikitext.py data/wiki/hi-2 hi
|
|
python postprocess_wikitext.py data/wiki/hi-100 hi
|
|
```
|