Add directory structure to readme.

This commit is contained in:
Piotr Czapla
2018-11-15 23:47:48 +01:00
parent 6ba3e68f9e
commit d5d61d31d2
2 changed files with 39 additions and 1 deletions
+38
View File
@@ -1,6 +1,44 @@
# ulmfit-multilingual
Temporary repository used for collaboration on application of for multiple languages.
## data directory strucutre
Directory structure after changes to the way we process wiki dumps.
```
data
├── imdb
│   ├── aclImdb
│   ├── imdb_lm
│   └── tmp
├── wiki
│   ├── de-100
│   │   └── models
│   ├── de-100-unk
│   │   └── models
│   ├── de-2
│   │   └── models
│   ├── de-2-unk
│   │   └── models
│   ├── de-all
│   │   └── models
│   ├── _dumps
│   ├── _extr
│   │   └── de
│   │   ├── AA
│   │   ├── AB
...
│   │   └── CC
│   ├── wikitext-103
│   │   └── models
│   └── wikitext-2
│      └── models
└── xnli
├── XNLI-1.0
└── XNLI-MT-1.0
├── multinli
└── xnli
```
## how to contribute
We have a fork of fastai to propose changes to fastai.text, with a branch for this project:
https://github.com/n-waves/fastai/tree/ulmfit_multilingual
+1 -1
View File
@@ -72,7 +72,7 @@ def prepare_imdb(file_path: str, prepare_lm = False):
if not dir_path.exists():
print(f"Extracting {file_path} to {dir_path}. This may take a long time...")
tgz_file = tarfile.open(file_path)
tgz_file.extractall(path=dir_path.parent) # the aclImdb.tgz has aclImdb dir packed
tgz_file.extractall(path=dir_path.parent) # the aclImdb.tgz has aclImdb dir packed
assert dir_path.exists()
print(f"Extracted to {dir_path}")