Julian Eisenschlos
88d94f38ba
Merge pull request #16 from n-waves/sentencepiece_fixes
...
Sentencepiece Fixes
2018-11-20 17:14:41 -03:00
NAUSICAA\Julian
40a2990322
Add suggested changes to rules system
2018-11-20 09:06:03 -03:00
NAUSICAA\Julian
a36c0518c6
Merge branch 'master' of https://github.com/n-waves/ulmfit-multilingual into sentencepiece_fixes
2018-11-20 08:55:37 -03:00
Aayush
429a3fe4c2
Merge pull request #13 from n-waves/models_path_fix
...
Models path fix, default_rules and sentencepiece support for train_clas
2018-11-20 12:25:25 +05:30
NAUSICAA\Julian
3b5629c0c5
Removing files
2018-11-20 00:46:04 -03:00
NAUSICAA\Julian
0ecc6342a6
Fixes when running sentence piece end to end
2018-11-20 00:38:44 -03:00
NAUSICAA\Julian
a08881180c
Merge branch 'master' of https://github.com/n-waves/ulmfit-multilingual into polyglot-lm
2018-11-19 16:33:44 -03:00
NAUSICAA\Julian
470f984ca4
Script to merge two languages
2018-11-19 16:31:38 -03:00
Aayush
73403bdcb3
fix double EOS in read_imdb
2018-11-19 19:30:11 +05:30
Aayush
6e867811db
Support for running sentencepiece with train_clas
...
Committing into this branch. Only changes are in `read_xnli` and `read_imdb`.
2018-11-19 19:28:19 +05:30
Aayush
5ffbe8ba5c
add vocab_size to sentencepiece
2018-11-19 19:13:55 +05:30
Aayush
69ed7b4169
Added preliminary test scripts for sentencepiece
...
Other minor changes:
- Function renaming: `test_pretrain_lm` -> `test_ulmfit_default_end_to_end`
- Delete test models after completing each test.
2018-11-19 19:11:20 +05:30
Nirant
56a9feec7d
Removed dependency note, use requirements.txt
2018-11-19 12:22:41 +05:30
Nirant K
635fb1c113
Add prepare_xnli.py
2018-11-19 06:50:50 +00:00
NAUSICAA\Julian
6292412eff
Adding scripts to download and format RCV and Webis CLS X-Language datasets
2018-11-18 22:29:07 -03:00
Sebastian
d796610452
Removed deprecated default_rules
2018-11-17 14:48:06 +00:00
Sebastian
7fe059c9a2
Fixed models path for vocabulary
2018-11-17 14:47:27 +00:00
Piotr Czapla
64ad8d8fd9
Merge pull request #8 from n-waves/sentencepiece
...
[WIP] Sub-word tokenization with sentencepiece
2018-11-16 23:35:01 +01:00
Piotr Czapla
36b056a465
Fix issues discovered during execution of end-to-end test.
2018-11-16 23:31:16 +01:00
Piotr Czapla
a21f100ab2
Merge branch 'master' into sentencepiece
2018-11-16 23:23:49 +01:00
Piotr Czapla
45389be413
Use drop_mult properly
2018-11-16 20:02:32 +01:00
Piotr Czapla
23b17da61e
Add end to end test and improve the train_clas params
2018-11-16 19:51:29 +01:00
Piotr Czapla
977a506cfd
Fix bug where LM was always intialized as QRNN despite the paramters.
2018-11-16 19:50:36 +01:00
Aayush
e72b19499c
fix indent
2018-11-16 23:54:48 +05:30
aayush
59c8852b5d
sentencepiece for pretraining
...
modified: fastai_contrib/utils.py
modified: ulmfit/pretrain_lm.py
2018-11-16 23:14:24 +05:30
Aayush
fb2ed98ac6
Merge branch 'master' into sentencepiece
2018-11-16 23:09:06 +05:30
Piotr Czapla
f0e538f621
Add support for QRNN training to train_clas
2018-11-16 17:04:52 +01:00
Piotr Czapla
d4698e095c
Change back temporary folder stucture
...
from wiki/_dumps to wiki_dumps . I'm not sure why I've changed that.
2018-11-16 00:21:14 +01:00
Piotr Czapla
363130bcff
Update to the docs
2018-11-16 00:17:41 +01:00
Piotr Czapla
87afa54f63
Merge pull request #9 from n-waves/classifier-fixes
...
Classifier fixes
2018-11-15 23:48:16 +01:00
Piotr Czapla
d5d61d31d2
Add directory structure to readme.
2018-11-15 23:47:48 +01:00
Piotr Czapla
6ba3e68f9e
Minor tweaks.
2018-11-15 23:41:02 +01:00
Piotr Czapla
08d71dd45e
Add script that fetches and prepres WT-2 WT-103
2018-11-15 23:40:50 +01:00
Piotr Czapla
9a60ef2fbd
Add assertions to train_clas and ability to limit the dataset size
2018-11-15 23:40:10 +01:00
Piotr Czapla
d8b95430b8
Chain all scripts needed to prepare a wiki dump, put .unk to new folder
2018-11-15 23:39:27 +01:00
Piotr Czapla
111fc7e4c3
Add perpare_imdb script
2018-11-15 23:37:15 +01:00
NAUSICAA\Julian
b9c587d9bd
Adding SentencePieceTokenizer
2018-11-15 19:18:42 -03:00
Nirant K
43340d03bc
Fix minor typos in train_clas
2018-11-15 18:04:15 +00:00
Aayush
1cc49e666f
Quick fixes to imports etc.
2018-11-15 23:13:16 +05:30
Aayush
3e8f9f8b5f
[WIP] Sub-word tokenization with sentencepiece
...
For resolving issue #3 .
@eisenjulian let's use this branch. I've written some skeleton code (untested currently) to be used around your tokenizer. You can commit that into fastai_contrib for our purpose.
2018-11-15 23:01:49 +05:30
Nirant K
56fc150eb1
Merge xnli
2018-11-15 15:53:37 +00:00
Nirant K
653f6e96a0
Classifier fails at fit_one_cycle
2018-11-15 15:03:52 +00:00
Nirant K
6622458fe7
Add aclImdb extractor
2018-11-15 15:03:12 +00:00
Sebastian
94fa1a3ecb
Encapsulated data reading in utils method, removed fastai processing, added doc string
2018-11-15 14:36:50 +00:00
Sebastian
300d78ee44
Consolidated methods for reading classification data, added method to read XNLI data
2018-11-15 14:34:07 +00:00
Sebastian
d5e6849cd8
Shortened XNLI download script, removed unnecessary parts; extracts now to data folder
2018-11-15 14:33:21 +00:00
Nirant K
62e6777952
Add download and unzip script
2018-11-14 15:53:59 +00:00
Nirant
898c9255e0
Fix minor typos
2018-11-14 21:07:15 +05:30
Nirant K
0fb8161061
Merge branch 'master' of https://github.com/n-waves/ulmfit-multilingual
2018-11-14 13:56:58 +00:00
Nirant K
70680c21ce
Add better logs, add multi-language flag to pretrain
...
- Added lang variable to pretrain_lm
- Tried the QRNN model with Hindi
- Added better logging and consistent model saving paths
- Fixed minor bug in validate(model, ...) to to validate(learn.model, ...)
2018-11-14 13:54:31 +00:00
Julian Eisenschlos