Piotr Czapla
d269d53d7d
Use fastai tokens instead of <xxx>
...
f'xx{token_name}' are kept as one token by Moses tokenizer, which is sometimes required if you want to have moses in tokenizers pipeline, and we use that for imdb.
2019-01-01 14:40:59 +01:00
Piotr Czapla
514a9e6b86
Fix BiLM training after update to newest fastai
2018-12-31 12:13:53 +01:00
Piotr Czapla
82d6a30b11
Fix SentencePiece implementation, to get 94.5% on imdb
...
- train on whole articles when tokenizer is sentencepice
- add moses tokenizer to get the same tokens on imdb as on wt103
- apply pre / post processing rules to moses tokenzier so that sentencepiece works on correctly preprocessed text (lowercased with html removed)
- add alpha implementation on xnli (no tests)
- remove vocab adaptation when swiching from wiki to imdb. As otherwise we get 50% of missing words and 93% accuracy on imdb
2018-12-27 15:14:41 +01:00
Piotr Czapla
6a899015ed
Generate imdb unsp.csv
2018-12-05 16:27:41 +01:00
Piotr Czapla
887211137a
Add fastai tokenizer to pretrain_lm
2018-12-01 16:42:11 +01:00
Piotr Czapla
8da47324c2
Clean ups and fixes
2018-11-22 15:32:40 +01:00
Piotr Czapla
dbd4884228
Fixes after mergin with master and updateing to newset fastai
2018-11-22 01:14:38 +01:00
NAUSICAA\Julian
8cb867b066
Compatibility with new fastai version
2018-11-20 19:21:02 -03:00
NAUSICAA\Julian
79691791b3
Typo fix
2018-11-20 18:53:23 -03:00
NAUSICAA\Julian
40a2990322
Add suggested changes to rules system
2018-11-20 09:06:03 -03:00
NAUSICAA\Julian
a36c0518c6
Merge branch 'master' of https://github.com/n-waves/ulmfit-multilingual into sentencepiece_fixes
2018-11-20 08:55:37 -03:00
NAUSICAA\Julian
0ecc6342a6
Fixes when running sentence piece end to end
2018-11-20 00:38:44 -03:00
Aayush
73403bdcb3
fix double EOS in read_imdb
2018-11-19 19:30:11 +05:30
Aayush
6e867811db
Support for running sentencepiece with train_clas
...
Committing into this branch. Only changes are in `read_xnli` and `read_imdb`.
2018-11-19 19:28:19 +05:30
Sebastian
d796610452
Removed deprecated default_rules
2018-11-17 14:48:06 +00:00
Piotr Czapla
36b056a465
Fix issues discovered during execution of end-to-end test.
2018-11-16 23:31:16 +01:00
Piotr Czapla
a21f100ab2
Merge branch 'master' into sentencepiece
2018-11-16 23:23:49 +01:00
Piotr Czapla
23b17da61e
Add end to end test and improve the train_clas params
2018-11-16 19:51:29 +01:00
aayush
59c8852b5d
sentencepiece for pretraining
...
modified: fastai_contrib/utils.py
modified: ulmfit/pretrain_lm.py
2018-11-16 23:14:24 +05:30
Aayush
fb2ed98ac6
Merge branch 'master' into sentencepiece
2018-11-16 23:09:06 +05:30
Piotr Czapla
d5d61d31d2
Add directory structure to readme.
2018-11-15 23:47:48 +01:00
Piotr Czapla
9a60ef2fbd
Add assertions to train_clas and ability to limit the dataset size
2018-11-15 23:40:10 +01:00
Piotr Czapla
111fc7e4c3
Add perpare_imdb script
2018-11-15 23:37:15 +01:00
NAUSICAA\Julian
b9c587d9bd
Adding SentencePieceTokenizer
2018-11-15 19:18:42 -03:00
Nirant K
43340d03bc
Fix minor typos in train_clas
2018-11-15 18:04:15 +00:00
Nirant K
56fc150eb1
Merge xnli
2018-11-15 15:53:37 +00:00
Nirant K
6622458fe7
Add aclImdb extractor
2018-11-15 15:03:12 +00:00
Sebastian
300d78ee44
Consolidated methods for reading classification data, added method to read XNLI data
2018-11-15 14:34:07 +00:00
Piotr Czapla
40ca6c200e
Refactor and temporary fix the pretrain_lm by adding learn.lr_find
...
For some reason this fixes the training issue. I though I've nailed it down but i haven't. :/ .
2018-11-10 00:32:29 +01:00
Piotr Czapla