Piotr Czapla
5529ec385b
Add sentence piece char_coverage 0.99 for non european languages
...
Otherwise Zh won't train on 15k
2019-02-18 21:50:46 +01:00
Piotr Czapla
cd47b3b5dc
Fix use_moses=True for mldoc so that it is identical to wiki with uses_moses=False
...
The issue was that Moses was executed after pre_rules when use_moses = True, But when data set was pre tokenized with Moses (use_moses=False) the pre_rules were executed after.
So our wikipedia had the following processing:
- raw text
- Moses
- pre_rules
- split(' ') # fastai BaseTokenizer
- post_rules
- sentence piece
While mldoc had the following tokenziation
- raw text
- pre_rules
- Moses
- post_rules
- sentence piece
After fix I've retrained the classfiers (without finetuning) and I haven't notice huge changes in the performance. 4 languages received slight improvment 4 got a slight decrease in performance.
2019-02-14 22:35:20 +01:00
Piotr Czapla
d269d53d7d
Use fastai tokens instead of <xxx>
...
f'xx{token_name}' are kept as one token by Moses tokenizer, which is sometimes required if you want to have moses in tokenizers pipeline, and we use that for imdb.
2019-01-01 14:40:59 +01:00
Piotr Czapla
514a9e6b86
Fix BiLM training after update to newest fastai
2018-12-31 12:13:53 +01:00
Piotr Czapla
f1b49a0d34
fix imports in learner (fastai adpatation)
2018-12-27 15:15:09 +01:00
Piotr Czapla
82d6a30b11
Fix SentencePiece implementation, to get 94.5% on imdb
...
- train on whole articles when tokenizer is sentencepice
- add moses tokenizer to get the same tokens on imdb as on wt103
- apply pre / post processing rules to moses tokenzier so that sentencepiece works on correctly preprocessed text (lowercased with html removed)
- add alpha implementation on xnli (no tests)
- remove vocab adaptation when swiching from wiki to imdb. As otherwise we get 50% of missing words and 93% accuracy on imdb
2018-12-27 15:14:41 +01:00
Piotr Czapla
ff30fb5642
Fastai upgrade
2018-12-27 14:57:15 +01:00
Piotr Czapla
fac3ce343b
Fix BiLM implementation after upgrade of fastai
2018-12-12 00:29:23 +01:00
Piotr Czapla
2338618563
Make biclasifier head a hyperparameter.
2018-12-09 00:39:27 +01:00
Piotr Czapla
591393b96c
Change AvgPooling to BiPooling as a default
2018-12-08 23:48:02 +01:00
Piotr Czapla
6a899015ed
Generate imdb unsp.csv
2018-12-05 16:27:41 +01:00
Piotr Czapla
887211137a
Add fastai tokenizer to pretrain_lm
2018-12-01 16:42:11 +01:00
Piotr Czapla
6d6ebef1ca
Update to newst fastai
2018-12-01 10:57:20 +01:00
Piotr Czapla
be117abac4
Make the end to end test run correctly
2018-11-24 23:51:23 +01:00
Piotr Czapla
8da47324c2
Clean ups and fixes
2018-11-22 15:32:40 +01:00
Piotr Czapla
dbd4884228
Fixes after mergin with master and updateing to newset fastai
2018-11-22 01:14:38 +01:00
Piotr Czapla
934fc79384
Merge branch 'master' into bilm
2018-11-21 23:48:23 +01:00
Piotr Czapla
6e3ef21b1f
Add Avg BiClassifier
2018-11-21 18:44:49 +01:00
NAUSICAA\Julian
8cb867b066
Compatibility with new fastai version
2018-11-20 19:21:02 -03:00
NAUSICAA\Julian
79691791b3
Typo fix
2018-11-20 18:53:23 -03:00
NAUSICAA\Julian
40a2990322
Add suggested changes to rules system
2018-11-20 09:06:03 -03:00
NAUSICAA\Julian
a36c0518c6
Merge branch 'master' of https://github.com/n-waves/ulmfit-multilingual into sentencepiece_fixes
2018-11-20 08:55:37 -03:00
NAUSICAA\Julian
0ecc6342a6
Fixes when running sentence piece end to end
2018-11-20 00:38:44 -03:00
Aayush
73403bdcb3
fix double EOS in read_imdb
2018-11-19 19:30:11 +05:30
Aayush
6e867811db
Support for running sentencepiece with train_clas
...
Committing into this branch. Only changes are in `read_xnli` and `read_imdb`.
2018-11-19 19:28:19 +05:30
Piotr Czapla
7f1f8efcc3
Working version of biclassfier
2018-11-19 12:58:58 +01:00
Piotr Czapla
c821d2e783
first version of bi classifier
2018-11-19 09:59:08 +01:00
Sebastian
d796610452
Removed deprecated default_rules
2018-11-17 14:48:06 +00:00
Piotr Czapla
895a58c88d
Merge branch 'master' into bilm
2018-11-16 23:39:15 +01:00
Piotr Czapla
36b056a465
Fix issues discovered during execution of end-to-end test.
2018-11-16 23:31:16 +01:00
Piotr Czapla
a21f100ab2
Merge branch 'master' into sentencepiece
2018-11-16 23:23:49 +01:00
Piotr Czapla
ebbf09775f
Merge branch 'master' into bilm
2018-11-16 20:09:41 +01:00
Piotr Czapla
23b17da61e
Add end to end test and improve the train_clas params
2018-11-16 19:51:29 +01:00
aayush
59c8852b5d
sentencepiece for pretraining
...
modified: fastai_contrib/utils.py
modified: ulmfit/pretrain_lm.py
2018-11-16 23:14:24 +05:30
Aayush
fb2ed98ac6
Merge branch 'master' into sentencepiece
2018-11-16 23:09:06 +05:30
Piotr Czapla
d5d61d31d2
Add directory structure to readme.
2018-11-15 23:47:48 +01:00
Piotr Czapla
9a60ef2fbd
Add assertions to train_clas and ability to limit the dataset size
2018-11-15 23:40:10 +01:00
Piotr Czapla
111fc7e4c3
Add perpare_imdb script
2018-11-15 23:37:15 +01:00
NAUSICAA\Julian
b9c587d9bd
Adding SentencePieceTokenizer
2018-11-15 19:18:42 -03:00
Nirant K
43340d03bc
Fix minor typos in train_clas
2018-11-15 18:04:15 +00:00
Nirant K
56fc150eb1
Merge xnli
2018-11-15 15:53:37 +00:00
Nirant K
6622458fe7
Add aclImdb extractor
2018-11-15 15:03:12 +00:00
Sebastian
300d78ee44
Consolidated methods for reading classification data, added method to read XNLI data
2018-11-15 14:34:07 +00:00
Piotr Czapla
33f9eb2cc7
Reuse RNNCore in implementation of BiLM, add accuracy
2018-11-14 21:16:16 +01:00
Piotr Czapla
5ba83b1d4e
Working version of BILM - probably won't train well yet
2018-11-14 14:55:56 +01:00
Piotr Czapla
a85800610b
WIP Working Backward LM using our new LangaugeModelLoader
2018-11-14 13:22:03 +01:00
Piotr Czapla
6ee1a2b27d
Add BiLM LanguageModelLoader with tests
2018-11-14 12:48:29 +01:00
Sebastian
cba86ab5d4
Added initial classes and changes for BiLM implementation
2018-11-13 17:46:28 +00:00
Piotr Czapla
40ca6c200e
Refactor and temporary fix the pretrain_lm by adding learn.lr_find
...
For some reason this fixes the training issue. I though I've nailed it down but i haven't. :/ .
2018-11-10 00:32:29 +01:00
Piotr Czapla
eb1da0b630
Updated README.md
2018-11-08 21:05:48 +01:00
Piotr Czapla