aayush
03c4a2cd15
Fixed self attention, local tests passed
2019-02-15 02:10:18 +05:30
Aayush
8fe2891402
Make callable through bicls_head
2019-01-07 00:02:39 +05:30
Aayush
d0c15472d6
Self attention for pooling linear classifier
...
This PR will introduce a `BiAttentionPoolingClassifier` as in [Attention is all you need](https://arxiv.org/abs/1706.03762 ) following the discussion with @sebastianruder in Teams.
I ran out of memory on my 1060 while testing the attention module, but was able to at least verify that it is functionally correct. Some changes might be required to ensure that the tensor passed to `self.layers` is of the right shape (but I'm not quite sure as of now).
I'll shift all the stuff to Collab for testing and see if it's any help.
2019-01-06 23:58:09 +05:30
Piotr Czapla
d269d53d7d
Use fastai tokens instead of <xxx>
...
f'xx{token_name}' are kept as one token by Moses tokenizer, which is sometimes required if you want to have moses in tokenizers pipeline, and we use that for imdb.
2019-01-01 14:40:59 +01:00
Piotr Czapla
514a9e6b86
Fix BiLM training after update to newest fastai
2018-12-31 12:13:53 +01:00
Piotr Czapla
f1b49a0d34
fix imports in learner (fastai adpatation)
2018-12-27 15:15:09 +01:00
Piotr Czapla
82d6a30b11
Fix SentencePiece implementation, to get 94.5% on imdb
...
- train on whole articles when tokenizer is sentencepice
- add moses tokenizer to get the same tokens on imdb as on wt103
- apply pre / post processing rules to moses tokenzier so that sentencepiece works on correctly preprocessed text (lowercased with html removed)
- add alpha implementation on xnli (no tests)
- remove vocab adaptation when swiching from wiki to imdb. As otherwise we get 50% of missing words and 93% accuracy on imdb
2018-12-27 15:14:41 +01:00
Piotr Czapla
ff30fb5642
Fastai upgrade
2018-12-27 14:57:15 +01:00
Piotr Czapla
fac3ce343b
Fix BiLM implementation after upgrade of fastai
2018-12-12 00:29:23 +01:00
Piotr Czapla
2338618563
Make biclasifier head a hyperparameter.
2018-12-09 00:39:27 +01:00
Piotr Czapla
591393b96c
Change AvgPooling to BiPooling as a default
2018-12-08 23:48:02 +01:00
Piotr Czapla
6a899015ed
Generate imdb unsp.csv
2018-12-05 16:27:41 +01:00
Piotr Czapla
887211137a
Add fastai tokenizer to pretrain_lm
2018-12-01 16:42:11 +01:00
Piotr Czapla
6d6ebef1ca
Update to newst fastai
2018-12-01 10:57:20 +01:00
Piotr Czapla
be117abac4
Make the end to end test run correctly
2018-11-24 23:51:23 +01:00
Piotr Czapla
8da47324c2
Clean ups and fixes
2018-11-22 15:32:40 +01:00
Piotr Czapla
dbd4884228
Fixes after mergin with master and updateing to newset fastai
2018-11-22 01:14:38 +01:00
Piotr Czapla
934fc79384
Merge branch 'master' into bilm
2018-11-21 23:48:23 +01:00
Piotr Czapla
6e3ef21b1f
Add Avg BiClassifier
2018-11-21 18:44:49 +01:00
NAUSICAA\Julian
8cb867b066
Compatibility with new fastai version
2018-11-20 19:21:02 -03:00
NAUSICAA\Julian
79691791b3
Typo fix
2018-11-20 18:53:23 -03:00
NAUSICAA\Julian
40a2990322
Add suggested changes to rules system
2018-11-20 09:06:03 -03:00
NAUSICAA\Julian
a36c0518c6
Merge branch 'master' of https://github.com/n-waves/ulmfit-multilingual into sentencepiece_fixes
2018-11-20 08:55:37 -03:00
NAUSICAA\Julian
0ecc6342a6
Fixes when running sentence piece end to end
2018-11-20 00:38:44 -03:00
Aayush
73403bdcb3
fix double EOS in read_imdb
2018-11-19 19:30:11 +05:30
Aayush
6e867811db
Support for running sentencepiece with train_clas
...
Committing into this branch. Only changes are in `read_xnli` and `read_imdb`.
2018-11-19 19:28:19 +05:30
Piotr Czapla
7f1f8efcc3
Working version of biclassfier
2018-11-19 12:58:58 +01:00
Piotr Czapla
c821d2e783
first version of bi classifier
2018-11-19 09:59:08 +01:00
Sebastian
d796610452
Removed deprecated default_rules
2018-11-17 14:48:06 +00:00
Piotr Czapla
895a58c88d
Merge branch 'master' into bilm
2018-11-16 23:39:15 +01:00
Piotr Czapla
36b056a465
Fix issues discovered during execution of end-to-end test.
2018-11-16 23:31:16 +01:00
Piotr Czapla
a21f100ab2
Merge branch 'master' into sentencepiece
2018-11-16 23:23:49 +01:00
Piotr Czapla
ebbf09775f
Merge branch 'master' into bilm
2018-11-16 20:09:41 +01:00
Piotr Czapla
23b17da61e
Add end to end test and improve the train_clas params
2018-11-16 19:51:29 +01:00
aayush
59c8852b5d
sentencepiece for pretraining
...
modified: fastai_contrib/utils.py
modified: ulmfit/pretrain_lm.py
2018-11-16 23:14:24 +05:30
Aayush
fb2ed98ac6
Merge branch 'master' into sentencepiece
2018-11-16 23:09:06 +05:30
Piotr Czapla
d5d61d31d2
Add directory structure to readme.
2018-11-15 23:47:48 +01:00
Piotr Czapla
9a60ef2fbd
Add assertions to train_clas and ability to limit the dataset size
2018-11-15 23:40:10 +01:00
Piotr Czapla
111fc7e4c3
Add perpare_imdb script
2018-11-15 23:37:15 +01:00
NAUSICAA\Julian
b9c587d9bd
Adding SentencePieceTokenizer
2018-11-15 19:18:42 -03:00
Nirant K
43340d03bc
Fix minor typos in train_clas
2018-11-15 18:04:15 +00:00
Nirant K
56fc150eb1
Merge xnli
2018-11-15 15:53:37 +00:00
Nirant K
6622458fe7
Add aclImdb extractor
2018-11-15 15:03:12 +00:00
Sebastian
300d78ee44
Consolidated methods for reading classification data, added method to read XNLI data
2018-11-15 14:34:07 +00:00
Piotr Czapla
33f9eb2cc7
Reuse RNNCore in implementation of BiLM, add accuracy
2018-11-14 21:16:16 +01:00
Piotr Czapla
5ba83b1d4e
Working version of BILM - probably won't train well yet
2018-11-14 14:55:56 +01:00
Piotr Czapla
a85800610b
WIP Working Backward LM using our new LangaugeModelLoader
2018-11-14 13:22:03 +01:00
Piotr Czapla
6ee1a2b27d
Add BiLM LanguageModelLoader with tests
2018-11-14 12:48:29 +01:00
Sebastian
cba86ab5d4
Added initial classes and changes for BiLM implementation
2018-11-13 17:46:28 +00:00
Piotr Czapla
40ca6c200e
Refactor and temporary fix the pretrain_lm by adding learn.lr_find
...
For some reason this fixes the training issue. I though I've nailed it down but i haven't. :/ .
2018-11-10 00:32:29 +01:00
aayush