wassname
8d9f6cb60c
misc
2019-11-26 22:37:36 +08:00
wassname
67da707648
tidy and metrics
2019-11-26 21:03:17 +08:00
wassname
6deed0107b
remove binary databunch
2019-11-26 15:49:19 +08:00
wassname
46ec76ec78
multiclass (but single classes don't work)
2019-11-26 12:19:32 +08:00
wassname
7557805043
made a binary classification version making binary category label list
2019-11-26 10:17:33 +08:00
Piotr Czapla
400e54dec1
Compatiblity with fastai 1.0.47
2019-11-06 10:45:48 +01:00
Piotr Czapla
f846bf8a4b
Add OpenFIle preproc to SentencePiece preproc
2019-10-14 17:20:26 +02:00
Piotr Czapla
1ff9e01766
Remove unused code
2019-09-07 21:17:32 +02:00
Piotr Czapla
8162cc5648
Clean up the old multfit training code
2019-09-07 15:22:01 +02:00
Piotr Czapla
2fe5c8a588
Update to fastai v1.0.57 - use new sentence piece implementaiton & sizes of hidden layers
2019-08-29 15:51:17 +02:00
Piotr Czapla
a7ac4f5170
missing file
2019-06-10 20:14:57 +02:00
Marcin
3125ce7d8b
Fix seeds
2019-05-11 16:59:59 +02:00
Marcin
27a720b105
Make decoder bias optional
...
These changes were made during poleval'19 competition to try to
reproduce results from fastai v0.7 in v1.0.
2019-05-01 20:57:20 +02:00
Piotr Czapla
5529ec385b
Add sentence piece char_coverage 0.99 for non european languages
...
Otherwise Zh won't train on 15k
2019-02-18 21:50:46 +01:00
Piotr Czapla
cd47b3b5dc
Fix use_moses=True for mldoc so that it is identical to wiki with uses_moses=False
...
The issue was that Moses was executed after pre_rules when use_moses = True, But when data set was pre tokenized with Moses (use_moses=False) the pre_rules were executed after.
So our wikipedia had the following processing:
- raw text
- Moses
- pre_rules
- split(' ') # fastai BaseTokenizer
- post_rules
- sentence piece
While mldoc had the following tokenziation
- raw text
- pre_rules
- Moses
- post_rules
- sentence piece
After fix I've retrained the classfiers (without finetuning) and I haven't notice huge changes in the performance. 4 languages received slight improvment 4 got a slight decrease in performance.
2019-02-14 22:35:20 +01:00
Piotr Czapla
d269d53d7d
Use fastai tokens instead of <xxx>
...
f'xx{token_name}' are kept as one token by Moses tokenizer, which is sometimes required if you want to have moses in tokenizers pipeline, and we use that for imdb.
2019-01-01 14:40:59 +01:00
Piotr Czapla
514a9e6b86
Fix BiLM training after update to newest fastai
2018-12-31 12:13:53 +01:00
Piotr Czapla
f1b49a0d34
fix imports in learner (fastai adpatation)
2018-12-27 15:15:09 +01:00
Piotr Czapla
82d6a30b11
Fix SentencePiece implementation, to get 94.5% on imdb
...
- train on whole articles when tokenizer is sentencepice
- add moses tokenizer to get the same tokens on imdb as on wt103
- apply pre / post processing rules to moses tokenzier so that sentencepiece works on correctly preprocessed text (lowercased with html removed)
- add alpha implementation on xnli (no tests)
- remove vocab adaptation when swiching from wiki to imdb. As otherwise we get 50% of missing words and 93% accuracy on imdb
2018-12-27 15:14:41 +01:00
Piotr Czapla
ff30fb5642
Fastai upgrade
2018-12-27 14:57:15 +01:00
Piotr Czapla
fac3ce343b
Fix BiLM implementation after upgrade of fastai
2018-12-12 00:29:23 +01:00
Piotr Czapla
2338618563
Make biclasifier head a hyperparameter.
2018-12-09 00:39:27 +01:00
Piotr Czapla
591393b96c
Change AvgPooling to BiPooling as a default
2018-12-08 23:48:02 +01:00
Piotr Czapla
6a899015ed
Generate imdb unsp.csv
2018-12-05 16:27:41 +01:00
Piotr Czapla
887211137a
Add fastai tokenizer to pretrain_lm
2018-12-01 16:42:11 +01:00
Piotr Czapla
6d6ebef1ca
Update to newst fastai
2018-12-01 10:57:20 +01:00
Piotr Czapla
be117abac4
Make the end to end test run correctly
2018-11-24 23:51:23 +01:00
Piotr Czapla
8da47324c2
Clean ups and fixes
2018-11-22 15:32:40 +01:00
Piotr Czapla
dbd4884228
Fixes after mergin with master and updateing to newset fastai
2018-11-22 01:14:38 +01:00
Piotr Czapla
934fc79384
Merge branch 'master' into bilm
2018-11-21 23:48:23 +01:00
Piotr Czapla
6e3ef21b1f
Add Avg BiClassifier
2018-11-21 18:44:49 +01:00
NAUSICAA\Julian
8cb867b066
Compatibility with new fastai version
2018-11-20 19:21:02 -03:00
NAUSICAA\Julian
79691791b3
Typo fix
2018-11-20 18:53:23 -03:00
NAUSICAA\Julian
40a2990322
Add suggested changes to rules system
2018-11-20 09:06:03 -03:00
NAUSICAA\Julian
a36c0518c6
Merge branch 'master' of https://github.com/n-waves/ulmfit-multilingual into sentencepiece_fixes
2018-11-20 08:55:37 -03:00
NAUSICAA\Julian
0ecc6342a6
Fixes when running sentence piece end to end
2018-11-20 00:38:44 -03:00
Aayush
73403bdcb3
fix double EOS in read_imdb
2018-11-19 19:30:11 +05:30
Aayush
6e867811db
Support for running sentencepiece with train_clas
...
Committing into this branch. Only changes are in `read_xnli` and `read_imdb`.
2018-11-19 19:28:19 +05:30
Piotr Czapla
7f1f8efcc3
Working version of biclassfier
2018-11-19 12:58:58 +01:00
Piotr Czapla
c821d2e783
first version of bi classifier
2018-11-19 09:59:08 +01:00
Sebastian
d796610452
Removed deprecated default_rules
2018-11-17 14:48:06 +00:00
Piotr Czapla
895a58c88d
Merge branch 'master' into bilm
2018-11-16 23:39:15 +01:00
Piotr Czapla
36b056a465
Fix issues discovered during execution of end-to-end test.
2018-11-16 23:31:16 +01:00
Piotr Czapla
a21f100ab2
Merge branch 'master' into sentencepiece
2018-11-16 23:23:49 +01:00
Piotr Czapla
ebbf09775f
Merge branch 'master' into bilm
2018-11-16 20:09:41 +01:00
Piotr Czapla
23b17da61e
Add end to end test and improve the train_clas params
2018-11-16 19:51:29 +01:00
aayush
59c8852b5d
sentencepiece for pretraining
...
modified: fastai_contrib/utils.py
modified: ulmfit/pretrain_lm.py
2018-11-16 23:14:24 +05:30
Aayush
fb2ed98ac6
Merge branch 'master' into sentencepiece
2018-11-16 23:09:06 +05:30
Piotr Czapla
d5d61d31d2
Add directory structure to readme.
2018-11-15 23:47:48 +01:00
Piotr Czapla
9a60ef2fbd
Add assertions to train_clas and ability to limit the dataset size
2018-11-15 23:40:10 +01:00
wassname