mirror of
https://github.com/wassname/multifit.git
synced 2026-09-12 12:32:46 +08:00
Merge pull request #55 from n-waves/multifit
New scripts to train multifit that are easier to work with
This commit is contained in:
Piotr Czapla
+1
-1
@@ -18,4 +18,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
SOFTWARE.
|
||||
@@ -1,108 +1,78 @@
|
||||
# ulmfit-multilingual
|
||||
Repository used for collaboration on application of ulmfit for multiple languages, it helps with pertraining and uses the
|
||||
fastai v1 . (The version in n-waves/fastai:ulmfit_multilingual)
|
||||
|
||||
# How to train classifier
|
||||
# MultiFiT: Efficient Multi-lingual Language Model Fine-tuning
|
||||
Code to reproduce the paper "[MultiFiT: Efficient Multi-lingual Language Model Fine-tuning](https://arxiv.org/abs/1909.04761)".
|
||||
|
||||
Here is a blog post with an introducing to our paper: http://nlp.fast.ai/classification/2019/09/10/multifit.html
|
||||
|
||||
This repository contains a small framework on top of fastai v1.0; the code is compatible with v1.0.47 up to v1.0.59 (the current as of 2019.11.03).
|
||||
The results between fastai versions may differ due to optimizations added to fastai. Our models were trained using 1.0.47.
|
||||
|
||||
The framework was rewritten to make it easier to use with the newest fastai.
|
||||
|
||||
We released 7 language models trained on corresponding Wikipedia dumps:
|
||||
- de_multifit_paper_version
|
||||
- es_multifit_paper_version
|
||||
- fr_multifit_paper_version
|
||||
- it_multifit_paper_version
|
||||
- ja_multifit_paper_version
|
||||
- ru_multifit_paper_version
|
||||
- zh_multifit_paper_version
|
||||
|
||||
To fetch the model just use `multifit.from_pretrained` function.
|
||||
Here are some example notebook showing how to train a classifier using a pretrained models.
|
||||
- [./notebooks/CLS-JA.ipynb](./notebooks/CLS-JA.ipynb) - example of classifier trained on amazon CLS JA music.
|
||||
- [./notebooks/MLDoc-JA-multifit_fp16.ipynb](./notebooks/MLDoc-JA-multifit_fp16.ipynb) - example of a faster multifit training using fp16 on MDLDoc.
|
||||
|
||||
## Results
|
||||
### MLDoc
|
||||
Document classification results on MLDoc dataset [Schwenk and Li, 2018](https://arxiv.org/abs/1805.09821)
|
||||
|
||||
| Model | de | es | fr | it | ja | ru | zh |
|
||||
|----------------|-----------|-----------|-----------|-----------|-----------|-----------|------------|
|
||||
|LASER | 92.70 | 88.75 | 90.80 | 85.93 | 85.15 | 84.65 | 88.98 |
|
||||
| MultiBERT | 94.0 | 95.15 | 93.20 | 85.82 | 87.48 | 86.85 | 90.72 |
|
||||
| MultiFiT | **95.90** | **96.07** | **94.77** | **90.25** | **90.03** | **87.65**| **92.52** |
|
||||
|
||||
### Amazon CLS
|
||||
Sentiment classification results on CLS dataset [Prettenhofer and Stein, 2010](https://dl.acm.org/citation.cfm?doid=2036264.2036277)
|
||||
|
||||
| | DE | FR | JA |
|
||||
|----------|-----------------------|-----------------------|----------------------|
|
||||
| MultiBERT| 86.05 / 84.90 / 82.00 | 86.15 / 86.90 / 86.65 | 80.87 / 82.83 / 79.95|
|
||||
| MultiFiT | 93.19 / 90.54 / 93.00 | 91.25 / 89.55 / 93.40 | 86.29 / 85.75 / 86.59|
|
||||
|
||||
|
||||
## How to use it with fastai v1.0
|
||||
You can use the pretrained models with fastai library as follows:
|
||||
```
|
||||
$ LANG=en
|
||||
$ python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='f' --nl 3 --name 'orig' --max-vocab 60000 \
|
||||
--lang ${LANG} --qrnn=False - train 10 --bs=50 --drop_mult=0 --label-smoothing-eps=0.0
|
||||
from fastai.text import *
|
||||
import multifit
|
||||
|
||||
exp = multifit.from_pretrained("name of the model")
|
||||
fa_config = exp.pretrain_lm.tokenizer.get_fastai_config(add_open_file_processor=True)
|
||||
data_lm = (TextList.from_folder(imdb_path, **fa_config)
|
||||
.filter_by_folder(include=['train', 'test', 'unsup'])
|
||||
.split_by_rand_pct(0.1)
|
||||
.label_for_lm()
|
||||
.databunch(bs=bs))
|
||||
learn = exp.finetune_lm.get_learner(data_lm)
|
||||
# learn is a preconfigured fastai learner with a pretrained model loaded
|
||||
learn.fit_one_cycle(10)
|
||||
learn.save_encoder("enc")
|
||||
...
|
||||
Model name: data/wiki/en-100/models/f60k/lstm_orig.m
|
||||
...
|
||||
|
||||
$ python -m ulmfit cls --dataset-path data/imdb --base-lm-path data/wiki/${LANG}-100/models/f60k/lstm_orig.m \
|
||||
--lang=${LANG} --name orig - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
```
|
||||
|
||||
You can re-evaluate classifiers by running
|
||||
## Reproducing the results
|
||||
This repository is a rewrite of the original training scripts so it lacks all the scripts used in the paper.
|
||||
We are working on a port to fastai v2.0 and then we will be adding the scripts that show how to reproduce the results.
|
||||
In case you need to use the scripts faster you can access the original scripts [here](https://github.com/n-waves/multifit/tree/ulmfit-multilingual-original-scripts).
|
||||
|
||||
## Citation
|
||||
```
|
||||
python -m ulmfit eval --glob="imdb/models/*/lstm_*.m"
|
||||
```
|
||||
|
||||
The same command can be used to quickly trian multiple classifiers, by adding the `--name` parameter:
|
||||
```
|
||||
python -m ulmfit eval --glob="imdb/models/*/lstm_nl3.m" --name "nl3-my-test1" --num-cls-epochs 4 --label-smoothing-eps=0.1 --lr_sched=1cycle
|
||||
```
|
||||
|
||||
To create a tar with model simply run
|
||||
```
|
||||
python -m ulmfit tar data/imdb/models/f60k/lstm_nl3.m
|
||||
```
|
||||
|
||||
## data directory strucutre
|
||||
|
||||
Directory structure after changes to the way we process wiki dumps.
|
||||
```
|
||||
data
|
||||
├── imdb
|
||||
│ ├── aclImdb
|
||||
│ ├── imdb_lm
|
||||
│ └── tmp
|
||||
├── wiki
|
||||
│ ├── de-100
|
||||
│ │ └── models
|
||||
│ ├── de-100-unk
|
||||
│ │ └── models
|
||||
│ ├── de-2
|
||||
│ │ └── models
|
||||
│ ├── de-2-unk
|
||||
│ │ └── models
|
||||
│ ├── de-all
|
||||
│ │ └── models
|
||||
│ ├── wikitext-103
|
||||
│ │ └── models
|
||||
│ └── wikitext-2
|
||||
│ └── models
|
||||
├── wiki_dumps
|
||||
├── wiki_extr
|
||||
│ └── de
|
||||
│ ├── AA
|
||||
│ ├── AB
|
||||
...
|
||||
└── CC
|
||||
└── xnli
|
||||
├── XNLI-1.0
|
||||
└── XNLI-MT-1.0
|
||||
├── multinli
|
||||
└── xnli
|
||||
```
|
||||
|
||||
## how to contribute
|
||||
We have a fork of fastai to propose changes to fastai.text, with a branch for this project:
|
||||
https://github.com/n-waves/fastai/tree/ulmfit_multilingual
|
||||
|
||||
Let us know that you want to start collaboration on fastai forum thread: [Multilingual ULMFIT](https://forums.fast.ai/t/multilingual-ulmfit/28117)
|
||||
and you will get access to both repositories.
|
||||
|
||||
- Follow the [developer installation of fastai](https://github.com/fastai/fastai#developer-install)
|
||||
- Add n-waves/fastai as additional remote as described here: https://help.github.com/articles/adding-a-remote/
|
||||
|
||||
Here is what I did:
|
||||
```bash
|
||||
$ cd fastai
|
||||
$ git remote add n-waves https://github.com/n-waves/fastai.git
|
||||
$ git remote -v
|
||||
n-waves https://github.com/n-waves/fastai.git (fetch)
|
||||
n-waves https://github.com/n-waves/fastai.git (push)
|
||||
origin https://github.com/fastai/fastai.git (fetch)
|
||||
origin https://github.com/fastai/fastai.git (push)
|
||||
|
||||
$ git fetch n-waves
|
||||
$ git checkout ulmfit_multilingual
|
||||
Branch 'ulmfit_multilingual' set up to track remote branch 'ulmfit_multilingual' from 'n-waves'.
|
||||
Switched to a new branch 'ulmfit_multilingual'
|
||||
|
||||
$ git push --set-upstream n-waves ulmfit_multilingual # to automatically push ulmfit_multilingual branch to the n-waves repo
|
||||
```
|
||||
|
||||
|
||||
## Running tests
|
||||
|
||||
To run the tests, the following data is necessary:
|
||||
|
||||
- wikitext-2 (prepared by `./prepare_wiki-en.sh`, along with wikitext-103)
|
||||
- imdb (prepared by `./prepare_imdb.sh`)
|
||||
|
||||
then simply run tests, e.g. `pytest .`
|
||||
@article{Eisenschlos2019MultiFit,
|
||||
title={MultiFiT: Efficient Multi-lingual Language Model Fine-tuning},
|
||||
author={Julian Eisenschlos, Sebastian Ruder, Piotr Czapla, Marcin Kardas, Sylvain Gugger, Jeremy Howard}
|
||||
journal={Proceedings of EMNLP-IJCNLP 2019},
|
||||
year={2019}
|
||||
}
|
||||
```
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,154 @@
|
||||
import pathlib
|
||||
from typing import Collection
|
||||
from pandas import DataFrame
|
||||
from sacremoses import MosesTokenizer
|
||||
import fastai
|
||||
from fastai.basic_data import DataBunch
|
||||
from fastai.core import ListRules, PathOrStr, IntsOrStrs, is_listy
|
||||
from fastai.data_block import ItemLists
|
||||
from fastai.text import *
|
||||
|
||||
|
||||
class MosesPreprocessingFunc():
|
||||
def __init__(self, lang: str):
|
||||
self.mt = MosesTokenizer(lang)
|
||||
|
||||
def __call__(self, t: str) -> str:
|
||||
return self.mt.tokenize(t, return_str=True, escape=True)
|
||||
|
||||
try:
|
||||
from fastai.text import SPProcessor
|
||||
except ImportError:
|
||||
|
||||
def _join_texts(texts:Collection[str], mark_fields:bool=False, include_bos:bool=True, include_eos:bool=False):
|
||||
if not isinstance(texts, np.ndarray): texts = np.array(texts)
|
||||
if is1d(texts): texts = texts[:,None]
|
||||
df = pd.DataFrame({i:texts[:,i] for i in range(texts.shape[1])})
|
||||
bos_tok = f'{BOS} ' if include_bos else ''
|
||||
text_col = f'{bos_tok}{FLD} {1} ' + df[0].astype(str) if mark_fields else f'{bos_tok}' + df[0].astype(str)
|
||||
for i in range(1,len(df.columns)):
|
||||
text_col += (f' {FLD} {i+1} ' if mark_fields else ' ') + df[i].astype(str)
|
||||
if include_eos: text_col = text_col + f' {EOS}'
|
||||
return text_col.values
|
||||
|
||||
def apply_rules(text, pre_rules=None, post_rules=None):
|
||||
"Apply `pre_rules` and `post_rules` to `text`"
|
||||
text = text.strip(' ')
|
||||
for r in ifnone(pre_rules, defaults.text_pre_rules): text = r(text)
|
||||
toks = text.split()
|
||||
for r in ifnone(post_rules, defaults.text_post_rules): toks = r(toks)
|
||||
return ' '.join(toks)
|
||||
|
||||
def get_default_size(texts, max_vocab_sz):
|
||||
"Either max_vocab_sz or one quarter of the number of unique words in `texts`"
|
||||
cnt = Counter()
|
||||
for t in texts:
|
||||
cnt.update(t.split())
|
||||
if len(cnt)//4 > max_vocab_sz: return max_vocab_sz
|
||||
res = len(cnt)//4
|
||||
while res%8 != 0: res+=1
|
||||
return res
|
||||
|
||||
full_char_coverage_langs = ["bg", "cs", "da", "de", "el", "en", "es", "et", "fi", "fr", "ga", "hr", "hu",
|
||||
"it","lt","lv","mt","nl","pl","pt","ro","sk","sl","sv"] # all European langs
|
||||
|
||||
def train_sentencepiece(texts:Collection[str], path:PathOrStr, pre_rules: ListRules=None, post_rules:ListRules=None,
|
||||
vocab_sz:int=None, max_vocab_sz:int=30000, model_type:str='unigram', max_sentence_len:int=20480, lang='en',
|
||||
char_coverage=None, tmp_dir='tmp', enc='utf8'):
|
||||
"Train a sentencepiece tokenizer on `texts` and save it in `path/tmp_dir`"
|
||||
from sentencepiece import SentencePieceTrainer
|
||||
cache_dir = Path(path)/tmp_dir
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
if vocab_sz is None: vocab_sz=get_default_size(texts, max_vocab_sz)
|
||||
raw_text_path = cache_dir / 'all_text.out'
|
||||
with open(raw_text_path, 'w', encoding=enc) as f: f.write("\n".join(texts))
|
||||
spec_tokens = ['\u2581'+s for s in defaults.text_spec_tok]
|
||||
SentencePieceTrainer.Train(" ".join([
|
||||
f"--input={raw_text_path} --max_sentence_length={max_sentence_len}",
|
||||
f"--character_coverage={ifnone(char_coverage, 0.99999 if lang in full_char_coverage_langs else 0.9998)}",
|
||||
f"--unk_id={len(defaults.text_spec_tok)} --pad_id=-1 --bos_id=-1 --eos_id=-1",
|
||||
f"--user_defined_symbols={','.join(spec_tokens)}",
|
||||
f"--model_prefix={cache_dir/'spm'} --vocab_size={vocab_sz} --model_type={model_type}"]))
|
||||
raw_text_path.unlink()
|
||||
return cache_dir
|
||||
|
||||
class SPProcessor(PreProcessor):
|
||||
"`PreProcessor` that tokenizes and numericalizes with `sentencepiece`"
|
||||
def __init__(self, ds:ItemList=None, pre_rules: ListRules=None, post_rules:ListRules=None, vocab_sz:int=None,
|
||||
max_vocab_sz:int=30000, model_type:str='unigram', max_sentence_len:int=20480, lang='en',
|
||||
char_coverage=None, tmp_dir='tmp', mark_fields:bool=False, include_bos:bool=True,
|
||||
include_eos:bool=False, sp_model=None, sp_vocab=None, n_cpus:int=None, enc='utf8'):
|
||||
try: from sentencepiece import SentencePieceTrainer,SentencePieceProcessor
|
||||
except ImportError:
|
||||
raise Exception('sentencepiece module is missing: run `pip install sentencepiece`')
|
||||
self.pre_rules,self.post_rules,self.enc = pre_rules,post_rules,enc
|
||||
self.mark_fields,self.include_bos,self.include_eos = mark_fields,include_bos,include_eos
|
||||
self.sp_model,self.sp_vocab,self.n_cpus = sp_model,sp_vocab,ifnone(n_cpus,defaults.cpus)
|
||||
self.train_func = partial(train_sentencepiece, pre_rules=pre_rules, post_rules=post_rules, vocab_sz=vocab_sz,
|
||||
max_vocab_sz=max_vocab_sz, model_type=model_type, max_sentence_len=max_sentence_len, lang=lang,
|
||||
char_coverage=char_coverage, tmp_dir=tmp_dir, enc=enc)
|
||||
|
||||
def process_one(self, item, join=True):
|
||||
if join: text = _join_texts([item], self.mark_fields, self.include_bos, self.include_eos)[0]
|
||||
text = apply_rules(text, pre_rules=self.pre_rules, post_rules=self.post_rules)
|
||||
return self._encode_batch([text])[0]
|
||||
|
||||
def process(self, ds):
|
||||
ds.items = _join_texts(ds.items, self.mark_fields, self.include_bos, self.include_eos)
|
||||
ds.items = [apply_rules(t, pre_rules=self.pre_rules, post_rules=self.post_rules)
|
||||
for t in progress_bar(ds.items, leave=False)]
|
||||
if self.sp_model is None or self.sp_vocab is None:
|
||||
cache_dir = self.train_func(ds.items, ds.path)
|
||||
self.sp_model,self.sp_vocab = cache_dir/'spm.model',cache_dir/'spm.vocab'
|
||||
if not getattr(self, 'vocab', False):
|
||||
with open(self.sp_vocab, 'r', encoding=self.enc) as f: self.vocab = Vocab([line.split('\t')[0] for line in f.readlines()])
|
||||
if self.n_cpus <= 1: ds.items = self._encode_batch(ds.items)
|
||||
else:
|
||||
with ProcessPoolExecutor(self.n_cpus) as e:
|
||||
ds.items = np.array(sum(e.map(self._encode_batch, partition_by_cores(ds.items, self.n_cpus)), []))
|
||||
ds.vocab = self.vocab
|
||||
|
||||
def _encode_batch(self, texts):
|
||||
from sentencepiece import SentencePieceProcessor
|
||||
tok = SentencePieceProcessor()
|
||||
tok.Load(str(self.sp_model))
|
||||
return [np.array(tok.EncodeAsIds(t)) for t in texts]
|
||||
|
||||
@classmethod
|
||||
def load(cls, path:PathOrStr, tmp_dir:PathOrStr='tmp', name:str='spm'):
|
||||
cache_dir = Path(path)/tmp_dir
|
||||
return cls(sp_model=cache_dir/f'{name}.model', sp_vocab=cache_dir/f'{name}.vocab')
|
||||
|
||||
class SPProcessor2(SPProcessor):
|
||||
def process(self, ds):
|
||||
super().process(ds)
|
||||
ds.vocab.sp_model = self.sp_model
|
||||
ds.vocab.sp_vocab = self.sp_vocab
|
||||
|
||||
# temporary loading function as from_df does not support processors
|
||||
def make_data_bunch_from_df(cls, path: PathOrStr, train_df: DataFrame, valid_df: DataFrame,
|
||||
tokenizer: Tokenizer = None, vocab: Vocab = None, classes: Collection[str] = None,
|
||||
text_cols: IntsOrStrs = 1,
|
||||
label_cols: IntsOrStrs = 0, label_delim: str = None, chunksize: int = 10000,
|
||||
max_vocab: int = 60000,
|
||||
min_freq: int = 2, mark_fields: bool = False, include_bos: bool = True,
|
||||
include_eos: bool = False, processor=None, **kwargs) -> DataBunch:
|
||||
"Create a `TextDataBunch` from DataFrames. `kwargs` are passed to the dataloader creation."
|
||||
assert processor is None or tokenizer is None, "Processor and tokenizer are mutually exclusive."
|
||||
|
||||
if processor is None:
|
||||
processor = fastai.text.data._get_processor(tokenizer=tokenizer, vocab=vocab, chunksize=chunksize, max_vocab=max_vocab,
|
||||
min_freq=min_freq, mark_fields=mark_fields,
|
||||
include_bos=include_bos, include_eos=include_eos)
|
||||
|
||||
if classes is None and is_listy(label_cols) and len(label_cols) > 1: classes = label_cols
|
||||
src = ItemLists(path, TextList.from_df(train_df, path, cols=text_cols, processor=processor),
|
||||
TextList.from_df(valid_df, path, cols=text_cols, processor=processor))
|
||||
if cls == TextLMDataBunch:
|
||||
src = src.label_for_lm()
|
||||
else:
|
||||
if label_delim is not None:
|
||||
src = src.label_from_df(cols=label_cols, classes=classes, label_delim=label_delim)
|
||||
else:
|
||||
src = src.label_from_df(cols=label_cols, classes=classes)
|
||||
return src.databunch(**kwargs)
|
||||
@@ -0,0 +1,3 @@
|
||||
from .datasets import Dataset, ULMFiTDataset
|
||||
from .training import ULMFiT,from_pretrained
|
||||
from .configurations import *
|
||||
@@ -0,0 +1,19 @@
|
||||
from pathlib import Path
|
||||
|
||||
import fire
|
||||
|
||||
import multifit.configurations
|
||||
import multifit
|
||||
|
||||
class Experiment:
|
||||
def new(self):
|
||||
return {n: getattr(multifit.configurations,n) for n in multifit.configurations.__all__}
|
||||
|
||||
def load(self, model_path):
|
||||
return multifit.ULMFiT().load_(Path(model_path))
|
||||
|
||||
def from_pretrained(self):
|
||||
return multifit.from_pretrained
|
||||
|
||||
if __name__ == '__main__':
|
||||
fire.Fire(Experiment())
|
||||
@@ -0,0 +1,115 @@
|
||||
import inspect
|
||||
from .training import *
|
||||
__all__ = [
|
||||
'multifit_paper_version',
|
||||
'multifit1552_fp32', 'multifit_fp32',
|
||||
'multifit_fp32_nl3',
|
||||
|
||||
'multifit1552_fp16','multifit_fp16',
|
||||
'multifit_fp16_nl3',
|
||||
'multifit1552_fp16_nl3_large',
|
||||
|
||||
'multifit_lstm',
|
||||
'multifit1152_lstm_nl3',
|
||||
'multifit1152_lstm_nl3_fp16_large',
|
||||
]
|
||||
|
||||
def multifit1552_fp32(bs=64):
|
||||
self = ULMFiT()
|
||||
self.replace_(
|
||||
label_smoothing_eps=0.0,
|
||||
true_wd=True,
|
||||
wd=0.1,
|
||||
seed=0,
|
||||
fp16=False,
|
||||
bs=bs,
|
||||
use_adam_08=False,
|
||||
early_stopping=None,
|
||||
name=_use_caller_name()
|
||||
)
|
||||
self.arch.replace_(
|
||||
tokenizer_type='fsp',
|
||||
max_vocab=15000,
|
||||
qrnn=True,
|
||||
n_layers=4,
|
||||
n_hid=1552
|
||||
)
|
||||
self.pretrain_lm.replace_(num_epochs=10, drop_mult=0.5, lr=(1e-2 * bs / 48))
|
||||
self.finetune_lm.replace_(num_epochs=10, drop_mult=1.0, lr=(1e-3 * bs / 48))
|
||||
self.classifier.replace_(num_epochs=8, drop_mult=0.5, bs=20, label_smoothing_eps=0.1)
|
||||
return self
|
||||
|
||||
multifit_fp32 = multifit1552_fp32
|
||||
|
||||
def multifit_fp32_nl3():
|
||||
return multifit1552_fp32().replace_(n_layers=3, name=_use_caller_name())
|
||||
|
||||
# FP16
|
||||
|
||||
def multifit1552_fp16():
|
||||
return multifit1552_fp32(bs=128).replace_(fp16=True, name=_use_caller_name())
|
||||
|
||||
def multifit1552_fp16_nl3_large():
|
||||
return multifit1552_fp32(bs=448).replace_(fp16=True, n_layers=3, num_epochs=20, name=_use_caller_name())
|
||||
|
||||
multifit_fp16 = multifit1552_fp16
|
||||
|
||||
def multifit_lstm():
|
||||
return multifit1552_fp32(bs=128).replace_(qrnn=False, n_hid=1552, name=_use_caller_name())
|
||||
|
||||
def multifit1152_lstm_nl3(bs=128):
|
||||
return multifit1552_fp32(bs).replace_(qrnn=False, n_hid=1152, n_layers=3, name=_use_caller_name())
|
||||
|
||||
def multifit1152_lstm_nl3_fp16_large():
|
||||
return multifit1152_lstm_nl3(bs=448).replace_(fp16=True, num_epochs=20, name=_use_caller_name())
|
||||
|
||||
def multifit_fp16_nl3():
|
||||
return multifit1552_fp16().replace_(n_layers=3, name=_use_caller_name())
|
||||
|
||||
def multifit_paper_version():
|
||||
self = ULMFiT()
|
||||
dps = {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
self.replace_(
|
||||
label_smoothing_eps=0.0,
|
||||
label_smoothing_eps_norm_by_classes=True,
|
||||
true_wd=True,
|
||||
wd=0.01, ## important :)
|
||||
seed=0,
|
||||
fp16=False,
|
||||
bs=64,
|
||||
use_adam_08=False,
|
||||
early_stopping=None,
|
||||
clip=0.12,
|
||||
dropout_values=dps,
|
||||
name=_use_caller_name()
|
||||
)
|
||||
self.arch.replace_(
|
||||
tokenizer_type='sp',
|
||||
max_vocab=15000,
|
||||
qrnn=True,
|
||||
n_layers=4,
|
||||
n_hid=1550 # vs 1552
|
||||
)
|
||||
self.pretrain_lm.replace_(num_epochs=10, drop_mult=0.0, lr=5e-3, use_adam_08=True, true_wd=False, wd=1e-7, bs=50,)
|
||||
self.finetune_lm.replace_(num_epochs=20, drop_mult=0.3, lr=1e-3, true_wd=False, wd=1e-7, bs=20)
|
||||
self.classifier.replace_(num_epochs=8, drop_mult=0.5, bs=18, label_smoothing_eps=0.1, early_stopping=None)
|
||||
return self
|
||||
|
||||
def ulmfit_orig():
|
||||
self = multifit_paper_version()
|
||||
self.replace_(
|
||||
seed=None,
|
||||
name=_use_caller_name()
|
||||
)
|
||||
self.arch.replace_(
|
||||
tokenizer_type='f',
|
||||
max_vocab=60000,
|
||||
qrnn=False,
|
||||
n_layers=3,
|
||||
n_hid=1150
|
||||
)
|
||||
return self
|
||||
|
||||
|
||||
def _use_caller_name():
|
||||
return inspect.stack()[1].function
|
||||
@@ -0,0 +1 @@
|
||||
from .dataset import Dataset, ULMFiTDataset, read_clas_csv, read_wiki_articles, ULMFiTTokenizer
|
||||
@@ -7,6 +7,7 @@ Articles are tokenized using the Moses tokenizer. Articles with least than
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
import json
|
||||
import csv
|
||||
|
||||
from shutil import copyfile
|
||||
|
||||
@@ -24,10 +25,10 @@ def get_texts(root):
|
||||
if text.strip() == title:
|
||||
# print('No content continuing...')
|
||||
continue
|
||||
yield (f"={title}=\n"+text)
|
||||
yield text
|
||||
|
||||
|
||||
def write_wikitext(file_path, text_iter, mt, num_tokens, mode='w', num_tokens_article_min=100):
|
||||
def write_wikitext(file_path, text_iter, mt, num_tokens, mode='w'):
|
||||
total_num_tokens = 0
|
||||
print(f'Writing to {file_path}...')
|
||||
i = 0
|
||||
@@ -49,8 +50,8 @@ def write_wikitext(file_path, text_iter, mt, num_tokens, mode='w', num_tokens_ar
|
||||
# calculate length based on tokens; add 1 for newline
|
||||
num_tokens_article += len(tokens) + 1
|
||||
|
||||
if num_tokens_article < num_tokens_article_min:
|
||||
# only use articles that have at least num_tokens_article_min tokens
|
||||
if num_tokens_article < 100:
|
||||
# only use articles that have at least 100 tokens
|
||||
continue
|
||||
|
||||
for tokenized in tokenized_paragraphs:
|
||||
@@ -65,6 +66,42 @@ def write_wikitext(file_path, text_iter, mt, num_tokens, mode='w', num_tokens_ar
|
||||
file_path, i, total_num_tokens))
|
||||
|
||||
|
||||
def wiki2csv(file_path, text_iter, num_tokens):
|
||||
total_num_tokens = 0
|
||||
print(f'Writing to {file_path}...')
|
||||
i = 0
|
||||
|
||||
with open(file_path, 'w', encoding='utf-8') as csvfile:
|
||||
f_out = csv.writer(csvfile, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)
|
||||
for i, text in enumerate(text_iter):
|
||||
num_tokens_article = 0 # count the number of tokens in an article
|
||||
tokenized_paragraphs = []
|
||||
paragraphs = text.split('\n')
|
||||
|
||||
for paragraph in paragraphs:
|
||||
tokenized = paragraph.strip()
|
||||
tokenized_paragraphs.append(tokenized)
|
||||
|
||||
tokens = tokenized.split(' ') # split on whitespace to keep newlines
|
||||
# don't count empty lines
|
||||
tokens = [token for token in tokens if token]
|
||||
|
||||
# calculate length based on tokens; add 1 for newline
|
||||
num_tokens_article += len(tokens) + 1
|
||||
|
||||
if num_tokens_article < 100:
|
||||
# only use articles that have at least 100 tokens
|
||||
continue
|
||||
|
||||
f_out.writerow(['\n'.join(tokenized_paragraphs)])
|
||||
|
||||
total_num_tokens += num_tokens_article + 1
|
||||
if num_tokens is not None and total_num_tokens > num_tokens:
|
||||
break
|
||||
if i % 10000 == 0 and i > 0:
|
||||
print('Processed {:,} documents. Total # tokens: {:,}.'.format(i, total_num_tokens))
|
||||
|
||||
|
||||
def main(args):
|
||||
|
||||
input_path = Path(args.input)
|
||||
@@ -87,7 +124,7 @@ def main(args):
|
||||
token_nums = [2000000, 200000, 200000]
|
||||
for split, token_num in zip(splits, token_nums):
|
||||
sml_file_path = sml_wiki / f'{args.lang}.wiki.{split}.tokens'
|
||||
write_wikitext(sml_file_path, text_iter, mt, token_num, num_tokens_article_min=args.tokens_min)
|
||||
write_wikitext(sml_file_path, text_iter, mt, token_num)
|
||||
lrg_file_path = lrg_wiki / f'{args.lang}.wiki.{split}.tokens'
|
||||
all_file_path = all_wiki / f'{args.lang}.wiki.{split}.tokens'
|
||||
# copy the content of the small file to the large file
|
||||
@@ -97,10 +134,24 @@ def main(args):
|
||||
|
||||
# add the new articles to the existing ones
|
||||
lrg_wiki_train = lrg_wiki / f'{args.lang}.wiki.train.tokens'
|
||||
write_wikitext(lrg_wiki_train, text_iter, mt, 98000000, mode='a', num_tokens_article_min=args.tokens_min)
|
||||
write_wikitext(lrg_wiki_train, text_iter, mt, 98000000, mode='a')
|
||||
all_wiki_train = all_wiki / f'{args.lang}.wiki.train.tokens'
|
||||
copyfile(lrg_wiki_train, all_wiki_train)
|
||||
write_wikitext(all_wiki_train, text_iter, mt, None, mode='a', num_tokens_article_min=args.tokens_min)
|
||||
write_wikitext(all_wiki_train, text_iter, mt, None, mode='a')
|
||||
|
||||
# def main(args):
|
||||
#
|
||||
# input_path = Path(args.input)
|
||||
# output = Path(args.output)
|
||||
# assert input_path.exists(), f'Error: {input_path} does not exist.'
|
||||
# output.mkdir(exist_ok=True)
|
||||
#
|
||||
# lrg_wiki = output / f'{args.lang}-100'
|
||||
# lrg_wiki.mkdir(exist_ok=True)
|
||||
#
|
||||
# text_iter = get_texts(input_path)
|
||||
#
|
||||
# wiki2csv(lrg_wiki / "rawtexts.csv", text_iter, int(2e7))
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
@@ -115,8 +166,5 @@ if __name__ == '__main__':
|
||||
parser.add_argument('-l', '--lang', required=True,
|
||||
help='the iso code of the language of the Wikipedia '
|
||||
'documents, e.g. en, fr, de, etc.')
|
||||
parser.add_argument('-t', '--tokens_min', required=False, type=int, default=100,
|
||||
help='the minimal number of tokens in an article')
|
||||
args = parser.parse_args()
|
||||
main(args)
|
||||
|
||||
@@ -0,0 +1,390 @@
|
||||
import tempfile
|
||||
|
||||
from fastai.text import *
|
||||
from fastai_contrib.text_data import MosesPreprocessingFunc, \
|
||||
make_data_bunch_from_df, SPProcessor2
|
||||
|
||||
def read_wiki_articles(filename):
|
||||
def istitle(line):
|
||||
return len(re.findall(r'^ ?= [^=]* = ?$', line)) != 0
|
||||
articles = []
|
||||
with open(filename, encoding='utf8') as f:
|
||||
lines = f.readlines()
|
||||
current_article = []
|
||||
for i, line in enumerate(lines):
|
||||
current_article.append(line)
|
||||
if i < len(lines) - 2 and lines[i + 1].strip() == "" and istitle(lines[i + 2]):
|
||||
articles.append("".join(current_article))
|
||||
current_article = []
|
||||
articles.append("".join(current_article))
|
||||
print(f"Wiki text was split to {len(articles)} articles")
|
||||
df = pd.DataFrame({'0': np.zeros(len(articles)), 'texts': np.array(articles, dtype=np.object)})
|
||||
if len(df.columns) == 1:
|
||||
df.insert(0, 'label', 0)
|
||||
return df
|
||||
|
||||
|
||||
def read_clas_csv(fn):
|
||||
df = pd.read_csv(fn, header=None).fillna("na")
|
||||
if len(df.columns) == 1:
|
||||
df.insert(0, 'label', 0)
|
||||
return df
|
||||
|
||||
|
||||
@dataclass
|
||||
class Dataset:
|
||||
dataset_path: Path
|
||||
|
||||
noise: float = 0.0
|
||||
limit: int = None
|
||||
|
||||
ds_type: str = None
|
||||
lang: str = None
|
||||
uses_moses: bool = False
|
||||
add_trn_to_lm: bool = True
|
||||
use_tst_for_lm: bool = False
|
||||
label_column: int = 0
|
||||
read_data: Callable = read_clas_csv
|
||||
|
||||
trn_name: Path = 'train.csv'
|
||||
val_name: Path = 'dev.csv'
|
||||
tst_name: Path = 'test.csv'
|
||||
unsup_name: Path = 'unsup.csv'
|
||||
|
||||
def __post_init__(self):
|
||||
self.add_trn_to_lm = True
|
||||
self._trn_df = None
|
||||
self._tst_df = None
|
||||
self._val_df = None
|
||||
|
||||
path = str(self.dataset_path)
|
||||
if 'wiki' in path and len(list(self.dataset_path.glob('*.wiki.*.tokens'))) >= 2:
|
||||
self._post_init_tokenized_wiki()
|
||||
elif 'wiki' in path and len(list(self.dataset_path.glob('wiki.*.tokens'))) >= 2:
|
||||
self._post_init_tokenized_wiki(wiki103=True)
|
||||
elif 'reddit' in path:
|
||||
self._post_init_default_csv(
|
||||
lang='en',
|
||||
uses_moses=False,
|
||||
add_trn_to_lm=True,
|
||||
use_lang_as_prefix=False)
|
||||
elif 'xnli' in path:
|
||||
raise NotImplementedError("Support for XNLI is not implemented yet")
|
||||
elif 'imdb' in path:
|
||||
self._post_init_default_csv(
|
||||
lang='en',
|
||||
uses_moses=False,
|
||||
add_trn_to_lm=True,
|
||||
use_lang_as_prefix=False)
|
||||
elif 'mldoc' in path:
|
||||
self._post_init_default_csv(
|
||||
lang=self._language_from_dataset_path(),
|
||||
uses_moses=False,
|
||||
add_trn_to_lm=False,
|
||||
use_lang_as_prefix=True)
|
||||
elif 'cls' in path:
|
||||
self._post_init_default_csv(
|
||||
lang=self._language_from_dataset_path(),
|
||||
uses_moses=False,
|
||||
add_trn_to_lm=True,
|
||||
use_lang_as_prefix=True)
|
||||
elif 'hate' in path:
|
||||
self._post_init_default_csv(
|
||||
lang=self._language_from_dataset_path(),
|
||||
uses_moses=False,
|
||||
add_trn_to_lm=True,
|
||||
use_lang_as_prefix=True)
|
||||
else:
|
||||
self.read_data = read_clas_csv
|
||||
self.trn_path = self.dataset_path / self.trn_name
|
||||
self.val_path = self.dataset_path / self.val_name
|
||||
self.tst_path = self.dataset_path / self.tst_name
|
||||
self.unsup_path = self.dataset_path / self.unsup_name
|
||||
|
||||
def _post_init_default_csv(self, lang, uses_moses, add_trn_to_lm, use_lang_as_prefix):
|
||||
self.lang = lang
|
||||
self.uses_moses = uses_moses
|
||||
self.add_trn_to_lm = add_trn_to_lm
|
||||
self.use_tst_for_lm = False
|
||||
self.label_column = 0
|
||||
self.read_data = read_clas_csv
|
||||
|
||||
prefix = f"{self.lang}." if use_lang_as_prefix else ""
|
||||
self.trn_path = self.dataset_path / f'{prefix}train.csv'
|
||||
self.val_path = self.dataset_path / f'{prefix}dev.csv'
|
||||
self.tst_path = self.dataset_path / f'{prefix}test.csv'
|
||||
self.unsup_path = self.dataset_path / f'{prefix}unsup.csv'
|
||||
|
||||
def _post_init_tokenized_wiki(self, wiki103=False):
|
||||
self.uses_moses = True
|
||||
self.use_tst_for_lm = False
|
||||
self.add_trn_to_lm = True
|
||||
self.lang = self._language_from_dataset_path()
|
||||
|
||||
self.read_data = read_wiki_articles
|
||||
if wiki103:
|
||||
prefix=""
|
||||
else:
|
||||
prefix=f"{self.lang}."
|
||||
|
||||
self.trn_path = self.dataset_path / f'{prefix}wiki.train.tokens'
|
||||
self.val_path = self.dataset_path / f'{prefix}wiki.valid.tokens'
|
||||
self.tst_path = self.dataset_path / f'{prefix}wiki.test.tokens'
|
||||
self.unsup_path = self.dataset_path / f'{prefix}wiki.unsup.tokens'
|
||||
|
||||
def _language_from_dataset_path(self):
|
||||
#TODO: Duplicate with training function
|
||||
lang, *size = self.dataset_path.name.split('-')
|
||||
if lang == "wikitext":
|
||||
lang = "en"
|
||||
return lang
|
||||
|
||||
def _load_n_cache_supervised_data(self):
|
||||
if not self._trn_df is not None or not self._tst_df is not None or not self._val_df is not None:
|
||||
trn_df = self.read_data(self.trn_path)
|
||||
tst_df = self.read_data(self.tst_path)
|
||||
val_df = self.read_data(self.val_path) if self.val_path.exists() else None
|
||||
|
||||
if val_df is None:
|
||||
print("Validation set not found using 10% of trn")
|
||||
val_len = max(int(len(trn_df) * 0.1), 2)
|
||||
trn_len = len(trn_df) - val_len
|
||||
trn_df, val_df = trn_df[:trn_len], trn_df[trn_len:]
|
||||
|
||||
self._trn_df, self._val_df, self._tst_df = trn_df, val_df, tst_df
|
||||
|
||||
return self._trn_df, self._val_df, self._tst_df
|
||||
|
||||
def load_supervised_data(self):
|
||||
trn_df, val_df, tst_df = self._load_n_cache_supervised_data()
|
||||
if self.noise > 0.0 is not None:
|
||||
trn_df = self._add_noise(trn_df, self.noise)
|
||||
val_df = self._add_noise(val_df, self.noise)
|
||||
|
||||
if self.limit is not None:
|
||||
print("Limiting data set to:", self.limit)
|
||||
trn_df = trn_df[:self.limit]
|
||||
val_df = val_df[:self.limit]
|
||||
|
||||
return trn_df, val_df, tst_df
|
||||
|
||||
def load_unsupervised_data(self):
|
||||
trn_df, val_df, tst_df = self._load_n_cache_supervised_data()
|
||||
unsup_df = self.read_data(self.unsup_path) if self.unsup_path.exists() else None
|
||||
lm_trn_df = pd.concat(
|
||||
([trn_df] if self.add_trn_to_lm else []) +
|
||||
([unsup_df] if unsup_df is not None else []) +
|
||||
([tst_df] if self.use_tst_for_lm else []))
|
||||
|
||||
return lm_trn_df, val_df
|
||||
|
||||
def _add_noise(self, trn_df, noise):
|
||||
count = len(trn_df)
|
||||
labels = trn_df[0].unique()
|
||||
assert np.issubdtype(labels.dtype, np.integer), "noise only works on numerical numbers"
|
||||
modulo = labels.max() + 1
|
||||
idx_to_distrub = np.random.permutation(count)[:int(count * noise)]
|
||||
trn_df.loc[idx_to_distrub, [0]] = (np.random.randint(1, modulo - 1, size=len(idx_to_distrub)) +
|
||||
trn_df.loc[idx_to_distrub][0]) % modulo
|
||||
print(
|
||||
f"Added noise to {len(idx_to_distrub)} examples, only {(count - len(idx_to_distrub)) / count} have correct labels")
|
||||
return trn_df
|
||||
|
||||
|
||||
@dataclass
|
||||
class ULMFiTDataset(Dataset):
|
||||
tokenizer: Tokenizer = None
|
||||
cache_path: Path = None
|
||||
|
||||
def __post_init__(self):
|
||||
super().__post_init__()
|
||||
self._vocab = None
|
||||
|
||||
def load_lm_databunch(self, bs, bptt):
|
||||
lm_suffix = str(bptt) if bptt != 70 else ""
|
||||
lm_suffix += "" if self.use_tst_for_lm else "-notst"
|
||||
data_lm = self.load_n_cache_databunch(f"lm{lm_suffix}",
|
||||
bunch_class=TextLMDataBunch,
|
||||
data_loader=self.load_unsupervised_data,
|
||||
bptt=bptt,
|
||||
bs=bs)
|
||||
|
||||
with (self.cache_path / "itos.pkl").open('wb') as f:
|
||||
pickle.dump(data_lm.vocab.itos, f)
|
||||
self._vocab = data_lm.vocab
|
||||
|
||||
print('Size of vocabulary:', len(data_lm.vocab.itos))
|
||||
print('First 20 words in vocab:', data_lm.vocab.itos[:20])
|
||||
data_lm.lang = self.lang
|
||||
return data_lm
|
||||
|
||||
def _load_vocab(self):
|
||||
if self._vocab is None:
|
||||
self._vocab = self.load_lm_databunch(bs=20, bptt=70).vocab
|
||||
return self._vocab
|
||||
|
||||
def load_clas_databunch(self, bs):
|
||||
vocab = self._load_vocab()
|
||||
|
||||
cls_name = "cls"
|
||||
if self.limit is not None:
|
||||
cls_name = f'{cls_name}limit{self.limit}'
|
||||
if self.noise > 0.0:
|
||||
cls_name = f'{cls_name}noise{self.noise}'
|
||||
|
||||
args = dict(vocab=vocab, bunch_class=TextClasDataBunch, bs=bs)
|
||||
data_cls = self.load_n_cache_databunch(cls_name, data_loader=lambda: self.load_supervised_data()[:2], **args)
|
||||
# Hack to load test dataset with labels
|
||||
data_tst = self.load_n_cache_databunch('tst', data_loader=lambda: self.load_supervised_data()[1:], **args)
|
||||
data_cls.test_dl = data_tst.valid_dl # data_tst.valid_dl holds test data
|
||||
data_cls.lang = self.lang
|
||||
return data_cls
|
||||
|
||||
def load_n_cache_databunch(self, name, bunch_class, data_loader, bs, **args):
|
||||
bunch_path = self.cache_path / name
|
||||
databunch = None
|
||||
if bunch_path.exists():
|
||||
try:
|
||||
databunch = load_data(self.cache_path, name, bs=bs)
|
||||
except (AttributeError, ImportError):
|
||||
print("Unable to load data bunch from cache - pickle issue, running processing again.")
|
||||
if databunch is None:
|
||||
print(f"Running tokenization: '{name}' ...")
|
||||
train_df, valid_df = data_loader()
|
||||
databunch = self.databunch_from_df(bunch_class, train_df, valid_df, **args)
|
||||
databunch.save(name)
|
||||
print(f"Data {name}, trn: {len(databunch.train_ds)}, val: {len(databunch.valid_ds)}")
|
||||
return databunch
|
||||
|
||||
def databunch_from_df(self, bunch_class, train_df, valid_df, **args):
|
||||
args.update(**self.tokenizer.get_fastai_config(dataset_uses_moses=self.uses_moses)) # TODO depends on the previous model
|
||||
databunch = make_data_bunch_from_df(cls=bunch_class,
|
||||
path=self.cache_path,
|
||||
train_df=train_df,
|
||||
valid_df=valid_df,
|
||||
mark_fields=True,
|
||||
text_cols=list(train_df.columns.values)[1:],
|
||||
**args)
|
||||
return databunch
|
||||
|
||||
|
||||
@dataclass
|
||||
class ULMFiTTokenizer:
|
||||
arch: Any # should be ULMFiTArchitecture, we use Any to avoid circular dependencies between imports
|
||||
pretrained_path: Path = None
|
||||
|
||||
def __post_init__(self):
|
||||
self.temp_dir = None
|
||||
if self.pretrained_path is None:
|
||||
self.temp_dir = tempfile.TemporaryDirectory()
|
||||
self.pretrained_path = Path(self.temp_dir.name)
|
||||
|
||||
def save(self, new_path: Path, vocab: Vocab = None, learn: Learner = None):
|
||||
"""
|
||||
In case of subwoard vocabularies reuse the base model vocabulary during tokenization.
|
||||
For word tokenization we still generate new vocabulary for each dataset,
|
||||
and we expect finetuning to handle the conversion
|
||||
"""
|
||||
|
||||
def copy_sp(path):
|
||||
print(f"Copy sp model from {path} to {new_path}")
|
||||
shutil.copy(str(path / 'itos.pkl'), str(new_path))
|
||||
shutil.copy(str(path / 'spm.model'), str(new_path))
|
||||
shutil.copy(str(path / 'spm.vocab'), str(new_path))
|
||||
|
||||
# reuse base model sentencepiece vocabulary
|
||||
new_path.mkdir(exist_ok=True, parents=True)
|
||||
if self.pretrained_path is None or self.pretrained_path.resolve() == new_path.resolve():
|
||||
return
|
||||
#
|
||||
# if (self.pretrained_path.parent / 'spm.vocab').exists():
|
||||
# copy_sp(self.pretrained_path.parent)
|
||||
|
||||
if (self.pretrained_path / 'spm.vocab').exists():
|
||||
copy_sp(self.pretrained_path)
|
||||
|
||||
if learn is not None:
|
||||
vocab = learn.data.vocab
|
||||
if vocab is not None:
|
||||
with (new_path / "itos.pkl").open('wb') as f:
|
||||
pickle.dump(vocab.itos, f)
|
||||
|
||||
def get_processor(self, dataset_uses_moses=False):
|
||||
return {
|
||||
'fsp': self._get_processor_sentence_piece,
|
||||
'f': self._get_processor_pure_fastai,
|
||||
'm': self._get_processor_pure_moses,
|
||||
'mf': self._get_processor_moses_fastai,
|
||||
|
||||
'sp': self._get_processor_sentence_piece, # deprecated
|
||||
'v': self._get_processor_pure_moses, # deprecated
|
||||
'vf': self._get_processor_moses_fastai, # deprecated
|
||||
}.get(self.arch.tokenizer_type)(dataset_uses_moses)
|
||||
|
||||
def get_vocab(self): return Vocab.load(self.pretrained_path / 'itos.pkl')
|
||||
|
||||
def get_fastai_config(self, dataset_uses_moses=False, add_open_file_processor=False):
|
||||
processor = self.get_processor(dataset_uses_moses)
|
||||
openfile = [OpenFileProcessor()] if add_open_file_processor else []
|
||||
return {'processor': openfile + [processor]}
|
||||
|
||||
@property
|
||||
def prefix(self):
|
||||
return f"{self.arch.tokenizer}{self.arch.max_vocab // 1000}k"
|
||||
|
||||
def _get_processor_sentence_piece(self, ds_uses_moses):
|
||||
moses_preproc = [MosesPreprocessingFunc(self.arch.lang)] if not ds_uses_moses else []
|
||||
|
||||
sp_model = self.pretrained_path / 'spm.model'
|
||||
if not sp_model.is_file():
|
||||
sp_model = None
|
||||
sp_vocab = self.pretrained_path / 'spm.vocab'
|
||||
if not sp_vocab.is_file():
|
||||
sp_vocab = None
|
||||
processor = SPProcessor2(
|
||||
pre_rules=moses_preproc + defaults.text_pre_rules,
|
||||
mark_fields=True,
|
||||
vocab_sz=self.arch.max_vocab,
|
||||
sp_model=sp_model,
|
||||
sp_vocab=sp_vocab,
|
||||
lang=self.arch.lang,
|
||||
tmp_dir=self.pretrained_path.absolute() # absolute make sure that dataset path is not added as prefix
|
||||
)
|
||||
return processor
|
||||
|
||||
def _default_processor(self, fastai_tokenizer=None):
|
||||
if fastai_tokenizer is None:
|
||||
fastai_tokenizer = Tokenizer(SpacyTokenizer, self.arch.lang)
|
||||
return [TokenizeProcessor(tokenizer=fastai_tokenizer), NumericalizeProcessor(max_vocab=self.arch.max_vocab)]
|
||||
|
||||
def _get_processor_pure_moses(self, ds_uses_moses):
|
||||
#TODO make sure processor doesnot return openfile
|
||||
moses_preproc = [MosesPreprocessingFunc(self.arch.lang)] if not ds_uses_moses else []
|
||||
tokenizer = Tokenizer(tok_func=BaseTokenizer,
|
||||
lang=self.arch.lang,
|
||||
pre_rules=moses_preproc,
|
||||
post_rules=[])
|
||||
return self._default_processor(tokenizer)
|
||||
|
||||
def _get_processor_moses_fastai(self, ds_uses_moses):
|
||||
moses_preproc = [MosesPreprocessingFunc(self.arch.lang)] if not ds_uses_moses else []
|
||||
tokenizer = Tokenizer(tok_func=BaseTokenizer,
|
||||
lang=self.arch.lang,
|
||||
pre_rules=moses_preproc + defaults.text_pre_rules,
|
||||
post_rules=defaults.text_post_rules)
|
||||
return self._default_processor(tokenizer)
|
||||
|
||||
def _get_processor_pure_fastai(self, ds_uses_moses):
|
||||
if not ds_uses_moses:
|
||||
warn("Make sure your base model was not pretrained on moses tokenized Wikipedia (default for multifit).")
|
||||
tokenizer = Tokenizer(tok_func=SpacyTokenizer, lang=self.arch.lang)
|
||||
return self._default_processor(tokenizer)
|
||||
|
||||
def cleanup(self):
|
||||
if self.temp_dir is not None:
|
||||
self.temp_dir.cleanup()
|
||||
self.temp_dir = None
|
||||
|
||||
def __del__(self):
|
||||
self.cleanup()
|
||||
|
||||
@@ -10,7 +10,7 @@ from pathlib import Path
|
||||
|
||||
import fire
|
||||
|
||||
from fastai_contrib.utils import replace_number, UNK
|
||||
from .utils import replace_number, UNK
|
||||
|
||||
|
||||
def build_vocab(file_path, cutoff=3):
|
||||
@@ -1,25 +1,17 @@
|
||||
"""
|
||||
Utility methods for data processing.
|
||||
"""
|
||||
import pathlib
|
||||
import fire
|
||||
from fastai.text import *
|
||||
|
||||
import shutil
|
||||
import pathlib
|
||||
import tarfile
|
||||
from typing import Dict, Tuple, List
|
||||
|
||||
import fire
|
||||
from sacremoses import MosesTokenizer
|
||||
|
||||
from fastai.text import *
|
||||
|
||||
EOS = 'xxeos' # fastai does not use eos, but we do
|
||||
SEP = 'xxsep' # special separator token for NLI
|
||||
|
||||
def replace_std_toks(x:str) -> str:
|
||||
"Replace standard token names with fastai supported tokens"
|
||||
# We change tokens to f'xx{token_name}' as it is not split by Moses tokenizer,
|
||||
# while f'<{token_name}>' is being split to: '<' f'{token_name}' '>'
|
||||
return x.replace('<unk>', UNK).replace('<bos>', BOS).replace('<eos>', EOS)
|
||||
|
||||
PAD_TOKEN_ID = 1
|
||||
IMDB, XNLI, TRN, VAL, TST, EN = 'imdb', 'xnli', 'train', 'val', 'test', 'en'
|
||||
DATASETS = ['imdb', 'xnli']
|
||||
@@ -34,97 +26,6 @@ CLASSES = ['neg', 'pos', 'unsup']
|
||||
number_match_re = re.compile(r'^([0-9]+[,.]?)+$')
|
||||
number_split_re = re.compile(r'([,.])')
|
||||
|
||||
class MosesPreprocessingFunc():
|
||||
|
||||
def __init__(self, lang: str):
|
||||
self.mt = MosesTokenizer(lang)
|
||||
|
||||
def __call__(self, t: str) -> str:
|
||||
return self.mt.tokenize(t, return_str=True, escape=True)
|
||||
|
||||
class SentencePieceTokenizer(Tokenizer):
|
||||
"Put together rules and a tokenizer function to tokenize text with multiprocessing."
|
||||
def __init__(self, spm_model, lang:str='en', pre_rules:ListRules=None,
|
||||
post_rules:ListRules=None, special_cases:Collection[str]=None, n_cpus:int=None):
|
||||
# moses is added to preprocessing functions
|
||||
super().__init__(self.tok_fun_with_sp, lang, pre_rules, post_rules, special_cases, n_cpus)
|
||||
self.spm_model = spm_model
|
||||
|
||||
def tok_fun_with_sp(self, lang):
|
||||
try:
|
||||
import sentencepiece as spm
|
||||
except ImportError:
|
||||
raise Exception('sentencepiece module is missing: run `pip install sentencepiece`')
|
||||
tok = BaseTokenizer(lang)
|
||||
tok.sp = spm.SentencePieceProcessor()
|
||||
tok.sp.Load(str(self.spm_model))
|
||||
return tok
|
||||
|
||||
def process_text(self, t:str, tok:BaseTokenizer) -> List[str]:
|
||||
"Process one text `t` with tokenizer `tok`."
|
||||
toks = super().process_text(t, tok)
|
||||
toks = tok.sp.EncodeAsPieces(" ".join(toks))
|
||||
return toks
|
||||
full_char_coverage_langs = ["bg", "cs", "da", "de", "el", "en", "es", "et", "fi", "fr", "ga", "hr", "hu",
|
||||
"it","lt","lv","mt","nl","pl","pt","ro","sk","sl","sv"] # all European langs
|
||||
|
||||
def get_sentencepiece(cache_dir:PathOrStr, load_text, pre_rules: ListRules=None, post_rules:ListRules=None,
|
||||
vocab_size:int=30000, model_type:str='unigram', input_sentence_size:int=1E7, lang='en'):
|
||||
try:
|
||||
import sentencepiece as spm
|
||||
except ImportError:
|
||||
raise Exception('sentencepiece module is missing: run `pip install sentencepiece`')
|
||||
|
||||
cache_dir = pathlib.Path(cache_dir)
|
||||
pre_rules = pre_rules if pre_rules is not None else defaults.text_pre_rules
|
||||
post_rules = post_rules if post_rules is not None else defaults.text_post_rules
|
||||
|
||||
special_cases = defaults.text_spec_tok
|
||||
if not os.path.isfile(cache_dir / 'spm.model') or not os.path.isfile(cache_dir / f'itos.pkl'):
|
||||
# load the text from the train tokens file
|
||||
text = load_text()
|
||||
text = filter(lambda x: len(x.rstrip(" ")), text)
|
||||
text = (reduce(lambda t, rule: rule(t), pre_rules, line) for line in text)
|
||||
def cleanup_n_postprocess(t):
|
||||
t = t.split()
|
||||
for r in post_rules:
|
||||
t = r(t)
|
||||
return ' '.join(t)
|
||||
text = map(cleanup_n_postprocess, text)
|
||||
raw_text_path = cache_dir / 'all_text.txt'
|
||||
with open(raw_text_path, 'w') as f: f.write("\n".join(text))
|
||||
|
||||
char_coverage = 1 if lang in full_char_coverage_langs else 0.99
|
||||
|
||||
sp_params = [
|
||||
f"--input={raw_text_path}",
|
||||
f"--character_coverage={char_coverage}",
|
||||
f"--unk_id={len(defaults.text_spec_tok)}",
|
||||
f"--pad_id=-1",
|
||||
f"--bos_id=-1",
|
||||
f"--eos_id=-1",
|
||||
f"--max_sentence_length=20480",
|
||||
f"--input_sentence_size={int(input_sentence_size)}",
|
||||
f"--user_defined_symbols={','.join(special_cases)}",
|
||||
f"--model_prefix={cache_dir/'spm'}",
|
||||
f"--vocab_size={vocab_size} --model_type={model_type}"]
|
||||
spm.SentencePieceTrainer.Train(" ".join(sp_params))
|
||||
|
||||
with open(cache_dir / 'spm.vocab', 'r') as f:
|
||||
vocab = [line.split('\t')[0] for line in f.readlines()]
|
||||
|
||||
pickle.dump(vocab, open(cache_dir/ f'itos.pkl', 'wb'))
|
||||
# todo add post rules
|
||||
vocab = Vocab(pickle.load(open(cache_dir / f'itos.pkl', 'rb')))
|
||||
# We cannot use lambdas or local methods here, since `tok_func` needs to be
|
||||
# pickle-able in order to be called in subprocesses when multithread tokenizing
|
||||
tokenizer = SentencePieceTokenizer(cache_dir/'spm.model',
|
||||
lang=lang,
|
||||
pre_rules=pre_rules,
|
||||
post_rules=post_rules)
|
||||
return {'tokenizer': tokenizer, 'vocab': vocab}
|
||||
|
||||
|
||||
def clear_cache_directory(path:PathOrStr, cache_name:str='tmp'):
|
||||
path = pathlib.Path(path)
|
||||
shutil.rmtree(path / cache_name)
|
||||
@@ -137,7 +38,6 @@ def get_texts(path):
|
||||
labels.append(idx)
|
||||
return np.array(texts), np.array(labels)
|
||||
|
||||
|
||||
def ensure_paths_exists(*paths, message="One or more required files cannot be found."):
|
||||
error = False
|
||||
for path in paths:
|
||||
@@ -147,12 +47,6 @@ def ensure_paths_exists(*paths, message="One or more required files cannot be fo
|
||||
if error:
|
||||
raise FileNotFoundError(message)
|
||||
|
||||
def get_data_folder() -> Path:
|
||||
"""
|
||||
return data folder to use for future processing
|
||||
"""
|
||||
return (pathlib.Path(__file__).parent.parent / "data")
|
||||
|
||||
def get_scripts_folder():
|
||||
"""
|
||||
return data folder to use for future processing
|
||||
@@ -227,15 +121,7 @@ def read_xnli(dir_path, lang, split, spm_path=None) -> Tuple[List[List[str]], Li
|
||||
file_path = f'XNLI-MT-1.0/xnli/{file_name}'
|
||||
file_path = dir_path / file_path
|
||||
|
||||
if spm_path is not None:
|
||||
tokenizer = SentencePieceTokenizer(spm_path,
|
||||
use_moses=False,
|
||||
lang=lang)
|
||||
tok = tokenizer.tok_fun_with_sp(lang)
|
||||
tokenize = lambda x: tokenizer.process_text(x, tok)
|
||||
print("WARNING: Sentence Piece is not tested on XNLI yet")
|
||||
else:
|
||||
tokenize = lambda x: x.split(' ')
|
||||
tokenize = lambda x: x.split(' ')
|
||||
|
||||
toks, lbls = [], []
|
||||
print(f'Reading {file_path}...')
|
||||
@@ -305,14 +191,12 @@ def read_file(file_path, outname=None):
|
||||
with open(file_path, encoding='utf8') as f:
|
||||
text = f.readlines()
|
||||
df = pd.DataFrame(
|
||||
{'text': text, 'labels': np.zeros(len(text))},
|
||||
{'text': np.array(text), 'labels': np.zeros(len(text))},
|
||||
columns=['labels', 'text'])
|
||||
if outname is not None:
|
||||
df.to_csv(file_path.parent / f'{outname}.csv', header=False, index=False)
|
||||
return df
|
||||
|
||||
|
||||
|
||||
def read_whitespace_file(filepath):
|
||||
"""Reads a file and prepares the tokens."""
|
||||
tokens = []
|
||||
@@ -328,7 +212,6 @@ class DataStump:
|
||||
self.ids = ids
|
||||
self.loss_func = F.cross_entropy
|
||||
|
||||
|
||||
def validate(model, ids, bptt=2000):
|
||||
"""
|
||||
Return the validation loss and perplexity of a model
|
||||
@@ -341,7 +224,7 @@ def validate(model, ids, bptt=2000):
|
||||
model.eval()
|
||||
model.reset()
|
||||
total_loss, num_examples = 0., 0
|
||||
for inputs, targets in tqdm(data):
|
||||
for inputs, targets in data:
|
||||
outputs, raws, outs = model(to_device(inputs, None))
|
||||
p_vocab = F.softmax(outputs, 1)
|
||||
for i, pv in enumerate(p_vocab):
|
||||
@@ -351,7 +234,6 @@ def validate(model, ids, bptt=2000):
|
||||
mean = total_loss / num_examples # divide by total number of tokens
|
||||
return mean, np.exp(mean)
|
||||
|
||||
|
||||
class TextReader():
|
||||
""" Returns a language model iterator that iterates through batches that are of length N(bptt,5)
|
||||
The first batch returned is always bptt+25; the max possible width. This is done because of they way that pytorch
|
||||
@@ -0,0 +1,585 @@
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
import dataclasses
|
||||
from fastai.callbacks import CSVLogger, SaveModelCallback
|
||||
from fastai.text import *
|
||||
|
||||
from multifit.datasets import ULMFiTDataset,ULMFiTTokenizer
|
||||
|
||||
CLS_BEST = 'cls_best'
|
||||
LM_BEST = "lm_best"
|
||||
ENC_BEST = "enc_best"
|
||||
|
||||
|
||||
def detect_lang_from_dataset_path(dataset_path:Path):
|
||||
lang, *size = dataset_path.name.split('-')
|
||||
if lang == "wikitext":
|
||||
lang = "en"
|
||||
if len(lang) == 2:
|
||||
return lang
|
||||
return None
|
||||
|
||||
|
||||
@dataclass
|
||||
class Params:
|
||||
def replace_(self, _verbose_diff=False, **changes):
|
||||
for f in dataclasses.fields(self):
|
||||
if f.name in changes:
|
||||
v = changes[f.name]
|
||||
if f.type == Path and v is not None:
|
||||
v = Path(v)
|
||||
orig = getattr(self, f.name)
|
||||
if orig != v and _verbose_diff:
|
||||
print(f"{self.__class__.__name__} Replacing {f.name} '{orig}' with '{v}")
|
||||
setattr(self, f.name, v)
|
||||
return self
|
||||
|
||||
|
||||
@dataclass
|
||||
class ULMFiTArchitecture(Params):
|
||||
tokenizer_type: str = "f"
|
||||
max_vocab: int = 60000
|
||||
lang: str = None
|
||||
|
||||
emb_sz: int = awd_lstm_lm_config['emb_sz']
|
||||
n_hid: int = awd_lstm_lm_config['n_hid']
|
||||
n_layers: int = awd_lstm_lm_config['n_layers']
|
||||
qrnn: bool = awd_lstm_lm_config['qrnn']
|
||||
|
||||
def model_name(self, name=""):
|
||||
model_suffix = '' # if self.lmseed is None else f'_lmseed-{self.lmseed}'
|
||||
model_prefix = 'qrnn' if self.qrnn else 'lstm'
|
||||
|
||||
model_name = f"{model_prefix}_{name}{model_suffix}.m"
|
||||
return model_name
|
||||
|
||||
def dataset_cache_suffix(self):
|
||||
tokenizer_prefix = f"{self.tokenizer_type}{self.max_vocab // 1000}k"
|
||||
return f'models/{tokenizer_prefix}'
|
||||
|
||||
def dataset(self, dataset_path_or_object, tokenizer=None, **args):
|
||||
if hasattr(dataset_path_or_object, 'load_lm_databunch'):
|
||||
return dataset_path_or_object
|
||||
if dataset_path_or_object is None:
|
||||
return None
|
||||
ds_path = Path(dataset_path_or_object)
|
||||
cache_path = ds_path / self.dataset_cache_suffix()
|
||||
if tokenizer is not None:
|
||||
tokenizer.save(cache_path) # saving the tokenizer to the cache_path so that it can be reused later.
|
||||
# TODO add proper caching prefixed with tokenizer hash.
|
||||
tokenizer = self.new_tokenizer(cache_path)
|
||||
return ULMFiTDataset(dataset_path=ds_path, cache_path=cache_path, tokenizer=tokenizer, **args)
|
||||
|
||||
def new_tokenizer(self, pretrained_path=None):
|
||||
"gets untrained tokenizer in that stores its data in tmp, use .save once trained"
|
||||
return ULMFiTTokenizer(arch=self, pretrained_path=pretrained_path)
|
||||
|
||||
def set_seed(seed, name):
|
||||
if seed is not None:
|
||||
print(f"Setting {name} seed to {seed}")
|
||||
torch.manual_seed(seed)
|
||||
torch.backends.cudnn.deterministic = True
|
||||
torch.backends.cudnn.benchmark = False
|
||||
np.random.seed(seed)
|
||||
|
||||
|
||||
def to_json_serializable(d):
|
||||
n = {}
|
||||
for k, v in d.items():
|
||||
if isinstance(v, dict):
|
||||
n[k] = to_json_serializable(v)
|
||||
elif isinstance(v, (float, int, str, list, tuple)):
|
||||
n[k] = v
|
||||
elif v is None:
|
||||
n[k] = v
|
||||
else:
|
||||
n[k] = str(v)
|
||||
return n
|
||||
|
||||
|
||||
def rename_dict_keys(d, rename_func):
|
||||
for k in list(d.keys()):
|
||||
d[rename_func(k)] = d.pop(k)
|
||||
|
||||
def convert_old_models_keys_hook(state_dict, *_, **__):
|
||||
rename_dict_keys(state_dict, lambda k:
|
||||
k.replace('linear', 'layers.0.linear') if 'layers.0' not in k else k)
|
||||
|
||||
def convert_new_models_keys_hook(state_dict, *_, **__):
|
||||
rename_dict_keys(state_dict, lambda k: k.replace('layers.0.linear', 'linear'))
|
||||
|
||||
def patch_learner(learn):
|
||||
encoder = get_model(learn.model)[0]
|
||||
if hasattr(encoder, 'module'): encoder = encoder.module
|
||||
if hasattr(encoder.rnns[0], 'layers'):
|
||||
encoder._register_load_state_dict_pre_hook(convert_old_models_keys_hook)
|
||||
learn.model._register_load_state_dict_pre_hook(convert_old_models_keys_hook)
|
||||
else:
|
||||
encoder._register_load_state_dict_pre_hook(convert_new_models_keys_hook)
|
||||
learn.model._register_load_state_dict_pre_hook(convert_new_models_keys_hook)
|
||||
return learn
|
||||
|
||||
@dataclass
|
||||
class ULMFiTTrainingCommand(Params):
|
||||
seed: int = 0
|
||||
name: str = None
|
||||
arch: ULMFiTArchitecture = field(repr=False, default=None)
|
||||
experiment_path: Path = None
|
||||
dataset_path: Path = None
|
||||
|
||||
@property
|
||||
def model_name(self):
|
||||
return (self.name or self.arch.model_name()) + (
|
||||
"" if self.seed is None or self.seed == 0 or "seed" in self.name else f"seed{self.seed}")
|
||||
|
||||
@property
|
||||
def info_json(self):
|
||||
return self.__class__.__name__.lower().replace("ulmfit", "") + ".json"
|
||||
|
||||
def _set_dataset_(self, dataset_or_path, tokenizer):
|
||||
#TODO: refactor, this bit is unclear (set_dataset that does nothing when is None passed?)
|
||||
dataset_or_path = dataset_or_path or self.dataset_path or getattr(self, 'base', self).dataset_path
|
||||
dataset = self.arch.dataset(dataset_or_path, tokenizer=tokenizer)
|
||||
self.dataset_path = dataset.dataset_path
|
||||
return dataset
|
||||
|
||||
@property
|
||||
def dataset(self):
|
||||
return self.arch.dataset(self.dataset_path, self.tokenizer)
|
||||
|
||||
@property
|
||||
def tokenizer(self):
|
||||
if self.experiment_path is None:
|
||||
return None
|
||||
return ULMFiTTokenizer(arch=self.arch, pretrained_path=self.experiment_path)
|
||||
|
||||
def save_paramters(self):
|
||||
params = dataclasses.asdict(self)
|
||||
base_exp_path = params.pop('base', {}).pop('experiment_path', None)
|
||||
params['base'] = base_exp_path
|
||||
exp_path = params.get('experiment_path', None)
|
||||
if exp_path:
|
||||
fn = self.info_json
|
||||
print("Saving dump to", exp_path / fn)
|
||||
json_str = json.dumps(to_json_serializable(params), indent=2)
|
||||
with (exp_path / fn).open("w") as f:
|
||||
f.write(json_str)
|
||||
return json_str
|
||||
|
||||
def load_(self, experiment_path, tantetive=True, update_arch=True, silent=False):
|
||||
fn = experiment_path / self.info_json
|
||||
if not fn.exists():
|
||||
if not tantetive:
|
||||
warn(f"Unable to load experiment_path {experiment_path}")
|
||||
return False
|
||||
print(f"Loading {fn}")
|
||||
with fn.open('r') as f:
|
||||
d = json.load(f)
|
||||
base = d.pop('base', None)
|
||||
arch = d.pop('arch')
|
||||
if hasattr(self, 'base'):
|
||||
self.base.load_(Path(base), tantetive=True, update_arch=False, silent=silent)
|
||||
|
||||
# compatiblity with older info.json formats where lang was not stored
|
||||
self.name = experiment_path.name # V ./de-1/models/fsp15k/multifit_fp16 -> ./de-1
|
||||
dataset_path = Path(d.get('dataset_path', experiment_path.parent.parent.parent))
|
||||
d['dataset_path'] = dataset_path
|
||||
d['experiment_path'] = experiment_path
|
||||
arch['lang'] = arch.get('lang', None) or detect_lang_from_dataset_path(dataset_path)
|
||||
arch['tokenizer_type'] = arch.pop('tokenizer', arch.get('tokenizer_type', None))
|
||||
|
||||
if update_arch:
|
||||
self.arch.replace_(_verbose_diff=not silent, **arch)
|
||||
self.replace_(_verbose_diff=not silent, **d)
|
||||
return arch
|
||||
|
||||
|
||||
@dataclass
|
||||
class ULMFiTPretraining(ULMFiTTrainingCommand):
|
||||
num_epochs: int = 10
|
||||
bs: int = 20
|
||||
bptt: int = 70
|
||||
drop_mult: float = 1.0
|
||||
dropout_values: dict = field(default_factory=dict)
|
||||
label_smoothing_eps: float = 0.0
|
||||
label_smoothing_eps_norm_by_classes: bool = True
|
||||
use_adam_08: bool = False
|
||||
true_wd: bool = True
|
||||
wd: bool = 0.01
|
||||
clip: float = None
|
||||
fp16: bool = False
|
||||
lr: float = 5e-3
|
||||
|
||||
def get_learner(self, data_lm, **additional_trn_args):
|
||||
config = awd_lstm_lm_config.copy()
|
||||
config.update(emb_sz=self.arch.emb_sz, n_hid=self.arch.n_hid, n_layers=self.arch.n_layers, qrnn=self.arch.qrnn,
|
||||
**self.dropout_values)
|
||||
|
||||
trn_args = dict(drop_mult=self.drop_mult, true_wd=self.true_wd, wd=self.wd,
|
||||
pretrained=False, clip=self.clip)
|
||||
trn_args.update(**additional_trn_args)
|
||||
print("Training args: ", trn_args, "config: ", config)
|
||||
learn = language_model_learner(data_lm,
|
||||
AWD_LSTM,
|
||||
config=config,
|
||||
model_dir=self.model_name,
|
||||
**trn_args)
|
||||
learn = patch_learner(learn)
|
||||
# compared to standard Adam, we set beta_1 to 0.8
|
||||
if self.use_adam_08:
|
||||
learn.opt_func = partial(optim.Adam, betas=(0.8, 0.99))
|
||||
|
||||
learn.callback_fns += [partial(CSVLogger, filename=f"{learn.model_dir}/lm-history")]
|
||||
if self.label_smoothing_eps > 0.0:
|
||||
eps = self.label_smoothing_eps
|
||||
if self.label_smoothing_eps_norm_by_classes:
|
||||
eps = eps/ learn.data.c
|
||||
print("Using Label smoothing with eps = ", eps)
|
||||
learn.loss_func = FlattenedLoss(LabelSmoothingCrossEntropy, eps=eps)
|
||||
|
||||
set_seed(self.seed, "LM training seed")
|
||||
if self.fp16:
|
||||
learn.to_fp16()
|
||||
return learn
|
||||
|
||||
def _fit_schedule(self, learn):
|
||||
print("Training lm from random weights")
|
||||
learn.unfreeze()
|
||||
learn.fit_one_cycle(self.num_epochs, self.lr, (0.8, 0.7))
|
||||
|
||||
def train_(self, dataset_or_path, tokenizer=None, **train_config):
|
||||
if self.arch.lang is None:
|
||||
lang = detect_lang_from_dataset_path(Path(dataset_or_path))
|
||||
if lang is None:
|
||||
warn("Unable to detect language from dataset path assuming English, use replace_(lang='??') change it.")
|
||||
lang = 'en'
|
||||
self.arch.lang = lang
|
||||
self.replace_(**train_config, _strict=True)
|
||||
set_seed(self.seed, "LM weights seed")
|
||||
if tokenizer is None:
|
||||
if hasattr(self, 'base'):
|
||||
tokenizer = self.base.tokenizer
|
||||
else:
|
||||
tokenizer = self.arch.new_tokenizer()
|
||||
|
||||
dataset = self._set_dataset_(dataset_or_path, tokenizer)
|
||||
learn = self.get_learner(data_lm=dataset.load_lm_databunch(bs=self.bs, bptt=self.bptt))
|
||||
experiment_path = learn.path / learn.model_dir
|
||||
print("Experiment", experiment_path)
|
||||
if self.num_epochs > 0:
|
||||
self._fit_schedule(learn)
|
||||
|
||||
self.experiment_path = experiment_path
|
||||
tokenizer.save(self.experiment_path, learn=learn)
|
||||
learn.to_fp32()
|
||||
learn.save_encoder(ENC_BEST)
|
||||
learn.save(LM_BEST, with_opt=False)
|
||||
learn.destroy()
|
||||
self.save_paramters()
|
||||
print("Language model saved to", self.experiment_path)
|
||||
|
||||
def validate(self):
|
||||
raise NotImplementedError("The validation on the language model is not implemented.")
|
||||
|
||||
@property
|
||||
def model_fnames(self):
|
||||
if self.experiment_path:
|
||||
model_path = self.experiment_path.absolute()
|
||||
cache_path = (model_path if (model_path / "itos.pkl").exists() else model_path.parent)
|
||||
return [model_path / LM_BEST, cache_path /'itos']
|
||||
return None
|
||||
|
||||
@property
|
||||
def encoder_fname(self):
|
||||
if self.experiment_path:
|
||||
return (self.experiment_path / ENC_BEST).absolute()
|
||||
return None
|
||||
|
||||
|
||||
@dataclass
|
||||
class ULMFiTFinetuning(ULMFiTPretraining):
|
||||
base: ULMFiTPretraining = field(repr=False, default=None)
|
||||
|
||||
def __post_init__(self):
|
||||
self.lr = 1e-3
|
||||
|
||||
def get_learner(self, data_lm, **additional_trn_args):
|
||||
pretrained_fnames = None if self.base is None else self.base.model_fnames
|
||||
# data_lm.lang is added after dataloading
|
||||
if pretrained_fnames is None and data_lm.lang != 'en':
|
||||
warn(f"You are using fastai english langauge model for {data_lm.lang}, you might be better off with just random weights.")
|
||||
learn = super().get_learner(data_lm, **additional_trn_args)
|
||||
# we don't use pretrained_fnames param so that we can add load_state_dict_hook
|
||||
if pretrained_fnames is not None:
|
||||
print("Loading pretrained weights: ", pretrained_fnames)
|
||||
fnames = [learn.path / learn.model_dir / f'{fn}.{ext}' for fn, ext in zip(pretrained_fnames, ['pth', 'pkl'])]
|
||||
learn.load_pretrained(*fnames)
|
||||
learn.freeze()
|
||||
return learn
|
||||
|
||||
def _fit_schedule(self, learn):
|
||||
if self.base is not None and self.base.model_fnames:
|
||||
print("Fitting using 2 cycle fit schedule")
|
||||
learn.freeze_to(-1)
|
||||
learn.fit_one_cycle(1, self.lr * 10, moms=(0.8, 0.7))
|
||||
learn.unfreeze()
|
||||
learn.fit_one_cycle(self.num_epochs, self.lr, moms=(0.8, 0.7))
|
||||
else:
|
||||
super()._fit_schedule(learn)
|
||||
|
||||
@dataclass
|
||||
class ULMFiTClassifier(ULMFiTTrainingCommand):
|
||||
bs: int = 20
|
||||
num_epochs: int = 10
|
||||
drop_mult: float = 0.5
|
||||
dropout_values: dict = field(default_factory=dict)
|
||||
wd: float = 0.01
|
||||
clip: float = None
|
||||
label_smoothing_eps: float = 0.0
|
||||
label_smoothing_eps_norm_by_classes: bool = False
|
||||
weighted_cross_entropy: tuple = None
|
||||
early_stopping: str = 'accuracy'
|
||||
fit_schedule: str = '1cycle'
|
||||
base: ULMFiTFinetuning = field(repr=False, default=None)
|
||||
random_init: bool = False
|
||||
seed: int = 0
|
||||
bptt: int = 70
|
||||
fp16: bool = False
|
||||
arch: ULMFiTArchitecture = None
|
||||
|
||||
def get_learner(self, data_clas, eval_only=False, **additional_trn_args):
|
||||
assert self.weighted_cross_entropy is None or self.label_smoothing_eps == 0, "Label smoohting not implemented with weighted_cross_entropy"
|
||||
if self.weighted_cross_entropy is not None:
|
||||
loss_func = CrossEntropyFlat(weight=torch.tensor(self.weighted_cross_entropy, dtype=torch.float32).cuda())
|
||||
elif self.label_smoothing_eps > 0.0:
|
||||
eps = self.label_smoothing_eps
|
||||
if self.label_smoothing_eps_norm_by_classes:
|
||||
eps = eps / data_clas.c
|
||||
print("Using Label smoothing with eps = ", eps)
|
||||
loss_func = FlattenedLoss(LabelSmoothingCrossEntropy, eps=eps)
|
||||
else:
|
||||
loss_func = None
|
||||
|
||||
set_seed(self.seed, "Classifier weights seed")
|
||||
config = awd_lstm_clas_config.copy()
|
||||
config.update(emb_sz=self.arch.emb_sz, n_hid=self.arch.n_hid, n_layers=self.arch.n_layers, qrnn=self.arch.qrnn,
|
||||
**self.dropout_values)
|
||||
|
||||
trn_args = dict(drop_mult=self.drop_mult, wd=self.wd, pretrained=False, bptt=self.bptt,
|
||||
loss_func=loss_func, clip=self.clip)
|
||||
if hasattr(Learner, 'silent'):
|
||||
trn_args.update(silent=eval_only)
|
||||
|
||||
trn_args.update(**additional_trn_args)
|
||||
print("Training args: ", trn_args, "config: ", config)
|
||||
learn = text_classifier_learner(data_clas,
|
||||
AWD_LSTM,
|
||||
config=config,
|
||||
model_dir=self.model_name,
|
||||
**trn_args)
|
||||
learn = patch_learner(learn)
|
||||
if self.base.encoder_fname and not self.random_init:
|
||||
print("Loading pretrained model", self.base.encoder_fname)
|
||||
learn.load_encoder(self.base.encoder_fname)
|
||||
learn.freeze()
|
||||
else:
|
||||
warn("No pretrained encoder")
|
||||
|
||||
set_seed(self.seed, "Classifier training seed")
|
||||
if not eval_only:
|
||||
learn.callback_fns += [partial(CSVLogger, filename=f"{learn.model_dir}/cls-history")]
|
||||
if self.early_stopping:
|
||||
learn.callback_fns += [partial(SaveModelCallback, every='improvement',
|
||||
name='cls_best_tmp',
|
||||
monitor=self.early_stopping)]
|
||||
if self.fp16:
|
||||
learn.to_fp16()
|
||||
return learn
|
||||
|
||||
def train_(self, dataset_or_path=None, **train_config):
|
||||
self.replace_(**train_config, _strict=True)
|
||||
|
||||
base_tokenizer = self.base.tokenizer
|
||||
dataset = self._set_dataset_(dataset_or_path, base_tokenizer)
|
||||
data_clas = dataset.load_clas_databunch(bs=self.bs)
|
||||
learn = self.get_learner(data_clas=data_clas)
|
||||
print(f"Training: {learn.path / learn.model_dir}")
|
||||
learn.unfreeze()
|
||||
self._fit_schedule(learn)
|
||||
|
||||
self.experiment_path = learn.path / learn.model_dir
|
||||
base_tokenizer.save(self.experiment_path, learn=learn)
|
||||
learn.to_fp32()
|
||||
learn.save(CLS_BEST, with_opt=False)
|
||||
print("Classifier model saved to", self.experiment_path)
|
||||
self.save_paramters()
|
||||
learn.destroy()
|
||||
return
|
||||
|
||||
def _validate(self, learn, ds_type):
|
||||
ds_name = ds_type.name.lower()
|
||||
print(f"Model: {self.name}, ds_name: {ds_name}")
|
||||
results_dict = dict(zip(
|
||||
[f'{ds_name} loss'] + [f"{ds_name} {getattr(m, '__name__', m.__class__.__name__)}" for m in learn.metrics],
|
||||
map(float, learn.validate(learn.data.dl(ds_type)))))
|
||||
results_dict['name'] = self.name
|
||||
return results_dict
|
||||
|
||||
def validate(self, *splits, data_cls=None, save_name=CLS_BEST, use_cache=True, save_preds=False):
|
||||
"""Validates
|
||||
splits - Dataset Types to validate on default DatasetType.Test, DatasetType.Valid, DatasetType.Train
|
||||
"""
|
||||
if len(splits) == 0:
|
||||
splits = [DatasetType.Test, DatasetType.Valid, DatasetType.Train]
|
||||
cache_file = (self.experiment_path / f'results{"" if save_name == CLS_BEST else "-" + save_name}.json')
|
||||
if use_cache and cache_file.exists():
|
||||
with cache_file.open("r") as fp:
|
||||
return json.load(fp)
|
||||
|
||||
if data_cls is None:
|
||||
data_cls = self.dataset.load_clas_databunch(bs=self.bs)
|
||||
|
||||
learn = self.get_learner(data_cls, eval_only=True)
|
||||
# avg = 'binary' if learn.data.c == 2 else 'macro'
|
||||
# FBeta(beta=1.0, average=avg), Precision(average=avg), Recall(average=avg),
|
||||
learn.metrics = [accuracy]
|
||||
print(f"Loading model {save_name}")
|
||||
learn.load(save_name)
|
||||
if save_preds:
|
||||
probs, targets = learn.get_preds(ordered=True, ds_type=DatasetType.Test, activ=partial(F.softmax, dim=-1))
|
||||
np.save(str(self.experiment_path / f"preds-on-test.npy"), probs.cpu().numpy())
|
||||
|
||||
results_dict = {}
|
||||
for split in splits:
|
||||
results_dict.update(self._validate(learn, split))
|
||||
print(results_dict)
|
||||
with cache_file.open("w") as fp:
|
||||
json.dump(results_dict, fp)
|
||||
return results_dict
|
||||
|
||||
def _fit_schedule(self, learn):
|
||||
getattr(self, '_fit_schedule_' + self.fit_schedule)(learn)
|
||||
|
||||
def _fit_schedule_1cycle(self, learn):
|
||||
learn.unfreeze()
|
||||
learn.fit_one_cycle(self.num_epochs, slice(1e-2 / (2.6 ** 4), 2e-2), moms=(0.8, 0.7))
|
||||
|
||||
def _fit_schedule_layered(self, learn):
|
||||
learn.freeze_to(-1)
|
||||
learn.fit_one_cycle(1, 2e-2, moms=(0.8, 0.7))
|
||||
if self.num_epochs > 1:
|
||||
learn.freeze_to(-2)
|
||||
learn.fit_one_cycle(1, slice(1e-2 / (2.6 ** 4), 1e-2), moms=(0.8, 0.7))
|
||||
learn.freeze_to(-3)
|
||||
learn.fit_one_cycle(1, slice(5e-3 / (2.6 ** 4), 5e-3), moms=(0.8, 0.7))
|
||||
learn.unfreeze()
|
||||
if self.num_epochs > 5:
|
||||
learn.fit_one_cycle(self.num_epochs - 4, slice(1e-3 / (2.6 ** 4), 1e-3), moms=(0.8, 0.7))
|
||||
|
||||
def _fit_schedule_2cycle(self, learn):
|
||||
learn.freeze_to(-1)
|
||||
learn.fit_one_cycle(1, 2e-2, moms=(0.8, 0.7))
|
||||
learn.unfreeze()
|
||||
if self.num_epochs > 1:
|
||||
learn.fit_one_cycle(self.num_epochs - 1, slice(1e-2 / (2.6 ** 4), 1e-2), moms=(0.8, 0.7))
|
||||
|
||||
def _fit_schedule_reverse_2cycle(self, learn):
|
||||
learn.unfreeze()
|
||||
for g in learn.layer_groups[-1:]:
|
||||
for l in g:
|
||||
if not learn.train_bn or not isinstance(l, bn_types): requires_grad(l, False)
|
||||
learn.create_opt(defaults.lr)
|
||||
learn.fit_one_cycle(self.num_epochs, slice(1e-2 / (2.6 ** 4), 2e-2), moms=(0.8, 0.7))
|
||||
learn.unfreeze()
|
||||
learn.fit_one_cycle(self.num_epochs, slice(1e-3 / (2.6 ** 4), 2e-3), moms=(0.8, 0.7))
|
||||
|
||||
def _fit_schedule_false_wd(self, learn):
|
||||
learn.true_wd = False
|
||||
learn.fit_one_cycle(1, 5e-2, moms=(0.8, 0.7), wd=1e-7)
|
||||
if self.num_epochs > 1:
|
||||
learn.freeze_to(-2)
|
||||
learn.fit_one_cycle(1, slice(5e-2 / (2.6 ** 4), 5e-2), moms=(0.8, 0.7), wd=1e-7)
|
||||
learn.freeze_to(-3)
|
||||
learn.fit_one_cycle(1, slice(5e-4 / (2.6 ** 4), 5e-4), moms=(0.8, 0.7), wd=1e-7)
|
||||
learn.unfreeze()
|
||||
if self.num_epochs > 5:
|
||||
learn.fit_one_cycle(self.num_epochs - 4, slice(1e-2 / (2.6 ** 4), 1e-2), moms=(0.8, 0.7), wd=1e-7)
|
||||
|
||||
|
||||
def path_if_model_exists(path, weights_name):
|
||||
"""Return path to model if it exists"""
|
||||
model_path = path / (weights_name + ".pth")
|
||||
return path if model_path.exists() else None
|
||||
|
||||
|
||||
@dataclass
|
||||
class ULMFiT:
|
||||
arch: ULMFiTArchitecture = None
|
||||
pretrain_lm: ULMFiTPretraining = None
|
||||
finetune_lm: ULMFiTFinetuning = None
|
||||
classifier: ULMFiTClassifier = None
|
||||
|
||||
def __post_init__(self):
|
||||
self.arch = ULMFiTArchitecture()
|
||||
self.pretrain_lm = ULMFiTPretraining(arch=self.arch)
|
||||
self.finetune_lm = ULMFiTFinetuning(arch=self.arch, base=self.pretrain_lm)
|
||||
self.classifier = ULMFiTClassifier(arch=self.arch, base=self.finetune_lm)
|
||||
|
||||
def load_(self, experiment_path:Path, silent=False):
|
||||
success = (self.classifier.load_(experiment_path, silent=silent) or
|
||||
self.finetune_lm.load_(experiment_path, silent=silent) or
|
||||
self.pretrain_lm.load_(experiment_path, silent=silent) or
|
||||
self.load_legacy_(experiment_path, silent=silent))
|
||||
if not success:
|
||||
warn(f'Unable to load experiment {experiment_path}')
|
||||
return self
|
||||
|
||||
def load_legacy_(self, experiment_path, silent=True):
|
||||
if not (experiment_path / "info.json").exists():
|
||||
return False
|
||||
with (experiment_path / "info.json").open('r') as f:
|
||||
d = json.load(f)
|
||||
dataset_path = d.pop('dataset_path', "")
|
||||
d['n_hid'] = d['nh']
|
||||
d['n_layers'] = d['nl']
|
||||
d['lang'] = detect_lang_from_dataset_path(Path(dataset_path))
|
||||
if "wiki" in str(dataset_path):
|
||||
self.arch.replace_(**d)
|
||||
self.pretrain_lm.replace_(**d)
|
||||
self.pretrain_lm.experiment_path = path_if_model_exists(experiment_path, LM_BEST)
|
||||
self.pretrain_lm.dataset_path = dataset_path if dataset_path in str(experiment_path) else None
|
||||
else:
|
||||
self.replace_(**d)
|
||||
self.finetune_lm.experiment_path = path_if_model_exists(experiment_path, ENC_BEST)
|
||||
self.finetune_lm.dataset_path = dataset_path if dataset_path in str(experiment_path) else None
|
||||
self.classifier.experiment_path = path_if_model_exists(experiment_path, CLS_BEST)
|
||||
self.classifier.dataset_path = dataset_path if dataset_path in str(experiment_path) else None
|
||||
return True
|
||||
|
||||
def replace_(self, **kwargs):
|
||||
self.arch.replace_(**kwargs)
|
||||
self.pretrain_lm.replace_(**kwargs)
|
||||
self.finetune_lm.replace_(**kwargs)
|
||||
self.classifier.replace_(**kwargs)
|
||||
return self
|
||||
|
||||
def pprint(self):
|
||||
print(f"""ULMFiT(
|
||||
{self.arch},
|
||||
{self.pretrain_lm},
|
||||
{self.finetune_lm},
|
||||
{self.classifier},
|
||||
)""")
|
||||
|
||||
def from_pretrained_(self, name, repo="n-waves/multifit-models"):
|
||||
name = name.rstrip(".tgz") # incase someone put's tgz name the name
|
||||
url = f"https://github.com/{repo}/releases/download/{name}/{name}.tgz"
|
||||
path = untar_data(url.rstrip(".tgz"), data=False) # untar_data adds .tgz
|
||||
return self.load_(path)
|
||||
|
||||
|
||||
def from_pretrained(name):
|
||||
#TODO: Detect name and load configuration
|
||||
from . import configurations
|
||||
return configurations.multifit_paper_version().from_pretrained_(name)
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
+1
-1
@@ -7,5 +7,5 @@ echo "Saving data in $DATA_DIR"
|
||||
wget -c "http://files.fast.ai/data/aclImdb.tgz" -P "${DATA_DIR}"
|
||||
|
||||
echo "Imdb is raw text no preparation is done"
|
||||
python -m fastai_contrib.utils prepare_imdb "${DATA_DIR}/aclImdb.tgz"
|
||||
python -m multifit.datasets.utils prepare_imdb "${DATA_DIR}/aclImdb.tgz"
|
||||
|
||||
|
||||
@@ -1,2 +0,0 @@
|
||||
|
||||
# https://storage.googleapis.com/reddit_comments_polish/comments.csv.gz
|
||||
+4
-11
@@ -13,13 +13,6 @@ else
|
||||
fi
|
||||
echo "Chosen language: ""$LANG"
|
||||
|
||||
if [ "$2" == "" ] ; then
|
||||
read -p "Enter the minimal tokens per articles [100]: " tokens_min
|
||||
TOKENS_MIN=${tokens_min:-100}
|
||||
else
|
||||
TOKENS_MIN="$2"
|
||||
fi
|
||||
|
||||
DUMP_DIR="${ROOT}/wiki_dumps"
|
||||
EXTR_DIR="${ROOT}/wiki_extr"
|
||||
WIKI_DIR="${ROOT}/wiki"
|
||||
@@ -53,8 +46,8 @@ else
|
||||
echo "${EXTR_PATH} already exists. Skipping extraction."
|
||||
fi
|
||||
|
||||
python -m ulmfit.create_wikitext -i "${EXTR_PATH}" -l "${LANG}" -o "${WIKI_DIR}" -t "${TOKENS_MIN}"
|
||||
python -m multifit.create_wikitext -i "${EXTR_PATH}" -l "${LANG}" -o "${WIKI_DIR}"
|
||||
|
||||
python -m ulmfit.postprocess_wikitext "${WIKI_DIR}/${LANG}-2" $LANG
|
||||
python -m ulmfit.postprocess_wikitext "${WIKI_DIR}/${LANG}-100" $LANG
|
||||
#python -m ulmfit.postprocess_wikitext "${WIKI_DIR}/${LANG}-all" $LANG
|
||||
python -m multifit.postprocess_wikitext "${WIKI_DIR}/${LANG}-2" $LANG
|
||||
python -m multifit.postprocess_wikitext "${WIKI_DIR}/${LANG}-100" $LANG
|
||||
#python -m multifit.postprocess_wikitext "${WIKI_DIR}/${LANG}-all" $LANG
|
||||
|
||||
+3
-3
@@ -1,5 +1,5 @@
|
||||
fire>=0.1.3
|
||||
cupy>=5.0.0
|
||||
scikit-learn>=0.20
|
||||
sacremoses>=0.0.5
|
||||
sentencepiece
|
||||
sacremoses==0.0.35
|
||||
sentencepiece==0.1.83
|
||||
fastai >= 1.0.43
|
||||
@@ -1,6 +0,0 @@
|
||||
# Zero shot from CLS
|
||||
|
||||
|
||||
|
||||
|
||||
### zeroshoot
|
||||
@@ -1,105 +0,0 @@
|
||||
|
||||
## Supervised classification results on MLDoc
|
||||
| Model | en | de | es | fr | it | ja | ru | zh |
|
||||
|----------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|------------|
|
||||
|LASER 0 shot | 80.75 | 87.03 | 82.60 | 82.83 | 73.25 | 60.95 | 68.83 | 72.90 |
|
||||
|LASER | 90.73 | 92.70 | 88.75 | 90.80 | 85.93 | 85.15 | 84.65 | 88.98 |
|
||||
|MultiCCA | 92.2 | 93.70 | 94.45 | 92.05 | 85.55 | 85.35 | 85.65 | 87.30 |
|
||||
|Bert Multi | 93.23 | 94.0 | 95.15 | 93.20 | 85.82 | 87.48 | 86.85 | 90.72 |
|
||||
|ULMFiT L30k-100 | | 91.35 | 83.32 | 88.77 | 77.99 | 71.12 | 72.20 | |
|
||||
|ULMFiT L30k | | 95.4 | 95.15 | 93.67 | 88.42 | 89.20 | **87.27** | 90.20 |
|
||||
|ULMFiT sp-fixed | | **95.6** | 94.80 | 94.20 | 88.52 | 88.72 | 86.85 | 90.47 |
|
||||
|ULMFIT Q15k 1cyc| 94.62 | **95.65** | 95.15 | **94.42** | 89.92 | 89.60 | | 90.78/89.82 |
|
||||
|ULMFIT Q15k 1c l| **94.99** | | 95.64 | 94.34 | **90.32** | 89.67 | 87.67^ | **92.22** |
|
||||
|ULMFIT Q15k 1cfl| **95.55** | **96.10** | 95.97 | 94.77 | **90.07** | 89.87 | 87.17 | **92.40** |
|
||||
|ULMFIT L30k 1cyc| | **95.85** | **96.32** | **94.82** | 89.87 | **90.45** | **87.94** | 92.02/91.64 |
|
||||
|
||||
- L30k - LSTM sp30k trained using gradual unfreezing
|
||||
- L30k-100 - --||-- **on 100 samples**
|
||||
- ULMFiT sp-fixed - --||-- with fixed tokenization
|
||||
- Q15k 1cyc - QRNN sp15k trained using 1cycle learning rate schedule
|
||||
- L30k 1cyc - LSTM sp30k trained using 1cycle learning rate schedule
|
||||
- We checked LSTM on sp15k on DE and got 95.53% accuracy which is comparable to QRNN sp15k
|
||||
- ^ - 16 epochs qrnn_nl4sl-bs500
|
||||
|
||||
|
||||
## Zeroshot
|
||||
|
||||
| Model | de | es | fr | it | ru | zh |
|
||||
|----------------------|------------|------------|-----------|-----------|-----------|-----------|
|
||||
| LASER-en | 87.65 | 75.48 | 84.00 | 71.18 | 66.58 | 76.65 |
|
||||
| ULMFiT L on LASER-en | **92.95** | **80.50** | 88.78 | 76.20 | **70.05** | 80.45 |
|
||||
| ULMFiT Q on LASER-en | 91.34 | 78.92 | **89.45** | 76.00 | 68.19 | **82.45** |
|
||||
- L - 1k LSTM sp30k
|
||||
- Q - 1k QRNN sp15k
|
||||
|
||||
#### LSTM results
|
||||
| Model | de | es | fr | it | ru | zh |
|
||||
|----------------------|------------|------------|-----------|-----------|-----------|-----------|
|
||||
| LASER-de | | 81.40 | 81.50 | 74.53 | 64.58 | 73.20 |
|
||||
| LASER-fr | 88.75 | 80.12 | | 72.58 | 67.35 | 79.40 |
|
||||
| LASER-en | 87.65 | 75.48 | 84.00 | 71.18 | 66.58 | 76.65 |
|
||||
| | | | | | | |
|
||||
| ULMFiT on LASER-de | | **85.50** | 87.37 | **78.75** | 66.95 | 72.32 |
|
||||
| ULMFiT on LASER-fr | 92.22 | 81.00 | | 76.88 | 68.33 | **84.65** |
|
||||
| ULMFiT on LASER-en | **92.95** | 80.50 | **88.78** | 76.20 | **70.05** | 80.45 |
|
||||
| | | | | | | |
|
||||
| % impr over LASER-de | | 22% | 32% | 17% | 7% | *-3%* |
|
||||
| % impr over LASER-fr | 31% | 4% | | 16% | 3% | 25% |
|
||||
| % impr over LASER-en | 43% | 20% | 30% | 17% | 10% | 16% |
|
||||
| ULMFiT 100 for comp. | 91.35 | 83.32 | 88.77 | 77.99 | 71.12 | |
|
||||
| | | | | | | |
|
||||
| Bert Multilingual-EN | 74.50 | 61.85 | 69.77 | 57.73 | 51.10 | 64.08 |
|
||||
|
||||
|
||||
#### From Laser trained on French data
|
||||
| Model Name | de | es | fr | it | ru | zh |
|
||||
|---------------------------|-------|-------|----|-------|-------|-------|
|
||||
| LASER fr 10k | 91.65 | 81.05 | | 75.08 | 70.73 | 76.33 |
|
||||
| LASER fr 1k | 88.75 | 80.12 | | 72.58 | 67.35 | 79.4 |
|
||||
| ULMFiT 10k on LASER-fr10k | 94.48 | 84.10 | | 77.93 | 72.87 | 84.53 |
|
||||
| ULMFiT 10k on LASER-fr1k | 92.30 | 82.10 | | 75.52 | 69.52 | 85.55 |
|
||||
| ULMFiT 1k on LASER-fr1k | 92.22 | 81.00 | | 76.88 | 68.33 | 84.65 |
|
||||
| | | | | | | |
|
||||
| Impr 10k over 10k | 34% | 16% | | 11% | 7% | 35% |
|
||||
| Impr 10k over 1k | 32% | 10% | | 11% | 7% | 30% |
|
||||
| Impr 1k over 1k | 31% | 4% | | 16% | 3% | 25% |
|
||||
|
||||
#### From Laser trained on German data
|
||||
| Model Name | de | es | fr | it | ru | zh |
|
||||
|---------------------------|----|-------|-------|-------|-------|-------|
|
||||
| LASER de 10k | | 83.5 | 82.85 | 76.6 | 68.8 | 73.12 |
|
||||
| LASER de 1k | | 81.4 | 81.5 | 74.53 | 64.58 | 73.2 |
|
||||
| ULMFiT 10k on LASER-de10k | | 86.92 | 87.17 | 79.35 | 70.15 | 78.15 |
|
||||
| ULMFiT 10k on LASER-de1k | | 84.65 | 87.48 | 78.70 | 67.65 | 77.50 |
|
||||
| ULMFiT 1k on LASER-de1k | | 85.5 | 87.37 | 78.75 | 66.95 | 72.32 |
|
||||
| | | | | | | |
|
||||
| Impr 10k over 10k | | 21% | 25% | 12% | 4% | 19% |
|
||||
| Impr 10k over 1k | | 17% | 32% | 16% | 9% | 16% |
|
||||
| Impr 1k over 1k | | 22% | 32% | 17% | 7% | -3% |
|
||||
|
||||
#### From Laser trained on English data
|
||||
| Model Name | de | es | fr | it | ru | zh |
|
||||
|---------------------------|-------|-------|-------|-------|-------|-------|
|
||||
| LASER en 10k | 87.43 | 77.38 | 78.7 | 72.53 | 67.7 | 75.18 |
|
||||
| LASER en 1k | 87.65 | 75.48 | 84 | 71.18 | 66.58 | 76.65 |
|
||||
| ULMFiT 10k on LASER-en10k | 92.05 | 80.05 | 86.95 | 76.65 | 70.57 | 80.85 |
|
||||
| ULMFiT 10k on LASER-en1k | 91.80 | 80.10 | 88.67 | 77.32 | 70.25 | 82.73 |
|
||||
| ULMFiT 1k on LASER-en1k | 92.95 | 80.50 | 88.78 | 76.20 | 70.05 | 80.45 |
|
||||
| | | | | | | |
|
||||
| Impr 10k over 10k | 37% | 12% | 39% | 15% | 9% | 23% |
|
||||
| Impr 10k over 1k | 34% | 19% | 29% | 21% | 11% | 26% |
|
||||
| Impr 1k over 1k | 43% | 20% | 30% | 17% | 10% | 16% |
|
||||
| ULMFiT qrnn on 1k LSRen1k | 91.32 | 78.92 | 89.45 | 75.99 | | 82.45 |
|
||||
| ULMFiT qrnn on 10k LSRen1k| 91.90 | 78.79 | 88.47 | 76.05 | | |
|
||||
|
||||
|
||||
## Noise resistance
|
||||
|
||||
| Model | en | de | es | fr | it | ja | ru | zh |
|
||||
|---------------------------------|------------|-----------|-----------|-----------|-----------|-----------|-----------|------------|
|
||||
|LASER 0 shot | 80.75 (en) | 87.03 (fr)| 82.60 (it)| 82.83 (de)| 73.25 (de)| 60.95 (en)| 68.83 (it)| 72.90 (de) |
|
||||
|ULMFiT | | **95.4** | **95.15** | **93.67** | **88.42** | **89.20** | **87.27** | |
|
||||
| % of noise | 20% | 13% | 18% | 18% | 27% | 40% | 32% | 28% |
|
||||
|ULMFiT trained on 1k noisy exmp. | | 94.49 | 93.12 | 90.49 | 83.72 | 74.72 | 75.67 | |
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,24 +0,0 @@
|
||||
|
||||
## QRNN sp15k timing
|
||||
```
|
||||
time python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-2' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 1 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1
|
||||
Max vocab: 15000
|
||||
Cache dir: data/wiki/ru-100/models/sp15k
|
||||
Model dir: data/wiki/ru-100/models/sp15k/qrnn_nl4-2.m
|
||||
Wiki text was split to 193047 articles
|
||||
Wiki text was split to 460 articles
|
||||
Data lm, trn: 193047, val: 460
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.943105 3.860063 0.477620
|
||||
Total time: 1:05:03
|
||||
data/wiki/ru-100/models/sp15k
|
||||
Saving info data/wiki/ru-100/models/sp15k/qrnn_nl4-2.m/info.json
|
||||
|
||||
real 65m30,341s
|
||||
user 48m49,047s
|
||||
sys 16m40,688s
|
||||
```
|
||||
@@ -1,448 +0,0 @@
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/de-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/de-100/models/sp15k/qrnn_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.467852 2.558666 0.525457
|
||||
Total time: 02:25
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.722157 2.387366 0.548566
|
||||
2 2.477095 2.170018 0.580988
|
||||
3 2.182971 1.981363 0.609205
|
||||
4 2.078041 1.836848 0.629900
|
||||
5 1.975613 1.744062 0.642769
|
||||
6 1.866875 1.656678 0.655799
|
||||
7 1.831995 1.595655 0.665479
|
||||
8 1.768020 1.540487 0.673880
|
||||
9 1.751569 1.488140 0.682557
|
||||
10 1.647143 1.441723 0.690275
|
||||
11 1.712795 1.399652 0.697534
|
||||
12 1.529405 1.350384 0.706170
|
||||
13 1.549134 1.313349 0.713210
|
||||
14 1.585015 1.278395 0.719908
|
||||
15 1.475010 1.248854 0.725591
|
||||
16 1.532636 1.221373 0.731053
|
||||
17 1.445181 1.203350 0.734503
|
||||
18 1.396236 1.191440 0.737102
|
||||
19 1.316587 1.186497 0.738052
|
||||
20 1.374460 1.185027 0.738290
|
||||
Total time: 1:11:26
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4.m/info.json
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4.m
|
||||
Loss and accuracy using (cls_best): [1.3879647, tensor(0.2595)]
|
||||
Processing data/wiki/en-100/models/sp15k/qrnn_nl4.m
|
||||
|
||||
|
||||
------
|
||||
|
||||
|
||||
|
||||
|
||||
$ python -m ulmfit eval --glob="wiki/*-100/models/sp15k/qrnn_nl4.m" --name nl4 --dataset-template='../mldoc/${lang}-1' --num-lm-epochs=20 --num-cls-epochs=8 --bs=18 --lr_sched=1cycle
|
||||
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁the', '▁,', 's', '▁.', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/en-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/en-100/models/sp15k/qrnn_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.567319 3.820788 0.346054
|
||||
Total time: 02:26
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.889761 3.601505 0.374049
|
||||
2 3.570620 3.357854 0.406134
|
||||
3 3.389516 3.153452 0.432199
|
||||
4 3.217872 2.985234 0.452187
|
||||
5 3.063675 2.851744 0.468071
|
||||
6 3.023959 2.754062 0.480278
|
||||
7 2.907327 2.647027 0.493494
|
||||
8 2.786187 2.562560 0.505051
|
||||
9 2.737610 2.500068 0.513554
|
||||
10 2.696695 2.430095 0.523029
|
||||
11 2.658439 2.380829 0.530339
|
||||
12 2.598193 2.318927 0.539454
|
||||
13 2.558214 2.275014 0.546136
|
||||
14 2.520342 2.230543 0.553176
|
||||
15 2.475964 2.190341 0.559245
|
||||
16 2.370359 2.161100 0.564223
|
||||
17 2.430078 2.136685 0.568197
|
||||
18 2.383946 2.125458 0.569950
|
||||
19 2.389433 2.117541 0.571265
|
||||
20 2.297921 2.116168 0.571367
|
||||
Total time: 1:11:10
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.705876 0.241169 0.918000
|
||||
2 0.450546 0.239528 0.926000
|
||||
3 0.335179 0.221836 0.931000
|
||||
4 0.202048 0.208652 0.951000
|
||||
5 0.144956 0.223669 0.954000
|
||||
6 0.073117 0.277062 0.953000
|
||||
7 0.045186 0.258046 0.962000
|
||||
8 0.022987 0.265977 0.961000
|
||||
Total time: 02:33
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.29402012, tensor(0.9460)]
|
||||
Processing data/wiki/es-100/models/sp15k/qrnn_nl4.m
|
||||
../mldoc/es-1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4.m
|
||||
Evaluating previously trained model
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
|
||||
Loss and accuracy using (cls_last): [0.19222946, tensor(0.9515)]
|
||||
Processing data/wiki/fr-100/models/sp15k/qrnn_nl4.m
|
||||
../mldoc/fr-1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/fr.dev.csv
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Running tokenization cls...
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp15k/qrnn_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.375679 2.676224 0.454405
|
||||
Total time: 02:19
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.901917 2.540690 0.475910
|
||||
2 2.593614 2.370477 0.504601
|
||||
3 2.423170 2.205713 0.530328
|
||||
4 2.287688 2.083261 0.549087
|
||||
5 2.161118 1.984955 0.564804
|
||||
6 2.221017 1.912810 0.575434
|
||||
7 2.111272 1.837854 0.588076
|
||||
8 2.032289 1.775163 0.598341
|
||||
9 1.984161 1.720519 0.607980
|
||||
10 1.904775 1.668184 0.617407
|
||||
11 1.829098 1.621347 0.626292
|
||||
12 1.855409 1.577870 0.634512
|
||||
13 1.843696 1.536835 0.642584
|
||||
14 1.767968 1.496428 0.650317
|
||||
15 1.741591 1.463305 0.656908
|
||||
16 1.682118 1.438706 0.662601
|
||||
17 1.666425 1.418383 0.666283
|
||||
18 1.623713 1.406877 0.668710
|
||||
19 1.645482 1.401546 0.669716
|
||||
20 1.579352 1.399608 0.670167
|
||||
Total time: 1:06:50
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.614734 0.262120 0.906000
|
||||
2 0.377443 0.327852 0.917000
|
||||
3 0.296728 0.392655 0.903000
|
||||
4 0.179866 0.423420 0.928000
|
||||
5 0.114529 0.398973 0.935000
|
||||
6 0.082004 0.325470 0.944000
|
||||
7 0.047604 0.359636 0.945000
|
||||
8 0.032579 0.354014 0.944000
|
||||
Total time: 02:22
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.33020702, tensor(0.9450)]
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp15k/qrnn_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.524609 2.582580 0.512131
|
||||
Total time: 02:34
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.656181 2.315530 0.550262
|
||||
2 2.265949 2.019140 0.598469
|
||||
3 1.985897 1.769565 0.638234
|
||||
4 1.831071 1.617888 0.660760
|
||||
5 1.735492 1.509642 0.677379
|
||||
6 1.637924 1.427618 0.690664
|
||||
7 1.564483 1.363384 0.700825
|
||||
8 1.508054 1.318165 0.708210
|
||||
9 1.471599 1.267787 0.716080
|
||||
10 1.398376 1.232899 0.722340
|
||||
11 1.311976 1.199602 0.728811
|
||||
12 1.401354 1.162299 0.735328
|
||||
13 1.385588 1.132408 0.740850
|
||||
14 1.256193 1.106556 0.745935
|
||||
15 1.289892 1.083529 0.750840
|
||||
16 1.220951 1.063360 0.754845
|
||||
17 1.259715 1.050884 0.757371
|
||||
18 1.165468 1.042870 0.759241
|
||||
19 1.242660 1.038160 0.760036
|
||||
20 1.194239 1.037506 0.760167
|
||||
Total time: 1:13:55
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.760780 0.391838 0.872000
|
||||
2 0.592674 0.427392 0.870000
|
||||
3 0.446265 0.593488 0.838000
|
||||
4 0.318780 0.605533 0.858000
|
||||
5 0.226914 0.665538 0.872000
|
||||
6 0.135525 0.742310 0.891000
|
||||
7 0.063984 0.778616 0.892000
|
||||
8 0.039366 0.827663 0.884000
|
||||
Total time: 02:49
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.70555997, tensor(0.8960)]
|
||||
Processing data/wiki/zh-100/models/sp15k/qrnn_nl4.m
|
||||
../mldoc/zh-1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4.m
|
||||
Evaluating previously trained model
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有']
|
||||
Loss and accuracy using (cls_last): [0.30052844, tensor(0.8982)]
|
||||
OrderedDict([('data/mldoc/de-1/models/sp15k/qrnn_nl4.m', 0.2592499852180481),
|
||||
('data/mldoc/en-1/models/sp15k/qrnn_nl4.m', 0.9462500214576721),
|
||||
('data/mldoc/es-1/models/sp15k/qrnn_nl4.m', 0.9514999985694885),
|
||||
('data/mldoc/fr-1/models/sp15k/qrnn_nl4.m', 0.9442499876022339),
|
||||
('data/mldoc/ja-1/models/sp15k/qrnn_nl4.m', 0.8960000276565552),
|
||||
('data/mldoc/zh-1/models/sp15k/qrnn_nl4.m', 0.8982499837875366)])
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.610785 0.239165 0.923000
|
||||
2 0.392899 0.281254 0.937000
|
||||
3 0.268695 0.444383 0.909000
|
||||
4 0.162150 0.427744 0.931000
|
||||
5 0.109248 0.422351 0.948000
|
||||
6 0.061984 0.411351 0.947000
|
||||
7 0.033645 0.413174 0.951000
|
||||
8 0.018704 0.404264 0.947000
|
||||
Total time: 02:16
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.29712877, tensor(0.9565)]
|
||||
Processing data/wiki/en-100/models/sp15k/qrnn_nl4.m
|
||||
../mldoc/en-1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_nl4.m
|
||||
Evaluating previously trained model
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/en.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁the', '▁,', 's', '▁.', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed']
|
||||
Loss and accuracy using (cls_last): [0.29526812, tensor(0.9463)]
|
||||
Processing data/wiki/es-100/models/sp15k/qrnn_nl4.m
|
||||
../mldoc/es-1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4.m
|
||||
Evaluating previously trained model
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
|
||||
Loss and accuracy using (cls_last): [0.19222946, tensor(0.9515)]
|
||||
Processing data/wiki/fr-100/models/sp15k/qrnn_nl4.m
|
||||
../mldoc/fr-1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_nl4.m
|
||||
Evaluating previously trained model
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/fr.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à']
|
||||
Loss and accuracy using (cls_last): [0.33129737, tensor(0.9442)]
|
||||
Processing data/wiki/it-100/models/sp15k/qrnn_nl4.m
|
||||
../mldoc/it-1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/it.dev.csv
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Running tokenization cls...
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁di', '▁e', "▁&'", "'", '▁il', '▁la', '▁in', 'e']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/it-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/it-100/models/sp15k/qrnn_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.929510 2.916437 0.434550
|
||||
Total time: 01:20
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.231523 2.723415 0.461407
|
||||
2 2.823881 2.498599 0.496812
|
||||
3 2.586648 2.283846 0.530524
|
||||
4 2.417038 2.125690 0.553137
|
||||
5 2.278636 1.995558 0.572757
|
||||
6 2.199877 1.887804 0.589102
|
||||
7 2.090629 1.799082 0.603201
|
||||
8 2.046975 1.725273 0.615247
|
||||
9 1.935966 1.654829 0.626968
|
||||
10 1.921190 1.590797 0.638228
|
||||
11 1.894758 1.528087 0.649369
|
||||
12 1.792718 1.477532 0.658754
|
||||
13 1.679359 1.428426 0.668648
|
||||
14 1.723383 1.377170 0.678987
|
||||
15 1.597491 1.339658 0.686348
|
||||
16 1.620966 1.307664 0.692993
|
||||
17 1.568962 1.284500 0.697923
|
||||
18 1.533934 1.271438 0.700628
|
||||
19 1.496832 1.264714 0.701968
|
||||
20 1.486198 1.262870 0.702333
|
||||
Total time: 39:33
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.784124 0.414312 0.847000
|
||||
2 0.550873 0.413405 0.861000
|
||||
3 0.445371 0.363693 0.877000
|
||||
4 0.271702 0.426771 0.899000
|
||||
5 0.165902 0.556069 0.881000
|
||||
6 0.091403 0.628809 0.897000
|
||||
7 0.065516 0.693292 0.893000
|
||||
8 0.033616 0.675199 0.897000
|
||||
Total time: 01:24
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.7380945, tensor(0.8992)]
|
||||
Processing data/wiki/ja-100/models/sp15k/qrnn_nl4.m
|
||||
../mldoc/ja-1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_nl4.m
|
||||
Evaluating previously trained model
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']
|
||||
Loss and accuracy using (cls_last): [0.7049702, tensor(0.8953)]
|
||||
Processing data/wiki/zh-100/models/sp15k/qrnn_nl4.m
|
||||
../mldoc/zh-1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4.m
|
||||
Evaluating previously trained model
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有']
|
||||
Loss and accuracy using (cls_last): [0.30052844, tensor(0.8982)]
|
||||
OrderedDict([('data/mldoc/de-1/models/sp15k/qrnn_nl4.m', 0.9564999938011169),
|
||||
('data/mldoc/en-1/models/sp15k/qrnn_nl4.m', 0.9462500214576721),
|
||||
('data/mldoc/es-1/models/sp15k/qrnn_nl4.m', 0.9514999985694885),
|
||||
('data/mldoc/fr-1/models/sp15k/qrnn_nl4.m', 0.9442499876022339),
|
||||
('data/mldoc/it-1/models/sp15k/qrnn_nl4.m', 0.8992499709129333),
|
||||
('data/mldoc/ja-1/models/sp15k/qrnn_nl4.m', 0.8952500224113464),
|
||||
('data/mldoc/zh-1/models/sp15k/qrnn_nl4.m', 0.8982499837875366)])
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## DE
|
||||
----------------------------------------
|
||||
Training issues
|
||||
|
||||
|
||||
1/2nd -- That was without fine tuning !!! 0 shot:)
|
||||
```
|
||||
python -m ulmfit load_cls data/mldoc/de-1/models/sp15k/qrnn_nl4.m --lang=de - train 0 --num-cls-epochs 8 --bs=18 --lr-sched=1cycle ✘ 1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_None.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/de.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
Your compiler (c++) may be ABI-incompatible with PyTorch!
|
||||
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
|
||||
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
|
||||
|
||||
See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
|
||||
for instructions on how to install GCC 4.9 or higher.
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.310368 1.177105 0.520000
|
||||
2 1.096284 0.899281 0.739000
|
||||
3 0.860910 0.668378 0.864000
|
||||
4 0.676764 0.733304 0.868000
|
||||
5 0.573360 0.590983 0.885000
|
||||
6 0.438448 0.446631 0.918000
|
||||
7 0.397323 0.531330 0.919000
|
||||
8 0.339557 0.437841 0.922000
|
||||
Total time: 02:25
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_None.m
|
||||
Loss and accuracy using (cls_best): [0.3380329, tensor(0.9295)]
|
||||
0.33803290128707886
|
||||
0.9294999837875366
|
||||
```
|
||||
|
||||
3rd aproach
|
||||
```
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.610785 0.239165 0.923000
|
||||
2 0.392899 0.281254 0.937000
|
||||
3 0.268695 0.444383 0.909000
|
||||
4 0.162150 0.427744 0.931000
|
||||
5 0.109248 0.422351 0.948000
|
||||
6 0.061984 0.411351 0.947000
|
||||
7 0.033645 0.413174 0.951000
|
||||
8 0.018704 0.404264 0.947000
|
||||
Total time: 02:16
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.29712877, tensor(0.9565)]
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,577 +0,0 @@
|
||||
# Overall
|
||||
|
||||
## DE BOOKS LSTM
|
||||
|
||||
```bash
|
||||
(fastai) julian@dl-box-spot:~/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/cls/de-books --base-lm-path ../data/wiki/de-100/models/sp30k/lstm_nl4.m --tokenizer='sp' --lang=de --name 'nl4' - train 20 --bs 40 --lr-sched=1cycle --num-cls-epochs=8
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/julian/ulmfit-multilingual/data/cls/de-books/models/sp30k
|
||||
Model dir: /home/julian/ulmfit-multilingual/data/cls/de-books/models/sp30k/lstm_nl4.m
|
||||
Validation set not found using 10% of trn
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 30600, val: 3400
|
||||
Running tokenization cls...
|
||||
Data cls, trn: 1800, val: 200
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 200, val: 2000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', '▁und', '▁in', 'en', "▁&'", 's', '-']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/julian/data/wiki/de-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/julian/data/wiki/de-100/models/sp30k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.553707 3.077508 0.458865
|
||||
Total time: 09:49
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.233023 2.998775 0.469265
|
||||
2 3.130155 2.900699 0.481559
|
||||
3 3.038409 2.793384 0.494638
|
||||
4 2.970666 2.693831 0.506061
|
||||
5 2.899437 2.601532 0.515972
|
||||
6 2.803581 2.516531 0.526783
|
||||
7 2.732246 2.443080 0.536339
|
||||
8 2.675900 2.375012 0.544895
|
||||
9 2.636490 2.313508 0.553726
|
||||
10 2.604711 2.253531 0.562466
|
||||
11 2.550045 2.202728 0.570852
|
||||
12 2.501192 2.145478 0.579989
|
||||
13 2.484679 2.092014 0.588614
|
||||
14 2.409206 2.044224 0.596671
|
||||
15 2.344645 2.008057 0.603097
|
||||
16 2.346225 1.976991 0.608867
|
||||
17 2.313172 1.954794 0.612839
|
||||
18 2.269678 1.937210 0.615802
|
||||
19 2.277551 1.930322 0.617028
|
||||
20 2.246812 1.928822 0.617285
|
||||
Total time: 3:38:47
|
||||
/home/julian/ulmfit-multilingual/data/cls/de-books/models/sp30k
|
||||
Saving info /home/julian/ulmfit-multilingual/data/cls/de-books/models/sp30k/lstm_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.406082 0.238703 0.905000
|
||||
2 0.313938 0.477311 0.865000 3 0.255491 0.228014 0.890000
|
||||
4 0.155123 0.384204 0.900000
|
||||
5 0.107264 0.374567 0.905000
|
||||
6 0.070755 0.468389 0.900000
|
||||
7 0.035681 0.243386 0.945000
|
||||
8 0.022694 0.242060 0.920000
|
||||
Total time: 05:49
|
||||
Saving models at /home/julian/ulmfit-multilingual/data/cls/de-books/models/sp30k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.43779454, tensor(0.9170)]
|
||||
0.4377945363521576
|
||||
0.9169999957084656
|
||||
```
|
||||
|
||||
## FR BOOKS LSTM
|
||||
|
||||
```bash
|
||||
python -m ulmfit cls --dataset-path data/cls/${LANG}-books --base-lm-path data/wiki-m/${LANG}-100/models/sp30k/lstm_nl4.m --lang=${LANG} --name 'nl4' - train 20 --bs 20 --num-cls-epochs=8 --lr-sched=single
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k/lstm_nl4.m
|
||||
Validation set not found using 10% of trn
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 33183, val: 3687
|
||||
Running tokenization cls...
|
||||
Data cls, trn: 1800, val: 200
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 200, val: 2000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁.', "'", 's', '▁la', '▁le', '▁et', '▁l', '▁à']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki-m/fr-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki-m/fr-100/models/sp30k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.790325 3.409294 0.367234
|
||||
Total time: 06:02
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.526303 3.326439 0.378936
|
||||
2 3.466923 3.226977 0.392378
|
||||
3 3.342312 3.111874 0.406997
|
||||
4 3.244619 2.992510 0.422330
|
||||
5 3.156150 2.877498 0.437467
|
||||
6 3.070326 2.762509 0.453874
|
||||
7 2.956969 2.651613 0.471552
|
||||
8 2.878008 2.535935 0.491058
|
||||
9 2.790110 2.438724 0.508560
|
||||
10 2.684145 2.323467 0.528415
|
||||
11 2.633781 2.231418 0.547093
|
||||
12 2.535126 2.143523 0.564889
|
||||
13 2.464436 2.055402 0.582077
|
||||
14 2.330094 1.989257 0.596582
|
||||
15 2.372371 1.924338 0.610048
|
||||
16 2.190224 1.866912 0.621738
|
||||
17 2.176868 1.834098 0.629221
|
||||
18 2.168293 1.809196 0.633879
|
||||
19 2.151132 1.797144 0.636382
|
||||
20 2.130476 1.793351 0.637044
|
||||
Total time: 2:30:05
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k/lstm_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.314315 0.530879 0.865000
|
||||
2 0.336746 0.468635 0.865000
|
||||
3 0.255810 0.324242 0.870000
|
||||
4 0.149121 0.480570 0.885000
|
||||
5 0.093909 0.613743 0.890000
|
||||
6 0.091678 0.660452 0.885000
|
||||
7 0.049993 0.649642 0.910000
|
||||
8 0.034218 0.640008 0.910000
|
||||
Total time: 04:19
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books/models/sp30k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.5418505, tensor(0.9100)]
|
||||
0.5418505072593689
|
||||
0.9100000262260437
|
||||
```
|
||||
|
||||
## JA BOOKS LSTM
|
||||
|
||||
```bash
|
||||
(fastai) julian@dl-box-spot:~/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/cls/ja-books --base-lm-path ../data/wiki/ja-100/models/sp30k/lstm_nl4.m --tokenizer='sp' --lang=ja --name 'nl4' - train 20 --bs 40 --lr-sched=1cycle --num-cls-epochs=8
|
||||
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/julian/ulmfit-multilingual/data/cls/ja-books/models/sp30k
|
||||
Model dir: /home/julian/ulmfit-multilingual/data/cls/ja-books/models/sp30k/lstm_nl4.m
|
||||
Validation set not found using 10% of trn
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 30600, val: 3399
|
||||
Running tokenization cls...
|
||||
Data cls, trn: 1800, val: 200
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 200, val: 1999
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁の', '▁。', '▁に', '▁を', '▁は', '▁年', '▁が', '▁)', '▁(']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/julian/data/wiki/ja-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/julian/data/wiki/ja-100/models/sp30k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.847485 3.422460 0.376126
|
||||
|
||||
Total time: 06:28
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.588070 3.347799 0.386644
|
||||
2 3.481361 3.262498 0.398140
|
||||
3 3.393415 3.164333 0.410120
|
||||
4 3.305471 3.069857 0.421340
|
||||
5 3.246775 2.972239 0.433235
|
||||
6 3.127283 2.886817 0.443638
|
||||
7 3.085512 2.806101 0.454796
|
||||
8 3.016604 2.738343 0.463713
|
||||
9 2.947214 2.667280 0.473756
|
||||
10 2.919253 2.602734 0.483177
|
||||
12 2.799891 2.488073 0.501841
|
||||
13 2.772961 2.432371 0.511213
|
||||
14 2.706188 2.389203 0.518911
|
||||
15 2.653137 2.346985 0.526103
|
||||
16 2.624165 2.316532 0.531397
|
||||
17 2.578764 2.293599 0.535356
|
||||
18 2.568077 2.279164 0.537922
|
||||
19 2.529823 2.271825 0.539241
|
||||
20 2.558044 2.270341 0.539438
|
||||
|
||||
Total time: 2:35:18
|
||||
/home/julian/ulmfit-multilingual/data/cls/ja-books/models/sp30k
|
||||
Saving info /home/julian/ulmfit-multilingual/data/cls/ja-books/models/sp30k/lstm_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.492803 0.584423 0.770000
|
||||
2 0.415298 0.697332 0.675000
|
||||
3 0.321692 0.742086 0.705000
|
||||
4 0.281904 1.092880 0.730000
|
||||
5 0.168274 1.050856 0.820000
|
||||
6 0.112483 0.895169 0.795000
|
||||
7 0.065752 1.082333 0.795000
|
||||
8 0.038848 1.138289 0.805000
|
||||
Total time: 05:46
|
||||
Saving models at /home/julian/ulmfit-multilingual/data/cls/ja-books/models/sp30k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.7825211, tensor(0.8514)]
|
||||
0.78252112865448
|
||||
0.8514257073402405
|
||||
```
|
||||
|
||||
## DE DVD LSTM
|
||||
|
||||
```bash
|
||||
(fastai) julian@dl-box-spot:~/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/cls/de-dvd --base-lm-path ../data/wiki/de-100/models/sp30k/lstm_nl4.m --tokenizer='sp' --lang=de --name 'nl4' - train 20 --bs 40 --lr-sched=1cycle --num-cls-epochs=8
|
||||
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/julian/ulmfit-multilingual/data/cls/de-dvd/models/sp30k
|
||||
Model dir: /home/julian/ulmfit-multilingual/data/cls/de-dvd/models/sp30k/lstm_nl4.m
|
||||
Validation set not found using 10% of trn
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 30600, val: 3400
|
||||
|
||||
Running tokenization cls...
|
||||
Data cls, trn: 1800, val: 200
|
||||
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 200, val: 2000
|
||||
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep',
|
||||
'<unk>', '▁', '▁.', '▁,', '▁der', '▁die', '▁und', '▁in', 'en', "▁&'", 's', '-']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, '
|
||||
hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/julian/data/wiki/de-100/models/sp30k/lstm_nl4.m/lm_best'), Po
|
||||
sixPath('/home/julian/data/wiki/de-100/models/sp30k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.564329 3.126153 0.456060
|
||||
Total time: 09:33
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.270905 3.030011 0.467729
|
||||
2 3.127578 2.923014 0.481434
|
||||
3 3.027712 2.804503 0.495315
|
||||
4 2.922042 2.698406 0.507350
|
||||
5 2.833169 2.605587 0.518419
|
||||
6 2.765501 2.521508 0.528258
|
||||
7 2.684195 2.443519 0.538367
|
||||
8 2.644001 2.373817 0.547404
|
||||
9 2.586362 2.309439 0.556455
|
||||
10 2.554237 2.253083 0.564804
|
||||
11 2.500762 2.196377 0.573611
|
||||
12 2.469062 2.144791 0.581450
|
||||
13 2.423278 2.093090 0.590096
|
||||
14 2.343388 2.043406 0.598047
|
||||
15 2.321417 2.008692 0.604748
|
||||
16 2.265463 1.972947 0.610544
|
||||
17 2.248210 1.948689 0.615224
|
||||
18 2.222042 1.934402 0.617739
|
||||
19 2.184187 1.926367 0.619161
|
||||
20 2.225068 1.925283 0.619336
|
||||
Total time: 3:47:59
|
||||
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.494345 0.347763 0.875000
|
||||
2 0.424967 0.466457 0.810000
|
||||
3 0.321692 0.440244 0.870000
|
||||
4 0.212389 0.323907 0.895000
|
||||
5 0.142327 0.532973 0.900000
|
||||
6 0.080535 0.452185 0.885000
|
||||
7 0.039367 0.456267 0.895000
|
||||
8 0.021793 0.470200 0.890000
|
||||
Total time: 06:18
|
||||
Saving models at /home/julian/ulmfit-multilingual/data/cls/de-dvd/models/sp30k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.56412625, tensor(0.8835)]
|
||||
0.5641262531280518
|
||||
0.8834999799728394
|
||||
```
|
||||
|
||||
## FR DVD LSTM
|
||||
|
||||
```bash
|
||||
(fastai) julian@dl-box-spot:~/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/cls/fr-dvd --base-lm-path ../data/wiki/fr-100/models/sp30k/lstm_nl4.m --tokenizer='sp' --lang=fr --name 'nl4' - train 20 --bs 40 --lr-sched=1cycle --num-cls-epochs=8
|
||||
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/julian/ulmfit-multilingual/data/cls/fr-dvd/models/sp30k
|
||||
Model dir: /home/julian/ulmfit-multilingual/data/cls/fr-dvd/models/sp30k/lstm_nl4.m
|
||||
Validation set not found using 10% of trn
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 12021, val: 1335
|
||||
Running tokenization cls...
|
||||
Data cls, trn: 1800, val: 200
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 200, val: 2000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁.', "'", 's', '▁la', '▁le', '▁et', '▁l', '▁à']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/julian/data/wiki/fr-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/julian/data/wiki/fr-100/models/sp30k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.825279 3.341784 0.382891
|
||||
Total time: 01:36
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.548075 3.280657 0.390619
|
||||
2 3.489078 3.206246 0.400057
|
||||
3 3.378426 3.099167 0.414316
|
||||
4 3.278241 2.985728 0.428153
|
||||
5 3.164112 2.868728 0.442793
|
||||
6 3.081279 2.740803 0.459863
|
||||
7 2.951635 2.615327 0.477424
|
||||
8 2.860259 2.511515 0.493157
|
||||
9 2.761055 2.386526 0.512397
|
||||
10 2.628587 2.277270 0.531014
|
||||
11 2.572315 2.181750 0.548689
|
||||
12 2.452535 2.083487 0.566041
|
||||
13 2.389231 1.998139 0.581409
|
||||
14 2.313358 1.927491 0.594620
|
||||
15 2.263673 1.873754 0.605384
|
||||
16 2.196958 1.827021 0.614506
|
||||
17 2.169217 1.797702 0.619863
|
||||
18 2.126882 1.777056 0.623906
|
||||
19 2.116131 1.767786 0.625270
|
||||
20 2.090418 1.765665 0.625703
|
||||
Total time: 37:20
|
||||
/home/julian/ulmfit-multilingual/data/cls/fr-dvd/models/sp30k
|
||||
Saving info /home/julian/ulmfit-multilingual/data/cls/fr-dvd/models/sp30k/lstm_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.586308 0.504827 0.750000
|
||||
2 0.478227 0.411526 0.860000
|
||||
3 0.417158 0.314054 0.890000
|
||||
4 0.286224 0.263725 0.900000
|
||||
5 0.163930 0.387664 0.880000
|
||||
6 0.095715 0.282535 0.930000
|
||||
7 0.051098 0.294014 0.930000
|
||||
8 0.028741 0.301007 0.930000
|
||||
Total time: 02:57
|
||||
Saving models at /home/julian/ulmfit-multilingual/data/cls/fr-dvd/models/sp30k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.5228756, tensor(0.8920)]
|
||||
0.5228756070137024
|
||||
0.8920000195503235
|
||||
```
|
||||
|
||||
## JA DVD LSTM
|
||||
|
||||
```bash
|
||||
(fastai) julian@dl-box-spot:~/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/cls/ja-dvd --base-lm-path ../data/wiki/ja-100/models/sp30k/lstm_nl4.m --tokenizer='sp' --lang=ja --name 'nl4' - train 20 --bs 40 --lr-sched=1cycle --num-cls-epochs=8
|
||||
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/julian/ulmfit-multilingual/data/cls/ja-dvd/models/sp30k
|
||||
Model dir: /home/julian/ulmfit-multilingual/data/cls/ja-dvd/models/sp30k/lstm_nl4.m
|
||||
Validation set not found using 10% of trn
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 30600, val: 3400
|
||||
Running tokenization cls...
|
||||
Data cls, trn: 1800, val: 200
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 200, val: 2000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁の', '▁。', '▁に', '▁を', '▁は', '▁年', '▁が', '▁)', '▁(']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/julian/data/wiki/ja-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/julian/data/wiki/ja-100/models/sp30k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.904242 3.480109 0.375665
|
||||
Total time: 04:37
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.671750 3.403708 0.386209
|
||||
2 3.565445 3.310030 0.398628
|
||||
3 3.443218 3.204042 0.411568
|
||||
4 3.374648 3.098158 0.424107
|
||||
5 3.288731 3.005343 0.435621
|
||||
6 3.187409 2.913090 0.446806
|
||||
7 3.135114 2.829881 0.457120
|
||||
8 3.073141 2.753949 0.467695
|
||||
9 2.996589 2.682856 0.478041
|
||||
10 2.909743 2.613899 0.487629
|
||||
11 2.859827 2.550690 0.497565
|
||||
12 2.818285 2.492902 0.507112
|
||||
13 2.779268 2.435685 0.516448
|
||||
14 2.718145 2.387462 0.525241
|
||||
15 2.664007 2.346267 0.532140
|
||||
16 2.641343 2.312850 0.537994
|
||||
17 2.599257 2.288488 0.542193
|
||||
18 2.579481 2.274002 0.544809
|
||||
19 2.571687 2.267283 0.545827
|
||||
20 2.560343 2.265548 0.546052
|
||||
Total time: 1:47:55
|
||||
/home/julian/ulmfit-multilingual/data/cls/ja-dvd/models/sp30k
|
||||
Saving info /home/julian/ulmfit-multilingual/data/cls/ja-dvd/models/sp30k/lstm_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.525819 0.402837 0.830000
|
||||
2 0.453459 0.425072 0.820000
|
||||
3 0.401383 0.482119 0.770000
|
||||
4 0.337860 0.502686 0.775000
|
||||
5 0.234284 0.805287 0.805000
|
||||
6 0.134409 0.729153 0.815000
|
||||
7 0.072370 0.895428 0.805000
|
||||
8 0.040945 0.832303 0.800000
|
||||
Total time: 03:09
|
||||
Saving models at /home/julian/ulmfit-multilingual/data/cls/ja-dvd/models/sp30k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.72511286, tensor(0.8395)]
|
||||
0.7251128554344177
|
||||
0.8395000100135803
|
||||
```
|
||||
|
||||
## DE MUSIC LSTM
|
||||
|
||||
```bash
|
||||
(fastai) julian@dl-box-spot:~/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/cls/de-music --base-lm-path ../data/wiki/de-100/models/sp30k/lstm_nl4.m --tokenizer='sp' --lang=de --name 'nl4' - train 20 --bs 20 --lr-sched=1cycle --num-cls-epochs=8
|
||||
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/julian/ulmfit-multilingual/data/cls/de-music/models/sp30k
|
||||
Model dir: /home/julian/ulmfit-multilingual/data/cls/de-music/models/sp30k/lstm_nl4.m
|
||||
Validation set not found using 10% of trn
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 30600, val: 3400
|
||||
Running tokenization cls...
|
||||
Data cls, trn: 1800, val: 200
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 200, val: 2000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', '▁und', '▁in', 'en', "▁&'", 's', '-']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/julian/data/wiki/de-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/julian/data/wiki/de-100/models/sp30k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.002521 3.526757 0.376006
|
||||
Total time: 06:32
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.738713 3.425875 0.389128
|
||||
2 3.607711 3.316313 0.403729
|
||||
3 3.441798 3.188569 0.418975
|
||||
4 3.347294 3.070090 0.432434
|
||||
5 3.261581 2.959181 0.447016
|
||||
6 3.161360 2.850135 0.460998
|
||||
7 3.092968 2.747133 0.475559
|
||||
8 2.999860 2.661642 0.488256
|
||||
9 2.946614 2.572933 0.502634
|
||||
10 2.834917 2.477213 0.517815
|
||||
11 2.768194 2.394673 0.532000
|
||||
12 2.743433 2.325108 0.545050
|
||||
13 2.596613 2.255300 0.557315
|
||||
14 2.347057 1.965440 0.616312
|
||||
15 2.289672 1.907880 0.626826
|
||||
16 2.238047 1.870341 0.634285
|
||||
17 2.169917 1.829818 0.641958
|
||||
18 2.145997 1.811395 0.645616
|
||||
19 2.111888 1.800622 0.647706
|
||||
20 2.067247 1.797927 0.648219
|
||||
Total time: 3:50:18
|
||||
/home/julian/ulmfit-multilingual/data/cls/de-music/models/sp30k
|
||||
Saving info /home/julian/ulmfit-multilingual/data/cls/de-music/models/sp30k/lstm_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.392259 0.319764 0.875000
|
||||
2 0.373723 0.401961 0.850000
|
||||
3 0.315902 0.415566 0.850000
|
||||
4 0.185113 0.312382 0.890000
|
||||
5 0.122869 0.399712 0.865000
|
||||
6 0.084130 0.435429 0.910000
|
||||
7 0.057294 0.394715 0.890000
|
||||
8 0.028046 0.391238 0.900000
|
||||
Total time: 06:34
|
||||
Saving models at /home/julian/ulmfit-multilingual/data/cls/de-music/models/sp30k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.4154407, tensor(0.9210)]
|
||||
0.41544070839881897
|
||||
0.9210000038146973
|
||||
```
|
||||
|
||||
## JA MUSIC LSTM
|
||||
|
||||
```bash
|
||||
(fastai) julian@dl-box-spot:~/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/cls/ja-music --base-lm-path ../data/wiki/ja-100/models/sp30k/lstm_nl4.m --tokenizer='sp' --lang=ja --name 'nl4' - train 20 --bs 20 --lr-sched=1cycle --num-cls-epochs=8
|
||||
|
||||
Data lm, trn: 30600, val: 3399
|
||||
Running tokenization cls...
|
||||
Data cls, trn: 1800, val: 200
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 200, val: 1999
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁の', '▁。', '▁に', '▁を', '▁は', '▁年', '▁が', '▁)', '▁(']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/julian/data/wiki/ja-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/julian/data/wiki/ja-100/models/sp30k/lstm_nl4.m/../itos')]
|
||||
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.680415 3.162607 0.451358
|
||||
Total time: 09:43
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.258039 3.043514 0.467028
|
||||
2 3.082901 2.909904 0.482319
|
||||
3 3.009017 2.784519 0.496660
|
||||
4 2.900604 2.671213 0.510406
|
||||
5 2.776977 2.570581 0.522192
|
||||
6 2.789654 2.488552 0.532739
|
||||
7 2.710876 2.407913 0.543447
|
||||
8 2.655036 2.342028 0.553344
|
||||
9 2.571593 2.281001 0.562552
|
||||
10 2.539299 2.207963 0.574177
|
||||
11 2.466461 2.139726 0.585225
|
||||
12 2.441152 2.081656 0.595266
|
||||
13 2.434502 2.018719 0.606514
|
||||
14 2.576859 2.190329 0.569373
|
||||
15 2.543341 2.137856 0.579508
|
||||
16 2.467283 2.092796 0.587677
|
||||
17 2.417593 2.061508 0.593782
|
||||
18 2.375962 2.038786 0.598027
|
||||
19 2.391491 2.029075 0.599871
|
||||
20 2.352595 2.026604 0.600235
|
||||
Total time: 2:41:18
|
||||
/home/julian/ulmfit-multilingual/data/cls/ja-music/models/sp30k
|
||||
Saving info /home/julian/ulmfit-multilingual/data/cls/ja-music/models/sp30k/lstm_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.458052 0.371875 0.860000
|
||||
2 0.473817 0.539201 0.730000
|
||||
3 0.423880 0.390433 0.845000
|
||||
4 0.310703 0.402607 0.855000
|
||||
5 0.211760 0.607136 0.865000
|
||||
6 0.108535 0.845904 0.860000
|
||||
7 0.053125 0.897018 0.860000
|
||||
8 0.024544 0.891663 0.855000
|
||||
Total time: 04:29
|
||||
Saving models at /home/julian/ulmfit-multilingual/data/cls/ja-music/models/sp30k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.777393, tensor(0.8644)]
|
||||
0.7773929834365845
|
||||
0.8644322156906128
|
||||
```
|
||||
|
||||
## FR MUSIC LSTM
|
||||
|
||||
```bash
|
||||
(fastai) julian@dl-box-spot:~/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/cls/fr-music --base-lm-path ../data/wiki/fr-100/models/sp30k/lstm_nl4.m --tokenizer='sp' --lang=fr --name 'nl4' - train 20 --bs 40 --lr-sched=1cycle --num-cls-epochs=8
|
||||
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 200, val: 2000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁.', "'", 's', '▁la', '▁le', '▁et', '▁l', '▁à']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/julian/data/wiki/fr-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/julian/data/wiki/fr-100/models/sp30k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.686174 3.271906 0.394277
|
||||
Total time: 02:47
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.490278 3.197725 0.403639
|
||||
2 3.356067 3.100815 0.415938
|
||||
3 3.269037 2.970971 0.433533
|
||||
4 3.102959 2.819620 0.453009
|
||||
5 2.957009 2.647919 0.478246
|
||||
6 2.793691 2.481614 0.504928
|
||||
7 2.646251 2.316360 0.534853
|
||||
8 2.532166 2.140370 0.566817
|
||||
9 2.361445 1.982554 0.596333
|
||||
10 2.258446 1.855159 0.621765
|
||||
11 2.155252 1.772740 0.640348
|
||||
12 2.071291 1.668775 0.660405
|
||||
13 1.887608 1.579711 0.677771
|
||||
14 1.873631 1.493046 0.694815
|
||||
15 1.824689 1.438728 0.705296
|
||||
16 1.766544 1.398732 0.714093
|
||||
17 1.646138 1.372478 0.719408
|
||||
18 1.684073 1.350950 0.723633
|
||||
19 1.650602 1.344994 0.724889
|
||||
20 1.602114 1.341957 0.725314
|
||||
Total time: 1:04:53
|
||||
/home/julian/ulmfit-multilingual/data/cls/fr-music/models/sp30k
|
||||
Saving info /home/julian/ulmfit-multilingual/data/cls/fr-music/models/sp30k/lstm_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.415938 0.214479 0.930000
|
||||
2 0.373839 0.334263 0.880000
|
||||
3 0.317772 0.660272 0.795000
|
||||
4 0.207807 0.440546 0.880000
|
||||
5 0.146999 0.377026 0.890000
|
||||
6 0.095834 0.288273 0.925000
|
||||
7 0.048218 0.350355 0.895000
|
||||
8 0.023682 0.325598 0.915000
|
||||
Total time: 03:18
|
||||
Saving models at /home/julian/ulmfit-multilingual/data/cls/fr-music/models/sp30k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.32345334, tensor(0.9295)]
|
||||
0.3234533369541168
|
||||
0.9294999837875366
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,118 +0,0 @@
|
||||
# Results on books dataset
|
||||
|
||||
| | de | fr |
|
||||
|-------------------------|-------|-------|
|
||||
| laser zero shot from en | 84.15 | 83.90 |
|
||||
| with ULMFIT QRNN sp15k | 89.60 | 87.84 |
|
||||
|
||||
## Laser results
|
||||
|
||||
| | en | de | fr |
|
||||
|------|--------|-------|------|
|
||||
| en: | 84.55 | 84.15 | 83.90|
|
||||
| de: | 82.60 | 85.20 | 83.05|
|
||||
| fr: | 77.20 | 82.95 | 84.85|
|
||||
|
||||
## ULMFiT improvment
|
||||
```
|
||||
data/cls/de-books-laser-en1/models/sp15k/qrnn_nl4.m: 0.8960000276565552
|
||||
data/cls/fr-books-laser-en1/models/sp15k/qrnn_nl4.m: 0.8784999847412109
|
||||
```
|
||||
|
||||
### Execution log
|
||||
```
|
||||
python -m ulmfit eval --glob="cls/*-books/models/sp15k/qrnn_nl4.m" --name nl4 --dataset-template='${lang}-books-laser-en1' --num-lm-epochs=0 --num-cls-epochs=8 --bs=18 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
Processing data/cls/de-books/models/sp15k/qrnn_nl4.m
|
||||
de-books-laser-en1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books-laser-en1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books-laser-en1/models/sp15k/qrnn_nl4.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books-laser-en1/de.dev.csv
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 152523, val: 16947
|
||||
Running tokenization cls...
|
||||
Data cls, trn: 1800, val: 200
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 200, val: 2000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
Your compiler (c++) may be ABI-incompatible with PyTorch!
|
||||
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
|
||||
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
|
||||
|
||||
See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
|
||||
for instructions on how to install GCC 4.9 or higher.
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books-laser-en1/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books-laser-en1/models/sp15k/qrnn_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.541837 0.487596 0.870000
|
||||
2 0.498016 0.490300 0.885000
|
||||
3 0.442417 0.479205 0.875000
|
||||
4 0.395640 0.528897 0.855000
|
||||
5 0.369408 0.521830 0.855000
|
||||
6 0.361129 0.481892 0.880000
|
||||
7 0.351095 0.481634 0.885000
|
||||
8 0.343147 0.481654 0.880000
|
||||
Total time: 02:35
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/de-books-laser-en1/models/sp15k/qrnn_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.29498395, tensor(0.8960)]
|
||||
Processing data/cls/en-books/models/sp15k/qrnn_nl4.m
|
||||
en-books-laser-en1
|
||||
Processing data/cls/fr-books/models/sp15k/qrnn_nl4.m
|
||||
fr-books-laser-en1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books-laser-en1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books-laser-en1/models/sp15k/qrnn_nl4.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books-laser-en1/fr.dev.csv
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 33183, val: 3687
|
||||
Running tokenization cls...
|
||||
Data cls, trn: 1800, val: 200
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 200, val: 2000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books-laser-en1/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books-laser-en1/models/sp15k/qrnn_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.551931 0.532421 0.835000
|
||||
2 0.509913 0.524893 0.880000
|
||||
3 0.433385 0.502657 0.860000
|
||||
4 0.397314 0.487201 0.880000
|
||||
5 0.365447 0.467523 0.885000
|
||||
6 0.356587 0.520736 0.855000
|
||||
7 0.353801 0.487093 0.875000
|
||||
8 0.343812 0.484453 0.880000
|
||||
Total time: 01:45
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/cls/fr-books-laser-en1/models/sp15k/qrnn_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.32666296, tensor(0.8785)]
|
||||
Processing data/cls/ja-books/models/sp15k/qrnn_nl4.m
|
||||
ja-books-laser-en1
|
||||
OrderedDict([('data/cls/de-books-laser-en1/models/sp15k/qrnn_nl4.m',
|
||||
0.8960000276565552),
|
||||
('data/cls/fr-books-laser-en1/models/sp15k/qrnn_nl4.m',
|
||||
0.8784999847412109)])
|
||||
data/cls/de-books-laser-en1/models/sp15k/qrnn_nl4.m: 0.8960000276565552
|
||||
data/cls/fr-books-laser-en1/models/sp15k/qrnn_nl4.m: 0.8784999847412109
|
||||
```
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,697 +0,0 @@
|
||||
# DE
|
||||
## SP15k LSTM nl4
|
||||
```
|
||||
$ python -m ulmfit lm --dataset-path data/wiki/de-100 --cuda-id=0 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang de --qrnn=False - train 10 --bs=100 --drop_mult=0
|
||||
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der',
|
||||
'▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.519809 2.600072 0.529963
|
||||
2 2.436580 2.538897 0.534651
|
||||
3 2.402220 2.510569 0.537314
|
||||
4 2.305741 2.439347 0.546574
|
||||
5 2.265683 2.376482 0.553794
|
||||
6 2.210663 2.305362 0.562672
|
||||
7 2.134196 2.230041 0.572958
|
||||
8 2.085375 2.150917 0.584621
|
||||
9 2.037781 2.097170 0.593747
|
||||
10 1.986773 2.081469 0.595799
|
||||
Total time: 19:18:33
|
||||
data/wiki/de-100/models/sp15k
|
||||
Saving info data/wiki/de-100/models/sp15k/lstm_nl4.m/info.jso
|
||||
```
|
||||
### MLDoc
|
||||
```bash
|
||||
LANG=de
|
||||
python -m ulmfit cls --dataset-path data/mldoc-m/${LANG}-1 --base-lm-path data/wiki-m/${LANG}-100/models/sp15k/lstm_nl4.m --lang=${LANG} --name 'nl4' - train 20 --bs 20 --num-cls-
|
||||
epochs=8 --lr_sched=1cycle
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc-m/de-1/models/sp15k
|
||||
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc-m/de-1/models/sp15k/lstm_nl4.m
|
||||
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc-m/de-1/de.dev.csv
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Running tokenization cls...
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki-m/de-100/models/sp15k/lstm_nl4.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki-m/de-100/models/sp15k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.333600 2.005051 0.596875
|
||||
Total time: 07:40
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.120653 1.886799 0.615784
|
||||
2 1.980713 1.763139 0.636041
|
||||
3 1.805195 1.655620 0.654068
|
||||
4 1.729641 1.564017 0.668772
|
||||
5 1.681813 1.491185 0.680613
|
||||
6 1.682965 1.422562 0.692458
|
||||
7 1.580731 1.357177 0.703143
|
||||
8 1.506753 1.297219 0.714487
|
||||
9 1.515824 1.235473 0.725413
|
||||
10 1.427750 1.178680 0.737216
|
||||
11 1.371839 1.118909 0.749590
|
||||
12 1.342978 1.068754 0.760473
|
||||
13 1.286842 1.011940 0.772384
|
||||
14 1.254822 0.960727 0.784244
|
||||
15 1.195136 0.919377 0.793910
|
||||
16 1.118260 0.881799 0.802814
|
||||
17 1.071546 0.855769 0.809040
|
||||
18 1.079081 0.839280 0.812895
|
||||
19 1.052724 0.831323 0.814723
|
||||
20 1.024207 0.829737 0.815070
|
||||
Total time: 3:08:58
|
||||
/home/test/workspace/ulmfit-multilingual/data/mldoc-m/de-1/models/sp15k
|
||||
Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc-m/de-1/models/sp15k/lstm_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.539181 0.239851 0.938000
|
||||
2 0.326801 0.374512 0.917000
|
||||
3 0.225103 0.330872 0.945000
|
||||
4 0.121660 0.444890 0.938000
|
||||
5 0.078411 0.422513 0.948000
|
||||
6 0.061354 0.509489 0.949000
|
||||
7 0.029890 0.438118 0.949000
|
||||
8 0.014213 0.441808 0.949000
|
||||
Total time: 09:00
|
||||
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc-m/de-1/models/sp15k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.3710725, tensor(0.9553)]
|
||||
|
||||
0.3710725009441376
|
||||
0.9552500247955322
|
||||
```
|
||||
|
||||
|
||||
## VF60k LSTM nl 3
|
||||
### LM
|
||||
```
|
||||
python -m ulmfit lm --dataset-path data/wiki/de-100 --cuda-id=1 --tokenizer='vf' --nl 3 --name 'nl3' --max-vocab 60000 --lang de --qrnn=False - train 10 --bs=50 --drop_mult=0
|
||||
Max vocab: 60000
|
||||
Cache dir: data/wiki/de-100/models/vf60k
|
||||
Model dir: data/wiki/de-100/models/vf60k/lstm_nl3.m
|
||||
Running tokenization
|
||||
Wiki text was split to 175965 articles
|
||||
Wiki text was split to 110 articles
|
||||
Size of vocabulary: 60003
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', ',', 'der', '.', 'und', 'die', 'in', "&'", 'von', 'den', '(', 'im', ')']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': None, 'pretrained_model': None, 'drop_mult': 0} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.214624 3.573368 0.397312
|
||||
2 3.194401 3.549021 0.396143
|
||||
3 3.116934 3.535322 0.398108
|
||||
4 3.159205 3.498862 0.400490
|
||||
5 3.104538 3.454015 0.405504
|
||||
6 2.996653 3.410940 0.409791
|
||||
7 2.987909 3.359425 0.413711
|
||||
8 2.941863 3.311215 0.419416
|
||||
9 2.914403 3.285807 0.423674
|
||||
10 2.857530 3.278313 0.425131
|
||||
data/wiki/de-100/models/vf60k
|
||||
Saving info data/wiki/de-100/models/vf60k/lstm_nl3.m/info.json
|
||||
```
|
||||
### MLDocs
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/de-1 --base-lm-path data/wiki/de-100/models/vf60k/lstm_nl3.m --lang=de --name 'nl3' - train 20 --bs 40
|
||||
Max vocab: 60000
|
||||
Cache dir: data/mldoc/de-1/models/vf60k
|
||||
Model dir: data/mldoc/de-1/models/vf60k/lstm_nl3.m
|
||||
Loading validation data/mldoc/de-1/de.dev.csv
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 13500, cls.val 1500
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 39171
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '.', 'der', ',', 'die', ')', '(', 'in', 'und', 'auf', 'von', 'den', 'im']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('data/wiki/de-100/models/vf60k/lstm_nl3.m/lm_best'), PosixPath('data/wiki/de-100/models/vf60k/lstm_nl3.m/../itos')], 'pretrained_model': None, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 20582, first 100: ['"', 'vh', 'ös', 'brs', '&', 'geg', 'lpo', 'vormonat', 'fgc', 'waigel', 'tcs', 'mic', 'bund-future', 'ajs', 'brn', 'dih', 'analysten', 'mrd', 'rpk', 'notierten', 'dividende', 'feb', 'aktienmarkt', 'rev', 'rentenmarkt', 'basispunkte', 'müßten', 'gewinnmitnahmen', 'aktienbörse', 'rußland', 'volkswirte', 'fls', 'steuerreform', 'kontrakte', 'kps', 'mge', 'zählern', 'vortagesschluß', 'umsätzen', 'prozent.', 'snb', 'dow-jones-index', 'reingewinn', 'notierungen', "\\'", 'gesamtmarkt', 'industrieproduktion', 'akr', 'kjf', '49-69-7565', 'abl', 'hoh', 'finanzdienst', 'atx', 'feinunze', 'zinserhöhung', 'zugelegt', 'netanjahu', 'verbraucherpreise', 'pence', 'ticks', 'arafat', 'kursgewinne', 'ker', 'aktienindex', 'rlb', 'smi', 'vorbörslich', 'dst', 'mkl', 'kontrakten', 'calls', 'veraenderung', 'gwa', 'gesamtjahr', 'auftragseingang', 'überschuß', 'erwarte', 'verlautete', 'eju', 'tms', 'jahresvergleich', 'vorjahreszeitraum', 'werden.', 'betriebsergebnis', 'rin', 'bobl-future', 'puts', 'fri', '4.50', 'schluß', 'ewu', 'standardwerte', 'jahresüberschuß', 'rechne', '49-69-756525', '16.00', 'peh', 'hmh', 'dtb']
|
||||
Training lm from: [PosixPath('data/wiki/de-100/models/vf60k/lstm_nl3.m/lm_best'), PosixPath('data/wiki/de-100/models/vf60k/lstm_nl3.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.532079 3.059487 0.465283
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.219148 2.945357 0.475736
|
||||
2 3.014822 2.804256 0.494567
|
||||
3 2.896143 2.652700 0.513166
|
||||
4 2.756027 2.516747 0.528836
|
||||
5 2.629735 2.383480 0.543956
|
||||
6 2.515785 2.281831 0.556083
|
||||
7 2.422463 2.178855 0.567950
|
||||
8 2.351060 2.091266 0.579531
|
||||
9 2.297676 2.017783 0.590206
|
||||
10 2.205688 1.937085 0.601936
|
||||
11 2.155664 1.871271 0.612579
|
||||
12 2.065812 1.806647 0.623888
|
||||
13 2.038635 1.748420 0.634389
|
||||
14 1.957434 1.696571 0.643807
|
||||
15 1.895242 1.653865 0.651743
|
||||
16 1.910458 1.618776 0.658140
|
||||
17 1.843909 1.598143 0.662129
|
||||
18 1.837299 1.583182 0.664999
|
||||
19 1.788718 1.573136 0.666785
|
||||
20 1.780236 1.574308 0.666625
|
||||
data/mldoc/de-1/models/vf60k
|
||||
Saving info data/mldoc/de-1/models/vf60k/lstm_nl3.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.526303 0.328480 0.892000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.346665 0.238605 0.920000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.266841 0.285444 0.921000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.175013 0.280545 0.921000
|
||||
2 0.178333 0.286059 0.923000
|
||||
Saving models at data/mldoc/de-1/models/vf60k/lstm_nl3.m
|
||||
Loss and accuracy using (cls_best): [0.16954255, tensor(0.9475)]
|
||||
OrderedDict([('data/mldoc/de-1/models/vf60k/lstm_nl3.m', 0.9474999904632568)])
|
||||
```
|
||||
MultiCCA: 93.7% , ulmfit: 94.74%
|
||||
## VF60k QRNN nl 4
|
||||
```
|
||||
Bptt 70
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.131590 4.123428 0.460519
|
||||
2 4.073408 4.077561 0.461808
|
||||
3 4.038208 4.056053 0.464489
|
||||
4 4.007055 4.012294 0.469722
|
||||
5 3.992992 3.977304 0.473496
|
||||
6 3.903659 3.934043 0.480102
|
||||
7 3.897762 3.894066 0.484782
|
||||
8 3.877661 3.854888 0.492338
|
||||
9 3.831059 3.829723 0.497970
|
||||
10 3.810376 3.823137 0.499966
|
||||
Total time: 18:44:08
|
||||
data/wiki/de-100/models/vf60k
|
||||
Saving info data/wiki/de-100/models/vf60k/qrnn_nl4.m/info.json
|
||||
```
|
||||
|
||||
```bash
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/de-100/models/vf60k/qrnn_nl4.m --lang=${LANG} --name nl4 - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
Max vocab: 60000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/vf60k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/vf60k/qrnn_nl4.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/de.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 39171
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '.', 'der', ',', 'die', ')', '(', 'in', 'und', 'auf', 'von', 'den', 'im']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
Your compiler (c++) may be ABI-incompatible with PyTorch!
|
||||
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
|
||||
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
|
||||
|
||||
See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
|
||||
for instructions on how to install GCC 4.9 or higher.
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
|
||||
Loading pretrained model
|
||||
Unknown tokens 20300, first 100: ['"', 'vh', 'ös', 'brs', '&', 'geg', 'lpo', 'vormonat', 'fgc', 'mesz', 'waigel', 'tcs', 'bund-future', 'ajs', 'brn', 'dih', 'analysten', 'mrd', 'rpk', 'emu', 'notierten', 'feb', 'aktienmarkt', 'dor', 'rentenmarkt', 'basispunkte', 'müßten', 'gewinnmitnahmen', 'aktienbörse', 'jelzin', 'rußland', 'volkswirte', 'fls', 'steuerreform', 'kontrakte', 'kps', 'mge', 'vortagesschluß', 'umsätzen', 'prozent.', 'dow-jones-index', 'reingewinn', 'notierungen', "\\'", 'gesamtmarkt', 'akr', 'kjf', '49-69-7565', 'abl', 'hoh', 'finanzdienst', 'atx', 'feinunze', 'zinserhöhung', 'zugelegt', 'verbraucherpreise', 'ticks', 'kursgewinne', 'ker', 'rlb', 'smi', 'vorbörslich', 'dst', 'mkl', 'ale', 'kontrakten', 'calls', 'veraenderung', 'gwa', 'gesamtjahr', 'auftragseingang', 'überschuß', 'verlautete', 'eju', 'tms', 'jahresvergleich', 'vorjahreszeitraum', 'werden.', 'betriebsergebnis', 'bobl-future', 'puts', 'fri', '4.50', 'schluß', 'ewu', 'spanne', 'standardwerte', 'jahresüberschuß', 'rechne', 'lire', '49-69-756525', '16.00', 'peh', 'hmh', 'dtb', 'tagesgeld', 'us-notenbank', 'corp', 'vorstandschef', 'greenspan']
|
||||
Bptt 70
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/de-100/models/vf60k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/de-100/models/vf60k/qrnn_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.724948 4.178421 0.449862
|
||||
Total time: 02:19
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.312489 4.049916 0.466992
|
||||
2 4.197414 3.919862 0.488602
|
||||
3 4.000882 3.793147 0.510018
|
||||
4 3.960565 3.691311 0.524944
|
||||
5 3.841827 3.590782 0.539775
|
||||
6 3.756638 3.515933 0.551585
|
||||
7 3.738561 3.439131 0.563536
|
||||
8 3.623295 3.371250 0.575563
|
||||
9 3.585063 3.307532 0.586810
|
||||
10 3.523384 3.256143 0.596964
|
||||
11 3.484239 3.195987 0.610036
|
||||
12 3.439287 3.140971 0.622494
|
||||
13 3.385262 3.087693 0.634652
|
||||
14 3.308803 3.050615 0.644066
|
||||
15 3.242234 2.999897 0.656247
|
||||
16 3.229038 2.966996 0.664862
|
||||
17 3.203192 2.946324 0.670038
|
||||
18 3.169675 2.930080 0.674204
|
||||
19 3.140696 2.920569 0.676475
|
||||
20 3.207376 2.919055 0.676769
|
||||
Total time: 1:00:09
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/vf60k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/vf60k/qrnn_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.779320 0.638282 0.925000
|
||||
2 0.653313 0.592119 0.940000
|
||||
3 0.569095 0.577936 0.939000
|
||||
4 0.519593 0.568577 0.947000
|
||||
Total time: 00:50
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/vf60k/qrnn_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.19424936, tensor(0.9528)]
|
||||
0.19424936175346375
|
||||
0.952750027179718
|
||||
```
|
||||
|
||||
## SP30k LSTM nl 4
|
||||
### LM
|
||||
```
|
||||
python -m ulmfit lm --dataset-path data/wiki/de-100 --cuda-id=0 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 30000 --lang sp --qrnn=False - train 10 --bs=50 --drop_mult=0
|
||||
1,2.833101,3.174348,0.472863
|
||||
2,2.788717,3.171983,0.471377
|
||||
3,2.831292,3.187135,0.471068
|
||||
4,2.723390,3.133801,0.475572
|
||||
5,2.681617,3.064743,0.481984
|
||||
6,2.662792,2.984701,0.489080
|
||||
7,2.542035,2.892254,0.499275
|
||||
8,2.422225,2.806846,0.508663
|
||||
9,2.462655,2.736171,0.517994
|
||||
10,2.396778,2.714520,0.521145
|
||||
data/wiki/de-100/models/sp30k/lstm_nl4.m/lm-history.csv
|
||||
```
|
||||
|
||||
### MLDocs
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/de-1 --base-lm-path data/wiki/de-100/models/sp30k/lstm_nl4.m --lang=de --name 'nl4' - train 20 --bs 40 ✘ 1
|
||||
Max vocab: 30000
|
||||
Cache dir: data/mldoc/de-1/models/sp30k
|
||||
Model dir: data/mldoc/de-1/models/sp30k/lstm_nl4.m
|
||||
Loading validation data/mldoc/de-1/de.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Tokenized data loaded, cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', '▁und', '▁in', 'en', "▁&'", 's', '-']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('data/wiki/de-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('data/wiki/de-100/models/sp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('data/wiki/de-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('data/wiki/de-100/models/sp30k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.042075 2.457199 0.547201
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.581403 2.305440 0.565500
|
||||
2 2.366814 2.139165 0.589417
|
||||
3 2.187646 1.986698 0.612081
|
||||
4 2.054434 1.857322 0.630642
|
||||
5 1.948663 1.758499 0.644389
|
||||
6 1.850596 1.673632 0.655852
|
||||
7 1.813331 1.593225 0.668256
|
||||
8 1.738136 1.523946 0.678633
|
||||
9 1.683469 1.463405 0.688561
|
||||
10 1.609236 1.410462 0.697171
|
||||
11 1.599416 1.356008 0.706997
|
||||
12 1.526982 1.308399 0.715433
|
||||
13 1.487115 1.263120 0.723749
|
||||
14 1.430917 1.224060 0.731837
|
||||
15 1.410333 1.191501 0.738267
|
||||
16 1.385961 1.166404 0.743477
|
||||
17 1.349813 1.144801 0.747553
|
||||
18 1.345938 1.132679 0.750188
|
||||
19 1.311102 1.127321 0.751208
|
||||
20 1.355743 1.126064 0.751384
|
||||
Saving info data/mldoc/de-1/models/sp30k/lstm_nl4.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.490199 0.246640 0.940000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.302251 0.243051 0.932000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.211028 0.249550 0.932000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.159555 0.230822 0.947000
|
||||
2 0.144418 0.226450 0.943000
|
||||
Saving models at data/mldoc/de-1/models/sp30k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_last): [0.16306259, tensor(0.9540)]
|
||||
```
|
||||
MultiCCA: 93.7% , ulmfit: 95.4%
|
||||
```
|
||||
Max vocab: 30000
|
||||
Cache dir: data/mldoc/de-1/models/sp30k
|
||||
Model dir: data/mldoc/de-1/models/sp30k/lstm_nl4-2nd.m
|
||||
Loading validation data/mldoc/de-1/de.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Tokenized data loaded, cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', '▁und', '▁in', 'en', "▁&'", 's', '-']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('data/mldoc/de-1/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('data/mldoc/de-1/models/sp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
data/mldoc/de-1/models/sp30k
|
||||
Saving info data/mldoc/de-1/models/sp30k/lstm_nl4-2nd.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.464957 0.258905 0.928000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.284900 0.243053 0.937000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.298546 0.204188 0.948000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.159097 0.199651 0.952000
|
||||
2 0.112476 0.203827 0.953000
|
||||
Saving models at data/mldoc/de-1/models/sp30k/lstm_nl4-2nd.m
|
||||
Loss and accuracy using (cls_last): [0.1689675, tensor(0.9550)]
|
||||
```
|
||||
### examples limited to 100
|
||||
|
||||
#### 2x run
|
||||
first run
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/de-1 --base-lm-path data/mldoc/de-1/models/sp30k/lstm_nl4.m --lang=de --name 'nl4-100-2x' --cuda-id=1 - train 0 --bs 40 --limit=100 --drop-mult-cls=0.3
|
||||
Max vocab: 30000
|
||||
Cache dir: data/mldoc/de-1/models/sp30k
|
||||
Model dir: data/mldoc/de-1/models/sp30k/lstm_nl4-100-2x.m
|
||||
Loading validation data/mldoc/de-1/de.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Limiting data set to: 100
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 100, cls.val 100
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', '▁und', '▁in', 'en', "▁&'", 's', '-']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('data/mldoc/de-1/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('data/mldoc/de-1/models/sp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
data/mldoc/de-1/models/sp30k
|
||||
Saving info data/mldoc/de-1/models/sp30k/lstm_nl4-100-2x.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.181207 1.315258 0.300000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.749909 1.204297 0.660000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.558658 1.083666 0.830000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.486175 1.020435 0.850000
|
||||
2 0.485117 0.958238 0.880000
|
||||
Saving models at data/mldoc/de-1/models/sp30k/lstm_nl4-100-2x.m
|
||||
..? ..
|
||||
```
|
||||
2nd run
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/de-1 --base-lm-path data/mldoc/de-1/models/sp30k/lstm_nl4.m --lang=de --name 'nl4-100-2x' --cuda-id=1 - train 0 --bs 40 --limit=100
|
||||
Max vocab: 30000
|
||||
Cache dir: data/mldoc/de-1/models/sp30k
|
||||
Model dir: data/mldoc/de-1/models/sp30k/lstm_nl4-100-2x.m
|
||||
Loading validation data/mldoc/de-1/de.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Limiting data set to: 100
|
||||
Tokenized data loaded, cls.trn 100, cls.val 100
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', '▁und', '▁in', 'en', "▁&'", 's', '-']
|
||||
Loading last classifier
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.441340 0.716396 0.840000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.312035 0.532610 0.910000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.267714 0.462694 0.920000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.242031 0.430018 0.930000
|
||||
2 0.231161 0.398335 0.930000
|
||||
Saving models at data/mldoc/de-1/models/sp30k/lstm_nl4-100-2x.m
|
||||
Loss and accuracy using (cls_last): [0.33284584, tensor(0.9252)]
|
||||
```
|
||||
#### 8 epoches at the end
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/de-1 --base-lm-path data/mldoc/de-1/models/sp30k/lstm_nl4.m --lang=de --name 'nl4-100-e8' --cuda-id=1 - train 0 --bs 40 --limit=100 --num-cls-epochs=8 --drop-mult-cls=0.3
|
||||
Max vocab: 30000
|
||||
Cache dir: data/mldoc/de-1/models/sp30k
|
||||
Model dir: data/mldoc/de-1/models/sp30k/lstm_nl4-100-e8.m
|
||||
Loading validation data/mldoc/de-1/de.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Limiting data set to: 100
|
||||
Tokenized data loaded, cls.trn 100, cls.val 100
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', '▁und', '▁in', 'en', "▁&'", 's', '-']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('data/mldoc/de-1/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('data/mldoc/de-1/models/sp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
data/mldoc/de-1/models/sp30k
|
||||
Saving info data/mldoc/de-1/models/sp30k/lstm_nl4-100-e8.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.208816 1.324359 0.280000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.716811 1.195012 0.440000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.535809 1.075753 0.590000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.499198 1.018431 0.760000
|
||||
2 0.480971 0.948658 0.880000
|
||||
3 0.468659 0.866477 0.860000
|
||||
4 0.460322 0.770794 0.880000
|
||||
5 0.461138 0.704613 0.900000
|
||||
6 0.442423 0.623944 0.900000
|
||||
7 0.422423 0.568031 0.920000
|
||||
8 0.417041 0.527571 0.930000
|
||||
Saving models at data/mldoc/de-1/models/sp30k/lstm_nl4-100-e8.m
|
||||
Loss and accuracy using (cls_last): [0.47343642, tensor(0.9070)]
|
||||
```
|
||||
Dropout 0.6
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/de-1 --base-lm-path data/mldoc/de-1/models/sp30k/lstm_nl4.m --lang=de --name 'nl4-100-e8dp6' --cuda-id=1 - train 0 --bs 40 --limit=100 --num-cls-epochs=8 --drop-mult-cls=0.6
|
||||
Max vocab: 30000
|
||||
Cache dir: data/mldoc/de-1/models/sp30k
|
||||
Model dir: data/mldoc/de-1/models/sp30k/lstm_nl4-100-e8dp6.m
|
||||
Loading validation data/mldoc/de-1/de.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Limiting data set to: 100
|
||||
Tokenized data loaded, cls.trn 100, cls.val 100
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', '▁und', '▁in', 'en', "▁&'", 's', '-']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('data/mldoc/de-1/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('data/mldoc/de-1/models/sp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
data/mldoc/de-1/models/sp30k
|
||||
Saving info data/mldoc/de-1/models/sp30k/lstm_nl4-100-e8dp6.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.151162 1.323036 0.280000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.745338 1.160084 0.610000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.535118 1.041519 0.770000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.459913 0.995187 0.860000
|
||||
2 0.451289 0.949036 0.830000
|
||||
3 0.460395 0.885940 0.800000
|
||||
4 0.454847 0.848194 0.770000
|
||||
5 0.447404 0.788741 0.810000
|
||||
6 0.428524 0.748181 0.760000
|
||||
7 0.419571 0.696069 0.760000
|
||||
8 0.408938 0.661937 0.770000
|
||||
Saving models at data/mldoc/de-1/models/sp30k/lstm_nl4-100-e8dp6.m
|
||||
Loss and accuracy using (cls_last): [0.53202456, tensor(0.8830)]
|
||||
```
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/de-1 --base-lm-path data/mldoc/de-1/models/sp30k/lstm_nl4.m --lang=de --name 'nl4-100-e8dp6x2' --cuda-id=1 - train 0 --bs 40 --limit=100 --num-cls-epochs=8 --drop-mult-cls=0.6
|
||||
Max vocab: 30000
|
||||
Cache dir: data/mldoc/de-1/models/sp30k
|
||||
Model dir: data/mldoc/de-1/models/sp30k/lstm_nl4-100-e8dp6x2.m
|
||||
Loading validation data/mldoc/de-1/de.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Limiting data set to: 100
|
||||
Tokenized data loaded, cls.trn 100, cls.val 100
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', '▁und', '▁in', 'en', "▁&'", 's', '-']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('data/mldoc/de-1/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('data/mldoc/de-1/models/sp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
data/mldoc/de-1/models/sp30k
|
||||
Saving info data/mldoc/de-1/models/sp30k/lstm_nl4-100-e8dp6x2.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.126586 1.338514 0.470000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.751379 1.181071 0.550000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.559534 1.083532 0.810000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.444137 1.034607 0.870000
|
||||
2 0.438850 0.983929 0.830000
|
||||
3 0.436560 0.906958 0.840000
|
||||
4 0.447400 0.847952 0.840000
|
||||
5 0.431961 0.783818 0.850000
|
||||
6 0.422364 0.713126 0.850000
|
||||
7 0.414145 0.662799 0.840000
|
||||
8 0.407066 0.630168 0.840000
|
||||
Saving models at data/mldoc/de-1/models/sp30k/lstm_nl4-100-e8dp6x2.m
|
||||
Loss and accuracy using (cls_last): [0.46259913, tensor(0.9147)]
|
||||
```
|
||||
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/de-1 --base-lm-path data/mldoc/de-1/models/sp30k/lstm_nl4.m --lang=de --name 'nl4-100-e8dp2' --cuda-id=1 - train 0 --bs 40 --limit=100 --num-cls-epochs=8 --drop-mult-cls=0.2
|
||||
Max vocab: 30000
|
||||
Cache dir: data/mldoc/de-1/models/sp30k
|
||||
Model dir: data/mldoc/de-1/models/sp30k/lstm_nl4-100-e8dp2.m
|
||||
Loading validation data/mldoc/de-1/de.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Limiting data set to: 100
|
||||
Tokenized data loaded, cls.trn 100, cls.val 100
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', '▁und', '▁in', 'en', "▁&'", 's', '-']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('data/mldoc/de-1/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('data/mldoc/de-1/models/sp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
data/mldoc/de-1/models/sp30k
|
||||
Saving info data/mldoc/de-1/models/sp30k/lstm_nl4-100-e8dp2.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.172059 1.311985 0.280000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.721259 1.180611 0.720000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.495393 1.051538 0.770000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.445929 0.984670 0.830000
|
||||
2 0.430556 0.897431 0.870000
|
||||
3 0.442683 0.800808 0.900000
|
||||
4 0.427033 0.711604 0.880000
|
||||
5 0.411931 0.624835 0.890000
|
||||
6 0.397705 0.560819 0.900000
|
||||
7 0.387848 0.506201 0.900000
|
||||
8 0.380063 0.459507 0.900000
|
||||
Saving models at data/mldoc/de-1/models/sp30k/lstm_nl4-100-e8dp2.m
|
||||
Loss and accuracy using (cls_last): [0.41103342, tensor(0.9105)]
|
||||
```
|
||||
|
||||
#### 2x e8
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/de-1 --base-lm-path data/mldoc/de-1/models/sp30k/lstm_nl4.m --lang=de --name 'nl4-100-2nd-2x' --cuda-id=1 - train 0 --bs 40 --limit=100 --num-cls-epochs=8 --drop-mult-cls=0.2
|
||||
Max vocab: 30000
|
||||
Cache dir: data/mldoc/de-1/models/sp30k
|
||||
Model dir: data/mldoc/de-1/models/sp30k/lstm_nl4-100-2nd-2x.m
|
||||
Loading validation data/mldoc/de-1/de.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Limiting data set to: 100
|
||||
Tokenized data loaded, cls.trn 100, cls.val 100
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', '▁und', '▁in', 'en', "▁&'", 's', '-']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('data/mldoc/de-1/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('data/mldoc/de-1/models/sp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
data/mldoc/de-1/models/sp30k
|
||||
Saving info data/mldoc/de-1/models/sp30k/lstm_nl4-100-2nd-2x.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.122616 1.290291 0.300000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.746805 1.166377 0.730000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.535163 1.058924 0.900000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.437773 1.022764 0.850000
|
||||
2 0.449220 0.949963 0.820000
|
||||
3 0.443732 0.854293 0.860000
|
||||
4 0.432721 0.746918 0.900000
|
||||
5 0.423113 0.718745 0.840000
|
||||
6 0.403492 0.671295 0.820000
|
||||
7 0.399091 0.539798 0.900000
|
||||
8 0.394950 0.508265 0.900000
|
||||
Saving models at data/mldoc/de-1/models/sp30k/lstm_nl4-100-2nd-2x.m
|
||||
Loading validation data/mldoc/de-1/de.dev.csv
|
||||
Loss and accuracy using (cls_last): [0.44688165, tensor(0.9062)]
|
||||
Loss and accuracy using (cls_best): [0.44688165, tensor(0.9062)]
|
||||
```
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/de-1 --base-lm-path data/mldoc/de-1/models/sp30k/lstm_nl4.m --lang=de --name 'nl4-100-2nd-2x' --cuda-id=1 - train 0 --bs 40 --limit=100 --num-cls-epochs=8 --drop-mult-cls=0.2
|
||||
Max vocab: 30000
|
||||
Cache dir: data/mldoc/de-1/models/sp30k
|
||||
Model dir: data/mldoc/de-1/models/sp30k/lstm_nl4-100-2nd-2x.m
|
||||
Loading validation data/mldoc/de-1/de.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Limiting data set to: 100
|
||||
Tokenized data loaded, cls.trn 100, cls.val 100
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', '▁und', '▁in', 'en', "▁&'", 's', '-']
|
||||
Loading last classifier
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.422151 0.797943 0.720000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.357166 0.736997 0.780000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.238496 1.497305 0.660000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.312356 1.522862 0.660000
|
||||
2 0.267801 1.518249 0.660000
|
||||
3 0.244077 1.110030 0.680000
|
||||
4 0.264972 0.798898 0.770000
|
||||
5 0.236816 0.398245 0.860000
|
||||
6 0.251284 0.415783 0.860000
|
||||
7 0.244988 0.417737 0.860000
|
||||
8 0.240362 0.415114 0.860000
|
||||
Saving models at data/mldoc/de-1/models/sp30k/lstm_nl4-100-2nd-2x.m
|
||||
Loss and accuracy using (cls_last): [0.27954015, tensor(0.9125)]
|
||||
Loss and accuracy using (cls_best): [0.27954015, tensor(0.9125)]
|
||||
```
|
||||
### Adding noise
|
||||
#### 40%
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/de-1 --base-lm-path data/mldoc/de-1/models/sp30k/lstm_nl4.m --lang=de --name 'nl4-noise0.4' --cuda-id=1 - train 0 --bs 40 --noise=0.4 --num-cls-epochs=8 --drop-mult-cls=0.2
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4-noise0.4.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/de.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Added noise to 400 examples, only 0.6 have correct labels
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', '▁und', '▁in', 'en', "▁&'", 's', '-']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4-noise0.4.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.053928 0.938391 0.535000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.941778 0.599400 0.836000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.858363 0.675211 0.760000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.768678 0.645293 0.788000
|
||||
2 0.758538 0.636551 0.780000
|
||||
3 0.753799 0.673323 0.708000
|
||||
4 0.731245 0.638630 0.736000
|
||||
5 0.691206 0.659491 0.717000
|
||||
6 0.691426 0.682510 0.696000
|
||||
7 0.672320 0.668610 0.702000
|
||||
8 0.653569 0.669633 0.694000
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4-noise0.4.m
|
||||
Loss and accuracy using (cls_last): [0.62477165, tensor(0.7717)]
|
||||
```
|
||||
#### 15%
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/de-1 --base-lm-path data/mldoc/de-1/models/sp30k/lstm_nl4.m --lang=de --name 'nl4-noise0.15' --cuda-id=1 - train 0 --bs 40 --noise=0.15 --num-cls-epochs=2 --drop-mult-cls=0.2
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4-noise0.15.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/de.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Added noise to 150 examples, only 0.85 have correct labels
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', '▁und', '▁in', 'en', "▁&'", 's', '-']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4-noise0.15.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.836104 0.584330 0.897000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.692108 0.303470 0.930000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.653277 0.330520 0.924000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.541086 0.331944 0.922000
|
||||
2 0.523274 0.335986 0.922000
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/lstm_nl4-noise0.15.m
|
||||
Loss and accuracy using (cls_last): [0.28749043, tensor(0.9355)]
|
||||
```
|
||||
@@ -1,6 +0,0 @@
|
||||
# EN
|
||||
## SP30k LSTM nl 4
|
||||
### LM
|
||||
|
||||
### MLDoc
|
||||
|
||||
@@ -1,269 +0,0 @@
|
||||
# ES
|
||||
|
||||
## SP30k LSTM nl 4
|
||||
### LM
|
||||
````
|
||||
python -m ulmfit lm --dataset-path data/wiki/es-100 --cuda-id=0 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 30000 --lang es --qrnn=False - train 10 --bs=50 --drop_mult=0
|
||||
Running tokenization
|
||||
Wiki text was split to 96224 articles
|
||||
Wiki text was split to 105 articles
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': None, 'pretrained_model': None, 'drop_mult': 0} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.269541 3.451855 0.387471
|
||||
2 3.161740 3.423016 0.386158
|
||||
3 3.187431 3.419638 0.388626
|
||||
4 3.115763 3.357066 0.393877
|
||||
5 2.996527 3.291787 0.402488
|
||||
6 3.021759 3.202183 0.410873
|
||||
7 2.998267 3.104373 0.422624
|
||||
8 2.827225 3.006537 0.436010
|
||||
9 2.784576 2.937735 0.446654
|
||||
10 2.789913 2.918509 0.450055
|
||||
data/wiki/es-100/models/sp30k
|
||||
Saving info data/wiki/es-100/models/sp30k/lstm_nl4.m/info.json
|
||||
````
|
||||
|
||||
### MLDoc
|
||||
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/es-1 --base-lm-path data/wiki/es-100/models/sp30k/lstm_nl4.m --lang=es --name 'nl4' --cuda-id=1 - train 20 --bs 40 --num-cls-epochs=2
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 13013, cls.val 1445
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/es-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/es-100/models/sp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/es-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/es-100/models/sp30k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.805415 2.188974 0.537779
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.429727 1.989691 0.569048
|
||||
2 2.218828 1.794969 0.603721
|
||||
3 2.015097 1.644815 0.629609
|
||||
4 1.877210 1.537773 0.646898
|
||||
5 1.775648 1.450283 0.660861
|
||||
6 1.749334 1.377085 0.672146
|
||||
7 1.601073 1.311101 0.684400
|
||||
8 1.564420 1.251074 0.694900
|
||||
9 1.532728 1.197607 0.704779
|
||||
10 1.391921 1.145408 0.716044
|
||||
11 1.379958 1.093550 0.726937
|
||||
12 1.324111 1.048308 0.735890
|
||||
13 1.344113 1.007926 0.745691
|
||||
14 1.243085 0.969521 0.754591
|
||||
15 1.230809 0.937330 0.762675
|
||||
16 1.162501 0.913408 0.768044
|
||||
17 1.170092 0.894892 0.773239
|
||||
18 1.110860 0.884449 0.775603
|
||||
19 1.115907 0.880448 0.776671
|
||||
20 1.083033 0.878421 0.776931
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.621574 0.391042 0.856000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.411668 0.215625 0.935000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.340519 0.222422 0.935000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.281729 0.192193 0.949000
|
||||
2 0.262074 0.202975 0.945000
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.1749019, tensor(0.9515)]
|
||||
```
|
||||
|
||||
|
||||
|
||||
## ES optimization
|
||||
|
||||
|
||||
|
||||
### Smaler vocab 15k
|
||||
#### LM
|
||||
```
|
||||
python -m ulmfit lm --dataset-path data/wiki-m/es-100 --cuda-id=0 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 \ ✘ 1
|
||||
--lang es --qrnn=False - train 10 --bs=50 --drop_mult=0
|
||||
Max vocab: 15000
|
||||
Cache dir: data/wiki-m/es-100/models/sp15k
|
||||
Model dir: data/wiki-m/es-100/models/sp15k/lstm_nl4.m
|
||||
Tokenized data loaded
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.961702 3.187043 0.403149
|
||||
Better model found at epoch 1 with val_loss value: 3.1870434284210205.
|
||||
2 2.928991 3.170802 0.402026
|
||||
Better model found at epoch 2 with val_loss value: 3.170802354812622.
|
||||
3 2.931906 3.128328 0.407816
|
||||
Better model found at epoch 3 with val_loss value: 3.128328323364258.
|
||||
4 2.869332 3.072160 0.414345
|
||||
Better model found at epoch 4 with val_loss value: 3.072160243988037.
|
||||
5 2.803377 2.997071 0.424847
|
||||
Better model found at epoch 5 with val_loss value: 2.997070550918579.
|
||||
6 2.758087 2.927369 0.432256
|
||||
Better model found at epoch 6 with val_loss value: 2.927368640899658.
|
||||
7 2.657733 2.825029 0.446440
|
||||
Better model found at epoch 7 with val_loss value: 2.8250293731689453.
|
||||
8 2.563273 2.728652 0.459271
|
||||
Better model found at epoch 8 with val_loss value: 2.7286524772644043.
|
||||
9 2.475741 2.654844 0.470864
|
||||
Better model found at epoch 9 with val_loss value: 2.654844045639038.
|
||||
10 2.428898 2.634355 0.474821
|
||||
Better model found at epoch 10 with val_loss value: 2.634355306625366.
|
||||
Total time: 17:53:59
|
||||
data/wiki-m/es-100/models/sp15k
|
||||
Saving info data/wiki-m/es-100/models/sp15k/lstm_nl4.m/info.json
|
||||
```
|
||||
|
||||
#### MLDoc
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/es-1 --base-lm-path data/wiki-m/es-100/models/sp15k/lstm_nl4.m --lang=es --name 'nl4' --cuda-id=0 - train 20 --bs 20 --num-cls-epochs=8
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/lstm_nl4.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 13013, cls.val 1445
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki-m/es-100/models/sp15k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki-m/es-100/models/sp15k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.355042 1.850417 0.589128
|
||||
Better model found at epoch 1 with val_loss value: 1.850416898727417.
|
||||
Total time: 03:25
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.087364 1.677666 0.619909
|
||||
Better model found at epoch 1 with val_loss value: 1.6776657104492188.
|
||||
2 1.880730 1.522996 0.648134
|
||||
Better model found at epoch 2 with val_loss value: 1.5229955911636353.
|
||||
3 1.767530 1.403644 0.668117
|
||||
Better model found at epoch 3 with val_loss value: 1.4036436080932617.
|
||||
4 1.659950 1.309353 0.684900
|
||||
Better model found at epoch 4 with val_loss value: 1.3093526363372803.
|
||||
5 1.546585 1.232220 0.699358
|
||||
Better model found at epoch 5 with val_loss value: 1.2322196960449219.
|
||||
6 1.592862 1.161846 0.713034
|
||||
Better model found at epoch 6 with val_loss value: 1.1618456840515137.
|
||||
7 1.444965 1.098108 0.726811
|
||||
Better model found at epoch 7 with val_loss value: 1.0981075763702393.
|
||||
8 1.340874 1.029193 0.741337
|
||||
Better model found at epoch 8 with val_loss value: 1.0291931629180908.
|
||||
9 1.351407 0.974317 0.753408
|
||||
Better model found at epoch 9 with val_loss value: 0.9743167757987976.
|
||||
10 1.231713 0.915328 0.767088
|
||||
Better model found at epoch 10 with val_loss value: 0.9153280854225159.
|
||||
11 1.151926 0.852391 0.782414
|
||||
Better model found at epoch 11 with val_loss value: 0.852391242980957.
|
||||
12 1.163565 0.794699 0.797228
|
||||
Better model found at epoch 12 with val_loss value: 0.7946987152099609.
|
||||
13 1.054929 0.743652 0.810518
|
||||
Better model found at epoch 13 with val_loss value: 0.74365234375.
|
||||
14 0.974651 0.695024 0.823344
|
||||
Better model found at epoch 14 with val_loss value: 0.6950243711471558.
|
||||
15 0.869718 0.651691 0.834510
|
||||
Better model found at epoch 15 with val_loss value: 0.6516908407211304.
|
||||
16 0.889763 0.615112 0.844947
|
||||
Better model found at epoch 16 with val_loss value: 0.6151121258735657.
|
||||
17 0.843503 0.590130 0.851694
|
||||
Better model found at epoch 17 with val_loss value: 0.5901297926902771.
|
||||
18 0.752870 0.575217 0.855496
|
||||
Better model found at epoch 18 with val_loss value: 0.5752172470092773.
|
||||
19 0.807087 0.567187 0.857605
|
||||
Better model found at epoch 19 with val_loss value: 0.5671872496604919.
|
||||
20 0.784531 0.566082 0.857827
|
||||
Better model found at epoch 20 with val_loss value: 0.5660821199417114.
|
||||
Total time: 1:26:48
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/lstm_nl4.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.616963 0.275447 0.923000
|
||||
Better model found at epoch 1 with val_loss value: 0.27544698119163513.
|
||||
Total time: 00:24
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.425890 0.201535 0.932000
|
||||
Better model found at epoch 1 with val_loss value: 0.2015346735715866.
|
||||
Total time: 00:27
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.265563 0.186434 0.951000
|
||||
Better model found at epoch 1 with val_loss value: 0.18643426895141602.
|
||||
Total time: 00:32
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.162097 0.180026 0.955000
|
||||
Better model found at epoch 1 with val_loss value: 0.1800260841846466.
|
||||
2 0.161748 0.187014 0.957000
|
||||
3 0.142789 0.166486 0.961000
|
||||
Better model found at epoch 3 with val_loss value: 0.1664857715368271.
|
||||
4 0.105920 0.173207 0.963000
|
||||
5 0.078164 0.184849 0.962000
|
||||
6 0.070540 0.189451 0.962000
|
||||
7 0.051490 0.208019 0.959000
|
||||
8 0.047614 0.193699 0.962000
|
||||
Total time: 05:20
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.1623594, tensor(0.9538)]
|
||||
0.16235940158367157
|
||||
0.9537500143051147
|
||||
```
|
||||
|
||||
### Larger dropout - no luck
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/es-1 --base-lm-path data/wiki-m/es-100/models/sp30k/lstm_nl4.m --lang=es --name 'nl4-drop' --cuda-id=0 - train 0 --bs 20 --num-cls-epochs=8 --drop-mul-lm=0.5 --drop-mul-cls=0.8
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4-drop.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 13013, cls.val 1445
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.310594 0.903076 0.724000
|
||||
Better model found at epoch 1 with val_loss value: 0.9030755758285522.
|
||||
Total time: 00:22
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.209348 0.714140 0.756000
|
||||
Better model found at epoch 1 with val_loss value: 0.714139997959137.
|
||||
Total time: 00:23
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.122636 0.627526 0.797000
|
||||
Better model found at epoch 1 with val_loss value: 0.6275263428688049.
|
||||
Total time: 00:29
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.102433 0.593338 0.801000
|
||||
Better model found at epoch 1 with val_loss value: 0.5933384895324707.
|
||||
2 1.096480 0.543266 0.818000
|
||||
Better model found at epoch 2 with val_loss value: 0.5432664155960083.
|
||||
3 1.082919 0.501089 0.837000
|
||||
Better model found at epoch 3 with val_loss value: 0.5010889172554016.
|
||||
4 1.069694 0.518807 0.812000
|
||||
5 1.040208 0.508399 0.825000
|
||||
6 1.032841 0.512187 0.838000
|
||||
7 1.031225 0.504557 0.825000
|
||||
8 1.016486 0.502335 0.837000
|
||||
Total time: 04:56
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/lstm_nl4-drop.m
|
||||
Loss and accuracy using (cls_best): [0.52473265, tensor(0.8160)]
|
||||
0.5247326493263245
|
||||
0.8159999847412109
|
||||
```
|
||||
@@ -1,76 +0,0 @@
|
||||
python -m ulmfit eval --glob="mldoc/*-1/models/sp15k/qrnn_nl4.m" --name nl4-8e-single --num-cls-epochs=8 --bs=18 --single=True
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-8e-single.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
Your compiler (c++) may be ABI-incompatible with PyTorch!
|
||||
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
|
||||
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
|
||||
|
||||
See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
|
||||
for instructions on how to install GCC 4.9 or higher.
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-8e-single.m/info.json
|
||||
Starting classifier training
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.676591 0.205210 0.947000
|
||||
2 0.402020 0.461279 0.912000
|
||||
3 0.287975 0.496294 0.921000
|
||||
4 0.258515 0.243489 0.954000
|
||||
5 0.219352 0.274136 0.949000
|
||||
6 0.149339 0.352294 0.956000
|
||||
7 0.092821 0.378696 0.962000
|
||||
8 0.055485 0.367379 0.963000
|
||||
9 0.042695 0.367151 0.964000
|
||||
10 0.034858 0.386749 0.961000
|
||||
11 0.021245 0.392899 0.963000
|
||||
Total time: 02:38
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_nl4-8e-single.m
|
||||
Traceback (most recent call last):
|
||||
File "/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/runpy.py", line 193, in _run_module_as_main
|
||||
"__main__", mod_spec)
|
||||
File "/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/runpy.py", line 85, in _run_code
|
||||
exec(code, run_globals)
|
||||
File "/home/pczapla/workspace/ulmfit-multilingual/ulmfit/__main__.py", line 73, in <module>
|
||||
fire.Fire(ULMFiT())
|
||||
File "/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/fire/core.py", line 127, in Fire
|
||||
component_trace = _Fire(component, args, context, name)
|
||||
File "/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/fire/core.py", line 366, in _Fire
|
||||
component, remaining_args)
|
||||
File "/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/fire/core.py", line 542, in _CallCallable
|
||||
result = fn(*varargs, **kwargs)
|
||||
File "/home/pczapla/workspace/ulmfit-multilingual/ulmfit/__main__.py", line 55, in eval
|
||||
results[key] = params.train_cls(num_lm_epochs=num_lm_epochs, **trn_params)[1]
|
||||
File "/home/pczapla/workspace/ulmfit-multilingual/ulmfit/train_clas.py", line 76, in train_cls
|
||||
return self.validate_cls('cls_best', bs=bs, data_tst=data_tst, learn=learn)
|
||||
File "/home/pczapla/workspace/ulmfit-multilingual/ulmfit/train_clas.py", line 84, in validate_cls
|
||||
learn.load(save_name)
|
||||
File "/home/pczapla/workspace/_oss/fastai/fastai/fastai/basic_train.py", line 243, in load
|
||||
if purge: self.purge(clear_opt=ifnone(with_opt, False))
|
||||
File "/home/pczapla/workspace/_oss/fastai/fastai/fastai/basic_train.py", line 293, in purge
|
||||
self.opt = OptimWrapper.load_with_state_and_layer_group(state['opt'], self.layer_groups)
|
||||
File "/home/pczapla/workspace/_oss/fastai/fastai/fastai/callback.py", line 130, in load_with_state_and_layer_group
|
||||
res.load_state_dict(state['opt_state'])
|
||||
File "/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/optim/optimizer.py", line 108, in load_state_dict
|
||||
raise ValueError("loaded state dict contains a parameter group "
|
||||
ValueError: loaded state dict contains a parameter group that doesn't match the size of optimizer's group
|
||||
@@ -1,180 +0,0 @@
|
||||
# FR
|
||||
|
||||
## SP15k QRNN nl 4
|
||||
```
|
||||
python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0
|
||||
Max vocab: 15000
|
||||
Cache dir: data/wiki/fr-100/models/sp15k
|
||||
Model dir: data/wiki/fr-100/models/sp15k/qrnn_nl4.m
|
||||
Wiki text was split to 174227 articles
|
||||
Wiki text was split to 491 articles
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 174227, val: 491
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le',
|
||||
'▁et', '▁l', '▁à']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.881558 2.790402 0.465847
|
||||
2 2.824942 2.732005 0.471660
|
||||
3 2.758845 2.672040 0.478273
|
||||
4 2.715069 2.602380 0.489159
|
||||
5 2.677029 2.553575 0.494752
|
||||
6 2.602514 2.476142 0.507337
|
||||
7 2.564386 2.388670 0.518902
|
||||
8 2.470835 2.304033 0.532000
|
||||
9 2.366890 2.243269 0.542781
|
||||
10 2.390439 2.223538 0.546622
|
||||
Total time: 9:09:26
|
||||
data/wiki/fr-100/models/sp15k
|
||||
Saving info data/wiki/fr-100/models/sp15k/qrnn_nl4.m/info.json
|
||||
|
||||
```
|
||||
|
||||
## SP30k LSTM nl 4
|
||||
### LM
|
||||
```
|
||||
python -m ulmfit lm --dataset-path data/wiki/fr-100 --cuda-id=1 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 30000 \ ✘ 130
|
||||
--lang fr --qrnn=False - train 10 --bs=50 --drop_mult=0
|
||||
Max vocab: 30000
|
||||
Cache dir: data/wiki/fr-100/models/sp30k
|
||||
Model dir: data/wiki/fr-100/models/sp30k/lstm_nl4.m
|
||||
Running tokenization
|
||||
Wiki text was split to 113288 articles
|
||||
Wiki text was split to 88 articles
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁.', "'", 's', '▁la', '▁le', '▁et', '▁l', '▁à']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': None, 'pretrained_model': None, 'drop_mult': 0} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.120035 3.449028 0.383274
|
||||
2 3.070353 3.431372 0.382520
|
||||
3 3.092097 3.406521 0.384604
|
||||
4 3.035016 3.356502 0.391155
|
||||
5 2.936505 3.297572 0.396365
|
||||
6 2.926953 3.192980 0.407546
|
||||
7 2.841542 3.115280 0.417741
|
||||
8 2.805254 3.008793 0.429512
|
||||
9 2.681713 2.944207 0.439959
|
||||
10 2.644920 2.923765 0.442415
|
||||
data/wiki/fr-100/models/sp30k
|
||||
Saving info data/wiki/fr-100/models/sp30k/lstm_nl4.m/info.json
|
||||
```
|
||||
|
||||
### MLDocs
|
||||
#### First run
|
||||
MultiCCA 92.05, ulmfit 93.90
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/fr-1 --base-lm-path data/wiki/fr-100/models/sp30k/lstm_nl4.m --lang=fr --name 'nl4' --cuda-id=1 - train 20 --bs 40
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k/lstm_nl4.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/fr.dev.csv
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 13500, cls.val 1500
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁.', "'", 's', '▁la', '▁le', '▁et', '▁l', '▁à']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp30k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.072937 2.621444 0.468314
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.737065 2.485359 0.486546
|
||||
2 2.625827 2.353188 0.507669
|
||||
3 2.408049 2.224600 0.527609
|
||||
4 2.332804 2.113603 0.544255
|
||||
5 2.242967 2.016229 0.560002
|
||||
6 2.162170 1.925214 0.574050
|
||||
7 2.094163 1.843778 0.587944
|
||||
8 2.011285 1.773228 0.599802
|
||||
9 1.931492 1.708201 0.611245
|
||||
10 1.883735 1.643842 0.623145
|
||||
11 1.793858 1.583394 0.635366
|
||||
12 1.759305 1.526640 0.646132
|
||||
13 1.741412 1.474198 0.657485
|
||||
14 1.675670 1.430597 0.666407
|
||||
15 1.624235 1.390453 0.674829
|
||||
16 1.588415 1.359892 0.681364
|
||||
17 1.594124 1.336594 0.686985
|
||||
18 1.567758 1.322139 0.689745
|
||||
19 1.536472 1.315883 0.690974
|
||||
20 1.530144 1.314872 0.691084
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k/lstm_nl4.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.528480 0.428756 0.853000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.365323 0.224117 0.928000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.300881 0.199623 0.936000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.217855 0.198016 0.937000
|
||||
2 0.206357 0.212208 0.938000
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.18914989, tensor(0.9390)]
|
||||
```
|
||||
|
||||
#### Second run
|
||||
MultiCCA 92.05, ulmfit 93.67
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/fr-1 --base-lm-path data/wiki/fr-100/models/sp30k/lstm_nl4.m --lang=fr --name 'nl4-2nd' --cuda-id=1 - train 20 --bs 40 --num-cls-epochs=8
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k/lstm_nl4-2nd.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/fr.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Tokenized data loaded, cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁.', "'", 's', '▁la', '▁le', '▁et', '▁l', '▁à']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/fr-100/models/sp30k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.080331 2.625329 0.467306
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.750554 2.485271 0.486776
|
||||
2 2.578659 2.353940 0.507637
|
||||
3 2.422981 2.224983 0.527749
|
||||
4 2.342781 2.113364 0.545006
|
||||
5 2.254575 2.007775 0.560709
|
||||
6 2.124016 1.920536 0.575680
|
||||
7 2.068470 1.847463 0.586699
|
||||
8 2.013289 1.775580 0.599840
|
||||
9 1.929649 1.705369 0.612201
|
||||
10 1.916013 1.646228 0.623175
|
||||
11 1.825515 1.586714 0.634298
|
||||
12 1.795780 1.529771 0.645840
|
||||
13 1.725532 1.476651 0.656197
|
||||
14 1.673942 1.429790 0.666030
|
||||
15 1.639384 1.392116 0.674128
|
||||
16 1.605681 1.359316 0.681356
|
||||
17 1.560283 1.337794 0.686116
|
||||
18 1.543926 1.323153 0.689276
|
||||
19 1.531950 1.318164 0.690415
|
||||
20 1.494068 1.316459 0.690586
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k/lstm_nl4-2nd.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.537720 0.395818 0.886000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.332603 0.232112 0.930000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.267323 0.230307 0.927000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.216402 0.226042 0.930000
|
||||
2 0.231040 0.232696 0.936000
|
||||
3 0.182048 0.217882 0.934000
|
||||
4 0.170389 0.212531 0.937000
|
||||
5 0.148332 0.214293 0.937000
|
||||
6 0.124968 0.210322 0.936000
|
||||
7 0.117591 0.234207 0.936000
|
||||
8 0.109146 0.218597 0.938000
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp30k/lstm_nl4-2nd.m
|
||||
Loss and accuracy using (cls_best): [0.21502711, tensor(0.9367)]
|
||||
```
|
||||
@@ -1,106 +0,0 @@
|
||||
# FR
|
||||
## SP15k QRNN NL4
|
||||
### LM
|
||||
|
||||
```
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
LANG=it
|
||||
python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.171145 3.516659 0.359233
|
||||
2 3.045057 3.472802 0.359628
|
||||
3 3.023009 3.401181 0.367101
|
||||
4 2.985105 3.351916 0.372709
|
||||
5 2.858441 3.280903 0.380848
|
||||
6 2.862504 3.210976 0.390263
|
||||
7 2.758775 3.122354 0.402106
|
||||
8 2.683234 3.035321 0.413798
|
||||
9 2.593757 2.964551 0.424886
|
||||
10 2.535500 2.947672 0.427958
|
||||
Total time: 11:30:03
|
||||
data/wiki/it-100/models/sp15k
|
||||
Saving info data/wiki/it-100/models/sp15k/qrnn_nl4.m/info.json
|
||||
```
|
||||
## xx
|
||||
|
||||
## SP30k LSTM nl 4
|
||||
### LM
|
||||
```
|
||||
python -m ulmfit lm --dataset-path data/wiki/it-100 --lang=it --bidir=False --qrnn=False --max-vocab 30000 --nl 4 --tokenizer=sp --name 'nl4bs100' - train 10 --bs 100 --dropout-mult=0
|
||||
Wiki text was split to 164583 articles
|
||||
Wiki text was split to 98 articles
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁di', "▁&'", "'", '▁e', '▁il', '▁la', 'e', '▁in']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': None, 'pretrained_model': None, 'drop_mult': 0.0} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.306743 3.717148 0.353641
|
||||
2 3.126413 3.606443 0.360839
|
||||
3 3.062586 3.545493 0.365721
|
||||
4 3.055600 3.474823 0.373451
|
||||
5 2.927211 3.406635 0.380311
|
||||
6 2.924096 3.321370 0.389487
|
||||
7 2.779998 3.233350 0.399968
|
||||
8 2.722100 3.147745 0.410365
|
||||
9 2.615910 3.087420 0.419097
|
||||
10 2.565747 3.075364 0.420906
|
||||
data/wiki/it-100/models/sp30k
|
||||
Saving info data/wiki/it-100/models/sp30k/lstm_nl4bs100.m/info.json
|
||||
```
|
||||
|
||||
|
||||
### MLDoc
|
||||
MultiCCA: 85.55%, ULMFiT 88.42%
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/it-1 --base-lm-path data/wiki/it-100/models/sp30k/lstm_nl4bs100.m --lang=it --name 'nl4bs100' --cuda-id=1 - train 20 --bs 40 --num-cls-epochs=2
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k/lstm_nl4bs100.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/it.dev.csv
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 13500, cls.val 1500
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁di', "▁&'", "'", '▁e', '▁il', '▁la', 'e', '▁in']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/it-100/models/sp30k/lstm_nl4bs100.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/it-100/models/sp30k/lstm_nl4bs100.m/../itos')], 'pretrained_model': None, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/it-100/models/sp30k/lstm_nl4bs100.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/it-100/models/sp30k/lstm_nl4bs100.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.826957 2.518636 0.492175
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.606302 2.397623 0.509596
|
||||
2 2.470586 2.260363 0.531301
|
||||
3 2.334087 2.113640 0.554089
|
||||
4 2.176830 1.988222 0.572687
|
||||
5 2.123101 1.869944 0.591537
|
||||
6 2.011187 1.770606 0.606682
|
||||
7 1.934953 1.676852 0.622504
|
||||
8 1.889363 1.592609 0.637525
|
||||
9 1.774590 1.517665 0.652233
|
||||
10 1.725905 1.435543 0.666759
|
||||
11 1.670903 1.365167 0.681168
|
||||
12 1.610080 1.302561 0.694462
|
||||
13 1.522876 1.242124 0.708201
|
||||
14 1.478528 1.193259 0.718366
|
||||
15 1.423993 1.150854 0.728324
|
||||
16 1.389901 1.115550 0.735836
|
||||
17 1.365959 1.094267 0.740730
|
||||
18 1.347579 1.079465 0.744019
|
||||
19 1.321906 1.074090 0.745281
|
||||
20 1.332676 1.073143 0.745453
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k/lstm_nl4bs100.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.632703 0.463210 0.831000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.527650 0.390041 0.858000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.436223 0.326409 0.871000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.361738 0.321380 0.875000
|
||||
2 0.340658 0.315946 0.877000
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp30k/lstm_nl4bs100.m
|
||||
Loss and accuracy using (cls_best): [0.32998973, tensor(0.8842)]
|
||||
```
|
||||
@@ -1,456 +0,0 @@
|
||||
# JA
|
||||
## SP30k LSTM nl 4
|
||||
### LM
|
||||
```
|
||||
python -m ulmfit lm --dataset-path data/wiki/ja-100 --cuda-id=0 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 30000 \
|
||||
--lang ja --qrnn=False - train 10 --bs=50 --drop_mult=0
|
||||
Max vocab: 30000
|
||||
Cache dir: data/wiki/ja-100/models/sp30k
|
||||
Model dir: data/wiki/ja-100/models/sp30k/lstm_nl4.m
|
||||
Running tokenization
|
||||
Wiki text was split to 98375 articles
|
||||
Wiki text was split to 138 articles
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁の', '▁。', '▁に', '▁を', '▁は', '▁年', '▁が', '▁)', '▁(']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': None, 'pretrained_model': None, 'drop_mult': 0} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.211025 3.328014 0.396197
|
||||
2 3.119410 3.286294 0.395946
|
||||
3 3.042064 3.247161 0.403915
|
||||
4 3.023840 3.161323 0.413816
|
||||
5 2.944752 3.102044 0.423163
|
||||
6 2.907167 3.015610 0.434095
|
||||
7 2.796073 2.927566 0.447088
|
||||
8 2.715568 2.828766 0.461556
|
||||
9 2.717255 2.747889 0.473289
|
||||
10 2.619846 2.731164 0.477403
|
||||
data/wiki/ja-100/models/sp30k
|
||||
Saving info data/wiki/ja-100/models/sp30k/lstm_nl4.m/info.json
|
||||
```
|
||||
|
||||
### MLDoc
|
||||
|
||||
#### CLS 1
|
||||
MultiCCA 85.35%, ULMFiT 89.20%
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/ja-1 --base-lm-path data/wiki/ja-100/models/sp30k/lstm_nl4.m --lang=ja --name 'nl4' --cuda-id=1 - train 20 --bs 40 --num-cls-epochs=8
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 13500, cls.val 1500
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁の', '▁。', '▁に', '▁を', '▁は', '▁年', '▁が', '▁)', '▁(']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp30k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.828645 2.386208 0.518716
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.462274 2.191761 0.549783
|
||||
2 2.229925 1.982564 0.586238
|
||||
3 2.043816 1.805435 0.616238
|
||||
4 1.885779 1.674736 0.637964
|
||||
5 1.773445 1.575366 0.653925
|
||||
6 1.713029 1.490263 0.667570
|
||||
7 1.660558 1.419641 0.680072
|
||||
8 1.579792 1.357093 0.690826
|
||||
9 1.459628 1.298609 0.701452
|
||||
10 1.433604 1.251296 0.710232
|
||||
11 1.439143 1.202794 0.719104
|
||||
12 1.399083 1.158469 0.728430
|
||||
13 1.310390 1.120877 0.736382
|
||||
14 1.322389 1.085479 0.744013
|
||||
15 1.272924 1.056051 0.750401
|
||||
16 1.235312 1.034233 0.755225
|
||||
17 1.227864 1.016682 0.759288
|
||||
18 1.209589 1.007038 0.761234
|
||||
19 1.173158 1.001694 0.762281
|
||||
20 1.189994 1.000854 0.762526
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.745803 0.554439 0.819000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.620647 0.392026 0.856000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.489173 0.369560 0.869000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.406491 0.365988 0.872000
|
||||
2 0.392645 0.351823 0.876000
|
||||
3 0.386403 0.331737 0.880000
|
||||
4 0.361338 0.333245 0.882000
|
||||
5 0.319456 0.347253 0.879000
|
||||
6 0.295419 0.350348 0.885000
|
||||
7 0.286144 0.348592 0.879000
|
||||
8 0.278896 0.358145 0.877000
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.29789856, tensor(0.8920)]
|
||||
|
||||
|
||||
$ mv /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4.m /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4x8.m
|
||||
|
||||
$ python -m ulmfit eval --glob="mldoc/ja-1/models/sp30k/lstm_nl4x8.m" --name nl4 --cuda-id=0
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Tokenized data loaded, cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁の', '▁。', '▁に', '▁を', '▁は', '▁年', '▁が', '▁)', '▁(']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4x8.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4x8.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.757615 0.562652 0.825000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.609298 0.382412 0.870000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.544682 0.379602 0.871000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.435453 0.360421 0.885000
|
||||
2 0.426099 0.350480 0.885000
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.32738593, tensor(0.8905)]
|
||||
OrderedDict([('data/mldoc/ja-1/models/sp30k/lstm_nl4.m', 0.890500009059906)])
|
||||
|
||||
python -m ulmfit eval --glob="mldoc/ja-1/models/sp30k/lstm_nl4x8.m" --name nl4x2 --cuda-id=0
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4x2.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Tokenized data loaded, cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁の', '▁。', '▁に', '▁を', '▁は', '▁年', '▁が', '▁)', '▁(']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4x8.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4x8.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4x2.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.778722 0.690746 0.805000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.574789 0.386483 0.862000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.518843 0.361983 0.869000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.435260 0.350808 0.869000
|
||||
2 0.386701 0.352221 0.875000
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4x2.m
|
||||
Loss and accuracy using (cls_best): [0.31783763, tensor(0.8892)]
|
||||
OrderedDict([('data/mldoc/ja-1/models/sp30k/lstm_nl4x2.m', 0.8892499804496765)])
|
||||
|
||||
```
|
||||
|
||||
### JA on 100 elements
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/ja-1 --base-lm-path data/wiki/ja-100/models/sp30k/lstm_nl4.m --lang=ja --name 'nl4-100' --cuda-id=1 - train 20 --bs 40 --num-cls-epochs=8 --limit=100
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4-100.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Limiting data set to: 100
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 100, cls.val 100
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁の', '▁。', '▁に', '▁を', '▁は', '▁年', '▁が', '▁)', '▁(']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp30k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.837937 2.387255 0.518590
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.466900 2.193583 0.549492
|
||||
2 2.232762 1.983981 0.586658
|
||||
3 2.026505 1.810167 0.615649
|
||||
4 1.918111 1.679784 0.636613
|
||||
5 1.748909 1.577095 0.653108
|
||||
6 1.708709 1.491436 0.667657
|
||||
7 1.640415 1.420449 0.679619
|
||||
8 1.577434 1.359511 0.690194
|
||||
9 1.551961 1.302819 0.700306
|
||||
10 1.475623 1.252393 0.710039
|
||||
11 1.435565 1.208159 0.718740
|
||||
12 1.354910 1.161781 0.727927
|
||||
13 1.351157 1.123244 0.736009
|
||||
14 1.299070 1.086383 0.743896
|
||||
15 1.258739 1.055745 0.750383
|
||||
16 1.210775 1.035209 0.754965
|
||||
17 1.228421 1.018373 0.758963
|
||||
18 1.179444 1.007714 0.761158
|
||||
19 1.197443 1.003041 0.762068
|
||||
20 1.163223 1.001939 0.762211
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4-100.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.269222 1.360420 0.340000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.969350 1.314497 0.400000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.832396 1.263416 0.550000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.780991 1.225439 0.600000
|
||||
2 0.765755 1.183010 0.600000
|
||||
3 0.749420 1.139053 0.600000
|
||||
4 0.731800 1.093319 0.610000
|
||||
5 0.711152 1.054695 0.610000
|
||||
6 0.694611 1.029465 0.580000
|
||||
7 0.680276 1.004366 0.580000
|
||||
8 0.668421 0.984848 0.590000
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4-100.m
|
||||
Loss and accuracy using (cls_best): [0.81621724, tensor(0.7437)]
|
||||
```
|
||||
|
||||
|
||||
### Japanese fixed sentence piece
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/ja-1 --base-lm-path data/mldoc/ja-1/models/bsp30k/lstm_nl4.m --lang=ja --name 'nl4' --cuda-id=0 - train 1 --bs 40 --num-cls-epochs=2
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Tokenized data loaded, cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁の', '▁。', '▁に', '▁を', '▁は', '▁年', '▁が', '▁)', '▁(']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/bsp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/bsp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/bsp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/bsp30k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.222162 0.986234 0.765830
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.237291 0.983976 0.766659
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.759230 0.619306 0.826000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.599281 0.423162 0.841000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.485808 0.360609 0.869000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.415960 0.390202 0.872000
|
||||
2 0.371651 0.365374 0.876000
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.330675, tensor(0.8873)]
|
||||
0.33067500591278076
|
||||
0.8872500061988831
|
||||
(fastaiv1) pczapla@galatea ~/w/ulmfit-multilingual ❯❯❯ python -m ulmfit eval --glob="mldoc/ja-1/models/sp30k/lstm_nl4.m" --name nl4-2nd --cuda-id=0
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4-2nd.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Tokenized data loaded, cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁の', '▁。', '▁に', '▁を', '▁は', '▁年', '▁が', '▁)', '▁(']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4-2nd.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.844110 0.700549 0.743000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.610796 0.400912 0.853000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.449974 0.358793 0.870000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.407609 0.397678 0.868000
|
||||
2 0.367383 0.373168 0.869000
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_nl4-2nd.m
|
||||
Loss and accuracy using (cls_best): [0.33044776, tensor(0.8863)]
|
||||
OrderedDict([('data/mldoc/ja-1/models/sp30k/lstm_nl4-2nd.m',
|
||||
0.8862500190734863)])
|
||||
```
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/ja-1 --base-lm-path data/mldoc/ja-1/models/bsp30k/lstm_nl4.m --lang=ja --name '2nd-nl4' --cuda-id=0 - train 1 --bs 40 --num-cls-epochs=2
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_2nd-nl4.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Tokenized data loaded, cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁の', '▁。', '▁に', '▁を', '▁は', '▁年', '▁が', '▁)', '▁(']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/bsp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/bsp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/bsp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/bsp30k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.208774 0.984775 0.766183
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.198147 0.984786 0.766730
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_2nd-nl4.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.735084 0.613895 0.803000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.550159 0.406097 0.867000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.468788 0.404081 0.862000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.395969 0.380797 0.870000
|
||||
2 0.349470 0.386497 0.866000
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_2nd-nl4.m
|
||||
Loss and accuracy using (cls_best): [0.32550755, tensor(0.8857)]
|
||||
0.3255075514316559
|
||||
0.8857499957084656
|
||||
|
||||
```
|
||||
```bash
|
||||
python -m ulmfit cls --dataset-path data/mldoc/ja-1 --base-lm-path data/mldoc/ja-1/models/bsp30k/lstm_nl4.m --lang=ja --name '3nd-nl4' --cuda-id=0 - train 0 --bs 40 --num-cls-epochs=2
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_3nd-nl4.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Tokenized data loaded, cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁の', '▁。', '▁に', '▁を', '▁は', '▁年', '▁が', '▁)', '▁(']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/bsp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/bsp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_3nd-nl4.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.777661 0.617013 0.786000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.603897 0.388985 0.867000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.510845 0.374942 0.874000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.468642 0.379503 0.872000
|
||||
2 0.430415 0.365797 0.880000
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp30k/lstm_3nd-nl4.m
|
||||
Loss and accuracy using (cls_best): [0.33084384, tensor(0.8882)]
|
||||
0.33084383606910706
|
||||
0.8882499933242798
|
||||
|
||||
python -m ulmfit cls --dataset-path data/mldoc/ja-1 --base-lm-path data/mldoc/ja-1/models/bsp30k/lstm_nl4.m --lang=ja --tokenizer=bsp --name '3nd-nl4' --cuda-id=0 - train 0 --bs 40 --num-cls-epochs=2
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/bsp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/bsp30k/lstm_3nd-nl4.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Tokenized data loaded, cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁の', '▁。', '▁に', '▁を', '▁は', '▁年', '▁が', '▁)', '▁(']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/bsp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/bsp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/bsp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/bsp30k/lstm_3nd-nl4.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.824216 0.604706 0.825000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.606317 0.409647 0.854000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.500782 0.381826 0.862000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.403516 0.366863 0.866000
|
||||
2 0.394599 0.357580 0.874000
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/bsp30k/lstm_3nd-nl4.m
|
||||
Loss and accuracy using (cls_best): [0.32903105, tensor(0.8848)]
|
||||
0.3290310502052307
|
||||
0.8847500085830688
|
||||
|
||||
python -m ulmfit cls --dataset-path data/mldoc/ja-1 --base-lm-path data/mldoc/ja-1/models/bsp30k/lstm_nl4.m --lang=ja --tokenizer=bsp --name '3nd-nl4' --cuda-id=0 - train 0 --bs 40 --num-cls-epochs=2
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/bsp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/bsp30k/lstm_3nd-nl4.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Tokenized data loaded, cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁の', '▁。', '▁に', '▁を', '▁は', '▁年', '▁が', '▁)', '▁(']
|
||||
Loading last classifier
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.460803 0.451998 0.855000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.460069 0.421900 0.867000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.361791 0.447982 0.859000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.301233 0.404477 0.868000
|
||||
2 0.269350 0.406427 0.870000
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/bsp30k/lstm_3nd-nl4.m
|
||||
Loss and accuracy using (cls_best): [0.34159982, tensor(0.8925)]
|
||||
0.34159982204437256
|
||||
0.8924999833106995
|
||||
|
||||
```
|
||||
## SP60k
|
||||
```
|
||||
python -m ulmfit lm --dataset-path data/wiki/ja-100 --cuda-id=1 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 60000 \
|
||||
--lang ja --qrnn=False - train 10 --bs=50 --drop_mult=0
|
||||
|
||||
Running tokenization
|
||||
Wiki text was split to 98375 articles
|
||||
Wiki text was split to 138 articles
|
||||
Size of vocabulary: 60000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁の', '▁。', '▁に', '▁を', '▁は', '▁年', '▁が', '▁)', '▁(']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': None, 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.557822 3.682454 0.366108
|
||||
2 3.377493 3.614226 0.369889
|
||||
3 3.391634 3.562171 0.377114
|
||||
4 3.328160 3.497388 0.385236
|
||||
5 3.290285 3.424971 0.394655
|
||||
6 3.159867 3.337317 0.407095
|
||||
7 3.139091 3.250999 0.417750
|
||||
8 3.103923 3.153146 0.433443
|
||||
9 2.979789 3.092179 0.443405
|
||||
10 2.984099 3.077171 0.446887
|
||||
data/wiki/ja-100/models/sp60k
|
||||
Saving info data/wiki/ja-100/models/sp60k/lstm_nl4.m/info.json
|
||||
```
|
||||
## MLDoc
|
||||
````bash
|
||||
python -m ulmfit cls --dataset-path data/mldoc/ja-1 --base-lm-path data/wiki/ja-100/models/sp60k/lstm_nl4.m --lang=ja --name 'nl4' --cuda-id=1 - train 20 --bs 40 --num-cls-epochs=2
|
||||
Max vocab: 60000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp60k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp60k/lstm_nl4.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 60000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁の', '▁。', '▁に', '▁を', '▁は', '▁年', '▁が', '▁)', '▁(']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp60k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp60k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp60k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ja-100/models/sp60k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.019972 2.548868 0.503754
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.643698 2.363415 0.532341
|
||||
2 2.403588 2.149524 0.567359
|
||||
3 2.218298 1.969651 0.597484
|
||||
4 2.059648 1.829897 0.619758
|
||||
5 1.941803 1.722339 0.636215
|
||||
6 1.862969 1.630191 0.650293
|
||||
7 1.796515 1.551929 0.663782
|
||||
8 1.727768 1.481659 0.675489
|
||||
9 1.667709 1.417764 0.687287
|
||||
10 1.606343 1.357994 0.697264
|
||||
11 1.553344 1.303901 0.707811
|
||||
12 1.539182 1.251784 0.718038
|
||||
Traceback (most recent call last):
|
||||
````
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,39 +0,0 @@
|
||||
# Multifit Best results after label smoothing
|
||||
|
||||
| | de-1 | en-1 | es-1 | fr-1 | it-1 | ja-1 | ru-1 | zh-1|
|
||||
|-----|-------|-------|-------|-------|-------|-------|-------|------|
|
||||
|best | 95.90 | 95.17 | 96.07 | 94.75 | 90.25 | 90.03 | 87.65 | 92.52|
|
||||
|max | 95.90 | 95.55 | 96.07 | 94.75 | 90.38 | 90.03 | 87.65 | 92.52|
|
||||
|avg | 95.77 | 95.27 | 95.92 | 94.75 | 90.24 | 89.89 | 87.28 | 92.31|
|
||||
|
||||
|
||||
## Log
|
||||
```
|
||||
python -m ulmfit eval --glob="mldoc/ru-1/models/sp15k/qrnn_nl4.m" --lr_sched=1cycle --bs=18 --num-cls-epochs=8 --name "nl4_tls4" --label-smoothing-eps=0.1
|
||||
|
||||
name tst_accuracy tst_loss val_accuracy val_loss
|
||||
0 data/mldoc/de-1/models/sp15k/qrnn_nl4_tls.m 0.95850 0.254842 0.946 0.320358
|
||||
1 data/mldoc/de-1/models/sp15k/qrnn_nl4_tls2.m 0.95900 0.245983 0.947 0.303949
|
||||
2 data/mldoc/de-1/models/sp15k/qrnn_nl4_tls3.m 0.95550 0.270527 0.938 0.323216
|
||||
3 data/mldoc/en-1/models/sp15k/qrnn_nl4_tls.m 0.95550 0.246017 0.959 0.237861
|
||||
4 data/mldoc/en-1/models/sp15k/qrnn_nl4_tls2.m 0.95075 0.258219 0.959 0.235698
|
||||
5 data/mldoc/en-1/models/sp15k/qrnn_nl4_tls3.m 0.95175 0.249414 0.960 0.245007
|
||||
6 data/mldoc/es-1/models/sp15k/qrnn_nl4_tls.m 0.95875 0.258491 0.961 0.255865
|
||||
7 data/mldoc/es-1/models/sp15k/qrnn_nl4_tls2.m 0.95825 0.263527 0.959 0.274785
|
||||
8 data/mldoc/es-1/models/sp15k/qrnn_nl4_tls3.m 0.96075 0.253370 0.965 0.254268
|
||||
9 data/mldoc/fr-1/models/sp15k/qrnn_nl4_tls.m 0.94750 0.277039 0.942 0.295544
|
||||
10 data/mldoc/fr-1/models/sp15k/qrnn_nl4_tls2.m 0.94750 0.284394 0.943 0.288495
|
||||
11 data/mldoc/fr-1/models/sp15k/qrnn_nl4_tls3.m 0.94750 0.268739 0.938 0.274793
|
||||
12 data/mldoc/it-1/models/sp15k/qrnn_nl4_tls.m 0.90100 0.424416 0.899 0.386466
|
||||
13 data/mldoc/it-1/models/sp15k/qrnn_nl4_tls2.m 0.90375 0.410442 0.913 0.381761
|
||||
14 data/mldoc/it-1/models/sp15k/qrnn_nl4_tls3.m 0.90250 0.416314 0.917 0.378864
|
||||
15 data/mldoc/ja-1/models/sp15k/qrnn_nl4_tls.m 0.89850 0.456913 0.887 0.507895
|
||||
16 data/mldoc/ja-1/models/sp15k/qrnn_nl4_tls2.m 0.90025 0.426836 0.897 0.469335
|
||||
17 data/mldoc/ja-1/models/sp15k/qrnn_nl4_tls3.m 0.89800 0.449715 0.890 0.502422
|
||||
18 data/mldoc/ru-1/models/sp15k/qrnn_nl4_tls.m 0.86550 0.571294 0.870 0.548535
|
||||
19 data/mldoc/ru-1/models/sp15k/qrnn_nl4_tls2.m 0.87650 0.587116 0.877 0.585862
|
||||
20 data/mldoc/ru-1/models/sp15k/qrnn_nl4_tls3.m 0.87625 0.550317 0.866 0.574534
|
||||
21 data/mldoc/zh-1/models/sp15k/qrnn_nl4_tls.m 0.92525 0.347967 0.921 0.350878
|
||||
22 data/mldoc/zh-1/models/sp15k/qrnn_nl4_tls2.m 0.92175 0.377572 0.917 0.380295
|
||||
23 data/mldoc/zh-1/models/sp15k/qrnn_nl4_tls3.m 0.92225 0.350547 0.916 0.362135
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,370 +0,0 @@
|
||||
```
|
||||
python -m ulmfit eval_noise_resistance --lang=es --size=10 --prefix-name="val_" --model="sp30k/lstm_nl4.m"
|
||||
Noise: 0
|
||||
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
|
||||
es-10
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_0.m
|
||||
Evaluating previously trained model
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Data cls, trn: 9458, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
||||
Loss and accuracy using (cls_best): [0.32779965, tensor(0.9515)]
|
||||
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_0.m',
|
||||
0.9514999985694885)])
|
||||
Noise: 5
|
||||
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
|
||||
es-10
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_5.m
|
||||
Evaluating previously trained model
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Data cls, trn: 9458, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
||||
Loss and accuracy using (cls_best): [0.33051395, tensor(0.9488)]
|
||||
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_5.m',
|
||||
0.9487500190734863)])
|
||||
Noise: 10
|
||||
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
|
||||
es-10
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_10.m
|
||||
Evaluating previously trained model
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Data cls, trn: 9458, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
||||
Loss and accuracy using (cls_best): [0.22158922, tensor(0.9433)]
|
||||
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_10.m',
|
||||
0.9432500004768372)])
|
||||
Noise: 15
|
||||
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
|
||||
es-10
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_15.m
|
||||
Evaluating previously trained model
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Data cls, trn: 9458, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
||||
Loss and accuracy using (cls_best): [0.25426567, tensor(0.9358)]
|
||||
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_15.m',
|
||||
0.9357500076293945)])
|
||||
Noise: 20
|
||||
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
|
||||
es-10
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_20.m
|
||||
Evaluating previously trained model
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Data cls, trn: 9458, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
||||
Loss and accuracy using (cls_best): [0.32246214, tensor(0.9210)]
|
||||
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_20.m',
|
||||
0.9210000038146973)])
|
||||
Noise: 25
|
||||
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
|
||||
es-10
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_25.m
|
||||
Evaluating previously trained model
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Data cls, trn: 9458, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
||||
Loss and accuracy using (cls_best): [0.823559, tensor(0.9095)]
|
||||
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_25.m',
|
||||
0.909500002861023)])
|
||||
Noise: 30
|
||||
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
|
||||
es-10
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_30.m
|
||||
Evaluating previously trained model
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Data cls, trn: 9458, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
||||
Loss and accuracy using (cls_best): [0.5010365, tensor(0.8942)]
|
||||
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_30.m',
|
||||
0.8942499756813049)])
|
||||
Noise: 35
|
||||
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
|
||||
es-10
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_35.m
|
||||
Evaluating previously trained model
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Data cls, trn: 9458, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
||||
Loss and accuracy using (cls_best): [0.95638776, tensor(0.5853)]
|
||||
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_35.m',
|
||||
0.5852500200271606)])
|
||||
Noise: 40
|
||||
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
|
||||
es-10
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_40.m
|
||||
Evaluating previously trained model
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Data cls, trn: 9458, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
||||
Loss and accuracy using (cls_best): [1.1012905, tensor(0.5642)]
|
||||
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_40.m',
|
||||
0.5642499923706055)])
|
||||
Noise: 45
|
||||
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
|
||||
es-10
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_45.m
|
||||
Evaluating previously trained model
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Data cls, trn: 9458, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
||||
Loss and accuracy using (cls_best): [1.6009017, tensor(0.3072)]
|
||||
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_45.m',
|
||||
0.3072499930858612)])
|
||||
Noise: 50
|
||||
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
|
||||
es-10
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_50.m
|
||||
Evaluating previously trained model
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Data cls, trn: 9458, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
||||
Loss and accuracy using (cls_best): [1.5735056, tensor(0.3072)]
|
||||
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_50.m',
|
||||
0.3072499930858612)])
|
||||
Noise: 55
|
||||
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
|
||||
es-10
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_55.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
|
||||
Added noise to 5201 examples, only 0.4500951575385917 have correct labels
|
||||
Added noise to 550 examples, only 0.45 have correct labels
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Data clsnoise0.55tv, trn: 9458, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.150436 1.391014 0.321000
|
||||
2 1.190502 5.388964 0.313000
|
||||
3 1.216090 1.697217 0.221000
|
||||
4 1.221863 1.676644 0.221000
|
||||
5 1.213776 1.734900 0.221000
|
||||
6 1.195663 1.713853 0.221000
|
||||
7 1.211159 1.710040 0.221000
|
||||
8 1.197578 1.674693 0.221000
|
||||
Total time: 29:10
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_55.m
|
||||
Loss and accuracy using (cls_best): [1.5282942, tensor(0.3072)]
|
||||
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_55.m',
|
||||
0.3072499930858612)])
|
||||
Noise: 60
|
||||
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
|
||||
es-10
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_60.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
|
||||
Added noise to 5674 examples, only 0.4000845844787482 have correct labels
|
||||
Added noise to 600 examples, only 0.4 have correct labels
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Running tokenization clsnoise0.6tv...
|
||||
Data clsnoise0.6tv, trn: 9458, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_60.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.176071 1.396755 0.321000
|
||||
2 1.211001 1.619019 0.289000
|
||||
3 1.229442 1.733743 0.261000
|
||||
4 1.190156 1.545205 0.312000
|
||||
5 1.182274 1.369377 0.308000
|
||||
6 1.169403 1.352204 0.304000
|
||||
7 1.166997 1.332295 0.316000
|
||||
8 1.165893 1.370641 0.313000
|
||||
Total time: 30:23
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_60.m
|
||||
Loss and accuracy using (cls_best): [1.2368572, tensor(0.6102)]
|
||||
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_60.m',
|
||||
0.6102499961853027)])
|
||||
Noise: 65
|
||||
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
|
||||
es-10
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_65.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
|
||||
Added noise to 6147 examples, only 0.35007401141890465 have correct labels
|
||||
Added noise to 650 examples, only 0.35 have correct labels
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Running tokenization clsnoise0.65tv...
|
||||
Data clsnoise0.65tv, trn: 9458, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_65.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.179257 1.460323 0.295000
|
||||
2 1.220349 1.516707 0.222000
|
||||
3 1.211396 1.870125 0.242000
|
||||
4 1.187261 1.922184 0.308000
|
||||
5 1.201833 1.429372 0.300000
|
||||
6 1.187137 1.580070 0.264000
|
||||
7 1.162549 1.845004 0.294000
|
||||
8 1.162919 1.514930 0.313000
|
||||
Total time: 29:23
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_65.m
|
||||
Loss and accuracy using (cls_best): [1.1729655, tensor(0.6385)]
|
||||
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_65.m',
|
||||
0.6384999752044678)])
|
||||
Noise: 70
|
||||
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
|
||||
es-10
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_70.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
|
||||
Added noise to 6620 examples, only 0.30006343835906113 have correct labels
|
||||
Added noise to 700 examples, only 0.3 have correct labels
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Running tokenization clsnoise0.7tv...
|
||||
Data clsnoise0.7tv, trn: 9458, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_70.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.155135 1.479137 0.312000
|
||||
2 1.190364 1.649113 0.288000
|
||||
3 1.220965 3.919039 0.280000
|
||||
4 1.222588 1.696949 0.258000
|
||||
5 1.220919 1.669896 0.264000
|
||||
6 1.217906 2.003806 0.257000
|
||||
7 1.216235 1.654473 0.258000
|
||||
8 1.217084 1.675933 0.258000
|
||||
Total time: 29:04
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_70.m
|
||||
Loss and accuracy using (cls_best): [1.5526773, tensor(0.1828)]
|
||||
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_70.m',
|
||||
0.18275000154972076)])
|
||||
Noise: 75
|
||||
Processing data/mldoc/es-1/models/sp30k/lstm_nl4.m
|
||||
es-10
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_75.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/es.dev.csv
|
||||
Added noise to 7093 examples, only 0.2500528652992176 have correct labels
|
||||
Added noise to 750 examples, only 0.25 have correct labels
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Running tokenization clsnoise0.75tv...
|
||||
Data clsnoise0.75tv, trn: 9458, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', '▁en', '▁el', '▁y', 's', '▁a', '▁que']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_75.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.170507 1.421675 0.344000
|
||||
2 1.221764 1.700004 0.246000
|
||||
3 1.215101 2.358311 0.263000
|
||||
4 1.243265 1.551931 0.257000
|
||||
5 1.222902 1.756996 0.271000
|
||||
6 1.215993 1.677014 0.266000
|
||||
7 1.225945 4.560951 0.263000
|
||||
8 1.219151 2.939914 0.245000
|
||||
Total time: 29:52
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10/models/sp30k/lstm_nl4_val_75.m
|
||||
Loss and accuracy using (cls_best): [1.7072973, tensor(0.2465)]
|
||||
OrderedDict([('data/mldoc/es-10/models/sp30k/lstm_nl4_val_75.m',
|
||||
0.24650000035762787)])
|
||||
noise accuracy
|
||||
0 0.00 0.95150
|
||||
1 0.05 0.94875
|
||||
2 0.10 0.94325
|
||||
3 0.15 0.93575
|
||||
4 0.20 0.92100
|
||||
5 0.25 0.90950
|
||||
6 0.30 0.89425
|
||||
7 0.35 0.58525
|
||||
8 0.40 0.56425
|
||||
9 0.45 0.30725
|
||||
10 0.50 0.30725
|
||||
11 0.55 0.30725
|
||||
12 0.60 0.61025
|
||||
13 0.65 0.63850
|
||||
14 0.70 0.18275
|
||||
15 0.75 0.24650
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,627 +0,0 @@
|
||||
## Debugging random init
|
||||
````
|
||||
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4_rnd2_0.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
````
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## first attempt at random init
|
||||
```
|
||||
python -m ulmfit eval_noise_resistance --lang=de --size=10 --prefix-name="_rnd_" --model="sp15k/qrnn_rnd-nl4.m" --label-smoothing-eps=0.1
|
||||
Noise: 0
|
||||
Processing data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
de-10
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_0.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/de.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 10000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
Your compiler (c++) may be ABI-incompatible with PyTorch!
|
||||
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
|
||||
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
|
||||
|
||||
See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
|
||||
for instructions on how to install GCC 4.9 or higher.
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_0.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.787174 0.985413 0.701000
|
||||
2 0.679534 0.697764 0.875000
|
||||
3 0.630987 7.125103 0.873000
|
||||
4 0.588259 0.653497 0.915000
|
||||
5 0.568135 0.641379 0.942000
|
||||
6 0.529713 0.557198 0.948000
|
||||
7 0.500168 0.538946 0.958000
|
||||
8 0.505462 0.550917 0.954000
|
||||
Total time: 19:37
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_0.m
|
||||
Loss and accuracy using (cls_best): [0.21539633, tensor(0.9613)]
|
||||
OrderedDict([('data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_0.m',
|
||||
0.9612500071525574)])
|
||||
Noise: 5
|
||||
Processing data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
de-10
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_5.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/de.dev.csv
|
||||
Added noise to 500 examples, only 0.95 have correct labels
|
||||
Added noise to 50 examples, only 0.95 have correct labels
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data clsnoise0.05tv, trn: 10000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_5.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.917813 0.874308 0.790000
|
||||
2 0.821460 1.239760 0.671000
|
||||
3 0.747909 2.624352 0.667000
|
||||
4 0.713649 0.735327 0.893000
|
||||
5 0.689947 1.175884 0.844000
|
||||
6 0.639073 1.066042 0.862000
|
||||
7 0.617660 0.845634 0.875000
|
||||
8 0.620402 0.670972 0.901000
|
||||
Total time: 19:28
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_5.m
|
||||
Loss and accuracy using (cls_best): [0.25263783, tensor(0.9560)]
|
||||
OrderedDict([('data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_5.m',
|
||||
0.9559999704360962)])
|
||||
Noise: 10
|
||||
Processing data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
de-10
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_10.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/de.dev.csv
|
||||
Added noise to 1000 examples, only 0.9 have correct labels
|
||||
Added noise to 100 examples, only 0.9 have correct labels
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data clsnoise0.1tv, trn: 10000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_10.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.976570 1.054016 0.705000
|
||||
2 0.885775 0.813666 0.835000
|
||||
3 0.861244 0.968860 0.762000
|
||||
4 0.790501 0.815453 0.839000
|
||||
5 0.754292 0.805088 0.849000
|
||||
6 0.742595 0.770547 0.864000
|
||||
7 0.712961 0.771171 0.863000
|
||||
8 0.695449 0.787870 0.858000
|
||||
Total time: 19:48
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_10.m
|
||||
Loss and accuracy using (cls_best): [0.2744636, tensor(0.9510)]
|
||||
OrderedDict([('data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_10.m',
|
||||
0.9509999752044678)])
|
||||
Noise: 15
|
||||
Processing data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
de-10
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_15.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/de.dev.csv
|
||||
Added noise to 1500 examples, only 0.85 have correct labels
|
||||
Added noise to 150 examples, only 0.85 have correct labels
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data clsnoise0.15tv, trn: 10000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_15.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.081478 1.075213 0.674000
|
||||
2 0.989475 0.939438 0.779000
|
||||
3 0.963180 0.984908 0.723000
|
||||
4 0.915556 1.209332 0.662000
|
||||
5 0.884642 1.000015 0.786000
|
||||
6 0.844702 0.884871 0.794000
|
||||
7 0.793699 0.882503 0.802000
|
||||
8 0.797470 0.871922 0.802000
|
||||
Total time: 19:50
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_15.m
|
||||
Loss and accuracy using (cls_best): [0.32492134, tensor(0.9445)]
|
||||
OrderedDict([('data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_15.m',
|
||||
0.9445000290870667)])
|
||||
Noise: 20
|
||||
Processing data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
de-10
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_20.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/de.dev.csv
|
||||
Added noise to 2000 examples, only 0.8 have correct labels
|
||||
Added noise to 200 examples, only 0.8 have correct labels
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data clsnoise0.2tv, trn: 10000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_20.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.130177 1.651658 0.424000
|
||||
2 1.049187 1.508039 0.286000
|
||||
3 1.045260 1.976680 0.578000
|
||||
4 0.971859 1.121615 0.735000
|
||||
5 0.965327 2.376971 0.684000
|
||||
6 0.901961 1.089674 0.744000
|
||||
7 0.868971 1.082978 0.750000
|
||||
8 0.845376 1.019824 0.740000
|
||||
Total time: 19:51
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_20.m
|
||||
Loss and accuracy using (cls_best): [0.46514454, tensor(0.9438)]
|
||||
OrderedDict([('data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_20.m',
|
||||
0.9437500238418579)])
|
||||
Noise: 25
|
||||
Processing data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
de-10
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_25.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/de.dev.csv
|
||||
Added noise to 2500 examples, only 0.75 have correct labels
|
||||
Added noise to 250 examples, only 0.75 have correct labels
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data clsnoise0.25tv, trn: 10000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_25.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.204033 1.368233 0.500000
|
||||
2 1.121006 1.219437 0.586000
|
||||
3 1.057657 1.139297 0.659000
|
||||
4 1.054685 1.043641 0.700000
|
||||
5 1.023957 1.069890 0.706000
|
||||
6 0.992645 1.073037 0.708000
|
||||
7 0.948602 1.054931 0.699000
|
||||
8 0.945395 1.078187 0.703000
|
||||
Total time: 20:09
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_25.m
|
||||
Loss and accuracy using (cls_best): [0.4676742, tensor(0.9137)]
|
||||
OrderedDict([('data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_25.m',
|
||||
0.9137499928474426)])
|
||||
Noise: 30
|
||||
Processing data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
de-10
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_30.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/de.dev.csv
|
||||
Added noise to 3000 examples, only 0.7 have correct labels
|
||||
Added noise to 300 examples, only 0.7 have correct labels
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data clsnoise0.3tv, trn: 10000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_30.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.245468 1.243019 0.492000
|
||||
2 1.192702 1.644169 0.435000
|
||||
3 1.187665 4.143492 0.490000
|
||||
4 1.113116 20.139246 0.540000
|
||||
5 1.092624 1.189916 0.609000
|
||||
6 1.052626 1.264737 0.617000
|
||||
7 1.032403 1.317357 0.649000
|
||||
8 1.003000 1.187038 0.653000
|
||||
Total time: 20:02
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_30.m
|
||||
Loss and accuracy using (cls_best): [0.5831716, tensor(0.9215)]
|
||||
OrderedDict([('data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_30.m',
|
||||
0.921500027179718)])
|
||||
Noise: 35
|
||||
Processing data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
de-10
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_35.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/de.dev.csv
|
||||
Added noise to 3500 examples, only 0.65 have correct labels
|
||||
Added noise to 350 examples, only 0.65 have correct labels
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data clsnoise0.35tv, trn: 10000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_35.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.285933 1.719733 0.309000
|
||||
2 1.258459 1.465174 0.422000
|
||||
3 1.240111 1.205106 0.512000
|
||||
4 1.195793 2.153573 0.571000
|
||||
5 1.150691 3.427428 0.588000
|
||||
6 1.115649 1.933489 0.601000
|
||||
7 1.078265 1.214095 0.599000
|
||||
8 1.045297 1.140148 0.604000
|
||||
Total time: 19:55
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_35.m
|
||||
Loss and accuracy using (cls_best): [0.5903087, tensor(0.9105)]
|
||||
OrderedDict([('data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_35.m',
|
||||
0.9104999899864197)])
|
||||
Noise: 40
|
||||
Processing data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
de-10
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_40.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/de.dev.csv
|
||||
Added noise to 4000 examples, only 0.6 have correct labels
|
||||
Added noise to 400 examples, only 0.6 have correct labels
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data clsnoise0.4tv, trn: 10000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_40.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.313393 1.523955 0.316000
|
||||
2 1.303059 1.964390 0.418000
|
||||
3 1.291624 1.550615 0.458000
|
||||
4 1.263588 2.995128 0.390000
|
||||
5 1.206715 1.265662 0.524000
|
||||
6 1.191890 1.221754 0.536000
|
||||
7 1.162122 1.223106 0.527000
|
||||
8 1.150922 1.240103 0.531000
|
||||
Total time: 19:53
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_40.m
|
||||
Loss and accuracy using (cls_best): [0.7210464, tensor(0.8583)]
|
||||
OrderedDict([('data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_40.m',
|
||||
0.8582500219345093)])
|
||||
Noise: 45
|
||||
Processing data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
de-10
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_45.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/de.dev.csv
|
||||
Added noise to 4500 examples, only 0.55 have correct labels
|
||||
Added noise to 450 examples, only 0.55 have correct labels
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data clsnoise0.45tv, trn: 10000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_45.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.345056 1.343081 0.378000
|
||||
2 1.281120 11.283777 0.232000
|
||||
3 1.284114 14.679921 0.390000
|
||||
4 1.267963 2.869378 0.485000
|
||||
5 1.227434 1.466781 0.490000
|
||||
6 1.209261 1.634938 0.495000
|
||||
7 1.170042 1.372811 0.494000
|
||||
8 1.162168 2.157310 0.492000
|
||||
Total time: 20:05
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_45.m
|
||||
Loss and accuracy using (cls_best): [1.0457553, tensor(0.8635)]
|
||||
OrderedDict([('data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_45.m',
|
||||
0.8634999990463257)])
|
||||
Noise: 50
|
||||
Processing data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
de-10
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_50.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/de.dev.csv
|
||||
Added noise to 5000 examples, only 0.5 have correct labels
|
||||
Added noise to 500 examples, only 0.5 have correct labels
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data clsnoise0.5tv, trn: 10000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_50.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.362338 1.361343 0.339000
|
||||
2 1.343794 1.358407 0.328000
|
||||
3 1.326264 3.336083 0.325000
|
||||
4 1.321352 4.200035 0.254000
|
||||
5 1.289333 1.363007 0.408000
|
||||
6 1.275341 1.449265 0.405000
|
||||
7 1.245595 1.358157 0.423000
|
||||
8 1.234815 1.346797 0.411000
|
||||
Total time: 19:35
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_50.m
|
||||
Loss and accuracy using (cls_best): [1.3260584, tensor(0.7103)]
|
||||
OrderedDict([('data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_50.m',
|
||||
0.7102500200271606)])
|
||||
Noise: 55
|
||||
Processing data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
de-10
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_55.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/de.dev.csv
|
||||
Added noise to 5500 examples, only 0.45 have correct labels
|
||||
Added noise to 550 examples, only 0.45 have correct labels
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data clsnoise0.55tv, trn: 10000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_55.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.373838 1.385533 0.265000
|
||||
2 1.355375 2.033619 0.316000
|
||||
3 1.358652 2.010394 0.260000
|
||||
4 1.337999 7.118755 0.351000
|
||||
5 1.309082 3.053319 0.361000
|
||||
6 1.286589 19.251106 0.359000
|
||||
7 1.276432 1.328096 0.379000
|
||||
8 1.266364 1.324883 0.378000
|
||||
Total time: 19:34
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_55.m
|
||||
Loss and accuracy using (cls_best): [1.3107486, tensor(0.6503)]
|
||||
OrderedDict([('data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_55.m',
|
||||
0.6502500176429749)])
|
||||
Noise: 60
|
||||
Processing data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
de-10
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_60.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/de.dev.csv
|
||||
Added noise to 6000 examples, only 0.4 have correct labels
|
||||
Added noise to 600 examples, only 0.4 have correct labels
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data clsnoise0.6tv, trn: 10000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_60.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.378700 1.377784 0.309000
|
||||
2 1.359247 9.472390 0.250000
|
||||
3 1.343403 1.714557 0.321000
|
||||
4 1.336044 1.331355 0.357000
|
||||
5 1.322668 1.450317 0.332000
|
||||
6 1.283835 2.692688 0.349000
|
||||
7 1.261502 1.541230 0.335000
|
||||
8 1.230086 1.839382 0.340000
|
||||
Total time: 19:58
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_60.m
|
||||
Loss and accuracy using (cls_best): [1.1523782, tensor(0.5580)]
|
||||
OrderedDict([('data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_60.m',
|
||||
0.5580000281333923)])
|
||||
Noise: 65
|
||||
Processing data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
de-10
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_65.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/de.dev.csv
|
||||
Added noise to 6500 examples, only 0.35 have correct labels
|
||||
Added noise to 650 examples, only 0.35 have correct labels
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data clsnoise0.65tv, trn: 10000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_65.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.374414 1.361766 0.327000
|
||||
2 1.367130 1.353700 0.341000
|
||||
3 1.365781 1.421649 0.269000
|
||||
4 1.358339 1.385666 0.280000
|
||||
5 1.357855 3.068685 0.334000
|
||||
6 1.343958 1.586822 0.316000
|
||||
7 1.330202 2.436025 0.324000
|
||||
8 1.322320 1.743209 0.330000
|
||||
Total time: 19:52
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_65.m
|
||||
Loss and accuracy using (cls_best): [1.7415464, tensor(0.4467)]
|
||||
OrderedDict([('data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_65.m',
|
||||
0.4467499852180481)])
|
||||
Noise: 70
|
||||
Processing data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
de-10
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_70.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/de.dev.csv
|
||||
Added noise to 7000 examples, only 0.3 have correct labels
|
||||
Added noise to 700 examples, only 0.3 have correct labels
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data clsnoise0.7tv, trn: 10000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_70.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.386991 1.377855 0.302000
|
||||
2 1.370086 1.741303 0.298000
|
||||
3 1.371910 1.402328 0.316000
|
||||
4 1.349717 1.378567 0.277000
|
||||
5 1.360438 1.471136 0.298000
|
||||
6 1.345680 1.395034 0.312000
|
||||
7 1.327264 1.611867 0.312000
|
||||
8 1.327243 1.657344 0.312000
|
||||
Total time: 20:09
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_70.m
|
||||
Loss and accuracy using (cls_best): [2.7352421, tensor(0.2693)]
|
||||
OrderedDict([('data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_70.m',
|
||||
0.2692500054836273)])
|
||||
Noise: 75
|
||||
Processing data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
de-10
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_75.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/de.dev.csv
|
||||
Added noise to 7500 examples, only 0.25 have correct labels
|
||||
Added noise to 750 examples, only 0.25 have correct labels
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data clsnoise0.75tv, trn: 10000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_75.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.376097 1.392457 0.263000
|
||||
2 1.372446 1.367712 0.320000
|
||||
3 1.356304 1.354679 0.297000
|
||||
4 1.342981 1.350475 0.335000
|
||||
5 1.340915 1.337473 0.343000
|
||||
6 1.320882 1.904698 0.357000
|
||||
7 1.291946 1.368179 0.339000
|
||||
8 1.283206 1.466608 0.353000
|
||||
Total time: 19:50
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_75.m
|
||||
Loss and accuracy using (cls_best): [1.4704828, tensor(0.1248)]
|
||||
OrderedDict([('data/mldoc/de-10/models/sp15k/qrnn_nl4__rnd_75.m',
|
||||
0.12475000321865082)])
|
||||
noise accuracy
|
||||
0 0.00 0.96125
|
||||
1 0.05 0.95600
|
||||
2 0.10 0.95100
|
||||
3 0.15 0.94450
|
||||
4 0.20 0.94375
|
||||
5 0.25 0.91375
|
||||
6 0.30 0.92150
|
||||
7 0.35 0.91050
|
||||
8 0.40 0.85825
|
||||
9 0.45 0.86350
|
||||
10 0.50 0.71025
|
||||
11 0.55 0.65025
|
||||
12 0.60 0.55800
|
||||
13 0.65 0.44675
|
||||
14 0.70 0.26925
|
||||
15 0.75 0.12475
|
||||
```
|
||||
@@ -1,717 +0,0 @@
|
||||
|
||||
|
||||
### MLDoc laser zero shoot 10k
|
||||
Loss and accuracy using (cls_best): [0.58419716, tensor(0.8150)] [0.6386394, tensor(0.7850)]
|
||||
name tst_accuracy tst_loss val_accuracy val_loss
|
||||
0 data/mldoc/de-10-laser-en1/models/sp15k/qrnn_r... 0.90550 0.407956 0.917 0.378366
|
||||
1 data/mldoc/es-10-laser-en1/models/sp15k/qrnn_r... 0.69725 1.371628 0.747 1.040883
|
||||
2 data/mldoc/fr-10-laser-en1/models/sp15k/qrnn_r... 0.87350 0.533766 0.882 0.488216
|
||||
3 data/mldoc/it-10-laser-en1/models/sp15k/qrnn_r... 0.72750 1.168527 0.804 1.180750
|
||||
4 data/mldoc/ja-10-laser-en1/models/sp15k/qrnn_r... 0.67550 1.941633 0.780 0.991396
|
||||
5 data/mldoc/ru-10-laser-en1/models/sp15k/qrnn_r... 0.63675 2.139875 0.822 0.765918
|
||||
6 data/mldoc/zh-10-laser-en1/models/sp15k/qrnn_r... 0.81500 0.584197 0.785 0.638639
|
||||
ds de-10-laser- es-10-laser- fr-10-laser- it-10-laser- ja-10-laser- ru-10-laser- zh-10-laser-
|
||||
best 90.55 69.73 87.35 72.75 67.55 63.67 81.5
|
||||
max 90.55 69.73 87.35 72.75 67.55 63.67 81.5
|
||||
avg 90.55 69.73 87.35 72.75 67.55 63.67 81.5
|
||||
|
||||
```
|
||||
|
||||
Starting classifier from random weights
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.087360 1.237505 0.515000
|
||||
2 0.810132 1.667991 0.545000
|
||||
3 0.779427 1.003253 0.679000
|
||||
4 0.662206 2.510274 0.800000
|
||||
5 0.604669 2.394876 0.718000
|
||||
6 0.501023 0.866812 0.810000
|
||||
7 0.398415 0.639844 0.818000
|
||||
8 0.448071 0.922495 0.822000
|
||||
Total time: 11:52
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-10-laser-en1/models/sp15k/qrnn_rnd_nl4.m
|
||||
```
|
||||
|
||||
```
|
||||
python -m ulmfit eval --glob="mldoc/*-1/models/sp15k/qrnn_nl4.m" --dataset_template='${lang}-10-laser-en1' --name rnd_nl4 --num-cls-epochs=8 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=18 --random-init=True
|
||||
python -m ulmfit eval --glob="mldoc/*-1/models/sp15k/qrnn_nl4.m" --dataset_template='${lang}-10-laser-en1' --name rnd_nl4 --num-cls-epochs=8 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=18 --random-init=True
|
||||
Processing data/mldoc/de-1/models/sp15k/qrnn_nl4.m
|
||||
de-10-laser-en1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10-laser-en1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10-laser-en1/models/sp15k/qrnn_rnd_nl4.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10-laser-en1/de.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 10000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
Your compiler (c++) may be ABI-incompatible with PyTorch!
|
||||
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
|
||||
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
|
||||
|
||||
See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
|
||||
for instructions on how to install GCC 4.9 or higher.
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
|
||||
Starting classifier from random weights
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.864752 0.760891 0.844000
|
||||
2 0.734504 1.077554 0.676000
|
||||
3 0.681645 0.703327 0.885000
|
||||
4 0.670696 0.779010 0.898000
|
||||
5 0.620256 0.664871 0.910000
|
||||
6 0.591837 1.077103 0.915000
|
||||
7 0.550238 0.607863 0.913000
|
||||
8 0.543874 0.607274 0.918000
|
||||
Total time: 19:55
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-10-laser-en1/models/sp15k/qrnn_rnd_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.35624045, tensor(0.9053)]
|
||||
Processing data/mldoc/en-1/models/sp15k/qrnn_nl4.m
|
||||
en-10-laser-en1
|
||||
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
|
||||
es-10-laser-en1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10-laser-en1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10-laser-en1/models/sp15k/qrnn_rnd_nl4.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10-laser-en1/es.dev.csv
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Data cls, trn: 9458, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
|
||||
Starting classifier from random weights
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.755512 1.360725 0.500000
|
||||
2 0.760655 1.362604 0.399000
|
||||
3 0.760876 20.748863 0.607000
|
||||
4 0.730208 8.120344 0.369000
|
||||
5 0.707735 1.149775 0.700000
|
||||
6 0.679102 1.010318 0.746000
|
||||
7 0.639611 3.087066 0.713000
|
||||
8 0.608591 1.327793 0.750000
|
||||
Total time: 11:38
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-10-laser-en1/models/sp15k/qrnn_rnd_nl4.m
|
||||
Loss and accuracy using (cls_best): [1.3680531, tensor(0.6975)]
|
||||
Processing data/mldoc/fr-1/models/sp15k/qrnn_nl4.m
|
||||
fr-10-laser-en1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-10-laser-en1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-10-laser-en1/models/sp15k/qrnn_rnd_nl4.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-10-laser-en1/fr.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 10000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à']
|
||||
Starting classifier from random weights
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.922996 1.406875 0.519000
|
||||
2 0.833284 1.172545 0.640000
|
||||
3 0.749922 0.697733 0.863000
|
||||
4 0.724680 0.735842 0.837000
|
||||
5 0.652541 0.679455 0.876000
|
||||
6 0.641541 0.671731 0.868000
|
||||
7 0.577571 0.734958 0.868000
|
||||
8 0.579186 0.703696 0.883000
|
||||
Total time: 19:15
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-10-laser-en1/models/sp15k/qrnn_rnd_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.47713563, tensor(0.8740)]
|
||||
Processing data/mldoc/it-1/models/sp15k/qrnn_nl4.m
|
||||
it-10-laser-en1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-10-laser-en1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-10-laser-en1/models/sp15k/qrnn_rnd_nl4.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-10-laser-en1/it.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 10000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁di', '▁e', "▁&'", "'", '▁il', '▁la', '▁in', 'e']
|
||||
Starting classifier from random weights
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.991065 1.602189 0.381000
|
||||
2 0.935880 0.888808 0.734000
|
||||
3 0.860670 0.868564 0.781000
|
||||
4 0.818734 0.945302 0.791000
|
||||
5 0.751467 3.113552 0.808000
|
||||
6 0.687606 0.921033 0.795000
|
||||
7 0.677044 1.222023 0.807000
|
||||
8 0.645511 1.418593 0.805000
|
||||
Total time: 11:44
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-10-laser-en1/models/sp15k/qrnn_rnd_nl4.m
|
||||
Loss and accuracy using (cls_best): [1.1276722, tensor(0.7272)]
|
||||
Processing data/mldoc/ja-1/models/sp15k/qrnn_nl4.m
|
||||
ja-10-laser-en1
|
||||
Processing data/mldoc/zh-1/models/sp15k/qrnn_nl4.m
|
||||
zh-10-laser-en1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-10-laser-en1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-10-laser-en1/models/sp15k/qrnn_rnd_nl4.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-10-laser-en1/zh.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 10000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有']
|
||||
Starting classifier from random weights
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.920411 1.159853 0.629000
|
||||
2 0.937020 1.371089 0.527000
|
||||
3 0.892036 3.091183 0.615000
|
||||
4 0.839919 0.939323 0.724000
|
||||
5 0.797184 1.174206 0.735000
|
||||
6 0.774195 0.914951 0.733000
|
||||
7 0.744524 0.875888 0.762000
|
||||
8 0.721782 0.825969 0.788000
|
||||
Total time: 19:53
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-10-laser-en1/models/sp15k/qrnn_rnd_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.54084456, tensor(0.8145)]
|
||||
OrderedDict([('data/mldoc/de-10-laser-en1/models/sp15k/qrnn_rnd_nl4.m',
|
||||
0.9052500128746033),
|
||||
('data/mldoc/es-10-laser-en1/models/sp15k/qrnn_rnd_nl4.m',
|
||||
0.6974999904632568),
|
||||
('data/mldoc/fr-10-laser-en1/models/sp15k/qrnn_rnd_nl4.m',
|
||||
0.8740000128746033),
|
||||
('data/mldoc/it-10-laser-en1/models/sp15k/qrnn_rnd_nl4.m',
|
||||
0.7272499799728394),
|
||||
('data/mldoc/zh-10-laser-en1/models/sp15k/qrnn_rnd_nl4.m',
|
||||
0.8144999742507935)])
|
||||
data/mldoc/de-10-laser-en1/models/sp15k/qrnn_rnd_nl4.m: 0.9052500128746033
|
||||
data/mldoc/es-10-laser-en1/models/sp15k/qrnn_rnd_nl4.m: 0.6974999904632568
|
||||
data/mldoc/fr-10-laser-en1/models/sp15k/qrnn_rnd_nl4.m: 0.8740000128746033
|
||||
data/mldoc/it-10-laser-en1/models/sp15k/qrnn_rnd_nl4.m: 0.7272499799728394
|
||||
data/mldoc/zh-10-laser-en1/models/sp15k/qrnn_rnd_nl4.m: 0.8144999742507935
|
||||
```
|
||||
|
||||
|
||||
### MLDoc laser zero shoot 1k
|
||||
|
||||
data/mldoc/de-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m: 0.777999997138977
|
||||
data/mldoc/es-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m: 0.7049999833106995
|
||||
data/mldoc/fr-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m: 0.7565000057220459
|
||||
data/mldoc/it-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m: 0.6852499842643738
|
||||
data/mldoc/ru-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m: 0.6137499809265137
|
||||
data/mldoc/zh-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m: 0.7919999957084656
|
||||
```
|
||||
python -m ulmfit eval --glob="mldoc/*-1/models/sp15k/qrnn_nl4.m" --dataset_template='${lang}-1-laser-en1' --name nl4-rnd --num-cls-epochs=8 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=18 --random-init=True
|
||||
Processing data/mldoc/de-1/models/sp15k/qrnn_nl4.m
|
||||
de-1-laser-en1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1-laser-en1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1-laser-en1/de.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
Your compiler (c++) may be ABI-incompatible with PyTorch!
|
||||
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
|
||||
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
|
||||
|
||||
See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
|
||||
for instructions on how to install GCC 4.9 or higher.
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
|
||||
Starting classifier from random weights
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.409097 1.357696 0.412000
|
||||
2 1.270668 1.920674 0.259000
|
||||
3 1.121176 1.099281 0.539000
|
||||
4 1.034688 2.448050 0.263000
|
||||
5 0.893729 1.306312 0.560000
|
||||
6 0.794611 0.945334 0.742000
|
||||
7 0.711580 0.997155 0.703000
|
||||
8 0.668383 0.877867 0.784000
|
||||
Total time: 02:21
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m
|
||||
Loss and accuracy using (cls_best): [0.64768696, tensor(0.7780)]
|
||||
Processing data/mldoc/en-1/models/sp15k/qrnn_nl4.m
|
||||
en-1-laser-en1
|
||||
Processing data/mldoc/es-1/models/sp15k/qrnn_nl4.m
|
||||
es-1-laser-en1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1-laser-en1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1-laser-en1/es.dev.csv
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
|
||||
Starting classifier from random weights
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.390623 1.330758 0.365000
|
||||
2 1.226550 1.615878 0.511000
|
||||
3 1.097214 1.439728 0.532000
|
||||
4 0.988972 1.093623 0.688000
|
||||
5 0.872939 1.278118 0.623000
|
||||
6 0.811549 0.894074 0.779000
|
||||
7 0.703965 0.821635 0.818000
|
||||
8 0.630001 0.782405 0.823000
|
||||
Total time: 02:00
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m
|
||||
Loss and accuracy using (cls_best): [0.9085049, tensor(0.7050)]
|
||||
Processing data/mldoc/fr-1/models/sp15k/qrnn_nl4.m
|
||||
fr-1-laser-en1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1-laser-en1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1-laser-en1/fr.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à']
|
||||
Starting classifier from random weights
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.331783 1.457629 0.270000
|
||||
2 1.206117 1.568209 0.442000
|
||||
3 1.085947 1.397149 0.477000
|
||||
4 0.965919 1.025710 0.668000
|
||||
5 0.854985 0.915386 0.732000
|
||||
6 0.784643 0.922064 0.725000
|
||||
7 0.691292 0.896307 0.761000
|
||||
8 0.641204 0.867914 0.785000
|
||||
Total time: 02:22
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m
|
||||
Loss and accuracy using (cls_best): [0.66897815, tensor(0.7565)]
|
||||
Processing data/mldoc/it-1/models/sp15k/qrnn_nl4.m
|
||||
it-1-laser-en1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1-laser-en1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1-laser-en1/it.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁di', '▁e', "▁&'", "'", '▁il', '▁la', '▁in', 'e']
|
||||
Starting classifier from random weights
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.365793 1.365611 0.388000
|
||||
2 1.262174 2.729846 0.271000
|
||||
3 1.109978 1.754144 0.415000
|
||||
4 0.961007 0.922328 0.731000
|
||||
5 0.822061 0.961720 0.721000
|
||||
6 0.724208 0.979705 0.707000
|
||||
7 0.637849 0.910123 0.754000
|
||||
8 0.588057 0.915819 0.744000
|
||||
Total time: 01:25
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m
|
||||
Loss and accuracy using (cls_best): [0.9051443, tensor(0.6852)]
|
||||
Processing data/mldoc/ja-1/models/sp15k/qrnn_nl4.m
|
||||
ja-1-laser-en1
|
||||
Processing data/mldoc/ru-1/models/sp15k/qrnn_nl4.m
|
||||
ru-1-laser-en1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1-laser-en1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1-laser-en1/ru.dev.csv
|
||||
Data lm, trn: 9195, val: 1021
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Starting classifier from random weights
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.381033 1.389893 0.231000
|
||||
2 1.327294 1.474365 0.396000
|
||||
3 1.190571 2.075783 0.501000
|
||||
4 1.070966 1.025509 0.611000
|
||||
5 0.950332 0.956073 0.718000
|
||||
6 0.812717 1.165698 0.706000
|
||||
7 0.737636 0.924283 0.778000
|
||||
8 0.697346 0.947041 0.779000
|
||||
Total time: 03:16
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m
|
||||
Loss and accuracy using (cls_best): [1.2389272, tensor(0.6137)]
|
||||
Processing data/mldoc/zh-1/models/sp15k/qrnn_nl4.m
|
||||
zh-1-laser-en1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-en1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-en1/zh.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有']
|
||||
Starting classifier from random weights
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.382525 1.435203 0.328000
|
||||
2 1.171075 3.494349 0.342000
|
||||
3 1.044511 1.895226 0.475000
|
||||
4 0.954990 1.675222 0.473000
|
||||
5 0.880456 0.954913 0.747000
|
||||
6 0.808870 1.084140 0.669000
|
||||
7 0.762522 0.920228 0.770000
|
||||
8 0.710014 0.894134 0.769000
|
||||
Total time: 02:22
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m
|
||||
Loss and accuracy using (cls_best): [0.6264392, tensor(0.7920)]
|
||||
OrderedDict([('data/mldoc/de-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m',
|
||||
0.777999997138977),
|
||||
('data/mldoc/es-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m',
|
||||
0.7049999833106995),
|
||||
('data/mldoc/fr-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m',
|
||||
0.7565000057220459),
|
||||
('data/mldoc/it-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m',
|
||||
0.6852499842643738),
|
||||
('data/mldoc/ru-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m',
|
||||
0.6137499809265137),
|
||||
('data/mldoc/zh-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m',
|
||||
0.7919999957084656)])
|
||||
data/mldoc/de-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m: 0.777999997138977
|
||||
data/mldoc/es-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m: 0.7049999833106995
|
||||
data/mldoc/fr-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m: 0.7565000057220459
|
||||
data/mldoc/it-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m: 0.6852499842643738
|
||||
data/mldoc/ru-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m: 0.6137499809265137
|
||||
data/mldoc/zh-1-laser-en1/models/sp15k/qrnn_nl4-rnd.m: 0.7919999957084656
|
||||
```
|
||||
|
||||
### MLDoc Classification on 1k
|
||||
|
||||
data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m: 0.9024999737739563
|
||||
data/mldoc/en-1/models/sp15k/qrnn_rnd-nl4.m: 0.8149999976158142
|
||||
data/mldoc/es-1/models/sp15k/qrnn_rnd-nl4.m: 0.8964999914169312
|
||||
data/mldoc/fr-1/models/sp15k/qrnn_rnd-nl4.m: 0.8220000267028809
|
||||
data/mldoc/it-1/models/sp15k/qrnn_rnd-nl4.m: 0.7889999747276306
|
||||
data/mldoc/ja-1/models/sp15k/qrnn_rnd-nl4.m: 0.8302500247955322
|
||||
data/mldoc/ru-1/models/sp15k/qrnn_rnd-nl4.m: 0.7319999933242798
|
||||
data/mldoc/zh-1/models/sp15k/qrnn_rnd-nl4.m: 0.8452500104904175
|
||||
|
||||
|
||||
```
|
||||
python -m ulmfit eval --glob="wiki/*-100/models/sp15k/qrnn_rnd-nl4.m" --name rnd-nl4 --dataset-template='../mldoc/${lang}-1' --num-lm-epochs=0 --num-cls-epochs=8 --bs=18 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
Processing data/wiki/de-100/models/sp15k/qrnn_rnd-nl4.m
|
||||
../mldoc/de-1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/de.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
Your compiler (c++) may be ABI-incompatible with PyTorch!
|
||||
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
|
||||
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
|
||||
|
||||
See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
|
||||
for instructions on how to install GCC 4.9 or higher.
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.411651 1.386593 0.265000
|
||||
2 1.287022 1.488027 0.361000
|
||||
3 1.084153 2.390431 0.370000
|
||||
4 0.904227 0.936213 0.769000
|
||||
5 0.740495 1.311880 0.538000
|
||||
6 0.642756 0.754690 0.833000
|
||||
7 0.582816 0.661088 0.892000
|
||||
8 0.548893 0.683635 0.875000
|
||||
Total time: 02:13
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
Loss and accuracy using (cls_best): [0.33525154, tensor(0.9025)]
|
||||
Processing data/wiki/en-100/models/sp15k/qrnn_rnd-nl4.m
|
||||
../mldoc/en-1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/en.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁the', '▁,', 's', '▁.', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_rnd-nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.369466 1.356636 0.374000
|
||||
2 1.263357 2.515120 0.320000
|
||||
3 1.119744 1.250081 0.569000
|
||||
4 0.949653 1.033515 0.666000
|
||||
5 0.802069 0.875799 0.779000
|
||||
6 0.676997 0.842525 0.807000
|
||||
7 0.613777 0.794573 0.826000
|
||||
8 0.571342 0.781615 0.837000
|
||||
Total time: 02:26
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/en-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
Loss and accuracy using (cls_best): [0.5295334, tensor(0.8150)]
|
||||
Processing data/wiki/es-100/models/sp15k/qrnn_rnd-nl4.m
|
||||
../mldoc/es-1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 2621, first 100: ['▁sa', '▁i', 'ncia', '▁ka', '▁k', '▁tra', '▁fi', '▁volvi', '▁g', '▁man', '▁pasó', '▁tropas', 'pon', 'tuvieron', '▁x', '▁les', '▁empez', 'ieron', '▁bas', 'sco', '▁cam', '▁adapta', 'sion', '▁mol', 'pico', 'siones', '▁obstante', '▁!', '▁w', 'cular', 'puesta', '▁inten', '▁produj', 'clu', 'simismo', '▁pas', 'fla', '▁amerindio', 'aje', '▁deja', '▁fre', '▁jo', '▁2.', 'american', '▁cre', 'bajo', '▁medi', 'gla', '▁dirigi', 'hol', '▁aparición', 'aciones', 'vivi', 'eras', 'spe', '▁continu', '▁permaneci', '▁ber', 'usa', 'bió', '▁permitió', '▁municipios', '▁regres', 'rt', 'mbi', '▁pr', '▁ofreci', 'emi', 'misiones', '▁cap', '▁ram', 'icio', '▁wal', 'fru', '▁gen', '▁originalmente', '▁eva', '▁ferr', '▁descubri', '▁aparecen', '▁fon', 'capi', 'estre', 'pec', '▁vendi', 'iéndose', 'eja', 'liber', 'nsa', 'ológico', 'ío', 'blo', '▁tro', '▁aviones', 'cara', '▁activo', 'mostró', 'disciplina', '▁ara', 'estra']
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_rnd-nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.378275 1.354129 0.314000
|
||||
2 1.209560 1.333649 0.507000
|
||||
3 1.022200 0.820093 0.801000
|
||||
4 0.854187 1.782254 0.389000
|
||||
5 0.722861 1.031932 0.692000
|
||||
6 0.640640 0.762994 0.853000
|
||||
7 0.583225 0.677089 0.901000
|
||||
8 0.556481 0.652575 0.904000
|
||||
Total time: 01:59
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
Loss and accuracy using (cls_best): [0.35487488, tensor(0.8965)]
|
||||
Processing data/wiki/fr-100/models/sp15k/qrnn_rnd-nl4.m
|
||||
../mldoc/fr-1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/fr.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_rnd-nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.360408 1.298057 0.444000
|
||||
2 1.251207 2.454086 0.393000
|
||||
3 1.098488 1.152682 0.544000
|
||||
4 0.926239 1.256870 0.622000
|
||||
5 0.806994 0.911339 0.732000
|
||||
6 0.717139 0.945148 0.726000
|
||||
7 0.635195 0.781772 0.825000
|
||||
8 0.602214 0.763521 0.825000
|
||||
Total time: 02:17
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
Loss and accuracy using (cls_best): [0.52210134, tensor(0.8220)]
|
||||
Processing data/wiki/it-100/models/sp15k/qrnn_rnd-nl4.m
|
||||
../mldoc/it-1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/it.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁di', '▁e', "▁&'", "'", '▁il', '▁la', '▁in', 'e']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_rnd-nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.357973 1.353393 0.293000
|
||||
2 1.282061 1.535401 0.390000
|
||||
3 1.144333 1.480346 0.533000
|
||||
4 0.985930 1.360542 0.540000
|
||||
5 0.862014 1.285450 0.661000
|
||||
6 0.720891 1.140574 0.629000
|
||||
7 0.625376 0.840085 0.791000
|
||||
8 0.572435 0.828283 0.793000
|
||||
Total time: 01:21
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
Loss and accuracy using (cls_best): [0.5931235, tensor(0.7890)]
|
||||
Processing data/wiki/ja-100/models/sp15k/qrnn_rnd-nl4.m
|
||||
../mldoc/ja-1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/ja.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁、', '▁。', '▁の', '▁に', '▁を', '▁年', 'の', '▁は', '▁・', '▁)']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_rnd-nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.377756 1.402221 0.254000
|
||||
2 1.243837 7.998792 0.254000
|
||||
3 1.066383 2.645358 0.354000
|
||||
4 0.903686 1.348676 0.541000
|
||||
5 0.843216 0.945152 0.743000
|
||||
6 0.759674 0.801283 0.810000
|
||||
7 0.689767 0.786832 0.820000
|
||||
8 0.674777 0.778615 0.818000
|
||||
Total time: 02:48
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ja-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
Loss and accuracy using (cls_best): [0.5008588, tensor(0.8303)]
|
||||
Processing data/wiki/ru-100/models/sp15k/qrnn_rnd-nl4.m
|
||||
../mldoc/ru-1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 9195, val: 1021
|
||||
Running tokenization cls...
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_rnd-nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.394592 1.393761 0.265000
|
||||
2 1.381886 1.476836 0.293000
|
||||
3 1.258016 1.153065 0.555000
|
||||
4 1.105392 1.323574 0.556000
|
||||
5 0.948704 1.049486 0.703000
|
||||
6 0.848964 1.480141 0.605000
|
||||
7 0.757975 1.001765 0.723000
|
||||
8 0.684587 0.982136 0.741000
|
||||
Total time: 03:07
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
Loss and accuracy using (cls_best): [0.7138667, tensor(0.7320)]
|
||||
Processing data/wiki/zh-100/models/sp15k/qrnn_rnd-nl4.m
|
||||
../mldoc/zh-1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
Training
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_rnd-nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.341714 1.208123 0.569000
|
||||
2 1.059330 1.169385 0.662000
|
||||
3 0.926222 0.771242 0.824000
|
||||
4 0.843994 1.997928 0.524000
|
||||
5 0.800537 0.874480 0.756000
|
||||
6 0.710552 0.909481 0.758000
|
||||
7 0.657595 0.719883 0.854000
|
||||
8 0.617662 0.727267 0.852000
|
||||
Total time: 02:20
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_rnd-nl4.m
|
||||
Loss and accuracy using (cls_best): [0.48266637, tensor(0.8453)]
|
||||
OrderedDict([('data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m',
|
||||
0.9024999737739563),
|
||||
('data/mldoc/en-1/models/sp15k/qrnn_rnd-nl4.m',
|
||||
0.8149999976158142),
|
||||
('data/mldoc/es-1/models/sp15k/qrnn_rnd-nl4.m',
|
||||
0.8964999914169312),
|
||||
('data/mldoc/fr-1/models/sp15k/qrnn_rnd-nl4.m',
|
||||
0.8220000267028809),
|
||||
('data/mldoc/it-1/models/sp15k/qrnn_rnd-nl4.m',
|
||||
0.7889999747276306),
|
||||
('data/mldoc/ja-1/models/sp15k/qrnn_rnd-nl4.m',
|
||||
0.8302500247955322),
|
||||
('data/mldoc/ru-1/models/sp15k/qrnn_rnd-nl4.m',
|
||||
0.7319999933242798),
|
||||
('data/mldoc/zh-1/models/sp15k/qrnn_rnd-nl4.m',
|
||||
0.8452500104904175)])
|
||||
data/mldoc/de-1/models/sp15k/qrnn_rnd-nl4.m: 0.9024999737739563
|
||||
data/mldoc/en-1/models/sp15k/qrnn_rnd-nl4.m: 0.8149999976158142
|
||||
data/mldoc/es-1/models/sp15k/qrnn_rnd-nl4.m: 0.8964999914169312
|
||||
data/mldoc/fr-1/models/sp15k/qrnn_rnd-nl4.m: 0.8220000267028809
|
||||
data/mldoc/it-1/models/sp15k/qrnn_rnd-nl4.m: 0.7889999747276306
|
||||
data/mldoc/ja-1/models/sp15k/qrnn_rnd-nl4.m: 0.8302500247955322
|
||||
data/mldoc/ru-1/models/sp15k/qrnn_rnd-nl4.m: 0.7319999933242798
|
||||
data/mldoc/zh-1/models/sp15k/qrnn_rnd-nl4.m: 0.8452500104904175
|
||||
```
|
||||
@@ -1,174 +0,0 @@
|
||||
## 100 ex. pseudo labeling bootstrapping
|
||||
|
||||
|
||||
## Laser pseudo labeling bootstrapping
|
||||
```
|
||||
Processing data/mldoc/de-1-laser-en1/models/sp15k/qrnn_nl4.m
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1-laser-en1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1-laser-en1/models/sp15k/qrnn_nl4.m
|
||||
Generating pseduolabels data/mldoc/de-1-laser-en1-ps
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/de-1-laser-en1/de.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', 'en', '▁und', 's', '▁in', 'er', "▁&'"]
|
||||
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
Your compiler (c++) may be ABI-incompatible with PyTorch!
|
||||
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
|
||||
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
|
||||
|
||||
See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
|
||||
for instructions on how to install GCC 4.9 or higher.
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
|
||||
Generating train dataset of size 1000, the accuracy is 0.997
|
||||
0 1 preds
|
||||
0 3 Tokio (Reuter) - Der Dollar ist am Donnerstag ... 3
|
||||
1 3 Kairo (Reuter) - Die ägyptische Zentralbank se... 3
|
||||
2 2 Bonn (Reuter) - Wegen einer Bombendrohung ist ... 2
|
||||
3 0 Berlin (Reuter) - Die Bahn AG will mit Hilfe p... 0
|
||||
4 3 08.15 Uhr MEZ - Deutsche Aktien nach den Rekor... 3
|
||||
Generating dev dataset of size 1000, the accuracy is 0.91
|
||||
0 1 preds
|
||||
0 1 New York (Reuter) - Das Vertrauen der US-Verbr... 1
|
||||
1 2 Tokio (Reuter) - Russische Patrouillenboote ha... 2
|
||||
2 2 Paris (Reuter) - Bei der Volksabstimmung in Al... 2
|
||||
3 2 Belgrad (Reuter) - Die serbische Polizei hat n... 2
|
||||
4 0 München (Reuter) - Der Stuttgarter Bosch-Konze... 0
|
||||
Processing data/mldoc/es-1-laser-en1/models/sp15k/qrnn_nl4.m
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1-laser-en1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1-laser-en1/models/sp15k/qrnn_nl4.m
|
||||
Generating pseduolabels data/mldoc/es-1-laser-en1-ps
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1-laser-en1/es.dev.csv
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁la', '▁.', 's', '▁en', '▁el', '▁y', '▁a', '▁que']
|
||||
Generating train dataset of size 1000, the accuracy is 0.988
|
||||
0 1 preds
|
||||
0 3 LONDRES, 5 sep (Reuter) - El dólar se mantenía... 3
|
||||
1 2 MADRID, 30 dic (Reuter) - La Generalitat de Va... 2
|
||||
2 3 PARIS, 30 jun (Reuter) - La Bolsa de París neg... 3
|
||||
3 0 MADRID, 23 dic (Reuter) - La agencia de valore... 0
|
||||
4 0 MADRID, 4 Feb (Reuter) - El Banco Bilbao Vizca... 0
|
||||
Generating dev dataset of size 1000, the accuracy is 0.879
|
||||
0 1 preds
|
||||
0 0 NUEVA YORK, 11 abr (Reuter) - MCI Communicatio... 0
|
||||
1 3 FRANCFORT, 17 jun (Reuter) - La Bolsa de Franc... 3
|
||||
2 2 BONN, 3 jun (Reuter) - Un destacado miembro de... 1
|
||||
3 2 LONDRES, 3 sep (Reuter) - El secretario de Def... 2
|
||||
4 3 MADRID, 3 oct (Reuter) - Las acciones de Pryca... 3
|
||||
Processing data/mldoc/fr-1-laser-en1/models/sp15k/qrnn_nl4.m
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1-laser-en1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1-laser-en1/models/sp15k/qrnn_nl4.m
|
||||
Generating pseduolabels data/mldoc/fr-1-laser-en1-ps
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/fr-1-laser-en1/fr.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', 's', '▁.', "'", '▁la', '▁le', '▁et', '▁l', '▁à']
|
||||
Generating train dataset of size 1000, the accuracy is 0.993
|
||||
0 1 preds
|
||||
0 2 WASHINGTON, 13 septembre, Reuter - Les Etats-U... 2
|
||||
1 1 PARIS, 10 juillet, Reuter - L'audit des financ... 1
|
||||
2 2 MOSCOU, 29 mai, Reuter - Après l'accord interv... 2
|
||||
3 2 PARIS, 1er octobre, Reuter - Le groupe communi... 2
|
||||
4 0 LONDRES, 3 juin, Reuter - National Grid Group ... 0
|
||||
Generating dev dataset of size 1000, the accuracy is 0.887
|
||||
0 1 preds
|
||||
0 1 PARIS, 30 décembre, Reuter - Zodiac . Chiffre ... 0
|
||||
1 0 AJACCIO, 11 décembre, Reuter - Une charge de 7... 2
|
||||
2 0 BRUXELLES, 26 décembre, Reuter - 1997 s'annonc... 0
|
||||
3 0 PARIS, 26 septembre, Reuter - Alcatel Alsthom ... 0
|
||||
4 1 NEW YORK, 25 octobre, Reuter - La hausse plus ... 1
|
||||
Processing data/mldoc/it-1-laser-en1/models/sp15k/qrnn_nl4.m
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1-laser-en1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1-laser-en1/models/sp15k/qrnn_nl4.m
|
||||
Generating pseduolabels data/mldoc/it-1-laser-en1-ps
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1-laser-en1/it.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁di', '▁e', "▁&'", "'", '▁il', '▁la', '▁in', 'e']
|
||||
Generating train dataset of size 1000, the accuracy is 0.987
|
||||
0 1 preds
|
||||
0 3 MILANO, 6 nov (Reuter) - La lira recupera ai p... 3
|
||||
1 1 MILANO, 20 giugno (Reuter) - Lo stacco dividen... 1
|
||||
2 3 MILANO, 20 set (Reuter) - Olivetti entra nel t... 3
|
||||
3 1 LONDRA, 2 aprile (Reuter) - L'aggregato moneta... 1
|
||||
4 3 Oro Londra fix ore 10,30 - 4 nov - $378,65. (c... 3
|
||||
Generating dev dataset of size 1000, the accuracy is 0.819
|
||||
0 1 preds
|
||||
0 0 L'istituto prevede un aumento dell'utile opera... 0
|
||||
1 1 FRANCOFORTE, 18 dic (Reuter) - La Bundesbank a... 1
|
||||
2 1 TOKIO, 28 agosto (Reuter) - Il ministro delle ... 1
|
||||
3 1 ROMA, 23 luglio (Reuter) - Il presidente del C... 1
|
||||
4 1 MONACO, 19 marzo (Reuter) - Il ministro delle ... 1
|
||||
Processing data/mldoc/ru-1-laser-en1/models/sp15k/qrnn_nl4.m
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1-laser-en1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1-laser-en1/models/sp15k/qrnn_nl4.m
|
||||
Generating pseduolabels data/mldoc/ru-1-laser-en1-ps
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1-laser-en1/ru.dev.csv
|
||||
Data lm, trn: 9195, val: 1021
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Generating train dataset of size 1000, the accuracy is 0.996
|
||||
0 1 preds
|
||||
0 0 КИЕВ, 20 июн (Рейтер) - Нацбанк Украины планир... 0
|
||||
1 3 МИНСК, 13 фев (Рейтер) - Курс белорусского руб... 3
|
||||
2 0 САНКТ-ПЕТЕРБУРГ, 25 авг (Рейтер) - Астробанк (... 0
|
||||
3 0 MOSCOW, Feb 7 (Reuter) - U.S. plane-maker Boei... 0
|
||||
4 2 В данном обзоре казахстанской прессы приводитс... 2
|
||||
Generating dev dataset of size 1000, the accuracy is 0.837
|
||||
0 1 preds
|
||||
0 0 ТБИЛИСИ, 25 мар (Рейтер) - Партнерский Фонд, с... 0
|
||||
1 3 МОСКВА, 3 ноя (Рейтер) - Казахстанская Межбанк... 3
|
||||
2 1 КИЕВ, 25 июл (Рейтер) - Нацбанк Украины рассмо... 1
|
||||
3 0 МОСКВА, 2 дек (Рейтер) - АО Уралсвязьинформ пр... 0
|
||||
4 2 В данном обзоре киргизской прессы приводится к... 2
|
||||
Processing data/mldoc/zh-1-laser-en1/models/sp15k/qrnn_nl4.m
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-en1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-en1/models/sp15k/qrnn_nl4.m
|
||||
Generating pseduolabels data/mldoc/zh-1-laser-en1-ps
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1-laser-en1/zh.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有']
|
||||
Generating train dataset of size 1000, the accuracy is 0.992
|
||||
0 1 preds
|
||||
0 1 〔路透社紐約10日電〕 芝加哥聯邦準備銀行總裁墨斯克週四表示,他預期1997年國內生產總值... 1
|
||||
1 0 〔路透社台北14日電〕台灣合作金庫週四將2週､1個月及2個月內的附條件交易利率全開在5.20... 0
|
||||
2 1 〔路透社倫敦6日電〕 在英國工黨政府賦予央行利率自主權後,英國央行在其新的首次貨幣政策委員... 1
|
||||
3 3 〔路透社東京4日電〕 東京股市週一收盤下跌,但在短暫跌破關鍵支撐19,500點後縮減跌幅.... 3
|
||||
4 2 美國總統克林頓接受明報訪問時表示,美國是貫徹始終地支持中英''聯合聲明''作為香港未來的基石... 2
|
||||
Generating dev dataset of size 1000, the accuracy is 0.817
|
||||
0 1 preds
|
||||
0 0 〔路透社台北20日電〕 台灣塑膠類週一早盤上漲,經紀商表示,主要是因為近期原物料價格上漲及... 0
|
||||
1 2 〔路透社華盛頓2日電〕比利時央行總裁弗沛雷茲週三表示,義大利里拉被低估,但美元可望攀升. ... 1
|
||||
2 0 〔路透社吉隆坡29日電〕 吉隆坡股市周二收市微升.分析師指二線股有散戶吸納,助長市場升勢,... 3
|
||||
3 2 〔路透社香港26日電〕 香港明報周四報導,面對台灣當局的"務實外交",和"台獨"傾向,中國... 2
|
||||
4 0 [路透社上海6日電] 據上海証券報周五報導,有關專家就滬市四家上市公司法人股通過拍賣進... 0
|
||||
Python 3.7.0 (default, Oct 9 2018, 10:31:47)
|
||||
Type 'copyright', 'credits' or 'license' for more information
|
||||
```
|
||||
@@ -1,86 +0,0 @@
|
||||
# QRNN DE
|
||||
## SP30k nl
|
||||
### LM
|
||||
```
|
||||
python -m ulmfit lm --dataset-path data/wiki/de-100 --bidir=False --qrnn=True --nl 4 --tokenizer='sp' --max-vocab 30000 --lang de --name 'nl4' --cuda-id=0 - train 10 --drop-mult=0 --bs=50
|
||||
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', '▁und', '▁in', "▁&'", 'en', 's', '-']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.790653 2.867094 0.511392
|
||||
2 2.742032 2.843288 0.510885
|
||||
3 2.696114 2.833874 0.512062
|
||||
4 2.671780 2.786312 0.516448
|
||||
5 2.611292 2.725993 0.522723
|
||||
6 2.542737 2.655713 0.530968
|
||||
7 2.572076 2.582141 0.539928
|
||||
8 2.465960 2.509654 0.549987
|
||||
9 2.405682 2.448580 0.558674
|
||||
10 2.339395 2.428111 0.562502
|
||||
```
|
||||
|
||||
### MLDocs
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/de-1 --cuda-id=0 --base-lm-path data-filtered/data/wiki/de-100/models/sp30k/qrnn_nl4.m --lang=de --name 'nl4' - train 20 --bs 40 --cls-max-len 700
|
||||
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/marcin/github/n-waves/ulmfit-multilingual/data/mldoc/de-1/models/sp30k
|
||||
Model dir: /home/marcin/github/n-waves/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/qrnn_nl4.m
|
||||
Loading validation /home/marcin/github/n-waves/ulmfit-multilingual/data/mldoc/de-1/de.dev.csv
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 13500, cls.val 1500
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁.', '▁,', '▁der', '▁die', '▁und', '▁in', "▁&'", 'en', 's', '-']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/marcin/github/n-waves/ulmfit-multilingual/data-filtered/data/wiki/de-100/models/sp30k/qrnn_nl4.m/lm_best'), PosixPath('/home/marcin/github/n-waves/ulmfit-multilingual/data-filtered/data/wiki/de-100/models/sp30k/qrnn_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.450698 2.601732 0.527671
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.888087 2.477170 0.542949
|
||||
2 2.621279 2.300024 0.568743
|
||||
3 2.313220 2.120824 0.592728
|
||||
4 2.176746 1.973596 0.613343
|
||||
5 2.114441 1.857317 0.628628
|
||||
6 2.022593 1.765069 0.642017
|
||||
7 1.936942 1.696150 0.651549
|
||||
8 1.860200 1.622848 0.661923
|
||||
9 1.795039 1.549579 0.673416
|
||||
10 1.740739 1.500053 0.681305
|
||||
11 1.695835 1.448141 0.689201
|
||||
12 1.605702 1.402924 0.697096
|
||||
13 1.582328 1.354327 0.706123
|
||||
14 1.548034 1.316290 0.712870
|
||||
15 1.496170 1.282155 0.719413
|
||||
16 1.514243 1.255556 0.724801
|
||||
17 1.482411 1.236461 0.728380
|
||||
18 1.458308 1.223498 0.730708
|
||||
19 1.422691 1.218288 0.731713
|
||||
20 1.380592 1.217068 0.731893
|
||||
/home/marcin/github/n-waves/ulmfit-multilingual/data/mldoc/de-1/models/sp30k
|
||||
Saving info /home/marcin/github/n-waves/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/qrnn_nl4.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.529402 0.376163 0.900000
|
||||
Better model found at epoch 1 with val_loss value: 0.3761630356311798.
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.290838 0.252989 0.916000
|
||||
Better model found at epoch 1 with val_loss value: 0.25298893451690674.
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.184352 0.204892 0.941000
|
||||
Better model found at epoch 1 with val_loss value: 0.20489171147346497.
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.113328 0.204136 0.947000
|
||||
Better model found at epoch 1 with val_loss value: 0.20413607358932495.
|
||||
2 0.106220 0.200674 0.949000
|
||||
Better model found at epoch 2 with val_loss value: 0.20067360997200012.
|
||||
Saving models at /home/marcin/github/n-waves/ulmfit-multilingual/data/mldoc/de-1/models/sp30k/qrnn_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.15208693, tensor(0.9532)]
|
||||
0.15208692848682404
|
||||
0.953249990940094
|
||||
```
|
||||
@@ -1,141 +0,0 @@
|
||||
# QRNN EN
|
||||
|
||||
## SP15k nl 4
|
||||
```
|
||||
python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4' --m
|
||||
ax-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0
|
||||
Max vocab: 15000
|
||||
Cache dir: data/wiki/en-100/models/sp15k
|
||||
Model dir: data/wiki/en-100/models/sp15k/qrnn_nl4.m
|
||||
Wiki text was split to 28476 articles
|
||||
Wiki text was split to 60 articles
|
||||
Data lm, trn: 28476, val: 60
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁the', '▁,', 's', '▁.', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.080874 3.197244 0.431796
|
||||
2 3.021043 3.147150 0.433593
|
||||
3 2.933366 3.125982 0.435766
|
||||
4 2.905764 3.103272 0.437356
|
||||
5 2.867981 3.032923 0.445030
|
||||
6 2.815294 2.958662 0.453979
|
||||
7 2.733671 2.869483 0.466015
|
||||
8 2.744779 2.785220 0.475833
|
||||
9 2.717722 2.704370 0.487687
|
||||
10 2.666089 2.675301 0.493602
|
||||
Total time: 9:07:27
|
||||
data/wiki/en-100/models/sp15k
|
||||
Saving info data/wiki/en-100/models/sp15k/qrnn_nl4.m/info.json
|
||||
```
|
||||
|
||||
|
||||
|
||||
## SP30k nl 4
|
||||
### LM
|
||||
|
||||
```
|
||||
python -m ulmfit lm --dataset-path data/wiki/wikitext-103 --bidir=False --qrnn=True --nl 4 --tokenizer='sp' --max-vocab 30000 --lang en --name 'nl4' --cuda-id=1 - train 10 --drop-mult=0 --bs=50
|
||||
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁the', '▁,', '▁.', 's', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.5} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.184221 3.256314 0.438527
|
||||
2 3.084555 3.241498 0.435628
|
||||
3 3.099060 3.258447 0.435060
|
||||
4 3.119621 3.220939 0.437597
|
||||
5 3.073662 3.165012 0.445108
|
||||
6 2.938047 3.086962 0.452921
|
||||
7 2.920506 2.998151 0.462940
|
||||
8 2.920506 2.899240 0.474378
|
||||
9 2.862836 2.835098 0.485305
|
||||
10 2.891070 2.810929 0.489867
|
||||
```
|
||||
|
||||
### LM, BS=128, drop-mult=0.5
|
||||
```
|
||||
python -m ulmfit lm --dataset-path data/wiki/wikitext-103 --bidir=False --qrnn=True --nl 4 --tokenizer='sp' --max-vocab 30000 --lang en --name 'nl4-bs128' --cuda-id=1 - train 10 --drop-mult=0.5 --bs=128
|
||||
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁the', '▁,', '▁.', 's', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.5} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.413345 3.280860 0.433011
|
||||
2 3.219606 3.129479 0.444172
|
||||
3 3.136091 3.094905 0.448493
|
||||
4 3.145281 3.033001 0.452830
|
||||
5 3.100366 2.980189 0.458984
|
||||
6 3.062894 2.923044 0.464841
|
||||
7 3.001627 2.834753 0.475316
|
||||
8 2.979051 2.792044 0.480915
|
||||
9 2.933140 2.733279 0.488346
|
||||
10 2.964397 2.720861 0.490423
|
||||
```
|
||||
|
||||
### MLDocs
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/en-1 --cuda-id=0 --base-lm-path data-filtered/data/wiki/wikitext-103/models/sp30k/qrnn_nl4.m --lang=en --name 'nl4' - train 20 --bs 40 --cls-max-len 700
|
||||
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/marcin/github/n-waves/ulmfit-multilingual/data/mldoc/en-1/models/sp30k
|
||||
Model dir: /home/marcin/github/n-waves/ulmfit-multilingual/data/mldoc/en-1/models/sp30k/qrnn_nl4.m
|
||||
Loading validation /home/marcin/github/n-waves/ulmfit-multilingual/data/mldoc/en-1/en.dev.csv
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 13500, cls.val 1500
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁the', '▁,', '▁.', 's', '▁of', '▁and', '▁in', '▁to', '▁a', 'ed']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/marcin/github/n-waves/ulmfit-multilingual/data-filtered/data/wiki/wikitext-103/models/sp30k/qrnn_nl4.m/lm_best'), PosixPath('/home/marcin/github/n-waves/ulmfit-multilingual/data-filtered/data/wiki/wikitext-103/models/sp30k/qrnn_nl4.m/.
|
||||
./itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.459886 3.692770 0.364677
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.962907 3.560222 0.379027
|
||||
2 3.673292 3.378484 0.402066
|
||||
3 3.460093 3.191662 0.424295
|
||||
4 3.296515 3.030681 0.442995
|
||||
5 3.161650 2.891829 0.459052
|
||||
6 3.022674 2.776469 0.473280
|
||||
7 2.974365 2.686321 0.484403
|
||||
8 2.869587 2.593854 0.496297
|
||||
9 2.785321 2.509093 0.506853
|
||||
10 2.677728 2.440328 0.516178
|
||||
11 2.641243 2.371950 0.525810
|
||||
12 2.652385 2.320008 0.533105
|
||||
13 2.547195 2.261057 0.542046
|
||||
14 2.491570 2.216933 0.548810
|
||||
15 2.454437 2.179364 0.555077
|
||||
16 2.414449 2.147612 0.559972
|
||||
17 2.358593 2.125351 0.563405
|
||||
18 2.362696 2.111580 0.565614
|
||||
19 2.341626 2.104268 0.566749
|
||||
20 2.342680 2.102918 0.566966
|
||||
/home/marcin/github/n-waves/ulmfit-multilingual/data/mldoc/en-1/models/sp30k
|
||||
Saving info /home/marcin/github/n-waves/ulmfit-multilingual/data/mldoc/en-1/models/sp30k/qrnn_nl4.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.622379 0.422002 0.882000
|
||||
Better model found at epoch 1 with val_loss value: 0.42200201749801636.
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.313018 0.275563 0.908000
|
||||
Better model found at epoch 1 with val_loss value: 0.27556276321411133.
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.241521 0.174606 0.933000
|
||||
Better model found at epoch 1 with val_loss value: 0.1746061146259308.
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.125556 0.170286 0.940000
|
||||
Better model found at epoch 1 with val_loss value: 0.17028628289699554.
|
||||
2 0.107322 0.181366 0.939000
|
||||
Saving models at /home/marcin/github/n-waves/ulmfit-multilingual/data/mldoc/en-1/models/sp30k/qrnn_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.18917121, tensor(0.9388)]
|
||||
0.1891712099313736
|
||||
0.9387500286102295
|
||||
```
|
||||
@@ -1,127 +0,0 @@
|
||||
# QRNN ES
|
||||
|
||||
## SP15k nl 4
|
||||
``
|
||||
export CUDA_VISIBLE_DEVICES=1
|
||||
LANG=es
|
||||
python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0
|
||||
|
||||
Wiki text was split to 161509 articles
|
||||
Wiki text was split to 78 articles
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 161509, val: 78
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁.', '▁la', 's', '▁el', '▁en', '▁y', '▁a', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
|
||||
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.851575 3.398695 0.372940
|
||||
2 2.801543 3.353015 0.372648
|
||||
3 2.807216 3.290132 0.380787
|
||||
4 2.696361 3.220115 0.388937
|
||||
5 2.668488 3.132770 0.399528
|
||||
6 2.565685 3.062742 0.408880
|
||||
7 2.503054 2.985069 0.419262
|
||||
8 2.448338 2.895266 0.431797
|
||||
9 2.411213 2.829787 0.441973
|
||||
10 2.403536 2.811063 0.445468
|
||||
Total time: 11:52:32
|
||||
data/wiki/es-100/models/sp15k
|
||||
Saving info data/wiki/es-100/models/sp15k/qrnn_nl4.m/info.json
|
||||
``
|
||||
|
||||
```bash
|
||||
export CUDA_VISIBLE_DEVICES=1
|
||||
LANG=es
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_nl4.m --lang=${LANG} --name 'nl4' - train 20 --bs 20 --num-cls-epochs=8
|
||||
```
|
||||
|
||||
|
||||
## SP30k nl 4
|
||||
### LM
|
||||
```
|
||||
python -m ulmfit lm --dataset-path data/wiki/es-100 --bidir=False --qrnn=True --nl 4 --tokenizer='sp' --max-vocab 30000 --lang es --name 'nl4' --cuda-id=0 - train 10 --drop-mult=0 --bs=50
|
||||
|
||||
Wiki text was split to 161509 articles
|
||||
Wiki text was split to 78 articles
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁.', '▁la', '▁el', '▁en', '▁y', 's', '▁a', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.067289 3.640350 0.357276
|
||||
2 2.958243 3.619773 0.358111
|
||||
3 3.033412 3.587700 0.359495
|
||||
4 2.933573 3.525202 0.367685
|
||||
5 2.904549 3.467990 0.372583
|
||||
6 2.798806 3.409506 0.380045
|
||||
7 2.733132 3.303108 0.391922
|
||||
8 2.675272 3.224150 0.401143
|
||||
9 2.635299 3.166430 0.410160
|
||||
10 2.656724 3.145599 0.413176
|
||||
```
|
||||
|
||||
### MLDocs
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/es-1 --cuda-id=0 --base-lm-path data-filtered/data/wiki/es-100/models/sp30k/qrnn_nl4.m --lang=es --name 'nl4' - train 20 --bs 40 --cls-max-len 700
|
||||
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/marcin/github/n-waves/ulmfit-multilingual/data/mldoc/es-1/models/sp30k
|
||||
Model dir: /home/marcin/github/n-waves/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/qrnn_nl4.m
|
||||
Loading validation /home/marcin/github/n-waves/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 13013, cls.val 1445
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁de', '▁,', '▁.', '▁la', '▁el', '▁en', '▁y', 's', '▁a', "▁&'"]
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/marcin/github/n-waves/ulmfit-multilingual/data-filtered/data/wiki/es-100/models/sp30k/qrnn_nl4.m/lm_best'), PosixPath('/home/marcin/github/n-waves/ulmfit-multilingual/data-filtered/data/wiki/es-100/models/sp30k/qrnn_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.352874 2.367255 0.514858
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.796090 2.203233 0.536513
|
||||
2 2.515840 1.970145 0.576640
|
||||
3 2.198857 1.774013 0.610990
|
||||
4 2.035614 1.633484 0.633450
|
||||
5 1.944539 1.535505 0.649110
|
||||
6 1.848854 1.451618 0.661764
|
||||
7 1.788579 1.382675 0.673166
|
||||
8 1.675414 1.320675 0.683617
|
||||
9 1.614536 1.264944 0.694086
|
||||
10 1.618723 1.215493 0.702936
|
||||
11 1.504875 1.164356 0.712921
|
||||
12 1.411316 1.126858 0.721374
|
||||
13 1.421174 1.079897 0.731196
|
||||
14 1.352116 1.044965 0.738148
|
||||
15 1.318876 1.013755 0.745312
|
||||
16 1.268569 0.986391 0.751383
|
||||
17 1.273424 0.971129 0.754643
|
||||
18 1.256196 0.960661 0.757439
|
||||
19 1.233202 0.955790 0.758405
|
||||
20 1.230536 0.955070 0.758496
|
||||
/home/marcin/github/n-waves/ulmfit-multilingual/data/mldoc/es-1/models/sp30k
|
||||
Saving info /home/marcin/github/n-waves/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/qrnn_nl4.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.739119 0.438338 0.867000
|
||||
Better model found at epoch 1 with val_loss value: 0.438338041305542.
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.425376 0.207067 0.950000
|
||||
Better model found at epoch 1 with val_loss value: 0.20706671476364136.
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.311269 0.172416 0.956000
|
||||
Better model found at epoch 1 with val_loss value: 0.17241604626178741.
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.226164 0.166543 0.958000
|
||||
Better model found at epoch 1 with val_loss value: 0.1665433794260025.
|
||||
2 0.199775 0.167683 0.956000
|
||||
Saving models at /home/marcin/github/n-waves/ulmfit-multilingual/data/mldoc/es-1/models/sp30k/qrnn_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.18184493, tensor(0.9448)]
|
||||
0.18184493482112885
|
||||
0.9447500109672546
|
||||
```
|
||||
@@ -1,30 +0,0 @@
|
||||
# IT
|
||||
## SP30k QRNN nl 4
|
||||
### LM
|
||||
```
|
||||
python -m ulmfit lm --dataset-path data/wiki/it-100/ --cuda-id=0 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 30000 --lang it --qrnn=True - train 10 --bs=50 --drop_mult=0
|
||||
Max vocab: 30000
|
||||
Cache dir: data/wiki/it-100/models/sp30k
|
||||
Model dir: data/wiki/it-100/models/sp30k/qrnn_nl4.m
|
||||
Tokenized data loaded
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁di', "▁&'", "'", '▁e', '▁il', '▁la', 'e', '▁in']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.354224 3.749085 0.350236
|
||||
2 3.274838 3.697026 0.351104
|
||||
3 3.222462 3.680071 0.352152
|
||||
4 3.217652 3.628976 0.357922
|
||||
5 3.117965 3.563592 0.364370
|
||||
6 3.075397 3.483997 0.372794
|
||||
7 3.002098 3.394749 0.383217
|
||||
8 2.936974 3.316284 0.393616
|
||||
9 2.843549 3.258448 0.401605
|
||||
10 2.818070 3.240303 0.404684
|
||||
Total time: 10:49:44
|
||||
data/wiki/it-100/models/sp30k
|
||||
Saving info data/wiki/it-100/models/sp30k/qrnn_nl4.m/info.json
|
||||
```
|
||||
|
||||
### MLDoc
|
||||
@@ -1,204 +0,0 @@
|
||||
# QRNN RU
|
||||
## SP15k nl4
|
||||
## LM
|
||||
export CUDA_VISIBLE_DEVICES=3
|
||||
LANG=ru
|
||||
python -m ulmfit lm --dataset-path data/wiki/ru-100 --bidir=False --qrnn=True --nl 4 --tokenizer='sp' --max-vocab 15000 --lang ru --name 'nl4' - train 10 --drop-mult=0 --bs=50 --label-smoothing-eps=0.1
|
||||
|
||||
## SP15k nl8
|
||||
### LM
|
||||
```
|
||||
5 2.869308 2.905951 0.466976
|
||||
6 2.768955 2.782804 0.481852
|
||||
7 2.654484 2.676304 0.495593
|
||||
8 2.585963 2.591748 0.508447
|
||||
9 2.512042 2.526819 0.518860
|
||||
10 2.520543 2.509287 0.521890
|
||||
Total time: 18:46:01
|
||||
data/wiki/ru-100/models/sp15k
|
||||
Saving info data/wiki/ru-100/models/sp15k/qrnn_nl8.m/info.json
|
||||
```
|
||||
### MLDoc
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_nl8.m --lang=${LANG} --name 'nl8' - train 20 --bs 20 --num-cls-epochs=8 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Model dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl8.m
|
||||
Loading validation /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 9195, val: 1021
|
||||
|
||||
Running tokenization cls...
|
||||
Data cls, trn: 1000, val: 1000
|
||||
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 1000, val: 4000
|
||||
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl8.m/lm_best'), PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl8.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.923423 2.334532 0.529978
|
||||
|
||||
Total time: 02:46
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.462077 2.150593 0.563281
|
||||
|
||||
2 2.230013 1.972095 0.596198
|
||||
|
||||
3 2.118523 1.812012 0.623204
|
||||
|
||||
4 1.916368 1.690016 0.644060
|
||||
|
||||
5 1.842718 1.585770 0.661704
|
||||
|
||||
6 1.748630 1.513972 0.674130
|
||||
|
||||
7 1.675032 1.447667 0.686207
|
||||
|
||||
8 1.628485 1.393949 0.695972
|
||||
|
||||
9 1.564814 1.330838 0.707272
|
||||
|
||||
10 1.553933 1.283114 0.715716
|
||||
|
||||
11 1.441891 1.234810 0.726201
|
||||
|
||||
12 1.496388 1.185676 0.735977
|
||||
|
||||
13 1.383019 1.141014 0.745528
|
||||
|
||||
14 1.256620 1.094201 0.755120
|
||||
|
||||
15 1.306187 1.052457 0.764280
|
||||
|
||||
16 1.297933 1.028387 0.769747
|
||||
|
||||
17 1.319773 1.004256 0.775285
|
||||
|
||||
18 1.178073 0.989788 0.778480
|
||||
|
||||
19 1.252248 0.982740 0.780057
|
||||
|
||||
20 1.177640 0.981201 0.780267
|
||||
|
||||
Total time: 1:24:58
|
||||
/home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Saving info /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl8.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.882775 0.510930 0.826000
|
||||
|
||||
2 0.683476 0.513669 0.847000
|
||||
|
||||
3 0.556661 0.590375 0.839000
|
||||
|
||||
4 0.454019 0.757216 0.828000
|
||||
|
||||
5 0.344460 0.549675 0.870000
|
||||
|
||||
6 0.246039 0.630242 0.861000
|
||||
|
||||
7 0.173423 0.649066 0.858000
|
||||
|
||||
8 0.098640 0.638015 0.867000
|
||||
|
||||
Total time: 05:11
|
||||
Saving models at /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl8.m
|
||||
Loss and accuracy using (cls_best): [0.64393336, tensor(0.8683)]
|
||||
```
|
||||
|
||||
### MLDoc nl8 -2nd
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME}-2 - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0
|
||||
.1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Model dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl8-2.m
|
||||
Loading validation /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
|
||||
Data lm, trn: 9195, val: 1021
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl8.m/lm_best'), PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl8.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.966292 3.450758 0.527065
|
||||
Total time: 02:58
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.495761 3.276329 0.560047
|
||||
2 3.319947 3.102911 0.593742
|
||||
3 3.137904 2.955317 0.620171
|
||||
4 3.040286 2.839161 0.642270
|
||||
5 2.869962 2.753622 0.658331
|
||||
6 2.905739 2.680881 0.672860
|
||||
7 2.836454 2.620925 0.685026
|
||||
8 2.857271 2.569716 0.695722
|
||||
9 2.702872 2.520050 0.705589
|
||||
10 2.701559 2.473591 0.715346
|
||||
11 2.740815 2.429558 0.725597
|
||||
12 2.646513 2.389550 0.735010
|
||||
13 2.587685 2.349614 0.744885
|
||||
14 2.546527 2.311087 0.754463
|
||||
15 2.568136 2.278581 0.762980
|
||||
16 2.492115 2.252367 0.769275
|
||||
17 2.338561 2.230529 0.775072
|
||||
18 2.447506 2.218215 0.778437
|
||||
19 2.364424 2.212115 0.780085
|
||||
20 2.367132 2.210520 0.780424
|
||||
Total time: 1:30:47
|
||||
/home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Saving info /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl8-2.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.969255 0.769736 0.843000
|
||||
2 0.846340 0.813483 0.839000
|
||||
3 0.718175 0.705339 0.867000
|
||||
4 0.609513 0.726442 0.875000
|
||||
Total time: 02:54
|
||||
Saving models at /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl8-2.m
|
||||
Loss and accuracy using (cls_best): [0.4056449, tensor(0.8698)]
|
||||
0.40564489364624023
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## cls
|
||||
```
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
LANG=ru
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_nl8.m --lang=${LANG} --name 'nl8' - train 20 --bs 20 --num-cls-epochs=8 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
```
|
||||
|
||||
## SP30k nl4
|
||||
### LM
|
||||
```
|
||||
python -m ulmfit lm --dataset-path data/wiki/ru-100 --bidir=False --qrnn=True --nl 4 --tokenizer='sp' --max-vocab 30000 --lang ru --name 'nl4' --cuda-id=0 - train 10 --drop-mult=0 --bs=50
|
||||
|
||||
Wiki text was split to 193047 articles
|
||||
Wiki text was split to 460 articles
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', '▁и', 'е', 'и', 'й', '▁на', '▁с']
|
||||
Training args: {'clip': 0.12, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.273207 3.350111 0.429702
|
||||
2 3.169897 3.274238 0.433682
|
||||
3 3.162197 3.247077 0.435900
|
||||
4 3.131630 3.168798 0.445252
|
||||
5 3.042942 3.096774 0.453532
|
||||
6 2.950550 3.002989 0.465113
|
||||
7 2.833593 2.902871 0.478954
|
||||
8 2.829737 2.805592 0.492138
|
||||
9 2.746991 2.733609 0.503711
|
||||
10 2.687201 2.708546 0.508050
|
||||
```
|
||||
@@ -1,136 +0,0 @@
|
||||
|
||||
#
|
||||
```
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
LANG=zh
|
||||
python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0
|
||||
|
||||
Wiki text was split to 103929 articles
|
||||
Wiki text was split to 113 articles
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 103929, val: 113
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁是', '▁中', '▁有']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
Your compiler (c++) may be ABI-incompatible with PyTorch!
|
||||
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
|
||||
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
|
||||
|
||||
See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
|
||||
for instructions on how to install GCC 4.9 or higher.
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.489521 2.734049 0.482433
|
||||
2 2.427567 2.662464 0.488089
|
||||
3 2.415744 2.613971 0.494118
|
||||
4 2.334062 2.560180 0.501209
|
||||
5 2.343723 2.503271 0.507307
|
||||
6 2.260171 2.444533 0.516768
|
||||
7 2.198721 2.367407 0.526631
|
||||
8 2.161857 2.308182 0.535856
|
||||
9 2.142125 2.252678 0.544535
|
||||
10 2.087831 2.234440 0.548529
|
||||
Total time: 11:01:47
|
||||
data/wiki/zh-100/models/sp15k
|
||||
Saving info data/wiki/zh-100/models/sp15k/qrnn_nl4.m/info.json
|
||||
```
|
||||
|
||||
## MLDoc
|
||||
```bash
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
LANG=zh
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_nl4.m --lang=${LANG} --name 'nl4' - train 20 --bs 20 --num-cls-epochs=8
|
||||
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/zh-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/zh-100/models/sp15k/qrnn_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.723684 2.148748 0.571206
|
||||
Total time: 02:13
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.157829 1.937637 0.601026
|
||||
2 1.898958 1.712967 0.637379
|
||||
3 1.722818 1.547745 0.664276
|
||||
4 1.570266 1.427551 0.682546
|
||||
5 1.503477 1.344690 0.696379
|
||||
6 1.434701 1.289549 0.704813
|
||||
7 1.425267 1.217570 0.717714
|
||||
8 1.373606 1.174655 0.725217
|
||||
9 1.297397 1.116406 0.735997
|
||||
10 1.211259 1.062999 0.745848
|
||||
11 1.248108 1.024482 0.754134
|
||||
12 1.198918 0.980273 0.762664
|
||||
13 1.121848 0.937985 0.771961
|
||||
14 1.111386 0.898821 0.780796
|
||||
15 1.120596 0.866009 0.787908
|
||||
16 1.056925 0.836998 0.794833
|
||||
17 1.020636 0.816387 0.799694
|
||||
18 1.002068 0.802623 0.802859
|
||||
19 0.998480 0.796877 0.804212
|
||||
20 0.959919 0.794685 0.804594
|
||||
Total time: 1:02:57
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp15k/qrnn_nl4.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.666322 0.433893 0.855000
|
||||
Total time: 00:08
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.448371 0.317440 0.889000
|
||||
Total time: 00:09
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.336693 0.309876 0.900000
|
||||
Total time: 00:10
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.266735 0.302003 0.903000
|
||||
2 0.222821 0.294501 0.905000
|
||||
3 0.207295 0.293751 0.908000
|
||||
4 0.179668 0.296945 0.911000
|
||||
5 0.153803 0.293158 0.911000
|
||||
Traceback (most recent call last):
|
||||
File "/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/runpy.py", line 193, in _run_module_as_main
|
||||
"__main__", mod_spec)
|
||||
File "/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/runpy.py", line 85, in _run_code
|
||||
exec(code, run_globals)
|
||||
File "/home/pczapla/workspace/ulmfit-multilingual/ulmfit/__main__.py", line 73, in <module>
|
||||
fire.Fire(ULMFiT())
|
||||
File "/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/fire/core.py", line 127, in Fire
|
||||
component_trace = _Fire(component, args, context, name)
|
||||
File "/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/fire/core.py", line 366, in _Fire
|
||||
component, remaining_args)
|
||||
File "/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/fire/core.py", line 542, in _CallCallable
|
||||
result = fn(*varargs, **kwargs)
|
||||
File "/home/pczapla/workspace/ulmfit-multilingual/ulmfit/train_clas.py", line 54, in train_cls
|
||||
learn.fit_one_cycle(num_cls_epochs, slice(1e-3 / (2.6 ** 4), 1e-3), moms=(0.8, 0.7))
|
||||
File "/home/pczapla/workspace/_oss/fastai/fastai/fastai/train.py", line 22, in fit_one_cycle
|
||||
learn.fit(cyc_len, max_lr, wd=wd, callbacks=callbacks)
|
||||
File "/home/pczapla/workspace/_oss/fastai/fastai/fastai/basic_train.py", line 178, in fit
|
||||
callbacks=self.callbacks+callbacks)
|
||||
File "/home/pczapla/workspace/_oss/fastai/fastai/fastai/utils/mem.py", line 77, in wrapper
|
||||
return func(*args, **kwargs)
|
||||
File "/home/pczapla/workspace/_oss/fastai/fastai/fastai/basic_train.py", line 90, in fit
|
||||
loss = loss_batch(model, xb, yb, loss_func, opt, cb_handler)
|
||||
File "/home/pczapla/workspace/_oss/fastai/fastai/fastai/basic_train.py", line 20, in loss_batch
|
||||
out = model(*xb)
|
||||
File "/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 477, in __call__
|
||||
result = self.forward(*input, **kwargs)
|
||||
File "/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
|
||||
input = module(input)
|
||||
File "/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 477, in __call__
|
||||
result = self.forward(*input, **kwargs)
|
||||
File "/home/pczapla/workspace/_oss/fastai/fastai/fastai/text/learner.py", line 235, in forward
|
||||
return self.concat(raw_outputs), self.concat(outputs)
|
||||
File "/home/pczapla/workspace/_oss/fastai/fastai/fastai/text/learner.py", line 221, in concat
|
||||
return [torch.cat([l[si] for l in arrs], dim=1) for si in range_of(arrs[0])]
|
||||
File "/home/pczapla/workspace/_oss/fastai/fastai/fastai/text/learner.py", line 221, in <listcomp>
|
||||
return [torch.cat([l[si] for l in arrs], dim=1) for si in range_of(arrs[0])]
|
||||
RuntimeError: CUDA error: out of memory
|
||||
```
|
||||
@@ -1,319 +0,0 @@
|
||||
# RU
|
||||
## SP15k nl4 QRNN
|
||||
```
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.053061 3.070487 0.450466
|
||||
2 2.874137 2.999093 0.455027
|
||||
3 2.864496 2.969308 0.458116
|
||||
4 2.890568 2.903564 0.466970
|
||||
5 2.746530 2.839789 0.474205
|
||||
6 2.683900 2.750476 0.486806
|
||||
7 2.674458 2.658535 0.499701
|
||||
8 2.595780 2.573735 0.512515
|
||||
9 2.530827 2.512999 0.522372
|
||||
10 2.505664 2.491850 0.526431
|
||||
Total time: 10:43:03
|
||||
data/wiki/ru-100/models/sp15k
|
||||
Saving info data/wiki/ru-100/models/sp15k/qrnn_nl4.m/info.json
|
||||
```
|
||||
```bash
|
||||
python -m ulmfit cls --dataset-path data/mldoc/ru-1 --base-lm-path data/wiki/ru-100/models/sp30k/lstm_nl4.m --lang=ru --name 'nl4-100' --cuda-id=1 - train 20 --bs 40 --num-cls-epochs=2
|
||||
|
||||
```
|
||||
|
||||
## SP25k qrnn
|
||||
### LM
|
||||
```bash
|
||||
python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name
|
||||
'nl4' --max-vocab 25000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0 --label-smoothing-eps=0.10 --tokenizer='sp
|
||||
Max vocab: 25000
|
||||
Cache dir: data/wiki/ru-100/models/sp25k
|
||||
Model dir: data/wiki/ru-100/models/sp25k/qrnn_nl4.m
|
||||
Wiki text was split to 193047 articles
|
||||
Wiki text was split to 460 articles
|
||||
Data lm, trn: 193047, val: 460
|
||||
Size of vocabulary: 25000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', 'х', '▁на']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.154972 4.198218 0.447508
|
||||
2 4.030367 4.159642 0.449420
|
||||
3 4.138530 4.146010 0.451526
|
||||
4 3.997120 4.097048 0.457177
|
||||
5 3.999151 4.036350 0.465117
|
||||
6 3.935380 3.955517 0.476446
|
||||
7 3.912357 3.875987 0.487591
|
||||
8 3.785693 3.789099 0.501560
|
||||
9 3.743162 3.725730 0.512294
|
||||
10 3.690226 3.706929 0.516769
|
||||
Total time: 12:10:03
|
||||
data/wiki/ru-100/models/sp25k
|
||||
```
|
||||
|
||||
```bash
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG
|
||||
}-100/models/sp25k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
|
||||
Max vocab: 25000
|
||||
Cache dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp25k
|
||||
Model dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp25k/qrnn_nl4.m
|
||||
Loading validation /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 9195, val: 1021
|
||||
Running tokenization cls...
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 25000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', 'х', '▁на']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp25k/qrnn_nl4.m/lm_best'), PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp25k/qrnn_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.626971 3.868075 0.474742
|
||||
Total time: 01:58
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.821786 3.625366 0.519506
|
||||
2 3.570115 3.379288 0.566803
|
||||
3 3.517294 3.179166 0.599955
|
||||
4 3.160131 3.028985 0.626484
|
||||
5 3.135806 2.923198 0.644557
|
||||
6 3.055160 2.840300 0.659376
|
||||
7 3.005086 2.770163 0.672080
|
||||
8 2.811366 2.708846 0.684065
|
||||
9 2.818394 2.658951 0.694358
|
||||
10 2.881018 2.605373 0.705269
|
||||
11 2.793422 2.560091 0.715893
|
||||
12 2.708385 2.516373 0.725908
|
||||
13 2.690258 2.471159 0.735673
|
||||
14 2.748342 2.436113 0.744533
|
||||
15 2.601220 2.394404 0.754131
|
||||
16 2.616882 2.372301 0.760451
|
||||
17 2.602902 2.349164 0.766014
|
||||
18 2.560349 2.336217 0.769222
|
||||
19 2.549936 2.332076 0.770150
|
||||
20 2.546798 2.331103 0.770472
|
||||
Total time: 53:22
|
||||
/home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp25k
|
||||
Saving info /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp25k/qrnn_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.043533 0.961182 0.731000
|
||||
2 0.859086 0.837210 0.824000
|
||||
3 0.735276 0.724173 0.871000
|
||||
4 0.612012 0.711034 0.857000
|
||||
Total time: 01:15
|
||||
Saving models at /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp25k/qrnn_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.3957597, tensor(0.8720)]
|
||||
0.3957597017288208
|
||||
0.871999979019165
|
||||
```
|
||||
|
||||
## VF60k QRNN
|
||||
### LM
|
||||
|
||||
### MLDoc
|
||||
```bash
|
||||
Max vocab: 60000
|
||||
Cache dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/vf60k
|
||||
Model dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/vf60k/qrnn_nl4.m
|
||||
Loading validation /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 9195, val: 1021
|
||||
Running tokenization cls...
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 55567
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', ',', '.', '-', 'в', ')', '(', 'на', "&'", 'и', 'по', 'с', 'the']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 34364, first 100: ['рейтер', '941', '8520', '095', 'said', '\x7f', 'доллару', 'янв', 'погашение', '3272', 'reuter', 'xd0', 'фев', '509410', 'уставный', 'which', 'percent', 'объективность', 'торгах', 'купона', 'million', 'its', 'июл', '044', 'алма-атинское', 'валютной', 'триллиона', 'межбанковской', 'would', 'авг', 'government', 'котировки', 'балансовая', 'ртс', 'выплата', 'прц', '8832', 'yeltsin', '983', 'средневзвешенная', '961', 'president', 'дек', 'minister', '2264', 'нацбанка', 'цбр', 'июн', 'newsroom', 'ммвб', 'гособлигаций', 'стр.1', 'also', 'foreign', 'офз', 'заявленный', 'шестимесячных', 'дисконтных', '-сказал', 'предыдущему', 'тбилисское', 'размещенный', 'told', 'riga', 'лари', 'стр.2', 'kroons', 'окт', 'сиданко', '--московское', 'adr', 'мосэнерго', 'shares', 'пресс-релизе', 'дилеры', 'триллионов', 'акциям', 'billion', 'демченко', 'тнк', 'litas', 'lats', 'дилеров', '--алма-атинское', 'щелкните', 'tuesday', 'зинец', 'friday', 'умвб', 'thursday', 'онэксим', 'трейдеры', 'nato', 'feb', 'дивиденды', 'former', 'could', 'нацбанк', 'стр.6', 'economic']
|
||||
Bptt 70
|
||||
Training lm from: [PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/vf60k/qrnn_nl4.m/lm_best'), PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/vf60k/qrnn_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 5.637876 4.853484 0.379844
|
||||
Total time: 01:28
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.906714 4.683807 0.405109
|
||||
2 4.850066 4.490903 0.434562
|
||||
3 4.591409 4.284740 0.464436
|
||||
4 4.379681 4.103634 0.490118
|
||||
5 4.079576 3.954377 0.511206
|
||||
6 4.199800 3.811692 0.531036
|
||||
7 4.004812 3.694871 0.548372
|
||||
8 3.995378 3.584868 0.567285
|
||||
9 3.884090 3.499729 0.583162
|
||||
10 3.897333 3.416602 0.598120
|
||||
11 3.726276 3.338907 0.613920
|
||||
12 3.690300 3.263694 0.629643
|
||||
13 3.614015 3.192474 0.646335
|
||||
14 3.530548 3.136064 0.659729
|
||||
15 3.451486 3.100320 0.668686
|
||||
16 3.444497 3.058001 0.678824
|
||||
17 3.407755 3.024943 0.686764
|
||||
18 3.383617 3.008939 0.690451
|
||||
19 3.342304 2.999911 0.692378
|
||||
20 3.339514 2.998623 0.692671
|
||||
Total time: 36:01
|
||||
/home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/vf60k
|
||||
Saving info /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/vf60k/qrnn_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.946690 0.855268 0.805000
|
||||
2 0.808650 0.750561 0.866000
|
||||
3 0.701750 0.712251 0.884000
|
||||
4 0.596392 0.687266 0.884000
|
||||
Total time: 00:44
|
||||
Saving models at /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/vf60k/qrnn_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.37472174, tensor(0.8802)]
|
||||
0.3747217357158661
|
||||
0.8802499771118164
|
||||
```
|
||||
|
||||
|
||||
## SP30k LSTM nl4
|
||||
### LM
|
||||
```
|
||||
python -m ulmfit lm --dataset-path data/wiki/ru-100 --cuda-id=0 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 30000 --lang ru --qrnn=False - train 10 --bs=50 --drop_mult=0
|
||||
Size of vocabulary: 30000 [39/805]
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', '▁и', 'е', 'и', 'й', '▁на', 'х']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': None, 'pretrained_model': None, 'drop_mult': 0} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.200520 3.295865 0.436852
|
||||
2 3.027569 3.168700 0.445551
|
||||
3 3.007320 3.132495 0.450450
|
||||
4 2.940000 3.041745 0.459344
|
||||
5 2.876227 2.952338 0.469182
|
||||
6 2.742553 2.860888 0.480943
|
||||
7 2.684717 2.769994 0.492934
|
||||
8 2.569419 2.669971 0.507300
|
||||
9 2.525698 2.604086 0.516840
|
||||
10 2.495174 2.591011 0.519415
|
||||
data/wiki/ru-100/models/sp30k
|
||||
Saving info data/wiki/ru-100/models/sp30k/lstm_nl4.m/info.json
|
||||
```
|
||||
### MLDoc - bsp
|
||||
MultiCCA: 85.65% ulmfit: 87.27%
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/ru-1 --base-lm-path data/wiki/ru-100/models/sp30k/lstm_nl4.m --lang=ru --name 'nl4-100' --cuda-id=1 - train 20 --bs 40 --num-cls-epochs=2
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k/lstm_nl4.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 9195, cls.val 1021
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', '▁и', 'е', 'и', 'й', '▁на', 'х']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp30k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.764138 2.289755 0.552181
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.414295 2.161708 0.572407
|
||||
2 2.310551 2.013092 0.596075
|
||||
3 2.124479 1.864450 0.620103
|
||||
4 1.970015 1.723395 0.642392
|
||||
5 1.883664 1.623308 0.658949
|
||||
6 1.793856 1.513542 0.677954
|
||||
7 1.625767 1.424582 0.693092
|
||||
8 1.677054 1.335406 0.709802
|
||||
9 1.578936 1.264322 0.723626
|
||||
10 1.523383 1.194463 0.737942
|
||||
11 1.436643 1.129712 0.750586
|
||||
12 1.351507 1.072792 0.762524
|
||||
13 1.357552 1.020739 0.773266
|
||||
14 1.310516 0.975852 0.783653
|
||||
15 1.216484 0.940323 0.791262
|
||||
16 1.187942 0.909915 0.797675
|
||||
17 1.141316 0.885367 0.803305
|
||||
18 1.114629 0.871992 0.805929
|
||||
19 1.075366 0.867010 0.807009
|
||||
20 1.166387 0.865594 0.807241
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k/lstm_nl4.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.831180 0.610087 0.787000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.678307 0.435860 0.856000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.547668 0.399889 0.870000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.445839 0.396535 0.869000
|
||||
2 0.417901 0.369961 0.882000
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.38499942, tensor(0.8727)]
|
||||
```
|
||||
|
||||
### MLDoc run 2x sp
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/ru-1 --base-lm-path data/wiki/ru-100/models/sp30k/lstm_nl4.m --lang=ru --name 'nl4' --cuda-id=1 - train 20 --bs 40 --num-cls-epochs=2
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k/lstm_nl4.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 9195, cls.val 1021
|
||||
Running tokenization...
|
||||
Saving tokenized: cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', '▁и', 'е', 'и', 'й', '▁на', 'х']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp30k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.662225 2.284158 0.552927
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.436114 2.151187 0.574219
|
||||
2 2.260576 2.012279 0.595820
|
||||
3 2.067110 1.862512 0.620246
|
||||
4 2.000703 1.729883 0.641713
|
||||
5 1.860899 1.609955 0.661346
|
||||
6 1.751010 1.522195 0.676297
|
||||
7 1.705993 1.420628 0.694044
|
||||
8 1.592143 1.338552 0.708978
|
||||
9 1.524927 1.270614 0.722596
|
||||
10 1.475408 1.198585 0.736638
|
||||
11 1.438226 1.134858 0.749314
|
||||
12 1.408821 1.076875 0.761448
|
||||
13 1.345137 1.020660 0.773432
|
||||
14 1.321399 0.978076 0.783070
|
||||
15 1.235357 0.936674 0.791642
|
||||
16 1.204204 0.906822 0.798548
|
||||
17 1.198709 0.884949 0.803528
|
||||
18 1.176732 0.874523 0.805585
|
||||
19 1.111195 0.871806 0.806239
|
||||
20 1.031497 0.869280 0.806826
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k/lstm_nl4.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.834704 0.615589 0.786000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.679823 0.418461 0.851000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.555612 0.426877 0.861000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.468084 0.391777 0.873000
|
||||
2 0.434714 0.388670 0.882000
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp30k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.3987146, tensor(0.8680)]
|
||||
0.3987146019935608
|
||||
0.8679999709129333
|
||||
```
|
||||
|
||||
```
|
||||
Second execution
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.749340 2.284773 0.552775
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.418463 2.157943 0.572302
|
||||
```
|
||||
@@ -1,91 +0,0 @@
|
||||
```
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
LANG=ru
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_nl4.m --lang=${LANG} --name 'nl4-lm5' - train 5 --bs 20 --num-cls-epochs=8 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-lm5.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
|
||||
Data lm, trn: 9195, val: 1021
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
Your compiler (c++) may be ABI-incompatible with PyTorch!
|
||||
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
|
||||
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
|
||||
|
||||
See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
|
||||
for instructions on how to install GCC 4.9 or higher.
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.549059 3.763798 0.472608
|
||||
Total time: 02:05
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.543295 3.263310 0.567992
|
||||
2 3.166918 2.968566 0.619391
|
||||
3 3.057842 2.812808 0.648944
|
||||
4 2.842979 2.726823 0.665521
|
||||
5 2.872606 2.703771 0.670281
|
||||
Total time: 14:40
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-lm5.m/info.json
|
||||
```
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_nl4.m --lang=${LANG} --name 'nl4-lm5' - train 5 --bs 18 --num-cls-epochs=8 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-lm5.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
|
||||
Data lm, trn: 9195, val: 1021
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
Your compiler (c++) may be ABI-incompatible with PyTorch!
|
||||
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
|
||||
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
|
||||
|
||||
See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
|
||||
for instructions on how to install GCC 4.9 or higher.
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.021985 0.763919 0.822000
|
||||
2 0.903123 0.756099 0.849000
|
||||
3 0.831409 0.852466 0.832000
|
||||
4 0.744423 0.753127 0.858000
|
||||
5 0.669933 0.747895 0.862000
|
||||
6 0.607411 0.744035 0.869000
|
||||
7 0.554080 0.706676 0.872000
|
||||
8 0.532403 0.719503 0.870000
|
||||
Total time: 03:12
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-lm5.m
|
||||
Loss and accuracy using (cls_best): [0.41288647, tensor(0.8615)]
|
||||
0.41288647055625916
|
||||
0.8615000247955322
|
||||
```
|
||||
@@ -1,132 +0,0 @@
|
||||
## bptt140
|
||||
### CLS
|
||||
```
|
||||
LANG=ru
|
||||
python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-bptt140' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 14 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1
|
||||
|
||||
Max vocab: 15000
|
||||
Cache dir: data/wiki/ru-100/models/sp15k
|
||||
Model dir: data/wiki/ru-100/models/sp15k/qrnn_nl4-bptt140.m
|
||||
Wiki text was split to 193047 articles
|
||||
Wiki text was split to 460 articles
|
||||
Data lm, trn: 193047, val: 460
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
Your compiler (c++) may be ABI-incompatible with PyTorch!
|
||||
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
|
||||
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
|
||||
|
||||
See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
|
||||
for instructions on how to install GCC 4.9 or higher.
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
|
||||
Bptt 70
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.022324 4.046203 0.451453
|
||||
2 3.840025 3.935647 0.462081
|
||||
3 3.873172 3.940451 0.459741
|
||||
4 3.850415 3.918466 0.462763
|
||||
5 3.814188 3.898976 0.465359
|
||||
6 3.771836 3.857443 0.472302
|
||||
7 3.761032 3.801748 0.479811
|
||||
8 3.712323 3.755207 0.486181
|
||||
9 3.706044 3.707724 0.493604
|
||||
10 3.693287 3.650429 0.502407
|
||||
11 3.563701 3.588871 0.513251
|
||||
12 3.477192 3.538018 0.522175
|
||||
13 3.486541 3.504327 0.528571
|
||||
14 3.484132 3.495028 0.530480
|
||||
Total time: 19:53:42
|
||||
data/wiki/ru-100/models/sp15k
|
||||
Saving info data/wiki/ru-100/models/sp15k/qrnn_nl4-bptt140.m/info.json
|
||||
```
|
||||
|
||||
### MLDoc
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_nl4-bptt140.m --lang=${LANG} --name nl4-bptt140 --bptt=140 - train 20 --bs 18 --num-cls-epochs=8 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-bptt140.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
|
||||
Running tokenization lm140...
|
||||
Data lm140, trn: 9195, val: 1021
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
Your compiler (c++) may be ABI-incompatible with PyTorch!
|
||||
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
|
||||
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
|
||||
|
||||
See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
|
||||
for instructions on how to install GCC 4.9 or higher.
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 140
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4-bptt140.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4-bptt140.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.855653 2.852676 0.452966
|
||||
Total time: 01:56
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.052491 2.572216 0.504417
|
||||
2 2.565436 2.252638 0.557341
|
||||
3 2.238792 1.980807 0.599827
|
||||
4 1.990266 1.784574 0.629615
|
||||
5 1.851867 1.647570 0.651466
|
||||
6 1.800950 1.539561 0.668753
|
||||
7 1.692110 1.447140 0.684268
|
||||
8 1.546868 1.380541 0.696082
|
||||
9 1.618451 1.312476 0.708090
|
||||
10 1.478336 1.255234 0.718722
|
||||
11 1.477739 1.197032 0.729453
|
||||
12 1.418238 1.151929 0.738932
|
||||
13 1.384237 1.103246 0.748681
|
||||
14 1.245625 1.061356 0.757009
|
||||
15 1.289399 1.028937 0.763857
|
||||
16 1.280893 1.006447 0.768844
|
||||
17 1.268177 0.985106 0.773329
|
||||
18 1.251713 0.975138 0.775565
|
||||
19 1.288352 0.968812 0.776884
|
||||
20 1.164147 0.967133 0.777174
|
||||
Total time: 52:46
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-bptt140.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.905054 0.572781 0.811000
|
||||
2 0.747270 0.606469 0.806000
|
||||
3 0.644590 0.682804 0.810000
|
||||
4 0.457427 0.605931 0.863000
|
||||
5 0.351969 0.652187 0.842000
|
||||
6 0.286099 0.589351 0.860000
|
||||
7 0.218377 0.622760 0.857000
|
||||
8 0.185043 0.597372 0.860000
|
||||
Total time: 03:05
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-bptt140.m
|
||||
Loss and accuracy using (cls_best): [0.47860995, tensor(0.8737)] [0.48851612, tensor(0.8600)]
|
||||
val_loss: 0.48851612
|
||||
val_accuracy: 0.8600000143051147
|
||||
tst_loss: 0.47860995
|
||||
tst_accuracy: 0.8737499713897705
|
||||
```
|
||||
@@ -1,123 +0,0 @@
|
||||
|
||||
## BS=18, lr_mult=1.0
|
||||
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.427713 3.693394 0.484268
|
||||
Total time: 01:32
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.758918 3.446661 0.529820
|
||||
2 3.394254 3.199054 0.577411
|
||||
3 3.235364 3.014517 0.610520
|
||||
4 3.125459 2.871101 0.637153
|
||||
5 2.994313 2.773862 0.654470
|
||||
6 2.915075 2.693080 0.669942
|
||||
7 2.855732 2.622858 0.683629
|
||||
8 2.755074 2.572147 0.694145
|
||||
9 2.697898 2.517524 0.704816
|
||||
10 2.689881 2.468190 0.715927
|
||||
11 2.579573 2.432807 0.723324
|
||||
12 2.659464 2.387878 0.733931
|
||||
13 2.520637 2.344804 0.744233
|
||||
14 2.482952 2.315014 0.751855
|
||||
15 2.564730 2.279045 0.761163
|
||||
16 2.552707 2.255916 0.766971
|
||||
17 2.511244 2.240169 0.770991
|
||||
18 2.461429 2.228213 0.774309
|
||||
19 2.426440 2.222140 0.775745
|
||||
20 2.425955 2.221128 0.775836
|
||||
Total time: 1:14:17
|
||||
|
||||
## BS=500, lr_mult=1.0
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 5.536769 3.850831 0.444662
|
||||
Total time: 01:10
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.845898 3.781763 0.461471
|
||||
2 4.388605 3.643141 0.491225
|
||||
3 4.038255 3.464554 0.526143
|
||||
|
||||
## BS=500, lr_mult=27
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 7.234749 5.868155 0.312675
|
||||
Total time: 01:41
|
||||
|
||||
## BS=500, lr_mult=10 + BS=50 lr_mult=10 for cls
|
||||
/data/wiki/ru-100/models/sp15k/qrnn_nl4sl.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 5.052441 4.082105 0.439539
|
||||
Total time: 02:27
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.120197 3.712686 0.498216
|
||||
2 3.727043 3.373258 0.557896
|
||||
3 3.383009 3.109635 0.598970
|
||||
4 3.180799 2.938478 0.626816
|
||||
5 3.048913 2.812639 0.647257
|
||||
6 2.943903 2.727179 0.661784
|
||||
7 2.864300 2.650275 0.674248
|
||||
8 2.773810 2.583594 0.687063
|
||||
9 2.724850 2.529445 0.697573
|
||||
10 2.673996 2.473824 0.708698
|
||||
11 2.657637 2.431461 0.716904
|
||||
12 2.591277 2.372668 0.730318
|
||||
13 2.537707 2.323294 0.741157
|
||||
14 2.486507 2.280270 0.751768
|
||||
15 2.435933 2.238660 0.762545
|
||||
16 2.401303 2.208848 0.769561
|
||||
17 2.374117 2.184253 0.776400
|
||||
18 2.341421 2.169156 0.780388
|
||||
19 2.328202 2.163922 0.781700
|
||||
20 2.315462 2.161784 0.782105
|
||||
Total time: 1:07:09
|
||||
------------------- Checking the influence of number of epochs on the accuracy
|
||||
(multifit) test@test:~/workspace/ulmfit-multilingual$ rm /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4sl-bs500.m/cls*
|
||||
(multifit) test@test:~/workspace/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME}-bs500 - train 0 --bs 50 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1 --lr_mult=1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4sl-bs500.m
|
||||
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
|
||||
Data lm, trn: 9195, val: 1021
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.063845 1.111960 0.601000
|
||||
2 0.902245 0.766871 0.817000
|
||||
3 0.766261 0.707502 0.861000
|
||||
4 0.680053 0.694492 0.866000
|
||||
Total time: 01:22
|
||||
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4sl-bs500.m
|
||||
Loss and accuracy using (cls_best): [0.41532615, tensor(0.8630)]
|
||||
0.41532614827156067
|
||||
0.8629999756813049
|
||||
(multifit) test@test:~/workspace/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME}-bs500 - train 0 --bs 50 --num-cls-epochs=16 --lr_sched=1cycle --label-smoothing-eps=0.1 --lr_mult=1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4sl-bs500.m
|
||||
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
|
||||
Data lm, trn: 9195, val: 1021
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Loading last classifier
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.556688 0.706000 0.873000
|
||||
2 0.537578 0.717411 0.865000
|
||||
3 0.532326 0.775549 0.854000
|
||||
4 0.529178 0.767506 0.861000
|
||||
5 0.521306 0.797604 0.860000
|
||||
6 0.527344 0.736225 0.868000
|
||||
7 0.516393 0.724941 0.878000
|
||||
8 0.510422 0.716110 0.873000
|
||||
9 0.504320 0.701886 0.869000
|
||||
10 0.500323 0.676577 0.878000
|
||||
11 0.493490 0.682657 0.873000
|
||||
12 0.484450 0.682047 0.878000
|
||||
13 0.479248 0.682782 0.880000
|
||||
14 0.474778 0.688019 0.873000
|
||||
15 0.472664 0.685304 0.874000
|
||||
16 0.470747 0.677925 0.878000
|
||||
Total time: 07:57
|
||||
@@ -1,260 +0,0 @@
|
||||
CUDA_VISIBLE_DEVICES=0 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 5 --name 'nl5-merity' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 2500 - train 14 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1
|
||||
CUDA_VISIBLE_DEVICES=1 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-wide2' --max-vocab 15000 --lang ${LANG} --qrnn=True --nh 3100 - train 10 --bs=100 --drop_mult=0 --label-smoothing-eps=0.1
|
||||
CUDA_VISIBLE_DEVICES=2 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-merity' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 2500 - train 14 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1
|
||||
CUDA_VISIBLE_DEVICES=3 python -m ulmfit lm --dataset-path data/wiki/ru-100 --bidir=False --qrnn=True --nl 4 --tokenizer=sp --max-vocab 15000 --lang ru --name nl4sl - train 10 --drop-mult=0 --bs=50 --label-smoothing-eps=0.1
|
||||
|
||||
## 25vocab
|
||||
CUDA_VISIBLE_DEVICES=3 python -m ulmfit lm --dataset-path data/wiki/ru-100 --bidir=False --qrnn=True --nl 4 --tokenizer=sp --max-vocab 25000 --lang ru --name nl4 - train 10 --drop-mult=0 --bs=50 --label-smoothing-eps=0.1
|
||||
|
||||
LANG=ru
|
||||
CUDA_VISIBLE_DEVICES=0 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-merity-wide2' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 3100 - train 10 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1
|
||||
|
||||
##### CLS
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
LANG=ru
|
||||
NAME=nl5-merity
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
|
||||
export CUDA_VISIBLE_DEVICES=1
|
||||
LANG=ru
|
||||
NAME=nl4-wide2
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
|
||||
export CUDA_VISIBLE_DEVICES=2
|
||||
LANG=ru
|
||||
NAME=nl4-merity
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
|
||||
export CUDA_VISIBLE_DEVICES=3
|
||||
LANG=ru
|
||||
NAME=nl4sl
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
|
||||
|
||||
-----------------------CLS1
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
LANG=ru
|
||||
NAME=nl4
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
|
||||
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
LANG=ru
|
||||
NAME=nl8
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
|
||||
|
||||
python -m ulmfit cls --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 5 --name 'nl5-merity' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 2500 - train 14 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1
|
||||
|
||||
CUDA_VISIBLE_DEVICES=1 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-wide2' --max-vocab 15000 --lang ${LANG} --qrnn=True --nh 3100 - train 10 --bs=100 --drop_mult=0 --label-smoothing-eps=0.1
|
||||
CUDA_VISIBLE_DEVICES=2 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-merity' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 2500 - train 14 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1
|
||||
CUDA_VISIBLE_DEVICES=3 python -m ulmfit lm --dataset-path data/wiki/ru-100 --bidir=False --qrnn=True --nl 4 --tokenizer=sp --max-vocab 15000 --lang ru --name nl4sl - train 10 --drop-mult=0 --bs=50 --label-smoothing-eps=0.1
|
||||
##
|
||||
|
||||
------------------------
|
||||
|
||||
7 3.680504 3.678406 0.498396
|
||||
8 3.556062 3.596037 0.512345
|
||||
9 3.553716 3.535783 0.523509
|
||||
10 3.523366 3.515352 0.527935
|
||||
Total time: 20:03:59
|
||||
data/wiki/ru-100/models/sp15k
|
||||
Saving info data/wiki/ru-100/models/sp15k/qrnn_ nl4sl.m/info.json
|
||||
|
||||
### Ru
|
||||
```
|
||||
export CUDA_VISIBLE_DEVICES=3
|
||||
LANG=ru
|
||||
NAME=nl4sl
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4sl.m
|
||||
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
|
||||
Data lm, trn: 9195, val: 1021
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4sl.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4sl.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.427713 3.693394 0.484268
|
||||
Total time: 01:32
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.758918 3.446661 0.529820
|
||||
2 3.394254 3.199054 0.577411
|
||||
3 3.235364 3.014517 0.610520
|
||||
4 3.125459 2.871101 0.637153
|
||||
5 2.994313 2.773862 0.654470
|
||||
6 2.915075 2.693080 0.669942
|
||||
7 2.855732 2.622858 0.683629
|
||||
8 2.755074 2.572147 0.694145
|
||||
9 2.697898 2.517524 0.704816
|
||||
10 2.689881 2.468190 0.715927
|
||||
11 2.579573 2.432807 0.723324
|
||||
12 2.659464 2.387878 0.733931
|
||||
13 2.520637 2.344804 0.744233
|
||||
14 2.482952 2.315014 0.751855
|
||||
15 2.564730 2.279045 0.761163
|
||||
16 2.552707 2.255916 0.766971
|
||||
17 2.511244 2.240169 0.770991
|
||||
18 2.461429 2.228213 0.774309
|
||||
19 2.426440 2.222140 0.775745
|
||||
20 2.425955 2.221128 0.775836
|
||||
Total time: 1:14:17
|
||||
/home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4sl.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.022382 0.779370 0.822000
|
||||
2 0.866379 0.792353 0.832000
|
||||
3 0.715650 0.698579 0.865000
|
||||
4 0.603621 0.693501 0.884000
|
||||
Total time: 02:05
|
||||
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4sl.m
|
||||
Loss and accuracy using (cls_best): [0.3978519, tensor(0.8723)]
|
||||
0.3978519141674042
|
||||
0.8722500205039978
|
||||
```
|
||||
|
||||
----
|
||||
|
||||
```bash
|
||||
$ export CUDA_VISIBLE_DEVICES=0
|
||||
$ LANG=ru
|
||||
$ NAME=nl4
|
||||
$ python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Model dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4.m
|
||||
Loading validation /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
|
||||
Data lm, trn: 9195, val: 1021
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4.m/lm_best'), PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.531968 3.764185 0.474252
|
||||
Total time: 01:44
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.770046 3.506013 0.522443
|
||||
2 3.546580 3.251341 0.571620
|
||||
3 3.320569 3.055680 0.606364
|
||||
4 3.130226 2.912925 0.631395
|
||||
5 3.072772 2.809725 0.649728
|
||||
6 2.765424 2.731825 0.662963
|
||||
7 2.959237 2.662104 0.676203
|
||||
8 2.807999 2.600417 0.688423
|
||||
9 2.771271 2.548279 0.699473
|
||||
10 2.809488 2.501688 0.709020
|
||||
11 2.707221 2.454946 0.719196
|
||||
12 2.597226 2.417315 0.728432
|
||||
13 2.609972 2.376176 0.737923
|
||||
14 2.590427 2.341666 0.746216
|
||||
15 2.572995 2.306599 0.754747
|
||||
16 2.496636 2.285632 0.760806
|
||||
17 2.508584 2.266456 0.765147
|
||||
18 2.441373 2.253839 0.768449
|
||||
19 2.430915 2.249204 0.769536
|
||||
20 2.426130 2.247966 0.769886
|
||||
Total time: 47:33
|
||||
/home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Saving info /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.060299 0.890710 0.716000
|
||||
2 0.884965 0.769866 0.853000
|
||||
3 0.722994 0.723213 0.875000
|
||||
4 0.609488 0.730594 0.865000
|
||||
Total time: 01:14
|
||||
Saving models at /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.39589784, tensor(0.8692)]
|
||||
0.39589783549308777
|
||||
0.8692499995231628
|
||||
```
|
||||
|
||||
|
||||
## wide 2
|
||||
```bash
|
||||
$ CUDA_VISIBLE_DEVICES=1 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-wide2' --max-vocab 15000 --lang ${LANG} --qrnn=True --nh 3100 - train 10 --bs=100 --drop_mult=0 --label-smoothing-eps=0.1
|
||||
Max vocab: 15000
|
||||
Cache dir: data/wiki/ru-100/models/sp15k
|
||||
Model dir: data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m
|
||||
Wiki text was split to 193047 articles
|
||||
Wiki text was split to 460 articles
|
||||
Data lm, trn: 193047, val: 460
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.908233 3.950865 0.463469
|
||||
2 3.738863 3.815026 0.477703
|
||||
3 3.696502 3.779513 0.483625
|
||||
4 3.692592 3.720908 0.490143
|
||||
5 3.600519 3.652444 0.501671
|
||||
6 3.564568 3.582584 0.511550
|
||||
7 3.472859 3.493226 0.525943
|
||||
8 3.390483 3.407970 0.541749
|
||||
9 3.351620 3.344207 0.552758
|
||||
10 3.329683 3.330087 0.556380
|
||||
Total time: 51:05:43
|
||||
data/wiki/ru-100/models/sp15k
|
||||
Saving info data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/info.json
|
||||
```
|
||||
### MLDoc
|
||||
export CUDA_VISIBLE_DEVICES=1
|
||||
LANG=ru
|
||||
NAME=nl4-wide2
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
|
||||
|
||||
|
||||
## Merity nl4
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=2 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-merity' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 2500 - train 14 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1
|
||||
Max vocab: 15000
|
||||
Cache dir: data/wiki/ru-100/models/sp15k
|
||||
Model dir: data/wiki/ru-100/models/sp15k/qrnn_nl4-merity.m
|
||||
Wiki text was split to 193047 articles
|
||||
Wiki text was split to 460 articles
|
||||
Data lm, trn: 193047, val: 460
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.965899 3.977734 0.460046
|
||||
2 3.806082 3.858176 0.472396
|
||||
3 3.839230 3.874757 0.469224
|
||||
4 3.762105 3.868653 0.469943
|
||||
5 3.800827 3.833991 0.474116
|
||||
6 3.755466 3.796329 0.479868
|
||||
7 3.691958 3.747888 0.487367
|
||||
8 3.660529 3.702986 0.493545
|
||||
9 3.593282 3.635035 0.504086
|
||||
10 3.585948 3.579200 0.513631
|
||||
11 3.473865 3.512114 0.525391
|
||||
12 3.451973 3.455807 0.535520
|
||||
13 3.418731 3.417129 0.542943
|
||||
14 3.385637 3.407541 0.545545
|
||||
Total time: 51:32:09
|
||||
data/wiki/ru-100/models/sp15k
|
||||
Saving info data/wiki/ru-100/models/sp15k/qrnn_nl4-merity.m/info.json
|
||||
```
|
||||
|
||||
#### MLDoc
|
||||
|
||||
export CUDA_VISIBLE_DEVICES=2
|
||||
LANG=ru
|
||||
NAME=nl4-merity
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
@@ -1,101 +0,0 @@
|
||||
## LM
|
||||
|
||||
|
||||
### MLDoc 1
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-merity.m
|
||||
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
|
||||
Data lm, trn: 9195, val: 1021
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4-merity.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4-merity.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.411345 3.660489 0.486965
|
||||
Total time: 02:43
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.613334 3.372079 0.544188
|
||||
2 3.325245 3.100234 0.596406
|
||||
3 3.181919 2.906442 0.631586
|
||||
4 3.010830 2.767429 0.656378
|
||||
5 2.880418 2.663865 0.676339
|
||||
6 2.825526 2.571074 0.694140
|
||||
7 2.766901 2.483362 0.711652
|
||||
8 2.601965 2.417213 0.726853
|
||||
9 2.569160 2.341699 0.744193
|
||||
10 2.588142 2.272457 0.760294
|
||||
11 2.494011 2.198197 0.779175
|
||||
12 2.421921 2.135517 0.795854
|
||||
13 2.396429 2.075012 0.812815
|
||||
14 2.306572 2.019140 0.828851
|
||||
15 2.281730 1.966554 0.843595
|
||||
16 2.206670 1.927567 0.854515
|
||||
17 2.143836 1.901352 0.862114
|
||||
18 2.141715 1.884954 0.867003
|
||||
19 2.070353 1.876935 0.869214
|
||||
20 2.066195 1.874844 0.869665
|
||||
Total time: 2:12:21
|
||||
/home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-merity.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.994393 0.755689 0.844000
|
||||
2 0.859871 0.822650 0.856000
|
||||
3 0.678185 0.721333 0.859000
|
||||
4 0.586906 0.693618 0.878000
|
||||
Total time: 04:17
|
||||
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-merity.m
|
||||
Loss and accuracy using (cls_best): [0.3872361, tensor(0.8777)]
|
||||
0.387236088514328
|
||||
0.8777499794960022
|
||||
```
|
||||
|
||||
### MLDoc 2
|
||||
```
|
||||
(multifit) test@test:~/workspace/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/mldoc/${LANG}-1/models/sp15k/qrnn_${NAME}.
|
||||
m --lang=${LANG} --name ${NAME}-16 - train 0 --bs 18 --num-cls-epochs=16 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-merity-16.m
|
||||
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
|
||||
Data lm, trn: 9195, val: 1021
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
/home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-merity-16.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.081481 0.837775 0.781000
|
||||
2 0.901621 0.798574 0.858000
|
||||
3 0.778870 0.826576 0.859000
|
||||
4 0.693465 0.787875 0.833000
|
||||
5 0.639763 0.841092 0.861000
|
||||
6 0.595044 0.731504 0.853000
|
||||
7 0.576115 0.796013 0.819000
|
||||
8 0.544098 0.744034 0.875000
|
||||
9 0.531359 0.699035 0.879000
|
||||
10 0.513886 0.698310 0.879000
|
||||
11 0.495473 0.686897 0.864000
|
||||
12 0.489863 0.688584 0.881000
|
||||
13 0.481086 0.675660 0.881000
|
||||
14 0.479960 0.684917 0.883000
|
||||
15 0.490157 0.687865 0.882000
|
||||
16 0.486081 0.679104 0.882000
|
||||
Total time: 15:26
|
||||
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-merity-16.m
|
||||
Loss and accuracy using (cls_best): [0.4047818, tensor(0.8737)]
|
||||
0.4047817885875702
|
||||
0.8737499713897705
|
||||
```
|
||||
@@ -1,38 +0,0 @@
|
||||
## LM
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 5 --name 'nl5-mer
|
||||
ity' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 2500 - train 14 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1
|
||||
Max vocab: 15000
|
||||
Cache dir: data/wiki/ru-100/models/sp15k
|
||||
Model dir: data/wiki/ru-100/models/sp15k/qrnn_nl5-merity.m
|
||||
Wiki text was split to 193047 articles
|
||||
Wiki text was split to 460 articles
|
||||
Data lm, trn: 193047, val: 460
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.969639 4.024787 0.452887
|
||||
2 3.814622 3.834142 0.476612
|
||||
3 3.798372 3.846118 0.473666
|
||||
4 3.742609 3.835311 0.474612
|
||||
5 3.715114 3.790690 0.480469
|
||||
6 3.652987 3.748408 0.486146
|
||||
7 3.573350 3.697325 0.493774
|
||||
8 3.589853 3.637134 0.504189
|
||||
9 3.558110 3.583030 0.512137
|
||||
10 3.501382 3.510491 0.524148
|
||||
11 3.408982 3.437177 0.536634
|
||||
12 3.402717 3.373548 0.548113
|
||||
13 3.293624 3.331311 0.556288
|
||||
14 3.309859 3.322777 0.558426
|
||||
Total time: 68:05:15
|
||||
data/wiki/ru-100/models/sp15k
|
||||
Saving info data/wiki/ru-100/models/sp15k/qrnn_nl5-merity.m/info.json
|
||||
```
|
||||
|
||||
### MLDoc 1
|
||||
```
|
||||
|
||||
```
|
||||
@@ -1,78 +0,0 @@
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME}-2 - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Model dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl8-2.m
|
||||
Loading validation /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
|
||||
Data lm, trn: 9195, val: 1021
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl8.m/lm_best'), PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl8.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.966292 3.450758 0.527065
|
||||
Total time: 02:58
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.495761 3.276329 0.560047
|
||||
2 3.319947 3.102911 0.593742
|
||||
3 3.137904 2.955317 0.620171
|
||||
4 3.040286 2.839161 0.642270
|
||||
5 2.869962 2.753622 0.658331
|
||||
6 2.905739 2.680881 0.672860
|
||||
7 2.836454 2.620925 0.685026
|
||||
8 2.857271 2.569716 0.695722
|
||||
9 2.702872 2.520050 0.705589
|
||||
10 2.701559 2.473591 0.715346
|
||||
11 2.740815 2.429558 0.725597
|
||||
12 2.646513 2.389550 0.735010
|
||||
13 2.587685 2.349614 0.744885
|
||||
14 2.546527 2.311087 0.754463
|
||||
15 2.568136 2.278581 0.762980
|
||||
16 2.492115 2.252367 0.769275
|
||||
17 2.338561 2.230529 0.775072
|
||||
18 2.447506 2.218215 0.778437
|
||||
19 2.364424 2.212115 0.780085
|
||||
20 2.367132 2.210520 0.780424
|
||||
Total time: 1:30:47
|
||||
/home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Saving info /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl8-2.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.969255 0.769736 0.843000
|
||||
2 0.846340 0.813483 0.839000
|
||||
3 0.718175 0.705339 0.867000
|
||||
4 0.609513 0.726442 0.875000
|
||||
Total time: 02:54
|
||||
Saving models at /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl8-2.m
|
||||
Loss and accuracy using (cls_best): [0.4056449, tensor(0.8698)]
|
||||
0.40564489364624023
|
||||
|
||||
|
||||
|
||||
(fastaiv1) n-waves@GV100:~/workspace/ulmfit-multilingual$ CUDA_VISIBLE_DEVICES=0 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-merity-wide2' --max-vocab 15000 --lang ${LANG} --qrnn=True --bptt=140 --nh 3100 - train 10 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1
|
||||
Max vocab: 15000
|
||||
Cache dir: data/wiki/ru-100/models/sp15k
|
||||
Model dir: data/wiki/ru-100/models/sp15k/qrnn_nl4-merity-wide2.m
|
||||
Wiki text was split to 193047 articles
|
||||
Wiki text was split to 460 articles
|
||||
Data lm, trn: 193047, val: 460
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.871650 3.908779 0.467000
|
||||
2 3.834093 3.884629 0.467916
|
||||
3 3.741005 3.870331 0.469612
|
||||
4 3.785444 3.818511 0.476906
|
||||
5 3.741888 3.752743 0.486148
|
||||
6 3.678481 3.672177 0.499054
|
||||
7 3.570398 3.581498 0.512801
|
||||
8 3.455193 3.482614 0.530569
|
||||
9 3.379779 3.409405 0.543477
|
||||
10 3.384574 3.387195 0.548881
|
||||
Total time: 27:24:33
|
||||
data/wiki/ru-100/models/sp15k
|
||||
@@ -1,115 +0,0 @@
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --tokenizer sp --max-vocab 16000 --qrnn True --lang=${LANG} --name nl4 - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
Max vocab: 16000
|
||||
Cache dir: data/mldoc/ru-1/models/sp16k
|
||||
Model dir: data/mldoc/ru-1/models/sp16k/qrnn_nl4.m
|
||||
Loading validation data/mldoc/ru-1/ru.dev.csv
|
||||
/sentencepiece/src/sentencepiece_trainer.cc(185) LOG(INFO) Running command: --input=data/mldoc/ru-1/models/sp16k/all_text.txt --character_coverage=0.99 --unk_id=8 --pad_id=-1 --bos_id=-1 --eos_id=-1 --max_sentence_length=20480 --input_sentence_size=10000000 --user_defined_symbols=xxunk,xxpad,xxbos,xxfld,xxmaj,xxup,xxrep,xxwrep --model_prefix=data/mldoc/ru-1/models/sp16k/spm --vocab_size=16000 --model_type=unigram
|
||||
/sentencepiece/src/unigram_model_trainer.cc(481) LOG(INFO) Starts training with :
|
||||
input: "data/mldoc/ru-1/models/sp16k/all_text.txt"
|
||||
model_prefix: "data/mldoc/ru-1/models/sp16k/spm"
|
||||
model_type: UNIGRAM
|
||||
vocab_size: 16000
|
||||
character_coverage: 0.99
|
||||
input_sentence_size: 10000000
|
||||
max_sentence_length: 20480
|
||||
user_defined_symbols: "xxunk"
|
||||
user_defined_symbols: "xxpad"
|
||||
user_defined_symbols: "xxbos"
|
||||
user_defined_symbols: "xxfld"
|
||||
user_defined_symbols: "xxmaj"
|
||||
user_defined_symbols: "xxup"
|
||||
user_defined_symbols: "xxrep"
|
||||
user_defined_symbols: "xxwrep"
|
||||
unk_id: 8
|
||||
bos_id: -1
|
||||
eos_id: -1
|
||||
pad_id: -1
|
||||
|
||||
/sentencepiece/src/trainer_interface.cc(183) LOG(INFO) Loading corpus: data/mldoc/ru-1/models/sp16k/all_text.txt
|
||||
/sentencepiece/src/trainer_interface.cc(216) LOG(INFO) Loading: ▁ ▁киев▁,▁20▁июн▁(▁ ▁рейтер▁)▁-▁ ▁нацбанк▁ ▁украины▁планирует▁постепенно▁отказаться▁от▁кредитных▁аукционов▁и▁использовать▁для▁рефинансирования▁банков▁только▁операции▁репо▁и▁ломбардное▁кредитование▁,▁сказала▁директор▁департамента▁ ▁нбу▁ ▁наталия▁ ▁гребеник▁.▁&'▁ ▁от▁кредитных▁аукционов▁ ▁нбу▁будет▁в▁дальнейшем▁отказываться▁,▁используя▁репо▁и▁ломбардное▁кредитование▁&'▁,▁-▁сказала▁директор▁кредитно-эмиссионного▁департамента▁.▁ ▁по▁ее▁словам▁,▁в▁настоящее▁время▁ ▁нацбанк▁использует▁все▁три▁канала▁рефинансирования▁банков▁.▁ ▁удельный▁вес▁рефинансирования▁через▁операции▁репо▁составляет▁50▁процентов▁,▁через▁кредитные▁аукционы▁и▁ломбардное▁кредитование▁под▁залог▁гособлигаций▁по▁25▁процентов▁.▁в▁частности▁,▁с▁начала▁года▁были▁проведены▁четыре▁кредитных▁аукционах▁на▁которых▁банкам▁было▁продано▁560▁миллионов▁гривен▁кредитов▁,▁сказала▁ ▁гребеник▁.▁ ▁по▁ее▁словам▁,▁средняя▁ставка▁продажи▁ресурсов▁на▁кредитных▁аукционах▁на▁3-4▁процента▁превышала▁ставку▁рефинансирования▁,▁действующую▁на▁день▁проведения▁аукциона▁.▁ ▁действующая▁в▁настоящее▁время▁ставка▁рефинансирования▁ ▁нбу▁составляет▁21▁процент▁годовых▁,▁ломбардная▁ставка▁-▁31▁процент▁.▁ ▁по▁соглашениям▁репо▁ставка▁может▁быть▁ниже▁ставки▁рефинансирования▁,▁но▁не▁более▁,▁чем▁на▁5▁процентных▁пунктов▁,▁сказал▁ ▁гребеник▁.▁ ▁по▁ее▁словам▁,▁в▁будущем▁ ▁нбу▁также▁планирует▁освоить▁инструмент▁векселей▁при▁рефинансировании▁коммерческих▁банков▁.▁&'▁ ▁мы▁будем▁переходить▁к▁использованию▁векселей▁как▁залога▁,▁что▁даст▁нам▁возможность▁более▁четко▁определять▁стоимость▁денежных▁ресурсов▁&'▁,▁-▁сказала▁ ▁гребеник▁.▁-▁ ▁наталия▁ ▁зинец▁,▁ ▁киевское▁бюро▁,▁(▁044▁)▁244▁9150▁.▁(▁c▁)▁ ▁reuters▁ ▁limited▁1997▁. size=0
|
||||
/sentencepiece/src/trainer_interface.cc(200) LOG(INFO) Too long lines (>=20480 bytes (it can be changed with --max_sentence_length flag). Skipped.
|
||||
/sentencepiece/src/trainer_interface.cc(200) LOG(INFO) Too long lines (>=20480 bytes (it can be changed with --max_sentence_length flag). Skipped.
|
||||
/sentencepiece/src/trainer_interface.cc(240) LOG(INFO) Loaded 998 sentences
|
||||
/sentencepiece/src/trainer_interface.cc(241) LOG(INFO) Loaded 0 test sentences
|
||||
/sentencepiece/src/trainer_interface.cc(265) LOG(INFO) all chars count=1565524
|
||||
/sentencepiece/src/trainer_interface.cc(273) LOG(INFO) Done: 99.1426% characters are covered.
|
||||
/sentencepiece/src/trainer_interface.cc(283) LOG(INFO) Alphabet size=68
|
||||
/sentencepiece/src/trainer_interface.cc(284) LOG(INFO) Final character coverage=0.991426
|
||||
/sentencepiece/src/trainer_interface.cc(316) LOG(INFO) Done! 998 sentences are loaded
|
||||
/sentencepiece/src/unigram_model_trainer.cc(127) LOG(INFO) Using 998 sentences for making seed sentencepieces
|
||||
/sentencepiece/src/unigram_model_trainer.cc(155) LOG(INFO) Making suffix array...
|
||||
/sentencepiece/src/unigram_model_trainer.cc(159) LOG(INFO) Extracting frequent sub strings...
|
||||
/sentencepiece/src/unigram_model_trainer.cc(210) LOG(INFO) Initialized 67755 seed sentencepieces
|
||||
/sentencepiece/src/trainer_interface.cc(322) LOG(INFO) Tokenizing input sentences with whitespace: 998
|
||||
/sentencepiece/src/trainer_interface.cc(331) LOG(INFO) Done! 31975
|
||||
/sentencepiece/src/unigram_model_trainer.cc(502) LOG(INFO) Using 31975 sentences for EM training
|
||||
/sentencepiece/src/unigram_model_trainer.cc(518) LOG(INFO) EM sub_iter=0 size=22877 obj=16.6184 num_tokens=70560 num_tokens/piece=3.08432
|
||||
/sentencepiece/src/unigram_model_trainer.cc(518) LOG(INFO) EM sub_iter=1 size=19595 obj=14.256 num_tokens=71915 num_tokens/piece=3.67007
|
||||
/sentencepiece/src/unigram_model_trainer.cc(518) LOG(INFO) EM sub_iter=0 size=17579 obj=14.2013 num_tokens=73054 num_tokens/piece=4.15575
|
||||
/sentencepiece/src/unigram_model_trainer.cc(518) LOG(INFO) EM sub_iter=1 size=17469 obj=14.1655 num_tokens=73390 num_tokens/piece=4.20116
|
||||
/sentencepiece/src/trainer_interface.cc(387) LOG(INFO) Saving model: data/mldoc/ru-1/models/sp16k/spm.model
|
||||
/sentencepiece/src/trainer_interface.cc(411) LOG(INFO) Saving vocabs: data/mldoc/ru-1/models/sp16k/spm.vocab
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 9195, val: 1021
|
||||
Running tokenization cls...
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 16000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', ',', '▁.', 'и', 'е', '▁в', 'й', '▁-', 'а', ')', '(']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
/home/pczapla/anaconda3/envs/fastaiv1/lib/python3.7/site-packages/torch/utils/cpp_extension.py:152: UserWarning:
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
Your compiler (c++) may be ABI-incompatible with PyTorch!
|
||||
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
|
||||
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
|
||||
|
||||
See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
|
||||
for instructions on how to install GCC 4.9 or higher.
|
||||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
||||
|
||||
!! WARNING !!
|
||||
|
||||
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
|
||||
Bptt 70
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.311670 4.217185 0.471835
|
||||
2 3.735446 3.595414 0.552916
|
||||
3 3.553164 3.354127 0.581794
|
||||
4 3.363475 3.259169 0.593828
|
||||
5 3.514256 3.261860 0.590224
|
||||
6 3.413725 3.223500 0.597156
|
||||
7 3.453391 3.182702 0.601941
|
||||
8 3.317564 3.131130 0.610511
|
||||
9 3.398653 3.092810 0.616117
|
||||
10 3.276093 3.037282 0.624851
|
||||
11 3.207109 2.980038 0.634575
|
||||
12 3.141415 2.928465 0.643130
|
||||
13 3.164837 2.878245 0.653095
|
||||
14 3.093078 2.823911 0.662821
|
||||
15 3.026668 2.770853 0.673216
|
||||
16 2.968236 2.723534 0.682577
|
||||
17 2.983422 2.690081 0.689747
|
||||
18 2.862256 2.666973 0.694282
|
||||
19 2.876733 2.656204 0.696821
|
||||
20 2.853209 2.654935 0.696994
|
||||
Total time: 39:43
|
||||
data/mldoc/ru-1/models/sp16k
|
||||
Saving info data/mldoc/ru-1/models/sp16k/qrnn_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.993379 0.788644 0.807000
|
||||
2 0.832733 0.773031 0.864000
|
||||
3 0.706515 0.715565 0.864000
|
||||
4 0.618606 0.720445 0.868000
|
||||
Total time: 00:56
|
||||
Saving models at data/mldoc/ru-1/models/sp16k/qrnn_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.39357555, tensor(0.8685)]
|
||||
0.3935755491256714
|
||||
0.8684999942779541
|
||||
```
|
||||
@@ -1,13 +0,0 @@
|
||||
\toprule
|
||||
RNN type & Vocabluary Size & Tokenization & Language & MLDoc Accuracy\\
|
||||
\midrule
|
||||
LSTM 3 & 60k & moses & DE & 94.74 \\
|
||||
LSTM 4 & 30k & sentence piece & DE & 95.40 \\
|
||||
QRNN 4 & 60k & moses & DE & 95.28 \\
|
||||
QRNN 4 & 15k & sentence piece & DE & 96.10 \\
|
||||
\midrule
|
||||
LSTM 4 & 30k & sentence piece & RU & 87.27 \\
|
||||
LSTM 4 & 15k & sentence piece & RU & 86.47 \\
|
||||
QRNN 4 & 60k & moses & RU & 87.60 \\
|
||||
QRNN 4 & 25k & sentence piece & RU & 87.20 \\
|
||||
QRNN 4 & 15k & sentence piece & RU & 87.17 \\
|
||||
@@ -1,28 +0,0 @@
|
||||
```
|
||||
% python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name
|
||||
'nl4' --max-vocab 25000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0 --label-smoothing-eps=0.10 --tokenizer='sp
|
||||
Max vocab: 25000
|
||||
Cache dir: data/wiki/ru-100/models/sp25k
|
||||
Model dir: data/wiki/ru-100/models/sp25k/qrnn_nl4.m
|
||||
Wiki text was split to 193047 articles
|
||||
Wiki text was split to 460 articles
|
||||
Data lm, trn: 193047, val: 460
|
||||
Size of vocabulary: 25000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', 'х', '▁на']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.154972 4.198218 0.447508
|
||||
2 4.030367 4.159642 0.449420
|
||||
3 4.138530 4.146010 0.451526
|
||||
4 3.997120 4.097048 0.457177
|
||||
5 3.999151 4.036350 0.465117
|
||||
6 3.935380 3.955517 0.476446
|
||||
7 3.912357 3.875987 0.487591
|
||||
8 3.785693 3.789099 0.501560
|
||||
9 3.743162 3.725730 0.512294
|
||||
10 3.690226 3.706929 0.516769
|
||||
Total time: 12:10:03
|
||||
data/wiki/ru-100/models/sp25k
|
||||
Saving info data/wiki/ru-100/models/sp25k/qrnn_nl4.m/info.json
|
||||
```
|
||||
@@ -1,183 +0,0 @@
|
||||
3 2.812496 2.877055 0.468569
|
||||
|
||||
4 2.705551 2.792535 0.479420
|
||||
|
||||
5 2.649598 2.726415 0.487439
|
||||
|
||||
6 2.599835 2.635610 0.499679
|
||||
|
||||
7 2.574639 2.554657 0.512358
|
||||
|
||||
8 2.489573 2.475936 0.523280
|
||||
|
||||
9 2.396540 2.415555 0.534089
|
||||
|
||||
10 2.374290 2.401968 0.536601
|
||||
|
||||
Total time: 15:49:20
|
||||
data/wiki/ru-100/models/sp15k
|
||||
Saving info data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/info.json
|
||||
Fire trace:
|
||||
1. Initial component
|
||||
2. Accessed property "lm" (/home/test/workspace/ulmfit-multilingual/ulmfit/__main__.py:32)
|
||||
3. Called routine "LMHyperParams" (/home/test/workspace/ulmfit-multilingual/ulmfit/__main__.py:32)
|
||||
4. Accessed property "train" (/home/test/workspace/ulmfit-multilingual/ulmfit/pretrain_lm.py:174)
|
||||
5. Called routine "train_lm" (/home/test/workspace/ulmfit-multilingual/ulmfit/pretrain_lm.py:174)
|
||||
6. ('Could not consume arg:', '--nh')
|
||||
|
||||
Type: NoneType
|
||||
String form: None
|
||||
|
||||
Usage: __main__.py lm --dataset-path data/wiki/ru-100 --tokenizer=sp --nl 4 --name nl4-wide2 --max-vocab 15000 --lang ru --qrnn=True - train 10 --bs=100 --drop_mult=0 -
|
||||
(multifit) test@test:~/workspace/ulmfit-multilingual$ less data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/info.json
|
||||
(multifit) test@test:~/workspace/ulmfit-multilingual$ CUDA_VISIBLE_DEVICES=1 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-wide2' --max-vocab 15000 --lang ${LANG} --qrnn=True - train 10 --bs=100 --drop_mult=0 ^C100 --
|
||||
(multifit) test@test:~/workspace/ulmfit-multilingual$ mv data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/ data/wiki/ru-100/models/sp15k/qrnn_nl4-2.m/
|
||||
(multifit) test@test:~/workspace/ulmfit-multilingual$ less data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/info.json^C
|
||||
(multifit) test@test:~/workspace/ulmfit-multilingual$ CUDA_VISIBLE_DEVICES=1 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-wid
|
||||
e2' --max-vocab 15000 --lang ${LANG} --qrnn=True --nh 3100 - train 10 --bs=100 --drop_mult=0
|
||||
Max vocab: 15000
|
||||
Cache dir: data/wiki/ru-100/models/sp15k
|
||||
Model dir: data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m
|
||||
^CTraceback (most recent call last):
|
||||
File "/home/test/anaconda3/envs/multifit/lib/python3.7/runpy.py", line 193, in _run_module_as_main
|
||||
"__main__", mod_spec)
|
||||
File "/home/test/anaconda3/envs/multifit/lib/python3.7/runpy.py", line 85, in _run_code
|
||||
exec(code, run_globals)
|
||||
File "/home/test/workspace/ulmfit-multilingual/ulmfit/__main__.py", line 119, in <module>
|
||||
fire.Fire(ULMFiT())
|
||||
File "/home/test/anaconda3/envs/multifit/lib/python3.7/site-packages/fire/core.py", line 127, in Fire
|
||||
component_trace = _Fire(component, args, context, name)
|
||||
File "/home/test/anaconda3/envs/multifit/lib/python3.7/site-packages/fire/core.py", line 366, in _Fire
|
||||
component, remaining_args)
|
||||
File "/home/test/anaconda3/envs/multifit/lib/python3.7/site-packages/fire/core.py", line 542, in _CallCallable
|
||||
result = fn(*varargs, **kwargs)
|
||||
File "/home/test/workspace/ulmfit-multilingual/ulmfit/pretrain_lm.py", line 176, in train_lm
|
||||
data_lm = self.load_wiki_data(bs=bs) if data_lm is None else data_lm
|
||||
File "/home/test/workspace/ulmfit-multilingual/ulmfit/pretrain_lm.py", line 253, in load_wiki_data
|
||||
train_df=read_wiki_articles(trn_path),
|
||||
File "/home/test/workspace/ulmfit-multilingual/ulmfit/pretrain_lm.py", line 48, in read_wiki_articles
|
||||
if i < len(lines)-2 and lines[i+1].strip() == "" and istitle(lines[i+2]):
|
||||
File "/home/test/workspace/ulmfit-multilingual/ulmfit/pretrain_lm.py", line 39, in istitle
|
||||
return len(re.findall(r'^ ?= [^=]* = ?$', line)) != 0
|
||||
File "/home/test/anaconda3/envs/multifit/lib/python3.7/re.py", line 223, in findall
|
||||
return _compile(pattern, flags).findall(string)
|
||||
KeyboardInterrupt
|
||||
^C
|
||||
(multifit) test@test:~/workspace/ulmfit-multilingual$ CUDA_VISIBLE_DEVICES=1 python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name 'nl4-wide2' --max-vocab 15000 --lang ${LANG} --qrnn=True --nh 3100 - train 10 --bs=100 --drop_mult=0 --label-smoothing-eps=0.1
|
||||
Max vocab: 15000
|
||||
Cache dir: data/wiki/ru-100/models/sp15k
|
||||
Model dir: data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m
|
||||
Wiki text was split to 193047 articles
|
||||
Wiki text was split to 460 articles
|
||||
Data lm, trn: 193047, val: 460
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.908233 3.950865 0.463469
|
||||
2 3.738863 3.815026 0.477703
|
||||
3 3.696502 3.779513 0.483625
|
||||
4 3.692592 3.720908 0.490143
|
||||
5 3.600519 3.652444 0.501671
|
||||
6 3.564568 3.582584 0.511550
|
||||
7 3.472859 3.493226 0.525943
|
||||
8 3.390483 3.407970 0.541749
|
||||
9 3.351620 3.344207 0.552758
|
||||
10 3.329683 3.330087 0.556380
|
||||
Total time: 51:05:43
|
||||
data/wiki/ru-100/models/sp15k
|
||||
Saving info data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/info.json
|
||||
(multifit) test@test:~/workspace/ulmfit-multilingual$ export CUDA_VISIBLE_DEVICES=1
|
||||
(multifit) test@test:~/workspace/ulmfit-multilingual$ LANG=ru
|
||||
(multifit) test@test:~/workspace/ulmfit-multilingual$ NAME=nl4-wide2
|
||||
(multifit) test@test:~/workspace/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-wide2.m
|
||||
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
|
||||
Data lm, trn: 9195, val: 1021
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/lm_best'), PosixPath('/home/test/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/qrnn_nl4-wide2.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.777423 3.222261 0.564503
|
||||
Total time: 04:35
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.292465 3.029143 0.602257
|
||||
2 3.034045 2.858176 0.634576
|
||||
3 2.943366 2.710314 0.665116
|
||||
4 2.722069 2.596702 0.687515
|
||||
5 2.819853 2.508158 0.705020
|
||||
6 2.734984 2.417240 0.724748
|
||||
7 2.674353 2.332395 0.743694
|
||||
8 2.527344 2.251373 0.762892
|
||||
9 2.473972 2.168185 0.784043
|
||||
10 2.359504 2.093983 0.803255
|
||||
11 2.287590 2.019540 0.823566
|
||||
12 2.254421 1.943832 0.845138
|
||||
13 2.203321 1.884380 0.863381
|
||||
14 2.142532 1.824186 0.881509
|
||||
15 2.121573 1.777664 0.894901
|
||||
16 2.013238 1.740772 0.905824
|
||||
17 2.026189 1.715271 0.913569
|
||||
18 1.904322 1.700163 0.917917
|
||||
19 1.889113 1.692539 0.919811
|
||||
20 1.903118 1.691033 0.920319
|
||||
Total time: 3:10:09
|
||||
/home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-wide2.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.006922 0.880313 0.788000
|
||||
2 0.823572 0.782953 0.860000
|
||||
3 0.679078 0.749164 0.872000
|
||||
4 0.579215 0.707200 0.872000
|
||||
Total time: 06:30
|
||||
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-wide2.m
|
||||
Loss and accuracy using (cls_best): [0.3935929, tensor(0.8708)]
|
||||
0.393592894077301
|
||||
0.8707500100135803
|
||||
(multifit) test@test:~/workspace/ulmfit-multilingual$ python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/mldoc/${LANG}-1/models/sp15k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME}-16 - train 0 --bs 18 --num-cls-epochs=16 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Model dir: /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-wide2-16.m
|
||||
Loading validation /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
|
||||
Data lm, trn: 9195, val: 1021
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
/home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Saving info /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-wide2-16.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.067446 0.822965 0.824000
|
||||
2 0.897088 0.845636 0.826000
|
||||
3 0.778055 0.828693 0.847000
|
||||
4 0.685080 0.893327 0.823000
|
||||
5 0.620457 0.929057 0.800000
|
||||
6 0.587644 0.802154 0.859000
|
||||
7 0.570255 0.713434 0.872000
|
||||
8 0.543071 0.705259 0.871000
|
||||
9 0.517465 0.715090 0.867000
|
||||
10 0.498291 0.695459 0.876000
|
||||
11 0.497857 0.698052 0.862000
|
||||
12 0.486924 0.681911 0.878000
|
||||
13 0.479041 0.676714 0.874000
|
||||
14 0.475131 0.677843 0.878000
|
||||
15 0.467238 0.672065 0.876000
|
||||
16 0.476889 0.680850 0.875000
|
||||
Total time: 23:47
|
||||
Saving models at /home/test/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/qrnn_nl4-wide2-16.m
|
||||
Loss and accuracy using (cls_best): [0.41155785, tensor(0.8700)]
|
||||
0.4115578532218933
|
||||
0.8700000047683716
|
||||
@@ -1,280 +0,0 @@
|
||||
|
||||
TOK=sp15k
|
||||
NAME=e8avg
|
||||
for LANG in ru fr; do
|
||||
for i in 1 2 3 4 5 6 7 8 9; do python -m ulmfit eval --glob="mldoc/${LANG}-1/models/${TOK}/qrnn_nl4.m" --name ${NAME}_$i --num-cls-epochs=8 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=18; done
|
||||
done
|
||||
TOK=sp15k
|
||||
NAME=avg
|
||||
for LANG in ru fr; do
|
||||
for i in 1 2 3 4 5 6 7 8 9; do python -m ulmfit eval --glob="mldoc/${LANG}-1/models/${TOK}/qrnn_nl4.m" --name ${NAME}_$i --num-cls-epochs=4 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=18; done
|
||||
done
|
||||
|
||||
NAME=e8avg
|
||||
TOK=vf60k
|
||||
for LANG in ru fr; do
|
||||
for i in 1 2 3 4 5 6 7 8 9; do python -m ulmfit eval --glob="mldoc/${LANG}-1/models/${TOK}/qrnn_nl4.m" --name ${NAME}_$i --num-cls-epochs=8 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=18; done
|
||||
done
|
||||
NAME=avg
|
||||
TOK=vf60k
|
||||
for LANG in ru fr; do
|
||||
for i in 1 2 3 4 5 6 7 8 9; do python -m ulmfit eval --glob="mldoc/${LANG}-1/models/${TOK}/qrnn_nl4.m" --name ${NAME}_$i --num-cls-epochs=4 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=18; done
|
||||
done
|
||||
|
||||
|
||||
for TOK in vf60k sp15k; do
|
||||
for LANG in ru fr; do
|
||||
NAME=e8avg
|
||||
for i in 1 2 3 4 5 6 7 8 9; do python -m ulmfit eval --glob="mldoc/${LANG}-1/models/${TOK}/qrnn_nl4.m" --name ${NAME}_$i --num-cls-epochs=8 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=18; done
|
||||
done
|
||||
done
|
||||
|
||||
for TOK in vf60k sp15k; do
|
||||
for LANG in ru fr; do
|
||||
NAME=avg
|
||||
for i in 1 2 3 4 5 6 7 8 9; do python -m ulmfit eval --glob="mldoc/${LANG}-1/models/${TOK}/qrnn_nl4.m" --name ${NAME}_$i --num-cls-epochs=4 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=18; done
|
||||
done
|
||||
done
|
||||
LANG=es
|
||||
for i in 1 2 3 4 5 6 7 9; do python -m ulmfit eval --glob="mldoc/${LANG}-1/models/sp15k/qrnn_nl4.m" --name avg_$i --num-cls-epochs=4 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=18; done
|
||||
|
||||
LANG=de
|
||||
for i in 1 2 3 4 5 6 7 9; do python -m ulmfit eval --glob="mldoc/${LANG}-1/models/sp15k/qrnn_nl4.m" --name avg_$i --num-cls-epochs=4 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=18; done
|
||||
|
||||
## epoch 8
|
||||
vf60k
|
||||
ds de-1 es-1 fr-1 it-1 ru-1
|
||||
best 95.45 96.17 94.77 90.72 87.85
|
||||
max 95.63 96.43 95.32 91.05 88.30
|
||||
avg 95.42 96.05 95.07 90.59 87.80
|
||||
|
||||
sp15k
|
||||
ds de-1 es-1 fr-1 it-1 ru-1
|
||||
best 96.17 95.92 94.55 90.45 86.95
|
||||
max 96.28 96.03 95.10 90.72 87.45
|
||||
avg 96.01 95.72 94.63 90.37 86.95
|
||||
|
||||
|
||||
-0--
|
||||
0 data/mldoc/fr-1/models/vf60k/qrnn_e8avg_1.m 0.95325 0.211328 0.951 0.211664
|
||||
1 data/mldoc/fr-1/models/vf60k/qrnn_e8avg_2.m 0.95075 0.199939 0.947 0.198606
|
||||
2 data/mldoc/fr-1/models/vf60k/qrnn_e8avg_3.m 0.95125 0.217569 0.952 0.215529
|
||||
3 data/mldoc/fr-1/models/vf60k/qrnn_e8avg_4.m 0.95225 0.208047 0.951 0.203784
|
||||
4 data/mldoc/fr-1/models/vf60k/qrnn_e8avg_5.m 0.95025 0.206937 0.946 0.206194
|
||||
5 data/mldoc/fr-1/models/vf60k/qrnn_e8avg_6.m 0.95075 0.203967 0.951 0.204809
|
||||
6 data/mldoc/fr-1/models/vf60k/qrnn_e8avg_7.m 0.94775 0.211408 0.954 0.201543
|
||||
7 data/mldoc/fr-1/models/vf60k/qrnn_e8avg_8.m 0.95075 0.202703 0.952 0.197218
|
||||
8 data/mldoc/fr-1/models/vf60k/qrnn_e8avg_9.m 0.94925 0.207698 0.947 0.204896
|
||||
9 data/mldoc/ru-1/models/vf60k/qrnn_avg_1.m 0.88300 0.375823 0.876 0.364029
|
||||
10 data/mldoc/ru-1/models/vf60k/qrnn_avg_2.m 0.87675 0.386695 0.883 0.356660
|
||||
11 data/mldoc/ru-1/models/vf60k/qrnn_avg_3.m 0.87750 0.372321 0.879 0.374368
|
||||
12 data/mldoc/ru-1/models/vf60k/qrnn_avg_4.m 0.87400 0.379490 0.875 0.370343
|
||||
13 data/mldoc/ru-1/models/vf60k/qrnn_avg_5.m 0.87725 0.380067 0.877 0.367522
|
||||
14 data/mldoc/ru-1/models/vf60k/qrnn_avg_6.m 0.87525 0.393280 0.874 0.368825
|
||||
15 data/mldoc/ru-1/models/vf60k/qrnn_avg_7.m 0.87900 0.380393 0.882 0.373376
|
||||
16 data/mldoc/ru-1/models/vf60k/qrnn_avg_8.m 0.88025 0.376825 0.875 0.375059
|
||||
17 data/mldoc/ru-1/models/vf60k/qrnn_avg_9.m 0.88125 0.380887 0.884 0.367705
|
||||
18 data/mldoc/ru-1/models/vf60k/qrnn_e8avg_1.m 0.87850 0.385976 0.888 0.385302
|
||||
19 data/mldoc/ru-1/models/vf60k/qrnn_e8avg_2.m 0.87600 0.384469 0.879 0.384878
|
||||
20 data/mldoc/ru-1/models/vf60k/qrnn_e8avg_4.m 0.87600 0.386223 0.870 0.391646
|
||||
21 data/mldoc/ru-1/models/vf60k/qrnn_e8avg_5.m 0.87950 0.385152 0.885 0.371122
|
||||
22 data/mldoc/ru-1/models/vf60k/qrnn_e8avg_6.m 0.88175 0.383746 0.875 0.391054
|
||||
23 data/mldoc/ru-1/models/vf60k/qrnn_e8avg_7.m 0.87850 0.392120 0.874 0.382000
|
||||
24 data/mldoc/ru-1/models/vf60k/qrnn_e8avg_8.m 0.87250 0.394342 0.881 0.378663
|
||||
25 data/mldoc/ru-1/models/vf60k/qrnn_e8avg_9.m 0.87950 0.387625 0.881 0.380548
|
||||
ds fr-1 ru-1
|
||||
best 94.77 87.85
|
||||
max 95.32 88.30
|
||||
avg 95.07 87.80
|
||||
---
|
||||
|
||||
# epoch 4
|
||||
## SP15k
|
||||
name tst_accuracy tst_loss val_accuracy val_loss
|
||||
0 data/mldoc/de-1/models/sp15k/qrnn_avg_1.m 0.95700 0.142444 0.945 0.206329
|
||||
1 data/mldoc/de-1/models/sp15k/qrnn_avg_2.m 0.95975 0.141225 0.955 0.189209
|
||||
2 data/mldoc/de-1/models/sp15k/qrnn_avg_3.m 0.95925 0.140172 0.946 0.198159
|
||||
3 data/mldoc/de-1/models/sp15k/qrnn_avg_4.m 0.95650 0.145889 0.942 0.193202
|
||||
4 data/mldoc/de-1/models/sp15k/qrnn_avg_5.m 0.96000 0.137710 0.953 0.192368
|
||||
5 data/mldoc/de-1/models/sp15k/qrnn_avg_6.m 0.95975 0.145318 0.954 0.196413
|
||||
6 data/mldoc/de-1/models/sp15k/qrnn_avg_7.m 0.95925 0.141719 0.943 0.199226
|
||||
7 data/mldoc/de-1/models/sp15k/qrnn_avg_8.m 0.96100 0.140644 0.949 0.204398
|
||||
8 data/mldoc/de-1/models/sp15k/qrnn_avg_9.m 0.96050 0.143330 0.949 0.189472
|
||||
9 data/mldoc/es-1/models/sp15k/qrnn_avg_1.m 0.95700 0.149081 0.963 0.162808
|
||||
10 data/mldoc/es-1/models/sp15k/qrnn_avg_2.m 0.95875 0.141572 0.963 0.150970
|
||||
11 data/mldoc/es-1/models/sp15k/qrnn_avg_3.m 0.95900 0.151387 0.962 0.157047
|
||||
12 data/mldoc/es-1/models/sp15k/qrnn_avg_4.m 0.95375 0.165935 0.956 0.182161
|
||||
13 data/mldoc/es-1/models/sp15k/qrnn_avg_5.m 0.95850 0.151109 0.960 0.156376
|
||||
14 data/mldoc/es-1/models/sp15k/qrnn_avg_6.m 0.95800 0.150724 0.961 0.152761
|
||||
15 data/mldoc/es-1/models/sp15k/qrnn_avg_7.m 0.95875 0.142476 0.963 0.151585
|
||||
16 data/mldoc/es-1/models/sp15k/qrnn_avg_8.m 0.95525 0.165120 0.957 0.164723
|
||||
17 data/mldoc/es-1/models/sp15k/qrnn_avg_9.m 0.95725 0.151323 0.960 0.156123
|
||||
18 data/mldoc/it-1/models/sp15k/qrnn_avg_1.m 0.90175 0.312996 0.900 0.297118
|
||||
19 data/mldoc/it-1/models/sp15k/qrnn_avg_2.m 0.90250 0.316763 0.903 0.274423
|
||||
20 data/mldoc/it-1/models/sp15k/qrnn_avg_3.m 0.89900 0.329157 0.915 0.290098
|
||||
21 data/mldoc/it-1/models/sp15k/qrnn_avg_4.m 0.90100 0.322112 0.907 0.285727
|
||||
22 data/mldoc/it-1/models/sp15k/qrnn_avg_5.m 0.90100 0.308545 0.910 0.276683
|
||||
23 data/mldoc/it-1/models/sp15k/qrnn_avg_6.m 0.90275 0.323594 0.915 0.287004
|
||||
24 data/mldoc/it-1/models/sp15k/qrnn_avg_7.m 0.89925 0.295158 0.910 0.269167
|
||||
25 data/mldoc/it-1/models/sp15k/qrnn_avg_9.m 0.90325 0.312664 0.908 0.297471
|
||||
ds de-1 es-1 it-1
|
||||
best 95.97 95.70 89.90
|
||||
max 96.10 95.90 90.32
|
||||
avg 95.92 95.74 90.13
|
||||
|
||||
## VF60k
|
||||
name tst_accuracy tst_loss val_accuracy val_loss
|
||||
0 data/mldoc/de-1/models/vf60k/qrnn_avg.m 0.95250 0.193797 0.946 0.225316
|
||||
1 data/mldoc/de-1/models/vf60k/qrnn_avg_1.m 0.95575 0.157327 0.947 0.189885
|
||||
2 data/mldoc/de-1/models/vf60k/qrnn_avg_2.m 0.95400 0.174519 0.947 0.201792
|
||||
3 data/mldoc/de-1/models/vf60k/qrnn_avg_3.m 0.95325 0.180489 0.947 0.208106
|
||||
4 data/mldoc/de-1/models/vf60k/qrnn_avg_4.m 0.95425 0.161056 0.949 0.199169
|
||||
5 data/mldoc/de-1/models/vf60k/qrnn_avg_5.m 0.94775 0.182012 0.941 0.210262
|
||||
6 data/mldoc/de-1/models/vf60k/qrnn_avg_6.m 0.95375 0.164578 0.947 0.198632
|
||||
7 data/mldoc/de-1/models/vf60k/qrnn_avg_7.m 0.95575 0.152596 0.947 0.196844
|
||||
8 data/mldoc/de-1/models/vf60k/qrnn_avg_8.m 0.95350 0.167661 0.942 0.203538
|
||||
9 data/mldoc/es-1/models/vf60k/qrnn_avg.m 0.95950 0.146121 0.961 0.161852
|
||||
10 data/mldoc/es-1/models/vf60k/qrnn_avg_1.m 0.95500 0.154836 0.960 0.176217
|
||||
11 data/mldoc/es-1/models/vf60k/qrnn_avg_2.m 0.95850 0.154539 0.961 0.163008
|
||||
12 data/mldoc/es-1/models/vf60k/qrnn_avg_3.m 0.96100 0.151916 0.966 0.169869
|
||||
13 data/mldoc/es-1/models/vf60k/qrnn_avg_4.m 0.95825 0.144630 0.962 0.144410
|
||||
14 data/mldoc/es-1/models/vf60k/qrnn_avg_5.m 0.95675 0.155685 0.960 0.175439
|
||||
15 data/mldoc/es-1/models/vf60k/qrnn_avg_6.m 0.95900 0.143995 0.959 0.164156
|
||||
16 data/mldoc/es-1/models/vf60k/qrnn_avg_7.m 0.95800 0.144662 0.962 0.162957
|
||||
17 data/mldoc/es-1/models/vf60k/qrnn_avg_8.m 0.95850 0.149185 0.962 0.163159
|
||||
18 data/mldoc/it-1/models/vf60k/qrnn_avg.m 0.89925 0.320389 0.912 0.272104
|
||||
19 data/mldoc/it-1/models/vf60k/qrnn_avg_1.m 0.90525 0.305978 0.920 0.255507
|
||||
20 data/mldoc/it-1/models/vf60k/qrnn_avg_2.m 0.90725 0.287647 0.917 0.245568
|
||||
21 data/mldoc/it-1/models/vf60k/qrnn_avg_3.m 0.89925 0.313870 0.910 0.271480
|
||||
22 data/mldoc/it-1/models/vf60k/qrnn_avg_4.m 0.91125 0.285618 0.915 0.255942
|
||||
23 data/mldoc/it-1/models/vf60k/qrnn_avg_5.m 0.91100 0.288841 0.911 0.255724
|
||||
24 data/mldoc/it-1/models/vf60k/qrnn_avg_6.m 0.90525 0.287412 0.914 0.253394
|
||||
25 data/mldoc/it-1/models/vf60k/qrnn_avg_7.m 0.90000 0.308104 0.910 0.256991
|
||||
26 data/mldoc/it-1/models/vf60k/qrnn_avg_8.m 0.90450 0.301262 0.918 0.251368
|
||||
ds de-1 es-1 it-1
|
||||
best 95.42 96.10 90.53
|
||||
max 95.57 96.10 91.12
|
||||
avg 95.34 95.83 90.48
|
||||
|
||||
|
||||
|
||||
# IT
|
||||
## VF60k - 9 runs eval
|
||||
for i in 1 2 3 4 5 6 7 8; do python -m ulmfit eval --glob="mldoc/${LANG}-1/models/vf60k/qrnn_nl4.m" --name avg_$i --num-cls-epochs=4 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=10; done
|
||||
python -m ulmfit eval --glob="mldoc/${LANG}-1/models/vf60k/qrnn_nl4*.m" --train=False
|
||||
|
||||
name tst_accuracy tst_loss val_accuracy val_loss
|
||||
0 data/mldoc/it-1/models/vf60k/qrnn_nl4.m 0.89925 0.320389 0.912 0.272104
|
||||
1 data/mldoc/it-1/models/vf60k/qrnn_nl4_1.m 0.90525 0.305978 0.920 0.255507
|
||||
2 data/mldoc/it-1/models/vf60k/qrnn_nl4_2.m 0.90725 0.287647 0.917 0.245568
|
||||
3 data/mldoc/it-1/models/vf60k/qrnn_nl4_3.m 0.89925 0.313870 0.910 0.271480
|
||||
4 data/mldoc/it-1/models/vf60k/qrnn_nl4_4.m 0.91125 0.285618 0.915 0.255942
|
||||
5 data/mldoc/it-1/models/vf60k/qrnn_nl4_5.m 0.91100 0.288841 0.911 0.255724
|
||||
6 data/mldoc/it-1/models/vf60k/qrnn_nl4_6.m 0.90525 0.287412 0.914 0.253394
|
||||
7 data/mldoc/it-1/models/vf60k/qrnn_nl4_7.m 0.90000 0.308104 0.910 0.256991
|
||||
8 data/mldoc/it-1/models/vf60k/qrnn_nl4_8.m 0.90450 0.301262 0.918 0.251368
|
||||
ds it-1
|
||||
best 90.53
|
||||
max 91.12
|
||||
avg 90.48
|
||||
|
||||
## sp15k - 9 runs eval
|
||||
LANG=it
|
||||
for i in 1 2 3 4 5 6 7 9; do python -m ulmfit eval --glob="mldoc/${LANG}-1/models/sp15k/qrnn_nl4.m" --name avg_$i --num-cls-epochs=4 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=18; done
|
||||
python -m ulmfit eval --glob="mldoc/${LANG}-1/models/sp15k/qrnn_nl4_a*.m" --train=False
|
||||
name tst_accuracy tst_loss val_accuracy val_loss
|
||||
0 data/mldoc/it-1/models/sp15k/qrnn_avg_1.m 0.90175 0.312996 0.900 0.297118
|
||||
1 data/mldoc/it-1/models/sp15k/qrnn_avg_2.m 0.90250 0.316763 0.903 0.274423
|
||||
2 data/mldoc/it-1/models/sp15k/qrnn_avg_3.m 0.89900 0.329157 0.915 0.290098
|
||||
3 data/mldoc/it-1/models/sp15k/qrnn_avg_4.m 0.90100 0.322112 0.907 0.285727
|
||||
4 data/mldoc/it-1/models/sp15k/qrnn_avg_5.m 0.90100 0.308545 0.910 0.276683
|
||||
5 data/mldoc/it-1/models/sp15k/qrnn_avg_6.m 0.90275 0.323594 0.915 0.287004
|
||||
6 data/mldoc/it-1/models/sp15k/qrnn_avg_7.m 0.89925 0.295158 0.910 0.269167
|
||||
7 data/mldoc/it-1/models/sp15k/qrnn_avg_9.m 0.90325 0.312664 0.908 0.297471
|
||||
ds it-1
|
||||
best 89.90
|
||||
max 90.32
|
||||
avg 90.13
|
||||
|
||||
|
||||
# ES
|
||||
## VF60k - 8 runs eval
|
||||
for i in 1 2 3 4 5 6 7 8; do python -m ulmfit eval --glob="mldoc/${LANG}-1/models/vf60k/qrnn_nl4.m" --name avg_$i --num-cls-epochs=4 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=10; done
|
||||
python -m ulmfit eval --glob="mldoc/${LANG}-1/models/vf60k/qrnn_nl4*.m" --train=False
|
||||
|
||||
name tst_accuracy tst_loss val_accuracy val_loss
|
||||
0 data/mldoc/es-1/models/vf60k/qrnn_nl4.m 0.95950 0.146121 0.961 0.161852
|
||||
1 data/mldoc/es-1/models/vf60k/qrnn_nl4_1.m 0.95500 0.154836 0.960 0.176217
|
||||
2 data/mldoc/es-1/models/vf60k/qrnn_nl4_2.m 0.95850 0.154539 0.961 0.163008
|
||||
3 data/mldoc/es-1/models/vf60k/qrnn_nl4_3.m 0.96100 0.151916 0.966 0.169869
|
||||
4 data/mldoc/es-1/models/vf60k/qrnn_nl4_4.m 0.95825 0.144630 0.962 0.144410
|
||||
5 data/mldoc/es-1/models/vf60k/qrnn_nl4_5.m 0.95675 0.155685 0.960 0.175439
|
||||
6 data/mldoc/es-1/models/vf60k/qrnn_nl4_6.m 0.95900 0.143995 0.959 0.164156
|
||||
7 data/mldoc/es-1/models/vf60k/qrnn_nl4_7.m 0.95800 0.144662 0.962 0.162957
|
||||
8 data/mldoc/es-1/models/vf60k/qrnn_nl4_8.m 0.95850 0.149185 0.962 0.163159
|
||||
ds es-1
|
||||
best 96.10
|
||||
max 96.10
|
||||
avg 95.83
|
||||
|
||||
|
||||
## sp15k - 8 runs eval
|
||||
LANG=es
|
||||
for i in 1 2 3 4 5 6 7 9; do python -m ulmfit eval --glob="mldoc/${LANG}-1/models/sp15k/qrnn_nl4.m" --name avg_$i --num-cls-epochs=4 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=18; done
|
||||
python -m ulmfit eval --glob="mldoc/${LANG}-1/models/sp15k/qrnn_avg*.m" --train=False
|
||||
name tst_accuracy tst_loss val_accuracy val_loss
|
||||
0 data/mldoc/es-1/models/sp15k/qrnn_avg_1.m 0.95700 0.149081 0.963 0.162808
|
||||
1 data/mldoc/es-1/models/sp15k/qrnn_avg_2.m 0.95875 0.141572 0.963 0.150970
|
||||
2 data/mldoc/es-1/models/sp15k/qrnn_avg_3.m 0.95900 0.151387 0.962 0.157047
|
||||
3 data/mldoc/es-1/models/sp15k/qrnn_avg_4.m 0.95375 0.165935 0.956 0.182161
|
||||
4 data/mldoc/es-1/models/sp15k/qrnn_avg_5.m 0.95850 0.151109 0.960 0.156376
|
||||
5 data/mldoc/es-1/models/sp15k/qrnn_avg_6.m 0.95800 0.150724 0.961 0.152761
|
||||
6 data/mldoc/es-1/models/sp15k/qrnn_avg_7.m 0.95875 0.142476 0.963 0.151585
|
||||
7 data/mldoc/es-1/models/sp15k/qrnn_avg_8.m 0.95525 0.165120 0.957 0.164723
|
||||
8 data/mldoc/es-1/models/sp15k/qrnn_avg_9.m 0.95725 0.151323 0.960 0.156123
|
||||
ds es-1
|
||||
best 95.70
|
||||
max 95.90
|
||||
avg 95.74
|
||||
|
||||
# DE
|
||||
## VF60k - 9 runs eval
|
||||
for i in 1 2 3 4 5 6 7 8; do python -m ulmfit eval --glob="mldoc/${LANG}-1/models/vf60k/qrnn_nl4.m" --name avg_$i --num-cls-epochs=4 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=10; done
|
||||
python -m ulmfit eval --glob="mldoc/${LANG}-1/models/vf60k/qrnn_nl4*.m" --train=False
|
||||
name tst_accuracy tst_loss val_accuracy val_loss
|
||||
0 data/mldoc/de-1/models/vf60k/qrnn_nl4.m 0.95250 0.193797 0.946 0.225316
|
||||
1 data/mldoc/de-1/models/vf60k/qrnn_nl4_1.m 0.95575 0.157327 0.947 0.189885
|
||||
2 data/mldoc/de-1/models/vf60k/qrnn_nl4_2.m 0.95400 0.174519 0.947 0.201792
|
||||
3 data/mldoc/de-1/models/vf60k/qrnn_nl4_3.m 0.95325 0.180489 0.947 0.208106
|
||||
4 data/mldoc/de-1/models/vf60k/qrnn_nl4_4.m 0.95425 0.161056 0.949 0.199169
|
||||
5 data/mldoc/de-1/models/vf60k/qrnn_nl4_5.m 0.94775 0.182012 0.941 0.210262
|
||||
6 data/mldoc/de-1/models/vf60k/qrnn_nl4_6.m 0.95375 0.164578 0.947 0.198632
|
||||
7 data/mldoc/de-1/models/vf60k/qrnn_nl4_7.m 0.95575 0.152596 0.947 0.196844
|
||||
8 data/mldoc/de-1/models/vf60k/qrnn_nl4_8.m 0.95350 0.167661 0.942 0.203538
|
||||
ds de-1
|
||||
best 95.42
|
||||
max 95.57
|
||||
avg 95.34
|
||||
|
||||
## sp15k - 8 runs eval
|
||||
for i in 1 2 3 4 5 6 7 9; do python -m ulmfit eval --glob="mldoc/${LANG}-1/models/sp15k/qrnn_nl4.m" --name avg_$i --num-cls-epochs=4 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=18; done
|
||||
python -m ulmfit eval --glob="mldoc/${LANG}-1/models/sp15k/qrnn_avg*.m" --train=False
|
||||
name tst_accuracy tst_loss val_accuracy val_loss
|
||||
0 data/mldoc/de-1/models/sp15k/qrnn_avg_1.m 0.95700 0.142444 0.945 0.206329
|
||||
1 data/mldoc/de-1/models/sp15k/qrnn_avg_2.m 0.95975 0.141225 0.955 0.189209
|
||||
2 data/mldoc/de-1/models/sp15k/qrnn_avg_3.m 0.95925 0.140172 0.946 0.198159
|
||||
3 data/mldoc/de-1/models/sp15k/qrnn_avg_4.m 0.95650 0.145889 0.942 0.193202
|
||||
4 data/mldoc/de-1/models/sp15k/qrnn_avg_5.m 0.96000 0.137710 0.953 0.192368
|
||||
5 data/mldoc/de-1/models/sp15k/qrnn_avg_6.m 0.95975 0.145318 0.954 0.196413
|
||||
6 data/mldoc/de-1/models/sp15k/qrnn_avg_7.m 0.95925 0.141719 0.943 0.199226
|
||||
7 data/mldoc/de-1/models/sp15k/qrnn_avg_8.m 0.96100 0.140644 0.949 0.204398
|
||||
8 data/mldoc/de-1/models/sp15k/qrnn_avg_9.m 0.96050 0.143330 0.949 0.189472
|
||||
ds de-1
|
||||
best 95.97
|
||||
max 96.10
|
||||
avg 95.92
|
||||
|
||||
# RU
|
||||
## VF60k - 9 runs eval
|
||||
for i in 1 2 3 4 5 6 7 8; do python -m ulmfit eval --glob="mldoc/${LANG}-1/models/vf60k/qrnn_nl4.m" --name nl4_$i --num-cls-epochs=4 --label-smoothing-eps=0.1 --lr_sched=1cycle --bs=10; done
|
||||
python -m ulmfit eval --glob="mldoc/${LANG}-1/models/vf60k/qrnn_nl4*.m" --train=False
|
||||
@@ -1,241 +0,0 @@
|
||||
# FR
|
||||
## LM
|
||||
```
|
||||
LANG=fr
|
||||
python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='vf' --nl 4 --name 'nl4' --max-vocab 60000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0 --label-smoothing-eps=0.0
|
||||
Max vocab: 60000
|
||||
Cache dir: data/wiki/fr-100/models/vf60k
|
||||
Model dir: data/wiki/fr-100/models/vf60k/qrnn_nl4.m
|
||||
Wiki text was split to 174227 articles
|
||||
Wiki text was split to 491 articles
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 174227, val: 491
|
||||
Size of vocabulary: 60003
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', 'de', ',', '\n', '.', 'la', 'le', 'et', 'à', 'en', "l'", "&'", 'les']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Bptt 70
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.359852 3.022507 0.434433
|
||||
2 3.253006 2.955765 0.435078
|
||||
3 3.274156 2.917242 0.442870
|
||||
4 3.181276 2.850124 0.451273
|
||||
5 3.169587 2.813115 0.456411
|
||||
6 3.075235 2.773676 0.462836
|
||||
7 3.054632 2.723182 0.469485
|
||||
8 2.964262 2.661821 0.479831
|
||||
9 3.019209 2.631244 0.487013
|
||||
10 2.899521 2.618838 0.489004
|
||||
Total time: 10:48:33
|
||||
data/wiki/fr-100/models/vf60k
|
||||
Saving info data/wiki/fr-100/models/vf60k/qrnn_nl4.m/info.json
|
||||
```
|
||||
## CLS
|
||||
|
||||
|
||||
|
||||
# ES
|
||||
## LM
|
||||
```
|
||||
LANG=es
|
||||
python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='vf' --nl 4 --name 'nl4' --max-vocab 60000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0 --label-smoothing-eps=0.0
|
||||
Max vocab: 60000
|
||||
Cache dir: data/wiki/es-100/models/vf60k
|
||||
Model dir: data/wiki/es-100/models/vf60k/qrnn_nl4.m
|
||||
Wiki text was split to 161509 articles
|
||||
Wiki text was split to 78 articles
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 161509, val: 78
|
||||
Size of vocabulary: 60003
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', 'de', ',', '\n', '.', 'la', 'el', 'en', 'y', 'a', "&'", 'que', 'los']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Bptt 70
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.285345 3.884676 0.312458
|
||||
2 3.157721 3.832607 0.313905
|
||||
3 3.193605 3.800210 0.316862
|
||||
4 3.152273 3.747068 0.319891
|
||||
5 3.028921 3.713120 0.324912
|
||||
6 3.067516 3.652925 0.330345
|
||||
7 3.006576 3.571537 0.339488
|
||||
8 2.922181 3.529282 0.345483
|
||||
9 2.871947 3.497736 0.352535
|
||||
10 2.862057 3.491642 0.354063
|
||||
Total time: 14:46:42
|
||||
data/wiki/es-100/models/vf60k
|
||||
Saving info data/wiki/es-100/models/vf60k/qrnn_nl4.m/info.json
|
||||
```
|
||||
## MLDoc
|
||||
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/vf60k/qrnn_nl4.m --lang=${LANG} --name nl4 - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
Max vocab: 60000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/vf60k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/vf60k/qrnn_nl4.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/es.dev.csv
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 13013, val: 1445
|
||||
Running tokenization cls...
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 34317
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', 'de', ',', '.', 'el', 'la', 'a', 'en', ')', '(', 'y', 'los', 'que']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 17152, first 100: ['pct', 'reuter', 'corresponsalía', 'mln', 'indice', 'cotizaba', 'mlns', '585-8308', 'francfort', 'oct', 'jul', 'abr', '585-2154', 'ibex-35', 'feb', 'ibex', 'ago', '585-2152', 'bundesbank', 'ftse', '585-2196', 'interanual', '585-2159', 'cac-40', 'cotizaban', 'uem', 'm.m', '10a', 'alcista', 'bbv', 'anoche', 'argentaria', 'pagarés', 'btp', 'transferibles', 'c.l.p', 'bch', '8,80', '585-8315', 'corros', 'retevisión', '7,35', 'spread', 'bln', 'cnmv', 'decenal', 'opv', 'vespertina', 'greenspan', 'alzas', 'nikkei', 'cambista', 'tir', 'preapertura', 'mibtel', 'tabacalera', 'ptas', 'día-día', 'diff', '18-26', '6-12', 'dif.d.ant', 'max.año', 'min.año', 'spi', 'inem', 'indust', 'fecsa', 'securities', 'repos', 'fomc', 'obligs', 'mibor', 'descartaban', 'sepi', 'interbancario', 'tietmeyer', '5,50', 'piqué', '6,75', 'aprobacion', 'moscu', 'brutas', 'deficit', '0830', 'buba', 'g-7', 'waigel', 'stet', 'petróleo-químicas', '.ibex', '5,25', '6,00', '3m', '5,30', 'trimestrales', 'cauto', 'smi', 'ant-', 'facilitadas']
|
||||
Bptt 70
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/es-100/models/vf60k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/es-100/models/vf60k/qrnn_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.714449 2.774868 0.469673
|
||||
Total time: 01:42
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.239635 2.591123 0.496131
|
||||
2 2.935826 2.367645 0.535486
|
||||
3 2.631979 2.196012 0.564117
|
||||
4 2.640709 2.058490 0.582902
|
||||
5 2.434918 1.949251 0.599310
|
||||
6 2.293211 1.855961 0.613708
|
||||
7 2.224960 1.773834 0.626423
|
||||
8 2.188689 1.698404 0.639268
|
||||
9 2.024225 1.623230 0.653119
|
||||
10 2.041964 1.555204 0.665692
|
||||
11 1.925207 1.492332 0.677868
|
||||
12 1.864637 1.421467 0.693237
|
||||
13 1.779024 1.361629 0.706401
|
||||
14 1.817028 1.301509 0.719889
|
||||
15 1.719223 1.261717 0.730797
|
||||
16 1.573684 1.221963 0.740282
|
||||
17 1.583578 1.192796 0.747645
|
||||
18 1.590957 1.174528 0.751411
|
||||
19 1.546806 1.167247 0.753300
|
||||
20 1.514999 1.165146 0.753615
|
||||
Total time: 37:16
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/vf60k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/vf60k/qrnn_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.654116 0.368231 0.907000
|
||||
2 0.447137 0.287264 0.961000
|
||||
3 0.308758 0.285717 0.958000
|
||||
4 0.216707 0.275839 0.962000
|
||||
Total time: 00:42
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/es-1/models/vf60k/qrnn_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.14618756, tensor(0.9597)] [0.16216491, tensor(0.9620)]
|
||||
val_loss: 0.16216491
|
||||
val_accuracy: 0.9620000123977661
|
||||
tst_loss: 0.14618756
|
||||
tst_accuracy: 0.9597499966621399
|
||||
```
|
||||
|
||||
|
||||
# IT
|
||||
## LM
|
||||
```
|
||||
LANG=it
|
||||
python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='vf' --nl 4 --name 'nl4' --max-vocab 60000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0 --label-smoothing-eps=0.0
|
||||
|
||||
Max vocab: 60000
|
||||
Cache dir: data/wiki/it-100/models/vf60k
|
||||
Model dir: data/wiki/it-100/models/vf60k/qrnn_nl4.m
|
||||
Wiki text was split to 164583 articles
|
||||
Wiki text was split to 98 articles
|
||||
Data lm, trn: 164583, val: 98
|
||||
Size of vocabulary: 60003
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', ',', '\n', '.', 'di', 'e', "&'", 'il', 'la', 'in', 'a', 'del', 'che']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Bptt 70
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.629171 4.075579 0.290754
|
||||
2 3.496484 4.007234 0.291424
|
||||
3 3.541803 3.973911 0.294861
|
||||
4 3.431979 3.926369 0.299076
|
||||
5 3.432869 3.880250 0.303598
|
||||
6 3.356332 3.823208 0.309304
|
||||
7 3.256672 3.760301 0.316393
|
||||
8 3.312303 3.708765 0.323862
|
||||
9 3.240380 3.670833 0.329326
|
||||
10 3.240536 3.661237 0.331286
|
||||
Total time: 15:32:22
|
||||
data/wiki/it-100/models/vf60k
|
||||
Saving info data/wiki/it-100/models/vf60k/qrnn_nl4.m/info.json
|
||||
```
|
||||
|
||||
|
||||
```bash
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/vf60k/qrnn_nl4.m --lang=${LANG} --name nl4 - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
Max vocab: 60000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/vf60k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/vf60k/qrnn_nl4.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/it.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Running tokenization cls...
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 29600
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '.', ',', 'di', 'e', ')', '(', 'il', "'", 'a', 'in', 'la', 'del']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 13370, first 100: ['pct', 'reuter', 'mld', 'mln', 'societa', 'dealer', 'btp', 'ott', 'dlr', 'attivita', 'venerdi', 'nov', 'feb', 'dic', 'stet', 'mibtel', 'bundesbank', 'bankitalia', 'mib30', 'perche', 'ipsoa', 'comit', 'cct', 'nil', 'cedola', 'puo', 'possibilita', 'lunedi', 'tranche', 'stg', 'warrant', 'stamane', 'ctz', 'giovedi', 'citta', 'ord', 'consob', 'uem', 'martedi', 'spread', 'verra', 't-bond', 'mercoledi', 'risp', 'viv', 'ffr', 'avra', 'compart', 'gmn', 'dovra', 'potra', 'fib30', 'contrattazioni', 'gemina', 'frf', 'controvalore', 'overnight', 'cir', 'apr', 'consensus', 'tendenziale', 'nikkei', 'autorita', 'tus', 'pretasse', 'fib', 'rialzi', 'fomc', 'gilt', 'circ', 'destagionalizzati', 'prec', 'liquidita', 'ecu', 'destagionalizzato', 'cariplo', 'stamani', 'obbligazionario', 'bur', 'imi', 'aggiudicazione', 'treu', 'ambroveneto', 'fixing', 'hpi', 'rnc', 'capacita', 'dietimi', 'greenspan', 'tietmeyer', 'waigel', 'nasdaq', 'eltsin', 'redditivita', 'liffe', 'telematico', 'ifil', 'interpellati', '6,25', 'visco']
|
||||
Bptt 70
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/it-100/models/vf60k/qrnn_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/it-100/models/vf60k/qrnn_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.406445 3.675336 0.338066
|
||||
Total time: 01:10
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.870676 3.516882 0.355481
|
||||
2 3.633525 3.322235 0.383076
|
||||
3 3.454955 3.121748 0.408930
|
||||
4 3.210115 2.935245 0.433205
|
||||
5 3.112426 2.775076 0.452784
|
||||
6 2.991053 2.638768 0.471221
|
||||
7 2.904022 2.533667 0.485577
|
||||
8 2.808465 2.426029 0.501932
|
||||
9 2.713658 2.320023 0.518699
|
||||
10 2.580141 2.226892 0.533786
|
||||
11 2.532727 2.133867 0.549680
|
||||
12 2.449591 2.034733 0.567797
|
||||
13 2.387805 1.963019 0.583013
|
||||
14 2.337399 1.880745 0.598986
|
||||
15 2.217255 1.818780 0.612503
|
||||
16 2.175724 1.764977 0.623581
|
||||
17 2.057536 1.726874 0.631422
|
||||
18 2.093975 1.705599 0.635835
|
||||
19 2.030292 1.694430 0.637838
|
||||
20 2.057254 1.691360 0.638669
|
||||
Total time: 32:28
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/vf60k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/vf60k/qrnn_nl4.m/info.json
|
||||
|
||||
***OOTM**
|
||||
```
|
||||
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/vf60k/qrnn_nl4.m --lang=${LANG} --name nl4 - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/vf60k/qrnn_nl4.m --lang=${LANG} --name nl4 - train 20 --bs 10 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
Max vocab: 60000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/vf60k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/vf60k/qrnn_nl4.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/it.dev.csv
|
||||
Data lm, trn: 13500, val: 1500
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 29600
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '.', ',', 'di', 'e', ')', '(', 'il', "'", 'a', 'in', 'la', 'del']
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.736275 0.717692 0.837000
|
||||
2 0.593485 0.444027 0.876000
|
||||
3 0.376322 0.411704 0.907000
|
||||
4 0.244267 0.370927 0.915000
|
||||
Total time: 00:33
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/it-1/models/vf60k/qrnn_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.3200554, tensor(0.8997)] [0.27118126, tensor(0.9150)]
|
||||
val_loss: 0.27118126
|
||||
val_accuracy: 0.9150000214576721
|
||||
tst_loss: 0.3200554
|
||||
tst_accuracy: 0.8997499942779541
|
||||
```
|
||||
@@ -1,178 +0,0 @@
|
||||
|
||||
## SP25k
|
||||
```bash
|
||||
python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='sp' --nl 4 --name
|
||||
'nl4' --max-vocab 25000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0 --label-smoothing-eps=0.10 --tokenizer='sp
|
||||
Max vocab: 25000
|
||||
Cache dir: data/wiki/ru-100/models/sp25k
|
||||
Model dir: data/wiki/ru-100/models/sp25k/qrnn_nl4.m
|
||||
Wiki text was split to 193047 articles
|
||||
Wiki text was split to 460 articles
|
||||
Data lm, trn: 193047, val: 460
|
||||
Size of vocabulary: 25000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', 'х', '▁на']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.154972 4.198218 0.447508
|
||||
2 4.030367 4.159642 0.449420
|
||||
3 4.138530 4.146010 0.451526
|
||||
4 3.997120 4.097048 0.457177
|
||||
5 3.999151 4.036350 0.465117
|
||||
6 3.935380 3.955517 0.476446
|
||||
7 3.912357 3.875987 0.487591
|
||||
8 3.785693 3.789099 0.501560
|
||||
9 3.743162 3.725730 0.512294
|
||||
10 3.690226 3.706929 0.516769
|
||||
Total time: 12:10:03
|
||||
data/wiki/ru-100/models/sp25k
|
||||
Saving info data/wiki/ru-100/models/sp25k/qrnn_nl4.m/info.json
|
||||
```
|
||||
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp25k/qrnn_${NAME}.m --lang=${LANG} --name ${NAME} - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
|
||||
Max vocab: 25000
|
||||
Cache dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp25k
|
||||
Model dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp25k/qrnn_nl4.m
|
||||
Loading validation /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 9195, val: 1021
|
||||
Running tokenization cls...
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Running tokenization tst...
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 25000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', 'х', '▁на']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp25k/qrnn_nl4.m/lm_best'), PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp25k/qrnn_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.626971 3.868075 0.474742
|
||||
Total time: 01:58
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.821786 3.625366 0.519506
|
||||
2 3.570115 3.379288 0.566803
|
||||
3 3.517294 3.179166 0.599955
|
||||
4 3.160131 3.028985 0.626484
|
||||
5 3.135806 2.923198 0.644557
|
||||
6 3.055160 2.840300 0.659376
|
||||
7 3.005086 2.770163 0.672080
|
||||
8 2.811366 2.708846 0.684065
|
||||
9 2.818394 2.658951 0.694358
|
||||
10 2.881018 2.605373 0.705269
|
||||
11 2.793422 2.560091 0.715893
|
||||
12 2.708385 2.516373 0.725908
|
||||
13 2.690258 2.471159 0.735673
|
||||
14 2.748342 2.436113 0.744533
|
||||
15 2.601220 2.394404 0.754131
|
||||
16 2.616882 2.372301 0.760451
|
||||
17 2.602902 2.349164 0.766014
|
||||
18 2.560349 2.336217 0.769222
|
||||
19 2.549936 2.332076 0.770150
|
||||
20 2.546798 2.331103 0.770472
|
||||
Total time: 53:22
|
||||
/home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp25k
|
||||
Saving info /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp25k/qrnn_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.043533 0.961182 0.731000
|
||||
2 0.859086 0.837210 0.824000
|
||||
3 0.735276 0.724173 0.871000
|
||||
4 0.612012 0.711034 0.857000
|
||||
Total time: 01:15
|
||||
Saving models at /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp25k/qrnn_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.3957597, tensor(0.8720)]
|
||||
0.3957597017288208
|
||||
0.871999979019165
|
||||
```
|
||||
|
||||
## V60k
|
||||
## VF60k
|
||||
|
||||
```
|
||||
LANG=ru
|
||||
python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='vf' --nl 4 --name 'nl4' --max-vocab 60000 --lang ${LANG} --qrnn=True - train 10 --bs=50 --drop_mult=0 --label-smoothing-eps=0.1
|
||||
Max vocab: 60000
|
||||
Cache dir: data/wiki/ru-100/models/vf60k
|
||||
Model dir: data/wiki/ru-100/models/vf60k/qrnn_nl4.m
|
||||
Wiki text was split to 193047 articles
|
||||
Wiki text was split to 460 articles
|
||||
Running tokenization lm...
|
||||
Data lm, trn: 193047, val: 460
|
||||
Size of vocabulary: 60003
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', ',', '\n', '.', 'в', 'и', ')', '(', 'на', '—', '«', '»', 'с']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 4.586900 4.478803 0.413023
|
||||
2 4.483496 4.400461 0.418495
|
||||
3 4.484620 4.390928 0.418422
|
||||
4 4.373594 4.350045 0.422567
|
||||
5 4.350337 4.307665 0.427411
|
||||
6 4.314571 4.249700 0.436324
|
||||
7 4.232540 4.183857 0.446341
|
||||
8 4.252573 4.119820 0.455522
|
||||
9 4.136978 4.088805 0.462345
|
||||
10 4.116755 4.079840 0.465394
|
||||
Total time: 11:24:03
|
||||
data/wiki/ru-100/models/vf60k
|
||||
```
|
||||
|
||||
## SP15k LSTM nl 3
|
||||
```bash
|
||||
python -m ulmfit cls --dataset-path data/mldoc/${LANG}-1 --base-lm-path data/wiki/${LANG}-100/models/sp15k/lstm_nl4.m --lang=${LANG} --name nl4 - train 20 --bs 18 --num-cls-epochs=4 --lr_sched=1cycle --label-smoothing-eps=0.1
|
||||
Max vocab: 15000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/lstm_nl4.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/ru.dev.csv
|
||||
Data lm, trn: 9195, val: 1021
|
||||
Data cls, trn: 1000, val: 1000
|
||||
Data tst, trn: 1000, val: 4000
|
||||
Size of vocabulary: 15000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁.', '▁в', 'а', 'и', 'е', '▁и', 'й', '▁на', 'х']
|
||||
Training args: {'clip': 0.12, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: {'output_p': 0.25, 'hidden_p': 0.1, 'input_p': 0.2, 'embed_p': 0.02, 'weight_p': 0.15}
|
||||
Loading pretrained model
|
||||
Unknown tokens 0, first 100: []
|
||||
Bptt 70
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/ru-100/models/sp15k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.705343 3.261906 0.558008
|
||||
Total time: 05:27
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.243956 3.073661 0.594862
|
||||
2 3.139877 2.917376 0.625388
|
||||
3 2.941367 2.786331 0.650792
|
||||
4 2.846027 2.682831 0.671712
|
||||
5 2.796714 2.600119 0.687167
|
||||
6 2.841771 2.527643 0.702408
|
||||
7 2.726931 2.459425 0.717738
|
||||
8 2.619217 2.402231 0.729743
|
||||
9 2.626002 2.349137 0.742474
|
||||
10 2.535362 2.299844 0.753796
|
||||
11 2.501980 2.257779 0.764137
|
||||
12 2.427705 2.209901 0.776203
|
||||
13 2.393852 2.167961 0.787562
|
||||
14 2.340693 2.129181 0.797972
|
||||
15 2.307895 2.094267 0.807763
|
||||
16 2.330075 2.069201 0.814278
|
||||
17 2.232444 2.049109 0.820321
|
||||
18 2.306738 2.038069 0.823257
|
||||
19 2.232783 2.031799 0.825218
|
||||
20 2.227589 2.030583 0.825465
|
||||
Total time: 2:13:57
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/lstm_nl4.m/info.json
|
||||
Single training schedule
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 1.019235 0.894525 0.820000
|
||||
2 0.885900 0.831892 0.772000
|
||||
3 0.714437 0.711899 0.865000
|
||||
4 0.608688 0.706948 0.868000
|
||||
Total time: 05:07
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/ru-1/models/sp15k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.42021805, tensor(0.8648)]
|
||||
0.4202180504798889
|
||||
0.8647500276565552
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,163 +0,0 @@
|
||||
# ZH
|
||||
|
||||
## SP15k QRNN
|
||||
|
||||
## SP30k LSTM nl 4
|
||||
### LM
|
||||
```
|
||||
python -m ulmfit lm --dataset-path data/wiki/zh-100 --cuda-id=0 --tokenizer='sp' --nl 4 --name 'nl4' --max-vocab 30000 --lang zh --qrnn=False - train 10 --bs=50 --drop_mult=0
|
||||
Max vocab: 30000
|
||||
Cache dir: data/wiki/zh-100/models/sp30k
|
||||
Model dir: data/wiki/zh-100/models/sp30k/lstm_nl4.m
|
||||
Tokenized data loaded
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁中', '▁人', '▁是']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': None, 'pretrained_model': None, 'drop_mult': 0} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.736679 3.050473 0.428462
|
||||
2 2.664505 3.011505 0.432414
|
||||
3 2.607435 2.942389 0.439985
|
||||
4 2.561503 2.851523 0.451965
|
||||
5 2.499060 2.798222 0.459438
|
||||
6 2.387191 2.720054 0.471021
|
||||
7 2.356725 2.648299 0.479029
|
||||
8 2.301895 2.553860 0.493597
|
||||
9 2.275601 2.481724 0.505979
|
||||
10 2.187606 2.465159 0.509590
|
||||
```
|
||||
### MLDoc
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/zh-1 --base-lm-path data/wiki/zh-100/models/sp30k/lstm_nl4.m --lang=zh --name 'nl4' --cuda-id=0 - train 20 --bs 40 --num-cls-epochs=2
|
||||
Max vocab: 30000
|
||||
Cache dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k
|
||||
Model dir: /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4.m
|
||||
Loading validation /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Tokenized data loaded, cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 30000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁中', '▁人', '▁是']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/zh-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/zh-100/models/sp30k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'alpha': 2, 'beta': 1, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/zh-100/models/sp30k/lstm_nl4.m/lm_best'), PosixPath('/home/pczapla/workspace/ulmfit-multilingual/data/wiki/zh-100/models/sp30k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.604460 2.225315 0.546099
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.240892 2.020697 0.578796
|
||||
2 2.025043 1.816424 0.613192
|
||||
3 1.832658 1.646025 0.640532
|
||||
4 1.746628 1.530125 0.659058
|
||||
5 1.621672 1.425179 0.675305
|
||||
6 1.544814 1.345650 0.689195
|
||||
7 1.464704 1.271710 0.702200
|
||||
8 1.412583 1.204830 0.714764
|
||||
9 1.332440 1.147108 0.725389
|
||||
10 1.327941 1.092910 0.736447
|
||||
11 1.227284 1.039441 0.747662
|
||||
12 1.200814 0.991910 0.758105
|
||||
13 1.161579 0.947898 0.768121
|
||||
14 1.100010 0.908599 0.776732
|
||||
15 1.059006 0.872309 0.785161
|
||||
16 1.045412 0.844972 0.791998
|
||||
17 1.026688 0.824872 0.796891
|
||||
18 1.013831 0.812786 0.799699
|
||||
19 0.978586 0.807678 0.800954
|
||||
20 0.982473 0.805671 0.801201
|
||||
/home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k
|
||||
Saving info /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.637427 0.505143 0.836000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.471189 0.317678 0.887000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.384985 0.288901 0.904000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.316358 0.275456 0.906000
|
||||
2 0.295534 0.278589 0.907000
|
||||
Saving models at /home/pczapla/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp30k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_best): [0.28411642, tensor(0.9020)]
|
||||
0.2841164171695709
|
||||
0.9020000100135803
|
||||
```
|
||||
|
||||
|
||||
|
||||
## SP60k LSTM nl 4
|
||||
### LM
|
||||
```
|
||||
Wiki text was split to 153503 articles
|
||||
Wiki text was split to 145 articles
|
||||
Size of vocabulary: 60000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁中', '▁是', '▁人']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': None, 'pretrained_model': None, 'drop_mult': 0} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Training lm from random weights
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.312704 3.701317 0.334740
|
||||
2 3.212988 3.648671 0.336709
|
||||
3 3.060103 3.584413 0.344427
|
||||
4 3.108131 3.477978 0.356738
|
||||
5 2.952951 3.410785 0.365901
|
||||
6 2.919397 3.325265 0.376316
|
||||
7 2.839392 3.224750 0.391707
|
||||
8 2.750095 3.132644 0.404416
|
||||
9 2.805704 3.066595 0.415245
|
||||
10 2.653435 3.055314 0.417736
|
||||
data/wiki/zh-100/models/sp60k
|
||||
Saving info data/wiki/zh-100/models/sp60k/lstm_nl4.m/info.json
|
||||
```
|
||||
### MLDoc
|
||||
```
|
||||
python -m ulmfit cls --dataset-path data/mldoc/zh-1 --base-lm-path data/wiki/zh-100/models/sp60k/lstm_nl4.m --lang=zh --name 'nl4' --cu
|
||||
da-id=0 - train 20 --bs 40 --num-cls-epochs=2
|
||||
Max vocab: 60000
|
||||
Cache dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp60k
|
||||
Model dir: /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp60k/lstm_nl4.m
|
||||
Loading validation /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/zh-1/zh.dev.csv
|
||||
Tokenized data loaded, lm.trn 13500, lm.val 1500
|
||||
Tokenized data loaded, cls.trn 1000, cls.val 1000
|
||||
Size of vocabulary: 60000
|
||||
First 20 words in vocab: ['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', 'xxwrep', '<unk>', '▁', '▁,', '▁的', '▁。', '▁年', '▁、', '▁在', '▁一', '▁中', '▁是', '▁人']
|
||||
Training args: {'tie_weights': True, 'clip': 0.12, 'bptt': 70, 'pretrained_fnames': [PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/zh-100/models/sp60k/lstm_nl4.m/lm_best'), Po
|
||||
sixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/zh-100/models/sp60k/lstm_nl4.m/../itos')], 'pretrained_model': None, 'drop_mult': 0.3} dps: [0.25 0.1 0.2 0.02 0.15]
|
||||
Unknown tokens 0, first 100: []
|
||||
Training lm from: [PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/zh-100/models/sp60k/lstm_nl4.m/lm_best'), PosixPath('/home/n-waves/workspace/ulmfit-multilingual/data/wiki/zh-
|
||||
100/models/sp60k/lstm_nl4.m/../itos')]
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 3.055914 2.690310 0.467917
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 2.713421 2.464873 0.503386
|
||||
2 2.429520 2.215309 0.543961
|
||||
3 2.247576 2.010849 0.578106
|
||||
4 2.083628 1.853473 0.602419
|
||||
5 1.969939 1.734762 0.621440
|
||||
6 1.904438 1.624005 0.640240
|
||||
7 1.783416 1.526202 0.656981
|
||||
8 1.719215 1.445780 0.671753
|
||||
9 1.621891 1.366912 0.687187
|
||||
10 1.589463 1.295759 0.701207
|
||||
11 1.510032 1.223578 0.716387
|
||||
12 1.404720 1.160607 0.729603
|
||||
13 1.414636 1.107378 0.741273
|
||||
14 1.364716 1.056422 0.753112
|
||||
15 1.327804 1.011525 0.763934
|
||||
16 1.255990 0.976447 0.771864
|
||||
17 1.181438 0.951213 0.778309
|
||||
18 1.192709 0.936060 0.781858
|
||||
19 1.190164 0.928613 0.783513
|
||||
20 1.172130 0.927612 0.783722
|
||||
/home/n-waves/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp60k
|
||||
Saving info /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp60k/lstm_nl4.m/info.json
|
||||
Starting classifier training
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.646537 0.516221 0.836000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.441884 0.361802 0.873000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.376583 0.318426 0.893000
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.280910 0.314279 0.889000
|
||||
2 0.308887 0.309718 0.903000
|
||||
Saving models at /home/n-waves/workspace/ulmfit-multilingual/data/mldoc/zh-1/models/sp60k/lstm_nl4.m
|
||||
Loss and accuracy using (cls_last): [0.30276635, tensor(0.8978)]
|
||||
```
|
||||
@@ -1,50 +0,0 @@
|
||||
# Results
|
||||
|
||||
## Set-up.
|
||||
- Num Tokens 15K
|
||||
- GPU V100
|
||||
- LM BPTT = 70
|
||||
- LM BS = 64
|
||||
- CLAS BS = 32
|
||||
|
||||
| Model | LSTM | QRNN |
|
||||
|----------------|-----------|-----------|
|
||||
| LM ms/batch | 143ms | 71ms |
|
||||
| CLAS ms/batch | 467ms | 156ms |
|
||||
|
||||
|
||||
```
|
||||
> python results/time_benchmark/qrnn_benchmark.py
|
||||
|
||||
Vocab size 14513
|
||||
QRNN
|
||||
LM
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 6.326089
|
||||
Total time: 00:11
|
||||
Batch size torch.Size([64, 70])
|
||||
Params = 22 MM
|
||||
Training time is 71.0 ms per batch
|
||||
CLAS
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.712603
|
||||
Total time: 00:10
|
||||
Batch size torch.Size([32, 1445])
|
||||
Params = 22 MM
|
||||
Training time is 156.0 ms per batch
|
||||
LSTM
|
||||
LM
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 6.262911
|
||||
Total time: 00:21
|
||||
Batch size torch.Size([64, 70])
|
||||
Params = 37 MM
|
||||
Training time is 143.0 ms per batch
|
||||
CLAS
|
||||
epoch train_loss valid_loss accuracy
|
||||
1 0.706715
|
||||
Total time: 00:32
|
||||
Batch size torch.Size([32, 1445])
|
||||
Params = 37 MM
|
||||
Training time is 467.0 ms per batch
|
||||
```
|
||||
@@ -1,52 +0,0 @@
|
||||
import glob
|
||||
import shutil
|
||||
import time
|
||||
from fastai.text import *
|
||||
|
||||
orig_path = untar_data(URLs.IMDB)
|
||||
path = Path('data') / 'imdb_small'
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
for mode in ['train', 'test']:
|
||||
for label in ['pos', 'neg']:
|
||||
tgt_path = path / mode / label
|
||||
tgt_path.mkdir(parents=True, exist_ok=True)
|
||||
# Keep just 10% of the files
|
||||
pattern = str(orig_path / mode / label / '3*.txt')
|
||||
for file in glob.glob(pattern):
|
||||
shutil.copy(file, tgt_path)
|
||||
|
||||
data_lm = TextLMDataBunch.from_folder(path, valid='test')
|
||||
data_clas = TextClasDataBunch.from_folder(path, bs=32, vocab=data_lm.train_ds.vocab, valid='test')
|
||||
|
||||
print('Vocab size', len(data_lm.train_ds.vocab.itos))
|
||||
|
||||
|
||||
def count_parameters(model, requires_grad):
|
||||
return sum(p.numel() for p in model.parameters() if p.requires_grad == requires_grad)
|
||||
|
||||
|
||||
def test(qrnn, func, config, data, arch=AWD_LSTM):
|
||||
total = len(list(data.train_dl))
|
||||
config = config.copy()
|
||||
config['qrnn'] = qrnn
|
||||
|
||||
learn = func(data, AWD_LSTM, config=config, pretrained=False)
|
||||
learn.unfreeze()
|
||||
params = count_parameters(learn.model, True)
|
||||
total = len(list(data.train_dl))
|
||||
start_time = time.clock()
|
||||
learn.fit(1)
|
||||
diff = time.clock() - start_time
|
||||
|
||||
print('Batch size', data.one_batch()[0].shape)
|
||||
print(f'Params = {params // 1000000} MM')
|
||||
print(f'Training time is {1000 * diff // total} ms per batch')
|
||||
|
||||
|
||||
for qrnn in [True, False]:
|
||||
print('QRNN' if qrnn else 'LSTM')
|
||||
print('LM')
|
||||
test(qrnn, language_model_learner, config=awd_lstm_lm_config, data=data_lm)
|
||||
print('CLAS')
|
||||
test(qrnn, text_classifier_learner, config=awd_lstm_clas_config, data=data_clas)
|
||||
@@ -0,0 +1,8 @@
|
||||
#!/usr/bin/env python
|
||||
# -*- coding: utf-8 -*-
|
||||
from setuptools import setup, find_packages
|
||||
setup(
|
||||
name="multifit",
|
||||
version="1.0",
|
||||
packages=find_packages(),
|
||||
)
|
||||
@@ -0,0 +1,59 @@
|
||||
from multifit import ULMFiT
|
||||
from fastai.text import *
|
||||
from sotabencheval.language_modelling import WikiText103Evaluator
|
||||
from sotabencheval.utils import is_server
|
||||
|
||||
def iterate_over_batches(data, bs, bptt):
|
||||
def batched(Z, bptt):
|
||||
sz = Z.shape[-1]
|
||||
for s in range(0, sz, bptt):
|
||||
yield Z[..., s:s+bptt]
|
||||
size = data.numel()
|
||||
batched_size = ((size-1) // bs) * bs
|
||||
# filp - to be able to switch to batch_size 1 later and maintain trasnfoxl memory
|
||||
X = data[:batched_size].view(bs, -1).flip(0,)
|
||||
Y = data[1:batched_size+1].view(bs, -1).flip(0,)
|
||||
yield from zip(batched(X, bptt), batched(Y, bptt))
|
||||
X = data[None, batched_size:-1]
|
||||
Y = data[None, batched_size+1:]
|
||||
yield from zip(batched(X, bptt), batched(Y, bptt))
|
||||
|
||||
#TODO the tokenization removes new lines so te perplexity coalculation is off
|
||||
def evaluate(pretrained_name):
|
||||
model = ULMFiT().from_pretrained_(pretrained_name)
|
||||
if is_server():
|
||||
wikitext_folder = WikiText103Evaluator.dataset.get_path(local_root="unused")
|
||||
else:
|
||||
wikitext_folder = untar_data(URLs.WIKITEXT)
|
||||
ds = model.arch.dataset(wikitext_folder, tokenizer=model.pretrain_lm.tokenizer)
|
||||
|
||||
test_df = ds.read_data(ds.tst_path)
|
||||
data_lm = ds.databunch_from_df(TextLMDataBunch, test_df, test_df, bs=20, bptt=70)
|
||||
learn = model.finetune_lm.get_learner(data_lm)
|
||||
|
||||
full_data = np.concatenate(data_lm.valid_ds.items)
|
||||
|
||||
evaluator = WikiText103Evaluator(
|
||||
model_name="Multifit (slim)",
|
||||
model_description=pretrained_name,
|
||||
paper_arxiv_id="1909.04761",
|
||||
local_root=str(wikitext_folder)
|
||||
)
|
||||
|
||||
learn.loss_func = None
|
||||
|
||||
dev = torch.device("cuda")
|
||||
evaluator.reset()
|
||||
batches = iterate_over_batches(torch.tensor(full_data), bs=200, bptt=70)
|
||||
for x,y in progress_bar(batches, total=len(full_data)//200//70):
|
||||
logits = learn.pred_batch(batch=[x.to(dev), y.to(dev)])
|
||||
log_probs = torch.log_softmax(logits, -1)
|
||||
evaluator.add(log_probs, y)
|
||||
if evaluator.cache_exists:
|
||||
break
|
||||
evaluator.save()
|
||||
print(pretrained_name)
|
||||
evaluator.print_results()
|
||||
return evaluator.results
|
||||
|
||||
evaluate("en_multifit_nl3_wiki103")
|
||||
Executable
+10
@@ -0,0 +1,10 @@
|
||||
#!/usr/bin/env bash -x
|
||||
source /workspace/venv/bin/activate
|
||||
PYTHON=${PYTHON:-"python"}
|
||||
REPO="$( cd "$(dirname "$0")" ; cd .. ; pwd -P )"
|
||||
cd $REPO
|
||||
$PYTHON -m pip install -e .
|
||||
$PYTHON -m pip install torch
|
||||
$PYTHON -m pip install spacy
|
||||
#$PYTHON -m spacy download en
|
||||
$PYTHON -m pip install git+https://github.com/PiotrCzapla/sotabench-eval.git
|
||||
@@ -0,0 +1,52 @@
|
||||
import pandas as pd, numpy as np
|
||||
import fire
|
||||
from pathlib import Path
|
||||
from sys import stderr
|
||||
from sklearn.model_selection import train_test_split
|
||||
import re
|
||||
|
||||
def to_csv(df, path):
|
||||
df.to_csv(path, header=None, index=None)
|
||||
|
||||
def remove_rt(df):
|
||||
return df.assign(text=df.text.str.replace('^RT @anonymized_account ',''))
|
||||
|
||||
def remove_duplicates(df):
|
||||
exact = df[~df.duplicated('text')]
|
||||
prefixes = exact.text.map(lambda t: t.endswith('…') and exact.text.str.startswith(t[:-1]).sum()>1)
|
||||
return exact[~prefixes]
|
||||
|
||||
def cross_remove_duplicates(from_df, other_df):
|
||||
exact = from_df[~from_df.text.isin(other_df.text)]
|
||||
other_prefixes = other_df.text[other_df.text.str.endswith('…')].str[:-1]
|
||||
if len(other_prefixes):
|
||||
other_prefixes_re = re.compile('^'+'|'.join([f'({re.escape(t)})' for t in other_prefixes]))
|
||||
else:
|
||||
other_prefixes_re = re.compile('^$')
|
||||
prefixes = exact.text.map(lambda t:
|
||||
(t.endswith('…') and other_df.text.str.startswith(t[:-1]).any()) or
|
||||
other_prefixes_re.match(t) is not None
|
||||
)
|
||||
return exact[~prefixes]
|
||||
|
||||
def split(data_dir, dedup=False):
|
||||
data_dir = Path(data_dir)
|
||||
train = pd.read_csv(data_dir / "pl.unsup.csv", header=None, names=["label", "text"])
|
||||
val_ratio = 0.1
|
||||
train = remove_rt(train)
|
||||
trn, val = train_test_split(train, test_size=val_ratio, random_state=12345, stratify=train.label)
|
||||
|
||||
if dedup:
|
||||
trn = remove_duplicates(trn)
|
||||
val = remove_duplicates(val)
|
||||
val = cross_remove_duplicates(val, trn)
|
||||
l1, l2, l3 = len(remove_duplicates(train)), len(trn), len(val)
|
||||
if l1 != l2 + l3:
|
||||
print("Warning: some condition believed by me to be invariant is not hold")
|
||||
print(f"{l1} should be equal to {l2} + {l3} = {l2+l3}")
|
||||
|
||||
|
||||
to_csv(trn, data_dir / "pl.train.csv")
|
||||
to_csv(val, data_dir / "pl.dev.csv")
|
||||
|
||||
if __name__ == "__main__": fire.Fire(split)
|
||||
@@ -1,162 +0,0 @@
|
||||
import os
|
||||
import glob
|
||||
import fire
|
||||
import ulmfit.pretrain_lm
|
||||
import ulmfit.train_clas
|
||||
from fastai import *
|
||||
from fastai.text import *
|
||||
from fastai_contrib.utils import *
|
||||
|
||||
"""
|
||||
It is a mixture of a pytest unit test and woven together to compose an end to end functional test.
|
||||
"""
|
||||
|
||||
import fastai.core
|
||||
fastai.core.defaults.cpus = 1
|
||||
cuda_id=0
|
||||
def copy_head(src_fn, dst_fn, n=1000):
|
||||
with src_fn.open("r") as s, dst_fn.open("w") as d:
|
||||
for i in range(n):
|
||||
d.write(s.readline())
|
||||
|
||||
def get_test_data():
|
||||
data = get_data_folder()
|
||||
wt = data / "wiki" / "wikitext-2"
|
||||
imdb = data / "imdb"
|
||||
|
||||
test_data = data / "test"
|
||||
if test_data.exists():
|
||||
shutil.rmtree(test_data)
|
||||
|
||||
test_wt = test_data / 'wikitext-s'
|
||||
test_imdb = test_data / 'imdb'
|
||||
test_wt.mkdir(exist_ok=True, parents=True)
|
||||
test_imdb.mkdir(exist_ok=True, parents=True)
|
||||
|
||||
sz=1
|
||||
# we use the same text to see if models can overfit
|
||||
copy_head(wt / 'en.wiki.train.tokens', test_wt / 'en.wiki.train.tokens', n=1000*sz)
|
||||
copy_head(wt / 'en.wiki.train.tokens', test_wt / 'en.wiki.valid.tokens', n=600*sz)
|
||||
copy_head(wt / 'en.wiki.train.tokens', test_wt / 'en.wiki.test.tokens', n=600*sz)
|
||||
copy_head(imdb / 'train.csv', test_imdb / 'train.csv', n=10*sz)
|
||||
copy_head(imdb / 'train.csv', test_imdb / 'test.csv', n=6 * sz)
|
||||
copy_head(imdb / 'train.csv', test_imdb / 'dev.csv', n=6 * sz)
|
||||
copy_head(imdb / 'train.csv', test_imdb / 'unsup.csv', n=1*sz)
|
||||
|
||||
return test_data, test_wt
|
||||
|
||||
|
||||
def test_ulmfit_works_with_relative_paths():
|
||||
""" Test ulmfit with (default) Moses tokenizer on small wikipedia dataset.
|
||||
"""
|
||||
os.chdir(get_data_folder()/"..")
|
||||
|
||||
|
||||
test_data, wt2 = get_test_data()
|
||||
lm_name = 'end-to-end-test-default'
|
||||
cuda_id = 0
|
||||
exp = ulmfit.pretrain_lm.LMHyperParams(
|
||||
dataset_path=wt2.relative_to(Path.cwd()),
|
||||
lang='en',
|
||||
qrnn=False,
|
||||
max_vocab=1000,
|
||||
name=lm_name,
|
||||
cuda_id=cuda_id)
|
||||
|
||||
exp.train_lm(num_epochs=1, bs=2)
|
||||
|
||||
#assert exp.results['accuracy'] > 0.02
|
||||
|
||||
exp2 = ulmfit.train_clas.CLSHyperParams.from_lm(test_data / 'imdb', exp.model_dir)
|
||||
exp2.train_cls(num_lm_epochs=1, unfreeze=False, bs=4,)
|
||||
|
||||
# should work for the second time as well
|
||||
|
||||
exp2 = ulmfit.train_clas.CLSHyperParams.from_lm(test_data / 'imdb', exp.model_dir)
|
||||
exp2.train_cls(num_lm_epochs=0, unfreeze=False, bs=4, )
|
||||
|
||||
|
||||
def test_ulmfit_default_end_to_end():
|
||||
""" Test ulmfit with (default) Moses tokenizer on small wikipedia dataset.
|
||||
"""
|
||||
test_data, wt2 = get_test_data()
|
||||
lm_name = 'end-to-end-test-default'
|
||||
cuda_id = 0
|
||||
exp = ulmfit.pretrain_lm.LMHyperParams(
|
||||
dataset_path=wt2,
|
||||
lang='en',
|
||||
qrnn=False,
|
||||
max_vocab=1000,
|
||||
name=lm_name,
|
||||
cuda_id=cuda_id)
|
||||
|
||||
exp.train_lm(num_epochs=1, bs=2)
|
||||
|
||||
#assert exp.results['accuracy'] > 0.02
|
||||
|
||||
exp2 = ulmfit.train_clas.CLSHyperParams.from_lm(test_data / 'imdb', exp.model_dir)
|
||||
exp2.train_cls(num_lm_epochs=0, unfreeze=False, bs=4,)
|
||||
|
||||
def test_ulmfit_fastai_end_to_end():
|
||||
""" Test ulmfit with sentencepiece tokenizer on small wikipedia dataset.
|
||||
"""
|
||||
test_data, wt2 = get_test_data()
|
||||
lm_name = 'end-to-end-test-fastai'
|
||||
|
||||
exp = ulmfit.pretrain_lm.LMHyperParams(
|
||||
dataset_path=wt2,
|
||||
lang='en',
|
||||
cuda_id=cuda_id,
|
||||
qrnn=False,
|
||||
tokenizer='f',
|
||||
max_vocab=100,
|
||||
nl=1,
|
||||
name=lm_name,
|
||||
)
|
||||
exp.train_lm(num_epochs=1, bs=2)
|
||||
exp2 = ulmfit.train_clas.CLSHyperParams.from_lm(test_data / 'imdb', exp.model_dir)
|
||||
exp2.train_cls(num_lm_epochs=0, unfreeze=False, bs=4, )
|
||||
|
||||
def test_ulmfit_fastai_end_to_end_label_smoothing():
|
||||
""" Test ulmfit with sentencepiece tokenizer on small wikipedia dataset.
|
||||
"""
|
||||
test_data, wt2 = get_test_data()
|
||||
lm_name = 'end-to-end-test-fastai'
|
||||
|
||||
exp = ulmfit.pretrain_lm.LMHyperParams(
|
||||
dataset_path=wt2,
|
||||
lang='en',
|
||||
cuda_id=cuda_id,
|
||||
qrnn=False,
|
||||
tokenizer='f',
|
||||
max_vocab=100,
|
||||
name=lm_name,
|
||||
)
|
||||
exp.train_lm(num_epochs=1, bs=2, label_smoothing_eps=0.1)
|
||||
exp2 = ulmfit.train_clas.CLSHyperParams.from_lm(test_data / 'imdb', exp.model_dir)
|
||||
exp2.train_cls(num_lm_epochs=0, unfreeze=False, bs=4, label_smoothing_eps=0.1 )
|
||||
|
||||
def test_ulmfit_sentencepiece_end_to_end():
|
||||
""" Test ulmfit with sentencepiece tokenizer on small wikipedia dataset.
|
||||
"""
|
||||
test_data, wt2 = get_test_data()
|
||||
lm_name = 'end-to-end-test-spm'
|
||||
|
||||
exp = ulmfit.pretrain_lm.LMHyperParams(
|
||||
dataset_path=wt2,
|
||||
lang='en',
|
||||
cuda_id=cuda_id,
|
||||
qrnn=False,
|
||||
tokenizer=ulmfit.pretrain_lm.Tokenizers.SUBWORD,
|
||||
max_vocab=200,
|
||||
name=lm_name,
|
||||
)
|
||||
exp.train_lm(num_epochs=1, bs=2)
|
||||
# not supported yet
|
||||
exp2 = ulmfit.train_clas.CLSHyperParams.from_lm(test_data / 'imdb', exp.model_dir)
|
||||
exp2.train_cls(num_lm_epochs=0, unfreeze=False, bs=4, )
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
fire.Fire() # allows using all functions via CLI
|
||||
|
||||
@@ -1,41 +0,0 @@
|
||||
# Todo
|
||||
- [ ] Update these docs
|
||||
|
||||
Getting Started
|
||||
---
|
||||
|
||||
## Download and Extract the Wikipedia corpus
|
||||
|
||||
In Linux, you can do all the following steps automatically with [prepare_wiki.sh](./prepare_wiki.sh)
|
||||
|
||||
**Manual Instructions**
|
||||
|
||||
We use the [WikiExtractor.py](http://medialab.di.unipi.it/wiki/Wikipedia_Extractor). It is a Python script that extracts and cleans text from a [Wikipedia database dump](http://download.wikimedia.org/).
|
||||
|
||||
At the end of this step, you should have the following directory structure inside ulmfit:
|
||||
```bash
|
||||
|
||||
|- data
|
||||
|- wiki
|
||||
|- wiki_dumps
|
||||
|- wiki_extr
|
||||
|- wikiextractor
|
||||
```
|
||||
The extracted data should be in the folder `wiki_extr` -> language name e.g.`en` (english), `fr` (french) `hi` (hindi) and so on.
|
||||
|
||||
## Create and Post Process WikiText
|
||||
|
||||
### Create and Post-Process
|
||||
If you used the automated shell script from previous step, this might look something like
|
||||
```bash
|
||||
python create_wikitext.py -i data/wiki_extr/hi -o data/wiki/hi -l hi
|
||||
```
|
||||
for hindi (unicode: 'hi')
|
||||
|
||||
This should create two splits of your Wikimedia Dumps: a small and large one.
|
||||
|
||||
_**Then**_, use the [postprocess_wikitext.py](./postprocess_wikitext.py) script to finish post processing. This processes numbers, builds a vocab, and limits the vocabulary size. This might look following for Hindi (`hi`)
|
||||
```bash
|
||||
python postprocess_wikitext.py data/wiki/hi-2 hi
|
||||
python postprocess_wikitext.py data/wiki/hi-100 hi
|
||||
```
|
||||
@@ -1 +0,0 @@
|
||||
|
||||
@@ -1,188 +0,0 @@
|
||||
import gc
|
||||
import os
|
||||
import pprint
|
||||
import tarfile
|
||||
import shutil
|
||||
from collections import OrderedDict
|
||||
from functools import wraps
|
||||
import pandas as pd
|
||||
import fire
|
||||
from .pretrain_lm import LMHyperParams, json_save, json_load, np
|
||||
from .train_clas import CLSHyperParams
|
||||
from pathlib import Path
|
||||
from string import Template
|
||||
|
||||
class FireView:
|
||||
def __init__(self, **kwargs):
|
||||
for k,v in kwargs.items():
|
||||
setattr(self, k, v)
|
||||
|
||||
def get_lang_from_dataset_path(ds):
|
||||
lang,*_ = ds.name.split("-")
|
||||
if len(lang) == 2:
|
||||
return lang
|
||||
return "en"
|
||||
|
||||
def get_dataset_path(p, dataset_template):
|
||||
ds = [x for x in p.parents if x.name == "models"][0].parent
|
||||
lang = get_lang_from_dataset_path(ds)
|
||||
pattern = Template(dataset_template).substitute(lang=lang, ds_name=ds.name)
|
||||
print(pattern)
|
||||
for ds_path in ds.parent.glob(pattern):
|
||||
yield lang, ds_path
|
||||
|
||||
class ULMFiT:
|
||||
@wraps(LMHyperParams)
|
||||
def lm(self, dataset_path, **changes):
|
||||
changes['dataset_path'] = dataset_path
|
||||
params = LMHyperParams(**changes)
|
||||
return FireView(train=params.train_lm)
|
||||
|
||||
lm2 = LMHyperParams
|
||||
@wraps(CLSHyperParams)
|
||||
def cls(self, dataset_path, base_lm_path=None, **changes):
|
||||
if base_lm_path is not None:
|
||||
params = CLSHyperParams.from_lm(dataset_path, base_lm_path, **changes)
|
||||
else:
|
||||
params = CLSHyperParams(dataset_path=dataset_path, **changes)
|
||||
return FireView(train=params.train_cls, validate_cls=params.validate_cls)
|
||||
|
||||
@wraps(CLSHyperParams)
|
||||
def load_cls(self, model_path, **changes):
|
||||
params = CLSHyperParams.from_json(model_path, **changes)
|
||||
return FireView(train=params.train_cls, validate_cls=params.validate_cls)
|
||||
|
||||
|
||||
def eval_noise_resistance(self, lang="de", size=1, prefix_name="", model="sp15k/qrnn_nl4.m",
|
||||
num_cls_epochs=8, bs=18, lr_sched="1cycle", label_smoothing_eps=0.0, **kwargs):
|
||||
results= []
|
||||
for noise in range(0, 80, 5):
|
||||
print("Noise: ", noise)
|
||||
d = self.eval(glob=f"mldoc/{lang}-1/models/{model}",
|
||||
name=f"nl4_{prefix_name}{noise}",
|
||||
noise=noise/100,
|
||||
dataset_template='${lang}-'+str(size),
|
||||
num_cls_epochs=num_cls_epochs,
|
||||
bs=bs,
|
||||
lr_sched=lr_sched,
|
||||
label_smoothing_eps=label_smoothing_eps,
|
||||
return_df=True,
|
||||
**kwargs)
|
||||
val = d['tst_accuracy'][0]
|
||||
results.append((noise/100, val))
|
||||
df = pd.DataFrame(results, columns=["noise", "accuracy"])
|
||||
df.to_csv(f"noise_{lang}-{size}{prefix_name}.csv")
|
||||
print(df)
|
||||
|
||||
def tar(self, model_path):
|
||||
data_dir = (Path.cwd()/"data").resolve()
|
||||
params = CLSHyperParams.from_json(model_path)
|
||||
name = str(params.dataset_dir.resolve().relative_to(data_dir)).replace("/", "-")
|
||||
|
||||
tar_name = f"models/{name}-{params.tokenizer_prefix}-{params.model_name}.tar"
|
||||
print("Storing model in", tar_name)
|
||||
with tarfile.open(tar_name, mode="w") as tar:
|
||||
for g in map(params.model_dir.glob, ['*_best.pth', 'info.json', '../spm.*', '../itos.*',]):
|
||||
for f in g:
|
||||
dest = f.resolve().relative_to(Path.cwd())
|
||||
print("Adding", f, dest)
|
||||
tar.add(f, dest)
|
||||
|
||||
def generate_pseudo_labels(self, glob="mldoc/*-1-laser-en1/models/sp15k/qrnn_nl4.m", bs=20, dest_dataset_template='${ds_name}-ps'):
|
||||
for base_model in sorted(Path("data").glob(glob)):
|
||||
print("Processing", base_model)
|
||||
|
||||
dataset_path = [x for x in base_model.parents if x.name == "models"][0].parent
|
||||
lang = get_lang_from_dataset_path(dataset_path)
|
||||
dest_dataset_path = dataset_path.parent/Template(dest_dataset_template).substitute(ds_name=dataset_path.name)
|
||||
try:
|
||||
_name = base_model.name.replace(".m", "").replace("lstm_", "").replace("qrnn_", "")
|
||||
params = CLSHyperParams.from_lm(dataset_path, base_model, lang=lang, name=_name, cuda_id=0)
|
||||
key = str(params.model_dir.relative_to(Path.cwd()))
|
||||
if (params.model_dir / "results.npy").exists():
|
||||
d = np.load(params.model_dir / "results.npy")
|
||||
d = d.tolist() # magiacally convert to dict
|
||||
elif (params.model_dir / "cls_best.pth").exists():
|
||||
print("Evaluating previously trained model")
|
||||
d = params.validate_cls(label_smoothing_eps=0.1)
|
||||
else:
|
||||
print("The model is not trained ignoring")
|
||||
continue
|
||||
print("Generating pseduolabels", dest_dataset_path)
|
||||
params.generate_pseudo_labels(dest_dataset_path, bs=bs)
|
||||
del params
|
||||
except Exception as e:
|
||||
print("Error", e)
|
||||
raise e
|
||||
gc.collect()
|
||||
|
||||
def ls(self, glob="mldoc/*-1/models/sp30k/lstm_nl4.m"):
|
||||
for i, name in enumerate(sorted(Path("data").glob(glob))):
|
||||
print(i, name, "cls:", (name/"cls_best.pth").exists())
|
||||
|
||||
def eval(self, glob="mldoc/*-1/models/sp30k/lstm_nl4.m", dataset_template='${ds_name}', name=None,
|
||||
num_lm_epochs=0, cuda_id=0, train=True, to_csv=None, return_df=False, label_smoothing_eps=0.0,
|
||||
**trn_params):
|
||||
results = []
|
||||
|
||||
def extract_agg(group):
|
||||
best = group.loc[group["val_accuracy"].idxmax()]["tst_accuracy"]
|
||||
return pd.Series({'best': best* 100,
|
||||
'max': group['tst_accuracy'].max()* 100,
|
||||
'avg': group['tst_accuracy'].mean()* 100})
|
||||
def pivot_to_lang(df):
|
||||
df['ds'] = df['name'].str.extract(r'data/[a-z]*/([^/]{1,12})[^/]*/models')
|
||||
best = df.groupby('ds').apply(extract_agg)
|
||||
best = best.round(2)
|
||||
return best.T
|
||||
for base_model in sorted(Path("data").glob(glob)):
|
||||
print("Processing", base_model)
|
||||
for lang, dataset_path in sorted(get_dataset_path(base_model, dataset_template)):
|
||||
try:
|
||||
_name = name
|
||||
if name is None:
|
||||
_name = base_model.name.replace(".m","").replace("lstm_","").replace("qrnn_","")
|
||||
params = CLSHyperParams.from_lm(dataset_path, base_model, lang=lang, name=_name, cuda_id=cuda_id)
|
||||
key = str(params.model_dir.relative_to(Path.cwd()))
|
||||
if (params.model_dir / "results.npy").exists():
|
||||
d = np.load(params.model_dir / "results.npy")
|
||||
d = d.tolist() # magiacally convert to dict
|
||||
elif (params.model_dir/"cls_best.pth").exists():
|
||||
print("Evaluating previously trained model")
|
||||
d = params.validate_cls(label_smoothing_eps=label_smoothing_eps)
|
||||
elif train:
|
||||
print("Training")
|
||||
d = params.train_cls(num_lm_epochs=num_lm_epochs, label_smoothing_eps=label_smoothing_eps, **trn_params)
|
||||
else:
|
||||
print("Skipping", (params.model_dir/"cls_best.pth"))
|
||||
d = None
|
||||
if d is not None:
|
||||
d['name']=key
|
||||
np.save(params.model_dir / "results.npy", d)
|
||||
results.append(d)
|
||||
del params
|
||||
except Exception as e:
|
||||
print("Error", e)
|
||||
gc.collect()
|
||||
df = pd.DataFrame.from_records(results)
|
||||
print(df)
|
||||
print(pivot_to_lang(df))
|
||||
if to_csv is not None:
|
||||
print(f"Saving result to: {to_csv}")
|
||||
df.to_csv(to_csv)
|
||||
if return_df:
|
||||
return df
|
||||
|
||||
def remove_lm_saves(self):
|
||||
for lm_save in Path("data").glob("**/lm_*.pth"):
|
||||
num = lm_save.stem.split("_")[-1]
|
||||
if not num.isdigit():
|
||||
continue
|
||||
if int(num) not in [5, 10, 15]:
|
||||
print("rm ", lm_save)
|
||||
os.remove(lm_save)
|
||||
|
||||
# python -m ulmfit cls --dataset-path data/mldoc/de-1-laser --base-lm-path data/mldoc/de-1/models/sp30k/lstm_nl4.m --lang=de --name 'nl4' --cuda-id=1 - train 0 --bs 40 --num-cls-epochs=2
|
||||
|
||||
if __name__ == '__main__':
|
||||
fire.Fire(ULMFiT())
|
||||
@@ -1,35 +0,0 @@
|
||||
"""
|
||||
Script to merge WikiText files created with `create_wikitext.py`.
|
||||
"""
|
||||
import fire
|
||||
from pathlib import Path
|
||||
from contextlib import ExitStack
|
||||
|
||||
def merge_wikitext(paths, langs, dest_path, num_sentences):
|
||||
wiki_paths = [Path(path) for path in paths]
|
||||
for wiki_path in wiki_paths:
|
||||
assert wiki_path.exists(), f'Error: {wiki_path} does not exist.'
|
||||
dest_path = Path(dest_path)
|
||||
dest_path.mkdir(exist_ok=True)
|
||||
splits = ['train', 'valid', 'test']
|
||||
concat_langs = '-'.join(langs)
|
||||
for split in splits:
|
||||
with ExitStack() as stack:
|
||||
files = [stack.enter_context(open(
|
||||
wiki_path / f'{lang}.wiki.{split}.tokens', 'r', encoding='utf-8'))
|
||||
for lang, wiki_path in zip(langs, wiki_paths)]
|
||||
|
||||
output = stack.enter_context(open(dest_path / f'{concat_langs}.wiki.{split}.tokens', 'w', encoding='utf-8'))
|
||||
done = False
|
||||
while not done:
|
||||
for file in files:
|
||||
lines = [file.readline() for x in range(num_sentences)]
|
||||
size = len(lines)
|
||||
lines = [line for line in lines if line]
|
||||
if len(lines) < size:
|
||||
done = True
|
||||
for line in lines:
|
||||
output.write(line)
|
||||
|
||||
if __name__ == '__main__':
|
||||
fire.Fire(merge_wikitext)
|
||||
@@ -1,341 +0,0 @@
|
||||
"""
|
||||
Script to train a model on a preprocessed Wiki dataset. Note that the dataset is
|
||||
expected to have been tokenized with Moses and processed with `postprocess_wikitext.py`.
|
||||
That is, the data is expected to be white-space separated and numbers are expected
|
||||
to be split.
|
||||
"""
|
||||
|
||||
import fire
|
||||
|
||||
from fastai.callbacks import CSVLogger
|
||||
from fastai.text import *
|
||||
from fastai_contrib.utils import read_whitespace_file, \
|
||||
validate, UNK, get_sentencepiece, PAD_TOKEN_ID, \
|
||||
replace_std_toks, MosesPreprocessingFunc
|
||||
|
||||
LM_BEST = "lm_best"
|
||||
ENC_BEST = "enc_best"
|
||||
|
||||
|
||||
class Tokenizers(Enum):
|
||||
SUBWORD='sp'
|
||||
BROKENSUBWORD = 'bsp'
|
||||
MOSES='v'
|
||||
MOSES_FA='vf'
|
||||
FASTAI='f'
|
||||
|
||||
def istitle(line):
|
||||
return len(re.findall(r'^ ?= [^=]* = ?$', line)) != 0
|
||||
|
||||
def read_wiki_articles(filename):
|
||||
articles = []
|
||||
with open(filename, encoding='utf8') as f:
|
||||
lines = f.readlines()
|
||||
current_article = []
|
||||
for i,line in enumerate(lines):
|
||||
current_article.append(line)
|
||||
if i < len(lines)-2 and lines[i+1].strip() == "" and istitle(lines[i+2]):
|
||||
articles.append("".join(current_article))
|
||||
current_article = []
|
||||
articles.append("".join(current_article))
|
||||
print(f"Wiki text was split to {len(articles)} articles")
|
||||
return pd.DataFrame({'texts': np.array(articles, dtype=np.object)})
|
||||
|
||||
def json_save(f, d):
|
||||
with Path(f).open("w") as fp:
|
||||
json.dump(d, fp)
|
||||
|
||||
def json_load(f):
|
||||
with open(f, 'r') as f:
|
||||
return json.load(f)
|
||||
|
||||
@dataclass
|
||||
class LMHyperParams:
|
||||
dataset_path: Union[str, Path] # data_dir
|
||||
|
||||
base_lm_path: Union[str, Path] = None
|
||||
backwards: str = False
|
||||
bidir: bool =False
|
||||
qrnn: bool = True
|
||||
max_vocab: int = 60000
|
||||
tokenizer: Tokenizers = Tokenizers.MOSES
|
||||
pretrained_model: str = None
|
||||
|
||||
emb_sz:int = 400
|
||||
nh: int = None
|
||||
nl: int = 3
|
||||
|
||||
# these hyperparameters are for training on ~100M tokens (e.g. WikiText-103)
|
||||
# for training on smaller datasets, more dropout is necessary
|
||||
dps = dict(output_p=0.25, hidden_p=0.1, input_p=0.2, embed_p=0.02, weight_p=0.15) # consider removing dps & clip from the default hyperparams and put them to train
|
||||
clip: float = 0.12
|
||||
bptt: int = 70
|
||||
# alpha and beta - defaults like in fastai/text/learner.py:RNNLearner()
|
||||
rnn_alpha: float = 2 # activation regularization (AR)
|
||||
rnn_beta: float = 1 # temporal activation regularization (TAR)
|
||||
|
||||
lang: str = 'en'
|
||||
name: str = None
|
||||
cuda_id: InitVar[int] = 0
|
||||
|
||||
def __post_init__(self, cuda_id):
|
||||
if self.bidir and self.backwards:
|
||||
raise ValueError('Both "backwards" and "bidir" options cannot be enabled at the same time')
|
||||
if not torch.cuda.is_available():
|
||||
print('CUDA not available. Setting device=-1.')
|
||||
cuda_id = -1
|
||||
torch.cuda.set_device(cuda_id)
|
||||
self.dataset_path = Path(self.dataset_path)
|
||||
self.base_lm_path = Path(self.base_lm_path) if self.base_lm_path is not None else None
|
||||
self.tokenizer = Tokenizers(self.tokenizer) if isinstance(self.tokenizer, str) else self.tokenizer
|
||||
|
||||
assert self.dataset_path.exists()
|
||||
self.cache_dir = self.dataset_path / 'models' / self.tokenizer_prefix
|
||||
self.model_dir = self.cache_dir / self.model_name
|
||||
|
||||
print('Max vocab:', self.max_vocab)
|
||||
print('Cache dir:', self.cache_dir)
|
||||
print('Model dir:', self.model_dir)
|
||||
if self.nh is None: self.nh = 1550 if self.qrnn else 1150
|
||||
if self.name is None: self.name = self.lang
|
||||
|
||||
@property
|
||||
def tokenizer_prefix(self): return f"{self.tokenizer.value}{self.max_vocab // 1000}k"
|
||||
|
||||
@property
|
||||
def model_direction(self):
|
||||
if self.bidir:
|
||||
return 'bi'
|
||||
if self.backwards:
|
||||
return 'bwd'
|
||||
else:
|
||||
return ''
|
||||
|
||||
@property
|
||||
def model_prefix(self): return self.model_direction + ('qrnn' if self.qrnn else 'lstm')
|
||||
|
||||
@property
|
||||
def model_name(self): return f"{self.model_prefix}_{self.name}.m"
|
||||
|
||||
@property
|
||||
def pretrained_fnames(self): return [self.base_lm_path / LM_BEST, self.base_lm_path / '../itos'] if self.base_lm_path else None
|
||||
|
||||
def tokenizer_to_fastai_args(self, sp_data_func, use_moses):
|
||||
moses_preproc = [MosesPreprocessingFunc(self.lang)] if use_moses else []
|
||||
if self.tokenizer is Tokenizers.SUBWORD or self.tokenizer is Tokenizers.BROKENSUBWORD:
|
||||
if self.base_lm_path and not(self.cache_dir/"spm.model").exists(): # ensure we are using the same sentence piece model
|
||||
shutil.copy(self.base_lm_path / '..' / 'itos.pkl', self.cache_dir)
|
||||
shutil.copy(self.base_lm_path / '..' / 'spm.model', self.cache_dir)
|
||||
shutil.copy(self.base_lm_path / '..' / 'spm.vocab', self.cache_dir)
|
||||
args = get_sentencepiece(self.cache_dir,
|
||||
sp_data_func,
|
||||
vocab_size=self.max_vocab,
|
||||
lang=self.lang,
|
||||
pre_rules=moses_preproc + defaults.text_pre_rules,
|
||||
post_rules=defaults.text_post_rules)
|
||||
elif self.tokenizer is Tokenizers.MOSES:
|
||||
args = dict(tokenizer=Tokenizer(tok_func=BaseTokenizer,
|
||||
lang=self.lang,
|
||||
pre_rules=moses_preproc + [replace_std_toks],
|
||||
post_rules=[]))
|
||||
elif self.tokenizer is Tokenizers.MOSES_FA:
|
||||
args = dict(tokenizer=Tokenizer(tok_func=BaseTokenizer,
|
||||
lang=self.lang,
|
||||
pre_rules=moses_preproc + defaults.text_pre_rules,
|
||||
post_rules=defaults.text_post_rules))
|
||||
elif self.tokenizer is Tokenizers.FASTAI:
|
||||
args = dict()
|
||||
else:
|
||||
raise ValueError(
|
||||
f"self.tokenizer has wrong value {self.tokenizer}, Allowed values are taken from {Tokenizers}")
|
||||
return args
|
||||
|
||||
def save_info(self):
|
||||
from dataclasses import asdict
|
||||
vals = {k: (str(v) if isinstance(v, Path) else v) for k,v in asdict(self).items()}
|
||||
vals.pop('name', None)
|
||||
vals.pop('lang', None)
|
||||
vals['tokenizer'] = self.tokenizer.value
|
||||
json_save(self.model_dir/'info.json', vals)
|
||||
print("Saving info", self.model_dir / 'info.json')
|
||||
|
||||
def train_lm(self, num_epochs=20, data_lm=None, bs=70, true_wd=False, drop_mult=0.0, lr=5e-3, label_smoothing_eps=0.0):
|
||||
self.model_dir.mkdir(exist_ok=True, parents=True)
|
||||
data_lm = self.load_wiki_data(bs=bs) if data_lm is None else data_lm
|
||||
learn = self.create_lm_learner(data_lm, drop_mult=drop_mult, label_smoothing_eps=label_smoothing_eps)
|
||||
print("Bptt", data_lm.bptt)
|
||||
learn.true_wd = true_wd
|
||||
if num_epochs > 0:
|
||||
if self.pretrained_fnames or self.pretrained_model:
|
||||
print("Training lm from: ", self.pretrained_fnames or self.pretrained_model)
|
||||
if learn.true_wd:
|
||||
learn.freeze_to(-1)
|
||||
learn.fit_one_cycle(1, 1e-2, moms=(0.8, 0.7))
|
||||
learn.unfreeze()
|
||||
learn.fit_one_cycle(num_epochs, 1e-3, moms=(0.8, 0.7))
|
||||
else:
|
||||
learn.freeze_to(-1)
|
||||
learn.fit_one_cycle(1, 1e-2, moms=(0.8, 0.7), wd=1e-7) # TODO Fix the learning rates
|
||||
learn.unfreeze()
|
||||
learn.fit_one_cycle(num_epochs, 1e-3, moms=(0.8, 0.7), wd=1e-7)
|
||||
else:
|
||||
print("Training lm from random weights")
|
||||
learn.unfreeze()
|
||||
if not learn.true_wd: learn.fit_one_cycle(num_epochs, lr, (0.8, 0.7), wd=1e-7)
|
||||
else: learn.fit_one_cycle(num_epochs, lr, (0.8, 0.7)) # TODO find proper values
|
||||
learn.save("lm_best_with_opt", with_opt=True)
|
||||
learn.save_encoder(ENC_BEST)
|
||||
learn.save(LM_BEST, with_opt=False)
|
||||
print(learn.path)
|
||||
|
||||
self.save_info()
|
||||
# do we need to return `learn'? it adds noise to Fire output
|
||||
#return learn
|
||||
|
||||
def create_lm_learner(self, data_lm, dps=None, label_smoothing_eps=0.0, **kwargs):
|
||||
assert self.bidir == False, "bidirectional model is not yet supported"
|
||||
config = dict(emb_sz=self.emb_sz, n_hid=self.nh, n_layers=self.nl, pad_token=PAD_TOKEN_ID, qrnn=self.qrnn,
|
||||
tie_weights=True, out_bias=True)
|
||||
config.update(dps or self.dps)
|
||||
trn_args = dict(clip=self.clip, alpha=self.rnn_alpha, beta=self.rnn_beta)
|
||||
trn_args.update(kwargs)
|
||||
print ("Training args: ", trn_args, "dps: ", dps or self.dps)
|
||||
learn = language_model_learner(data_lm, AWD_LSTM, config=config, model_dir=self.model_dir.relative_to(data_lm.path), pretrained=False, **trn_args)
|
||||
if self.pretrained_model is not None:
|
||||
print("Loading pretrained model")
|
||||
model_path = untar_data(self.pretrained_model, data=False)
|
||||
fnames = [list(model_path.glob(f'*.{ext}'))[0] for ext in ['pth', 'pkl']]
|
||||
learn.load_pretrained(*fnames)
|
||||
learn.freeze()
|
||||
if self.pretrained_fnames is not None:
|
||||
print("Loading pretrained model")
|
||||
fnames = [f'{fn}.{ext}' for fn,ext in zip(self.pretrained_fnames, ['pth', 'pkl'])]
|
||||
learn.load_pretrained(*fnames)
|
||||
learn.freeze()
|
||||
# compared to standard Adam, we set beta_1 to 0.8
|
||||
learn.opt_fn = partial(optim.Adam, betas=(0.8, 0.99))
|
||||
learn.callback_fns += [partial(CSVLogger, filename=f"{learn.model_dir}/lm-history"),
|
||||
# partial(SaveModelCallback, every='improvement', name='lm') disabled due to Memory issues
|
||||
]
|
||||
if label_smoothing_eps > 0.0:
|
||||
learn.loss_func = FlattenedLoss(LabelSmoothingCrossEntropy, eps=label_smoothing_eps)
|
||||
return learn
|
||||
|
||||
def load_train_text(self):
|
||||
trn_path = self.dataset_path / f'{self.lang}.wiki.train.tokens'
|
||||
with open(trn_path) as f:
|
||||
return [line.rstrip('\n') for line in f]
|
||||
|
||||
def load_wiki_data(self, bs=70):
|
||||
self.model_dir.mkdir(exist_ok=True, parents=True)
|
||||
trn_path = self.dataset_path / f'{self.lang}.wiki.train.tokens'
|
||||
val_path = self.dataset_path / f'{self.lang}.wiki.valid.tokens'
|
||||
tst_path = self.dataset_path / f'{self.lang}.wiki.test.tokens'
|
||||
for path_ in [trn_path, val_path, tst_path]:
|
||||
assert path_.exists(), f'Error: {path_} does not exist.'
|
||||
|
||||
args = self.tokenizer_to_fastai_args(sp_data_func=self.load_train_text, use_moses=False)
|
||||
|
||||
data_lm = self.lm_databunch(f"lm{self.bptt if self.bptt != 70 else ''}",
|
||||
train_df=read_wiki_articles(trn_path),
|
||||
valid_df=read_wiki_articles(val_path),
|
||||
classes=None,
|
||||
bs=bs,
|
||||
text_cols='texts',
|
||||
bptt=self.bptt,
|
||||
**args)
|
||||
|
||||
itos, stoi, trn_path = data_lm.vocab.itos, data_lm.vocab.stoi, data_lm.path
|
||||
print('Size of vocabulary:', len(itos))
|
||||
print('First 20 words in vocab:', data_lm.vocab.itos[:20])
|
||||
return data_lm
|
||||
|
||||
def lm_databunch(self, name, *args, **kwargs):
|
||||
return self.databunch(name, bunch_class=TextLMDataBunch, *args, **kwargs)
|
||||
|
||||
def databunch(self, name, bunch_class, train_df, valid_df, bs, force=False, **args):
|
||||
bunch_path = self.cache_dir / name
|
||||
if force and bunch_path.exist():
|
||||
print("Forcefully recreating the databunch, removing previously stored data")
|
||||
for f in bunch_path.glob("*.npy"):
|
||||
f.unlink()
|
||||
if bunch_path.isdir():
|
||||
if name != ".":
|
||||
bunch_path.rmdir()
|
||||
else:
|
||||
bunch_path.unlink()
|
||||
|
||||
if (bunch_path / 'itos.pkl').exists():
|
||||
data = bunch_class.load(self.cache_dir, name, bs=bs)
|
||||
elif bunch_path.exists():
|
||||
data = load_data(self.cache_dir, file=name, bs=bs)
|
||||
else:
|
||||
print(f"Running tokenization {name}...")
|
||||
data = bunch_class.from_df(path=self.cache_dir,
|
||||
train_df=train_df,
|
||||
valid_df=valid_df,
|
||||
max_vocab=self.max_vocab,
|
||||
bs=bs,
|
||||
**args)
|
||||
data.save(name)
|
||||
with open(self.cache_dir/"itos.pkl", 'wb') as f:
|
||||
pickle.dump(data.vocab.itos, f)
|
||||
|
||||
|
||||
print(f"Data {name}, trn: {len(data.train_ds)}, val: {len(data.valid_ds)}")
|
||||
return data
|
||||
|
||||
@classmethod
|
||||
def from_lm(cls, dataset_path, base_lm_path, **kwargs) -> 'LMHyperParams':
|
||||
dataset_path = Path(dataset_path).resolve()
|
||||
base_lm_path = Path(base_lm_path).resolve()
|
||||
d = json_load(base_lm_path/'info.json')
|
||||
d['dataset_path'] = dataset_path
|
||||
d['base_lm_path'] = base_lm_path
|
||||
d.pop('bs', None)
|
||||
d.pop('drop_mult', None)
|
||||
subword = d.pop('subword', False)
|
||||
tokenizer = d.pop('tokenizer', None)
|
||||
if tokenizer is not None:
|
||||
d['tokenizer'] = Tokenizers(tokenizer)
|
||||
elif subword:
|
||||
d['tokenizer'] = Tokenizers.SUBWORD
|
||||
else:
|
||||
d['tokenizer'] = Tokenizers.MOSES
|
||||
|
||||
d.update(kwargs)
|
||||
return cls(**d)
|
||||
@classmethod
|
||||
def from_json(cls, model_path:Path, **kwargs):
|
||||
model_path = Path(model_path).resolve()
|
||||
name = re.search(r"[a-z]+_(.+).m", model_path.name).group(1)
|
||||
with open(model_path / 'info.json', 'r') as f:
|
||||
d = json.load(f)
|
||||
d.update(kwargs)
|
||||
d['name'] = name
|
||||
dataset_path = path_strip(model_path, "data", "models").parent
|
||||
d['dataset_path'] = str(dataset_path)
|
||||
d['lang'] = infer_lang_from_dataset(dataset_path.name)
|
||||
return cls(**d)
|
||||
|
||||
def infer_lang_from_dataset(name:str):
|
||||
return name.split("-")[0]
|
||||
|
||||
def path_strip(path, from_folder, to_folder):
|
||||
to_p = [p for p in path.parents if p.name == to_folder][0]
|
||||
from_p = [p for p in path.parents if p.name == from_folder][0]
|
||||
return to_p.relative_to(from_p.parent)
|
||||
|
||||
def validate_lm(self):
|
||||
if not self.exp.subword and self.exp.max_vocab is None:
|
||||
raise NotImplementedError("figure out how to validate and save results")
|
||||
# only if we use the unpreprocessed version and the full vocabulary
|
||||
# are the perplexity results comparable to previous work
|
||||
print(f"Validating model performance with test tokens from: {trn_path}")
|
||||
tst_tok = read_whitespace_file(trn_path)
|
||||
tst_ids = np.array([([stoi.get(w, stoi[UNK]) for w in s]) for s in tst_tok])
|
||||
logloss, perplexity = validate(learn.model, tst_ids, self.exp.bptt)
|
||||
print('Test logloss:', logloss.item(), 'perplexity:', perplexity.item())
|
||||
|
||||
if __name__ == '__main__':
|
||||
fire.Fire(LMHyperParams)
|
||||
@@ -1,109 +0,0 @@
|
||||
from dataclasses import dataclass
|
||||
from ulmfit.train_clas import LMHyperParams
|
||||
from fastai.text import TextLMDataBunch, TextClasDataBunch
|
||||
from fastai.basic_train import LearnerCallback
|
||||
from fastai.torch_core import PBar, Rank0Tensor
|
||||
from torch import nn, Tensor
|
||||
|
||||
from typing import List, Collection, Any
|
||||
from pathlib import Path
|
||||
import pandas as pd
|
||||
import fire
|
||||
import random
|
||||
|
||||
@dataclass
|
||||
class ParallelAlignmentCallback(LearnerCallback):
|
||||
"A `LearnerCallback` that adds parallel alignment between sentences."
|
||||
|
||||
data_src:TextClasDataBunch
|
||||
data_tgt:TextClasDataBunch
|
||||
alpha:float=0.1
|
||||
|
||||
def __post_init__(self):
|
||||
self.bs = self.data_src.bs
|
||||
self.loss = nn.CosineEmbeddingLoss(margin=0.5)
|
||||
self.ones = torch.cat((torch.ones(self.bs), -torch.ones(self.bs)))
|
||||
|
||||
def pool(self, x:Tensor, bs:int, is_max:bool):
|
||||
"Pool the tensor along the seq_len dimension."
|
||||
f = F.adaptive_max_pool1d if is_max else F.adaptive_avg_pool1d
|
||||
return f(x.transpose(1,2), (1,)).view(bs,-1)
|
||||
|
||||
def get_representation(batch):
|
||||
last_output = self.learn.model(batch)
|
||||
output = last_output[1][-1]
|
||||
bs,sl,_ = output.size()
|
||||
avgpool = self.pool(output, bs, False)
|
||||
mxpool = self.pool(output, bs, True)
|
||||
return torch.cat([output[:,-1], mxpool, avgpool], 1)
|
||||
|
||||
def on_train_begin(self, pbar:PBar, metrics_names:Collection[str], **kwargs:Any)->None:
|
||||
self.counter = 0
|
||||
|
||||
def on_backward_begin(self, last_loss:Rank0Tensor, last_input:Tensor, **kwargs):
|
||||
"Adjust the loss by adding similarity of parallel sentences"
|
||||
src_rep = self.get_representation(data_src.train_ds[self.counter])
|
||||
tgt_rep = self.get_representation(data_tgt.train_ds[self.counter])
|
||||
|
||||
offset = -random.randrange(1, self.bs)
|
||||
|
||||
src_rep = torch.cat((src_rep, src_rep))
|
||||
tgt_rep = torch.cat((tgt_rep, tgt_rep[range(offset, self.bs + offset)]))
|
||||
|
||||
parallel_loss = self.alpha * self.loss(src_rep, tgt_rep, self.y)
|
||||
|
||||
self.counter += 1
|
||||
self.counter %= len(data_src.train_ds)
|
||||
return last_loss + parallel_loss
|
||||
|
||||
|
||||
@dataclass
|
||||
class XLingualLMHyperParams(LMHyperParams):
|
||||
|
||||
parallel_data_path: str=None
|
||||
parallel_data_bs: int=32
|
||||
src_lang: str=None
|
||||
tgt_lang: str=None
|
||||
|
||||
def create_lm_learner(self, data_lm, dps=None, **kwargs):
|
||||
learner = super().create_lm_learner(data_lm, dps, **kwargs)
|
||||
if self.parallel_data_path is not None:
|
||||
src_trn_df = pd.read_csv(self.parallel_data_path / self.src_lang / 'train.csv', header=None)
|
||||
tgt_trn_df = pd.read_csv(self.parallel_data_path / self.tgt_lang / 'train.csv', header=None)
|
||||
bs = self.parallel_data_bs
|
||||
data_src = TextClasDataBunch.from_df(path=self.cache_dir, train_df=src_trn_df, lm_type=self.lm_type, bs=bs)
|
||||
data_tgt = TextClasDataBunch.from_df(path=self.cache_dir, train_df=tgt_trn_df, lm_type=self.lm_type, bs=bs)
|
||||
learner.callback_fns = [
|
||||
partial(ParallelAlignmentCallback, data_src=data_src, data_tgt=data_tgt)
|
||||
] + learner.callback_fns
|
||||
|
||||
def load_wiki_data(self, bs=70):
|
||||
trn_path = self.dataset_path / f'{self.lang}.wiki.train.tokens'
|
||||
val_path = self.dataset_path / f'{self.lang}.wiki.valid.tokens'
|
||||
tst_path = self.dataset_path / f'{self.lang}.wiki.test.tokens'
|
||||
for path_ in [trn_path, val_path, tst_path]:
|
||||
assert path_.exists(), f'Error: {path_} does not exist.'
|
||||
|
||||
args = self.tokenzier_to_fastai_args(trn_data_loading_func=self.load_train_text, add_moses=False)
|
||||
try:
|
||||
data_lm = TextLMDataBunch.load(self.cache_dir, '.', lm_type=self.lm_type, bs=bs)
|
||||
print("Tokenized data loaded")
|
||||
except FileNotFoundError:
|
||||
print("Running tokenization")
|
||||
data_lm = TextLMDataBunch.from_df(path=self.cache_dir, train_df=read_wiki_articles(trn_path),
|
||||
valid_df=read_wiki_articles(val_path),
|
||||
classes=None, lm_type=self.lm_type, max_vocab=self.max_vocab,
|
||||
bs=bs, text_cols='texts', **args)
|
||||
data_lm.save('.')
|
||||
|
||||
itos, stoi, trn_path = data_lm.vocab.itos, data_lm.vocab.stoi, data_lm.path
|
||||
print('Size of vocabulary:', len(itos))
|
||||
print('First 20 words in vocab:', data_lm.vocab.itos[:20])
|
||||
return data_lm
|
||||
|
||||
if __name__ == '__main__':
|
||||
fire.Fire(XLingualLMHyperParams)
|
||||
|
||||
|
||||
# python -m ulmfit.XLingualLMHyperParams --dataset-path data/wiki/wikitext-103 --bidir=True --qrnn=True --nl=4 --tokenizer=sp --name 'nl4' --bs 120 --cuda-id 0 - train 10 --drop-mult=0 --bs 40
|
||||
|
||||
@@ -1,271 +0,0 @@
|
||||
"""
|
||||
Train a classifier on top of a language model trained with `pretrain_lm.py`.
|
||||
Optionally fine-tune LM before.
|
||||
"""
|
||||
|
||||
import fire
|
||||
|
||||
from fastai.callbacks import CSVLogger
|
||||
from fastai.text import *
|
||||
from fastai_contrib.utils import PAD_TOKEN_ID
|
||||
from ulmfit.pretrain_lm import LMHyperParams, ENC_BEST
|
||||
|
||||
|
||||
class CLSHyperParams(LMHyperParams):
|
||||
# dir_path -> data/imdb/
|
||||
use_test_for_validation=False
|
||||
|
||||
bicls_head:str = 'BiPoolingLinearClassifier'
|
||||
|
||||
def __post_init__(self, *args, **kwargs):
|
||||
super().__post_init__(*args, **kwargs)
|
||||
self.dataset_dir=self.dataset_path
|
||||
|
||||
@property
|
||||
def need_fine_tune_lm(self): return not (self.model_dir/f"enc_best.pth").exists()
|
||||
|
||||
def lr_schedule_layered(self, learn, num_cls_epochs):
|
||||
learn.freeze_to(-1)
|
||||
learn.fit_one_cycle(1, 2e-2, moms=(0.8, 0.7))
|
||||
if num_cls_epochs > 1:
|
||||
learn.freeze_to(-2)
|
||||
learn.fit_one_cycle(1, slice(1e-2 / (2.6 ** 4), 1e-2), moms=(0.8, 0.7))
|
||||
learn.freeze_to(-3)
|
||||
learn.fit_one_cycle(1, slice(5e-3 / (2.6 ** 4), 5e-3), moms=(0.8, 0.7))
|
||||
learn.unfreeze()
|
||||
learn.fit_one_cycle(num_cls_epochs, slice(1e-3 / (2.6 ** 4), 1e-3), moms=(0.8, 0.7))
|
||||
|
||||
def lr_schedule_2cycle(self, learn, num_cls_epochs):
|
||||
print("2cycle training schedule")
|
||||
learn.freeze_to(-1)
|
||||
learn.fit_one_cycle(1, 2e-2, moms=(0.8, 0.7))
|
||||
learn.unfreeze()
|
||||
if num_cls_epochs > 1:
|
||||
learn.fit_one_cycle(num_cls_epochs -1, slice(1e-2 / (2.6 ** 4), 1e-2), moms=(0.8, 0.7))
|
||||
|
||||
def lr_schedule_1cycle(self, learn, num_cls_epochs):
|
||||
print("Single training schedule")
|
||||
learn.unfreeze()
|
||||
learn.fit_one_cycle(num_cls_epochs, slice(1e-2 / (2.6 ** 4), 2e-2), moms=(0.8, 0.7))
|
||||
|
||||
def lr_schedule_false_wd(self, learn, num_cls_epochs):
|
||||
learn.true_wd = False
|
||||
print("Starting classifier training")
|
||||
learn.fit_one_cycle(1, 5e-2, moms=(0.8, 0.7), wd=1e-7)
|
||||
if num_cls_epochs > 1:
|
||||
learn.freeze_to(-2)
|
||||
learn.fit_one_cycle(1, slice(5e-2 / (2.6 ** 4), 5e-2), moms=(0.8, 0.7), wd=1e-7)
|
||||
learn.freeze_to(-3)
|
||||
learn.fit_one_cycle(1, slice(5e-4 / (2.6 ** 4), 5e-4), moms=(0.8, 0.7), wd=1e-7)
|
||||
learn.unfreeze()
|
||||
if num_cls_epochs > 5:
|
||||
learn.fit_one_cycle(num_cls_epochs-4, slice(1e-2 / (2.6 ** 4), 1e-2), moms=(0.8, 0.7), wd=1e-7)
|
||||
|
||||
def train_cls(self, num_lm_epochs, unfreeze=True, num_cls_frozen_epochs=1, bs=40, drop_mul_lm=0.3, drop_mul_cls=0.5,
|
||||
use_test_for_validation=False, num_cls_epochs=2, limit=None, noise=0.0, cls_max_len=20*70, lr_sched='layered',
|
||||
label_smoothing_eps=0.0, random_init=False):
|
||||
assert use_test_for_validation == False, "use_test_for_validation=True is not supported"
|
||||
self.model_dir.mkdir(exist_ok=True, parents=True)
|
||||
|
||||
if not unfreeze:
|
||||
num_cls_epochs = 1
|
||||
|
||||
data_clas, data_lm, data_tst = self.load_cls_data(bs, limit=limit, noise=noise)
|
||||
|
||||
if self.need_fine_tune_lm and not random_init:
|
||||
if not (self.model_dir/(ENC_BEST+".pth")).exists():
|
||||
self.train_lm(num_lm_epochs, data_lm=data_lm, drop_mult=drop_mul_lm, label_smoothing_eps=label_smoothing_eps)
|
||||
else:
|
||||
print("Language model already exist, skipping finetuning")
|
||||
learn = self.create_cls_learner(data_clas, drop_mult=drop_mul_cls, max_len=cls_max_len,
|
||||
label_smoothing_eps=label_smoothing_eps, random_init=random_init)
|
||||
if not random_init:
|
||||
try:
|
||||
learn.load('cls_best')
|
||||
print("Loading last classifier")
|
||||
except FileNotFoundError:
|
||||
learn.load_encoder(ENC_BEST)
|
||||
else:
|
||||
print("Starting classifier from random weights")
|
||||
|
||||
|
||||
if hasattr(self, 'lr_schedule_'+lr_sched):
|
||||
learn.true_wd = True
|
||||
getattr(self, 'lr_schedule_'+lr_sched)(learn, num_cls_epochs)
|
||||
else:
|
||||
raise ValueError(f"Wrong lr_sched: {lr_sched}")
|
||||
|
||||
print(f"Saving models at {learn.path / learn.model_dir}")
|
||||
learn.save('cls_last', with_opt=False)
|
||||
learn.save('cls_best', with_opt=False) # we don't use early stopping for the time being
|
||||
del learn
|
||||
return self.validate_cls('cls_best', bs=bs, data_cls=data_clas, data_tst=data_tst, learn=None)
|
||||
|
||||
def validate_cls(self, save_name='cls_best', bs=40, data_cls=None, data_tst=None, learn=None, label_smoothing_eps=0.0):
|
||||
if data_tst is None:
|
||||
data_cls, _, data_tst = self.load_cls_data(bs)
|
||||
if learn is None:
|
||||
learn = self.create_cls_learner(data_tst, drop_mult=0.3, label_smoothing_eps=label_smoothing_eps)
|
||||
learn.unfreeze()
|
||||
learn.load(save_name)
|
||||
f1 = FBeta(beta=1, average='binary')
|
||||
f1.on_train_begin()
|
||||
learn.metrics += [f1]
|
||||
val_res=[-1, -1]
|
||||
if data_cls:
|
||||
val_res = learn.validate(data_cls.valid_dl)
|
||||
tst_res = learn.validate(data_tst.valid_dl)
|
||||
print(f"Loss and accuracy using ({save_name}):", tst_res, val_res)
|
||||
results = {'val_loss': val_res[0], 'val_accuracy': float(val_res[1]), 'tst_loss':tst_res[0], 'tst_accuracy': float(tst_res[1]) }
|
||||
return results
|
||||
|
||||
def generate_pseudo_labels(self, dest_folder, save_name='cls_best', bs=40, data_cls=None, learn=None, label_smoothing_eps=0.0):
|
||||
if data_cls is None:
|
||||
data_cls, _, _ = self.load_cls_data(bs)
|
||||
if learn is None:
|
||||
learn = self.create_cls_learner(data_cls, drop_mult=0.3, label_smoothing_eps=label_smoothing_eps)
|
||||
learn.unfreeze()
|
||||
learn.load(save_name)
|
||||
|
||||
def make_data_set(ds_type, name):
|
||||
probs, lbls = learn.get_preds(ds_type=ds_type, ordered=True)
|
||||
preds = torch.argmax(probs, 1)
|
||||
preds = to_np(preds)
|
||||
fn = self.dataset_path / f"{self.lang}.{name}.csv"
|
||||
if fn.exists():
|
||||
df = pd.read_csv(fn, header=None)
|
||||
df = df.iloc[(len(df) - len(preds)):] # account for the training files where first 10% elements were taken as validation
|
||||
else:
|
||||
df = pd.read_csv(self.dataset_path / f"{self.lang}.dev.csv", header=None)
|
||||
df = df.iloc[:len(preds)] # if using training only get first n for validatation
|
||||
|
||||
accuracy = (df[0] == preds).sum() / len(preds)
|
||||
print(f"Generating {name} dataset of size {len(preds)}, the accuracy is {accuracy}")
|
||||
df['preds'] = preds
|
||||
print(df.head())
|
||||
del df['preds']
|
||||
assert accuracy > 0.7, f"Accuracy is smaller than 0.7 {accuracy}"
|
||||
|
||||
df[0] = preds
|
||||
dest_folder.mkdir(parents=True, exist_ok=True)
|
||||
df.to_csv(dest_folder / f"{self.lang}.{name}.csv", index=None, header=None)
|
||||
|
||||
make_data_set(DatasetType.Train, "train")
|
||||
make_data_set(DatasetType.Valid, "dev")
|
||||
shutil.copy(self.dataset_path / f"{self.lang}.test.csv", dest_folder)
|
||||
shutil.copy(self.dataset_path / f"{self.lang}.unsup.csv", dest_folder)
|
||||
|
||||
def create_cls_learner(self, data_clas, dps=None, label_smoothing_eps=0.0, random_init=False, **kwargs):
|
||||
assert self.bidir == False, "bidirectional model is not yet supported"
|
||||
config = dict(emb_sz=self.emb_sz, n_hid=self.nh, n_layers=self.nl, pad_token=PAD_TOKEN_ID, qrnn=self.qrnn)
|
||||
config.update(dps or self.dps)
|
||||
trn_args=dict(bptt=self.bptt, clip=self.clip)
|
||||
trn_args.update(kwargs)
|
||||
learn = text_classifier_learner(data_clas, AWD_LSTM, config=config,
|
||||
pretrained=False, path=self.model_dir.parent, model_dir=self.model_dir.name, **trn_args)
|
||||
|
||||
if self.pretrained_model is not None and not random_init:
|
||||
print("Loading pretrained model", self.pretrained_model)
|
||||
model_path = untar_data(self.pretrained_model, data=False)
|
||||
fnames = [list(model_path.glob(f'*.{ext}'))[0] for ext in ['pth', 'pkl']]
|
||||
learn.load_pretrained(*fnames, strict=False)
|
||||
learn.freeze()
|
||||
|
||||
learn.callback_fns += [partial(CSVLogger, filename=f"{learn.model_dir}/cls-history"),
|
||||
#partial(SaveModelCallback, every='improvement', name='cls_best') disabled due to memory issues
|
||||
]
|
||||
if label_smoothing_eps > 0.0:
|
||||
learn.loss_func = FlattenedLoss(LabelSmoothingCrossEntropy, eps=label_smoothing_eps)
|
||||
return learn
|
||||
|
||||
def load_cls_data(self, bs, **kwargs):
|
||||
self.model_dir.mkdir(exist_ok=True, parents=True)
|
||||
add_trn_to_lm = True
|
||||
lang = self.lang
|
||||
use_moses = True
|
||||
if 'xnli' in str(self.dataset_dir):
|
||||
NotImplementedError("Support for Xnli is not implemented yet")
|
||||
if 'imdb' in self.dataset_dir.name:
|
||||
lang=''
|
||||
add_trn_to_lm = True
|
||||
if 'mldoc' in str(self.dataset_dir):
|
||||
add_trn_to_lm = False # False as trn_df is contained in unsup already
|
||||
lang = self.lang
|
||||
|
||||
data = self.load_data(lang=lang,
|
||||
add_trn_to_lm=add_trn_to_lm,
|
||||
use_moses=use_moses,
|
||||
**kwargs)
|
||||
return self.databunches(bs, **data)
|
||||
|
||||
def load_data(self, lang='', **kwargs):
|
||||
prefix = '' if lang == '' else lang+'.'
|
||||
trn_df = pd.read_csv(self.dataset_path / f'{prefix}train.csv', header=None)
|
||||
tst_df = pd.read_csv(self.dataset_path / f'{prefix}test.csv', header=None)
|
||||
val_fn = self.dataset_path / f'{prefix}dev.csv'
|
||||
if val_fn.exists():
|
||||
print("Loading validation", val_fn)
|
||||
val_df = pd.read_csv(val_fn, header=None)
|
||||
else:
|
||||
val_df = None
|
||||
unsup_fn = self.dataset_path / f'{prefix}unsup.csv'
|
||||
unsup_df = pd.read_csv(unsup_fn, header=None) if unsup_fn.exists() else trn_df[:0]
|
||||
|
||||
if val_df is None:
|
||||
print("Validation set not found using 10% of trn")
|
||||
val_len = max(int(len(trn_df) * 0.1), 2)
|
||||
trn_len = len(trn_df) - val_len
|
||||
trn_df, val_df = trn_df[:trn_len], trn_df[trn_len:]
|
||||
kwargs.update(dict(trn_df=trn_df, val_df=val_df, tst_df=tst_df, unsup_df=unsup_df))
|
||||
return kwargs
|
||||
|
||||
def add_noise(self, trn_df, noise):
|
||||
count = len(trn_df)
|
||||
labels = trn_df[0].unique()
|
||||
assert np.issubdtype(labels.dtype, np.integer), "noise only works on numerical numbers"
|
||||
modulo = labels.max() + 1
|
||||
idx_to_distrub = np.random.permutation(count)[:int(count * noise)]
|
||||
trn_df.loc[idx_to_distrub, [0]] = (np.random.randint(1, modulo - 1, size=len(idx_to_distrub)) +
|
||||
trn_df.loc[idx_to_distrub][0]) % modulo
|
||||
print(f"Added noise to {len(idx_to_distrub)} examples, only {(count - len(idx_to_distrub)) / count} have correct labels")
|
||||
return trn_df
|
||||
|
||||
def databunches(self, bs, trn_df, val_df, tst_df, unsup_df, add_trn_to_lm=True, use_moses=False, force=False, limit=None, noise=0.0):
|
||||
lm_trn_df = pd.concat([unsup_df, val_df, tst_df] + ([trn_df] if add_trn_to_lm else []))
|
||||
val_len = max(int(len(lm_trn_df) * 0.1), 2)
|
||||
lm_trn_df = lm_trn_df[val_len:]
|
||||
lm_val_df = lm_trn_df[:val_len]
|
||||
|
||||
cls_name="cls"
|
||||
if limit is not None:
|
||||
print("Limiting data set to:", limit)
|
||||
trn_df = trn_df[:limit]
|
||||
val_df = val_df[:limit]
|
||||
cls_name=f'{cls_name}limit{limit}'
|
||||
|
||||
if noise > 0.0:
|
||||
trn_df = self.add_noise(trn_df, noise)
|
||||
val_df = self.add_noise(val_df, noise)
|
||||
cls_name = f'{cls_name}noise{noise}tv'
|
||||
|
||||
args = self.tokenizer_to_fastai_args(sp_data_func=lambda: trn_df[1], use_moses=use_moses)
|
||||
args['text_cols'] = list(trn_df.columns.values)[1:]
|
||||
args['mark_fields'] = True
|
||||
lm_suffix = self.bptt if self.bptt != 70 else ""
|
||||
data_lm = self.lm_databunch(f'lm{lm_suffix}', train_df=lm_trn_df, valid_df=lm_val_df, bs=bs, force=force, bptt=self.bptt, **args)
|
||||
args['vocab'] = data_lm.vocab
|
||||
data_cls = self.cls_databunch(cls_name, train_df=trn_df, valid_df=val_df, bs=bs, force=force, **args)
|
||||
data_tst = self.cls_databunch('tst', train_df=val_df, valid_df=tst_df, bs=bs, force=force, **args) # Hack to load test dataset with labels
|
||||
|
||||
print('Size of vocabulary:', len(data_lm.vocab.itos))
|
||||
print('First 20 words in vocab:', data_lm.vocab.itos[:20])
|
||||
return data_cls, data_lm, data_tst
|
||||
|
||||
def cls_databunch(self, name, *args, **kwargs):
|
||||
return self.databunch(name, bunch_class=TextClasDataBunch, *args, **kwargs)
|
||||
|
||||
if __name__ == '__main__':
|
||||
fire.Fire(CLSHyperParams)
|
||||
|
||||
##
|
||||
|
||||
@@ -1,73 +0,0 @@
|
||||
from dataclasses import dataclass
|
||||
from ulmfit.train_clas import CLSHyperParams
|
||||
from fastai.text import TextLMDataBunch, TextClasDataBunch
|
||||
|
||||
from typing import List
|
||||
from pathlib import Path
|
||||
import pandas as pd
|
||||
import fire
|
||||
|
||||
@dataclass
|
||||
class XLingualCLSHyperParams(CLSHyperParams):
|
||||
csv_name: str='train.csv'
|
||||
target_paths: List[str]=None
|
||||
parallel_data_path: str=None
|
||||
|
||||
def __post_init__(self, *args, **kwargs):
|
||||
super().__post_init__(*args, **kwargs)
|
||||
self.target_paths = [] if self.target_paths is None else self.target_paths
|
||||
|
||||
def load_cls_data(self, bs, force=False, use_test_for_validation=False, **kwargs):
|
||||
args = self.tokenzier_to_fastai_args(trn_data_loading_func=lambda: trn_df[1], add_moses=True)
|
||||
src_path = self.dataset_path
|
||||
csv_name = self.csv_name
|
||||
tgt_paths = [Path(tgt_path) for tgt_path in self.target_paths]
|
||||
mixed_csv = pd.read_csv(src_path / csv_name, header=None)
|
||||
for tgt_path in tgt_paths:
|
||||
mixed_csv = pd.concat([mixed_csv, pd.read_csv(tgt_path / csv_name, header=None)])
|
||||
|
||||
xcvs_name = ('x_' + csv_name)
|
||||
mixed_csv.to_csv(src_path / xcvs_name, header=None, index=False)
|
||||
|
||||
try:
|
||||
if force: raise FileNotFoundError("Forcing reloading of caches")
|
||||
data_lm = TextLMDataBunch.load(src_path, 'xlm', lm_type=self.lm_type, bs=bs)
|
||||
print(f"Tokenized data loaded, xlm.trn {len(data_lm.train_ds)}, xlm.val {len(data_lm.valid_ds)}")
|
||||
except FileNotFoundError:
|
||||
print(f"Running tokenization...")
|
||||
data_lm = TextLMDataBunch.from_csv(path=src_path, csv_name=xcvs_name, bs=bs, lm_type=self.lm_type, **kwargs, **args)
|
||||
print(f"Saving tokenized: cls.trn {len(data_lm.train_ds)}, cls.val {len(data_lm.valid_ds)}")
|
||||
data_lm.save('xlm')
|
||||
|
||||
try:
|
||||
if force: raise FileNotFoundError("Forcing reloading of caches")
|
||||
data_cls = TextClasDataBunch.load(src_path, 'cls', bs=bs)
|
||||
print(f"Tokenized data loaded, cls.trn {len(data_cls.train_ds)}, cls.val {len(data_cls.valid_ds)}")
|
||||
except FileNotFoundError:
|
||||
args['vocab'] = data_lm.vocab # make sure we use the same vocab for classifcation
|
||||
print(f"Running tokenization...")
|
||||
data_cls = TextClasDataBunch.from_csv(path=src_path, csv_name=csv_name, bs=bs, **kwargs, **args)
|
||||
|
||||
print(f"Saving tokenized: cls.trn {len(data_cls.train_ds)}, cls.val {len(data_cls.valid_ds)}")
|
||||
data_cls.save('cls')
|
||||
|
||||
print('Size of vocabulary:', len(data_lm.vocab.itos))
|
||||
print('First 20 words in vocab:', data_lm.vocab.itos[:20])
|
||||
return data_cls, data_lm
|
||||
|
||||
def validate_cls(self, save_name='cls_last', bs=40):
|
||||
args = self.tokenzier_to_fastai_args(trn_data_loading_func=lambda: trn_df[1], add_moses=True)
|
||||
data_clas, data_lm = self.load_cls_data(bs, use_test_for_validation=True)
|
||||
data_eval = [
|
||||
TextClasDataBunch.from_csv(path=Path(tgt_path), csv_name=self.csv_name, **args)
|
||||
for tgt_path in self.target_paths
|
||||
]
|
||||
|
||||
for data in [data_clas] + data_eval:
|
||||
learn = self.create_cls_learner(data, drop_mult=0.1)
|
||||
learn.load(save_name)
|
||||
print(f"Loss and accuracy using ({save_name}) for dataset at {data.path}:", learn.validate())
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
fire.Fire(XLingualCLSHyperParams)
|
||||
Reference in New Issue
Block a user