mirror of https://github.com/wassname/phoneme2grapheme.git synced 2026-09-10 12:30:12 +08:00

Files

T

gitea c61836db48 Merge branch 'master' of github.com:wassname/phoneme2grapheme

Conflicts:
	readme.md

2017-06-01 13:46:44 +08:00

394 KiB

Raw Permalink Blame History

In [1]:

import os
# os.environ['THEANO_FLAGS']='mode=FAST_RUN,device=gpu,floatX=float32'
os.environ['KERAS_BACKEND']='tensorflow'
os.sys.path.append('..')

In [2]:

# keras
import keras
# from keras.engine.training import slice_X
from keras.layers import Activation, TimeDistributed, Dense, RepeatVector, recurrent
from keras.layers import InputLayer, Embedding, Lambda, Reshape, InputLayer, Dropout
from keras.models import Sequential
from keras.preprocessing import sequence

from keras.wrappers.scikit_learn import KerasClassifier
from keras.layers.wrappers import Bidirectional, TimeDistributed

# from keras.engine.training import slice_X
from keras.layers import Activation, TimeDistributed, Dense, RepeatVector, recurrent
from keras.layers import InputLayer, Embedding
from keras.models import Sequential

# helper
from keras_tqdm import TQDMNotebookCallback

Using TensorFlow backend.

In [3]:

# sklearn
import sklearn
# from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import train_test_split

In [18]:

# pylab imports
from matplotlib import pyplot as plt
import numpy as np
import pandas as pd
import scipy as sp
import seaborn as sns
%matplotlib inline

# io & utils
from path import Path
from pprint import pprint
import json
import arrow
from tqdm import tqdm_notebook as tqdm

import re
import string
import itertools
import collections

In [29]:

seed=1337 # for reproducibility
batch_size=64*3

In [30]:

import sklearn

import numpy as np
import scipy as sp
import pandas as pd
from matplotlib import pyplot as plt
%matplotlib inline

from tqdm import tqdm_notebook as tqdm

In [31]:

import os
os.sys.path.append('.')

In [32]:

%reload_ext autoreload
%autoreload 2
from helpers import dataset_isledict, show_results

About

What?

This project tried to build a spelling guesser, or technically a phoneme2grapheme translator.

What, what?

Image you hear someone say a word for the first time:

DYE-uh-REE-a

How is it spelt? We know the answer

DYE-uh-REE-a => "Diarrhoea"

But what if we didn't, here's a word you probobly don't know:

PHA-NG-AH-RAY =>

answer

So we are building something that can take the pronounciation and guess the spelling.

Data?

We want a dictionary of data like

pron	spelling
DYE-uh-REE-a	Diarrhoea
PHA-NG-AH-RAY	-
AR-kən-saw	Arkansas

The best data comes from the CMU Pronouncing Dictionary with 140k entries

Pronounciation|Spelling -------| ------- |----------- AA D|odd AE T|at HH AH T|hut AO T|ought K AW|cow HH AY D|hide B IY|be CH IY Z|cheese D IY|dee DH IY|thee EH D|Ed

Why?

It can be used to

generate spelling for words that are not in the dictionary
identify hard to spell words in english
make a consistent spelling scheme for english?
other things?
no one's done it

How

I'm not happy with my results, so what do you suggest?

Metrics

The first steps is to decide on a metric and generate dummy benchmarks. That way we know what we have to beat.

Our metric is

Charector error rate: CER
bad: 50% DYE-uh-REE-a =>	Dirrrhoaa
good: 0% DYE-uh-REE-a =>	Diarrhoea

In [33]:

import numpy as np
def CER(y_pred,y_test):
    """
    A keras metric for Charecter error rate
    
    inputs:
        y_pred - an array with shape samples*index*classes
        y_test - as above    
    """
    return 1-(y_pred.argmax(-1)==y_test.argmax(-1)).sum()/(y_test.shape[0]*y_test.shape[1])
# print ('Char error rate {:%}'.format(CER(y_pred,y_test)))

from keras.utils.np_utils import to_categorical
# good
assert CER(
    np.array([to_categorical([0,2,1,4,0],5)]),
    np.array([to_categorical([0,2,1,4,0],5)])
) ==0

# bad
assert CER(
    np.array([to_categorical([0,2,1,4],5)]),
    np.array([to_categorical([0,2,0,2],5)])
)==0.5

In [119]:

def WER(y_true,y_pred):
    # calc word error rate (compare 23.3-33.89 for grap2phon https://github.com/cmusphinx/g2p-seq2seq)
    # like https://github.com/cmusphinx/g2p-seq2seq/blob/master/g2p_seq2seq/g2p.py#L317
    c=(y_pred.argmax(-1)==y_true.argmax(-1)).all(-1).sum()
    return (1-c/len(y_true))

from leven import levenshtein 
def element_error_rate(y_true, y_pred, ytable=ytable):
    """Returns the word error rate of the supplied hypothesis with respect to
    the reference string."""
    c_test = [''.join(x).strip() for x in ytable.decode(y_true)]
    c_pred = [''.join(x).strip() for x in ytable.decode(y_pred)]
    distance = np.array([levenshtein(pred,test) for pred,test in zip(c_pred,c_test)])
    elems = np.array([len(x) for x in c_test])
    error_rate = distance / len(elems)
    return error_rate.sum()

# unit test
a=[[1,2,3],[1,2,3]]
b=[[1,2,3],[2,0,3]]
a=np.array([keras.utils.np_utils.to_categorical(x,4) for x in a])
b=np.array([keras.utils.np_utils.to_categorical(x,4) for x in b])
assert WER(a,b)==0.5
assert element_error_rate(a,b)==1

load data

My input data is the CMUDict which maps english to the 39 ARPAbet phonemes. I then convert this to IPA.

I wrote helper functions to load the data this is boring so I will leave it out - jeremy style

download
parse a text file
remove duplicates
turn charectors into numbers and back
- a=>0, 0=>a, b=>2, ... etc
turn phonemes into numbers and back
- AH => 0, 0=>AH, ... etc
split into 2 sets :( it should be three since we shouldn't use the test set for trying model archetectures

We will make it faster by restricting it to sequences of 10 or under

In [64]:

# grab a small subset of the data
(y_train, X_train),(y_test,X_test),(ytable,xtable) = dataset_isledict.get_data(
    verbose=1,
    max_phonemes=10,
    max_chars=10,
#     invert=False,
#     x_hot=True,
    unique_graphemes=True,
    unique_phonemes=True
    
)
X_train = X_train[:,::-1,:]
X_test = X_test[:,::-1,:]
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.1, random_state=seed)

nb_chars = len(ytable.chars)
nb_phons = len(xtable.chars)
maxlen_x = xtable.maxlen
maxlen_y = ytable.maxlen

loaded entries 282842
removed duplicate phonemes leaving 260707
removed duplicate graphemes leaving 236077
removed blacklisted entries leaving 158184
restricted to less than 10 phonemes leaving 113368 entries or 71.81%
X_train shape: (90879, 10, 29)
X_test shape: (22489, 10, 29)
y_train shape: (90879, 10, 43)
y_test shape: (22489, 10, 43)

In [65]:

# crop to batch size to prevent errors
train_crop = len(X_train)-(len(X_train)%batch_size)
test_crop = len(X_test)-(len(X_test)%batch_size)
val_crop = len(X_val)-(len(X_val)%batch_size)
print(train_crop,test_crop,val_crop)
X_train=X_train[:train_crop]
y_train=y_train[:train_crop]
X_test=X_test[:test_crop]
y_test=y_test[:test_crop]
X_val=X_val[:val_crop]
y_val=y_val[:val_crop]

81600 22464 9024

In [66]:

# lets just do a sanity test
assert (X_train.sum(-1)==1).all(),'all classes must sum to one'
assert (y_train.sum(-1)==1).all(),'all classes must sum to one'
assert (X_train.max(-1)==1).all(),'should have max value of 1'
assert (X_train.min(-1)==0).all(), 'should have min of 0'

In [67]:

# view data
a=[''.join(x[::-1]) for x in xtable.decode(X_train[:5])]
b=[''.join(y) for y in ytable.decode(y_train[:5])]
list(zip(a,b))

Out [67]:

[('mˈeiŋgl̩  ', 'mangel    '),
 ('fɹˈut˺ɪŋ  ', 'fruiting  '),
 ('tˈʌkəhˌoʊ ', 'tuckahoe  '),
 ('kloʊ      ', 'clos      '),
 ('fɪlˈɪpiæk ', 'filipiak  ')]

Dummy model

Our best dummy models get 60% CER or above

We have to a little massaging because scikit-learn works with 2d arrays while keras uses >2d. I use argmax, to_categorical, and reshape for this.

In [60]:

from keras.utils.np_utils import to_categorical
def to_categorical_3d(data2,n=None):
    data=data2.astype(np.int)
    return np.array([to_categorical(data[i],n) for i in range(len(data))])

In [61]:

from sklearn.dummy import DummyClassifier
for strategy in ['stratified', 'most_frequent', 'prior', 'uniform']:
    clf = DummyClassifier(strategy=strategy,random_state=0)
    clf.fit(X_train.reshape((-1,nb_phons)), y_train.reshape((-1,nb_chars)).argmax(-1))
    
    # on a simple classifier CER=1-score
    score = clf.score(X_test.reshape((-1,nb_phons)), y_test.reshape((-1,nb_chars)).argmax(-1))  
    print('{strategy:20.20s}: CER={CER:2.2%}'.format(strategy=strategy, CER=1-score))

stratified          : CER=87.76%
most_frequent       : CER=69.53%
prior               : CER=69.53%
uniform             : CER=96.44%

In [62]:

from sklearn.dummy import DummyRegressor
for strategy in ['mean', 'median', 'quantile', 'constant']:
    clf = DummyRegressor(strategy=strategy, constant=np.zeros((10,1)), quantile=0.555)
    clf.fit(X_train.argmax(-1), y_train.argmax(-1))
    score = clf.score(X_test.argmax(-1), y_test.argmax(-1))  
    y_pred_2d = clf.predict(X_test.argmax(-1))  
    y_pred_3d=to_categorical_3d(y_pred_2d)
    cer = CER(y_pred_3d, y_test)
    print('{strategy:20.20s}: CER={CER:2.2%}'.format(strategy=strategy, CER=cer))

mean                : CER=88.13%
median              : CER=71.09%
quantile            : CER=71.82%
constant            : CER=69.53%

Helpers

display
- something to show our results as as table
- progress logger to show predictons as we go
- tqdm_keras to have fast progress bars

PredictionLogger

With keras you can make custom callback so that you can see the prediction after every epoch. You will see it in action soon.

In [95]:

# PredictionLogger
import keras
from keras.callbacks import Callback
class PredictionLogger(keras.callbacks.Callback):
    def __init__(self, n=1, batch_size=batch_size):
        self.n = n
        self.batch_size = batch_size

    def on_epoch_end(self, batch, logs={}):
        y_pred = self.model.predict(self.model.validation_data[0][:self.batch_size], batch_size=batch_size)
        p_pred = ytable.decode(y_pred)
        p_test = ytable.decode(self.model.validation_data[1][:self.batch_size])
        text_pred=[''.join(ls) for ls in p_pred[:self.n]]
        text_true=[''.join(ls) for ls in p_test[:self.n]]
        print('')
        pprint(list(zip(text_true,text_pred)))

prediction_logger = PredictionLogger(n=3)

quick cnn

In [28]:

from keras.layers import Embedding, Convolution1D, GlobalMaxPooling1D, Reshape, Flatten, Dropout
import keras
model = Sequential()

# we start off with an efficient embedding layer which maps
# our vocab indices into embedding_dims dimensions
model.add(Embedding(nb_phons,
                    90,
                    input_length=maxlen_x,
                    dropout=0.2))

# we add a Convolution1D, which will learn nb_filter
# word group filters of size filter_length:
model.add(Convolution1D(nb_filter=256,
                        filter_length=3,
                        border_mode='valid',
                        activation='relu',
                        subsample_length=1))
model.add(Dropout(0.2))

# we add a Convolution1D, which will learn nb_filter
# word group filters of size filter_length:
model.add(Convolution1D(nb_filter=256,
                        filter_length=3,
                        border_mode='valid',
                        activation='relu',
                        subsample_length=1))

# classifier
model.add(Flatten())
model.add(Dense(maxlen_y*nb_chars))
model.add(Reshape((maxlen_y,nb_chars)))

model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy',
              optimizer='nadam', #keras.optimizers.Nadam(lr=0.2),
              metrics=['accuracy'])
# model.summary()

history = model.fit(
    X_train.argmax(-1), 
    y_train, 
    batch_size=batch_size, 
    nb_epoch=20, 
    verbose=0, 
    validation_data=[X_test.argmax(-1),y_test],
    callbacks=[
        keras.callbacks.EarlyStopping(patience=2),
        PredictionLogger(n=3),
        TQDMNotebookCallback()
    ]
)

[0;31m---------------------------------------------------------------------------[0m
[0;31mNameError[0m                                 Traceback (most recent call last)
[0;32m<ipython-input-28-0dea8e399d33>[0m in [0;36m<module>[0;34m()[0m
[1;32m     41[0m     [0mX_train[0m[0;34m.[0m[0margmax[0m[0;34m([0m[0;34m-[0m[0;36m1[0m[0;34m)[0m[0;34m,[0m[0;34m[0m[0m
[1;32m     42[0m     [0my_train[0m[0;34m,[0m[0;34m[0m[0m
[0;32m---> 43[0;31m     [0mbatch_size[0m[0;34m=[0m[0mbatch_size[0m[0;34m,[0m[0;34m[0m[0m
[0m[1;32m     44[0m     [0mnb_epoch[0m[0;34m=[0m[0;36m20[0m[0;34m,[0m[0;34m[0m[0m
[1;32m     45[0m     [0mverbose[0m[0;34m=[0m[0;36m0[0m[0;34m,[0m[0;34m[0m[0m

[0;31mNameError[0m: name 'batch_size' is not defined

In [73]:

pd.DataFrame(history.history).plot()

scores = model.evaluate(X_test.argmax(-1),y_test, verbose=0)
score = dict(zip(model.metrics_names,scores))

y_pred = model.predict(X_test.argmax(-1), verbose=0, batch_size=batch_size)

print ('Accuracy        {:%}'.format(score['acc']))
print ('Word error rate {:%}'.format(WER(y_pred,y_test)))
print ('Char error rate {:%}'.format(CER(y_pred,y_test)))
n=100
show_results(ytable=ytable, y_pred=y_pred[:n],y_test=y_test[:n],xtable=xtable,X_test=X_test[:n])

Out [73]:

Accuracy        77.108367%
Word error rate 80.812842%
Char error rate 22.891636%

pronunciation	guess	spelling
kˈɑkəsˌɔid	cocksoid	caucasoid
bɹˈæsfˌild	brassfildd	brassfield
flˈæʃbl̩b	flashblb	flashbulb
hˈɛstiə	hestia	hestia
kɹˈæbi	crabyy	crabby
tɹədˈus	troduse	traduce
təlˈulə	talulaa	tallulah
bˈitn̩	beatn	beaton
hˈɑlɪtʃɛk	halicck	holecek
beidˈɔiə	bedooy	bedoya

Using a sequence2sequence rnn with attention

This notebook uses the seq2seq translator with "attention" for alignment.

We choose this because this model has had great results (a 24% word error rate) for grapheme two phoneme translation. We are doing it the other way round: phoneme to grapheme. This is a bit harder but the same model should work well.

I tried lots of anttenion layer for keras but in the end I wrote by own by porting from tensorflow. This is based on tensorflows implementation.

In [43]:

from helpers.layers import Attention, SimplifiedAttention

In [44]:

# set model parameters
nb_chars = len(ytable.chars)
nb_phons = len(xtable.chars)
maxlen_x = xtable.maxlen
maxlen_y = ytable.maxlen
# batch_size = 300
hidden_nodes = 64
RNN=keras.layers.recurrent.GRU

In [45]:

import tensorflow
import keras

# you probobly need these exact versions
assert keras.__version__=='1.2.2'
assert tensorflow.__version__=='1.0.0'

In [ ]:

In [80]:

# rnn's in tensorflow sometimes need exact batches
assert X_val.shape[0]%batch_size==0
assert X_train.shape[0]%batch_size==0
batch_size

Out [80]:

In [ ]:

# # here's a pretrained model with 81% accuracy
# model = keras.models.load_model(
#     './models/phone_to_respelling_201705-31233850_acc-0.81.hdf',
#     custom_objects=dict(Attention=Attention)
# )

In [92]:

model = Sequential()

# Encode, bidirectional keeps the accuracy up at the end of the word
model.add(Bidirectional(RNN(hidden_nodes, return_sequences=True), batch_input_shape=(batch_size, maxlen_x, nb_phons)))
# model.add(RNN(hidden_nodes, return_sequences=True, batch_input_shape=(batch_size, maxlen_x, nb_phons)))
model.add(Dropout(0.2))

# Decode with attention
model.add(Attention(RNN(hidden_nodes, return_sequences=True, consume_less='mem')))
model.add(Dropout(0.2))

# # I could add some more RNN models
# model.add(RNN(hidden_nodes, return_sequences=True, consume_less='mem'))

# classifier
model.add(TimeDistributed(Dense(nb_chars)))

model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy',
              optimizer='nadam',
              metrics=['accuracy'])
model.summary()

____________________________________________________________________________________________________
Layer (type)                     Output Shape          Param #     Connected to                     
====================================================================================================
bidirectional_13 (Bidirectional) (192, 10, 128)        41472       bidirectional_input_13[0][0]     
____________________________________________________________________________________________________
dropout_26 (Dropout)             (192, 10, 128)        0           bidirectional_13[0][0]           
____________________________________________________________________________________________________
attention_13 (Attention)         (192, 10, 64)         57728       dropout_26[0][0]                 
____________________________________________________________________________________________________
dropout_27 (Dropout)             (192, 10, 64)         0           attention_13[0][0]               
____________________________________________________________________________________________________
timedistributed_13 (TimeDistribu (192, 10, 29)         1885        dropout_27[0][0]                 
____________________________________________________________________________________________________
activation_14 (Activation)       (192, 10, 29)         0           timedistributed_13[0][0]         
====================================================================================================
Total params: 101,085
Trainable params: 101,085
Non-trainable params: 0
____________________________________________________________________________________________________

In [97]:


history = model.fit(X_train, y_train, 
                    batch_size=batch_size, 
                    nb_epoch=60, 
                    verbose=0, 
                    validation_data=[X_val,y_val],
                    callbacks=[
                            keras.callbacks.EarlyStopping(monitor='val_acc', patience=2),
                            PredictionLogger(n=4),
                            TQDMNotebookCallback(),
                        ]
                   )