TTS/speaker_encoder/notebooks/PlotUmapLibriTTS.ipynb

Overview

This notebook can be used with both a single or multi- speaker corpus and allows the interactive plotting of speaker embeddings linked to underlying audio (see instructions in the repo's speaker_embedding directory)

Depending on the directory structure used for your corpus, you may need to adjust handling of speaker_to_utter and locations.

In [1]:

import os
import glob
import random
import numpy as np
import torch
import umap

from TTS.speaker_encoder.model import SpeakerEncoder
from TTS.utils.audio import AudioProcessor
from TTS.utils.generic_utils import load_config

from bokeh.io import output_notebook, show
from bokeh.plotting import figure
from bokeh.models import HoverTool, ColumnDataSource, BoxZoomTool, ResetTool, OpenURL, TapTool
from bokeh.transform import factor_cmap, factor_mark
from bokeh.palettes import Category10

For larger sets of speakers, you can use Category20, but you need to change it in the pal variable too

List of Bokeh palettes here: http://docs.bokeh.org/en/1.4.0/docs/reference/palettes.html

NB: if you have problems with other palettes, first see https://stackoverflow.com/questions/48333820/why-do-some-bokeh-palettes-raise-a-valueerror-when-used-in-factor-cmap

In [2]:

output_notebook()

Loading BokehJS ...

[JavaScript output - execution disabled for security]

You should also adjust all the path constants to point at the relevant locations for you locally

In [3]:

MODEL_RUN_PATH = "/home/neil/main/Projects/TTS3/speaker_encoder/libritts_360-half-September-28-2019_10+46AM-8565c50/"
MODEL_PATH = MODEL_RUN_PATH + "best_model.pth.tar"
CONFIG_PATH = MODEL_RUN_PATH + "config.json"

# My single speaker locations
#EMBED_PATH = "/home/neil/main/Projects/TTS3/embeddings/neil14/"
#AUDIO_PATH = "/home/neil/data/Projects/NeilTTS/neil14/wavs/"

# My multi speaker locations
EMBED_PATH = "/home/neil/main/Projects/TTS3/embeddings/libritts_360/"
AUDIO_PATH = "/home/neil/main/Projects/LibriTTS/LibriTTS/train-clean-360/"

In [4]:

!ls -1 $MODEL_RUN_PATH

best_model.pth.tar
config.json
events.out.tfevents.1569660396.erogol-desktop

In [5]:

CONFIG = load_config(CONFIG_PATH)
ap = AudioProcessor(**CONFIG['audio'])

 > Setting up Audio Processor...
 | > sample_rate:24000
 | > num_mels:40
 | > min_level_db:-100
 | > frame_shift_ms:12.5
 | > frame_length_ms:50
 | > ref_level_db:20
 | > num_freq:1025
 | > power:None
 | > preemphasis:0.98
 | > griffin_lim_iters:None
 | > signal_norm:True
 | > symmetric_norm:True
 | > mel_fmin:0
 | > mel_fmax:8000.0
 | > max_norm:4.0
 | > clip_norm:True
 | > do_trim_silence:False
 | > sound_norm:False
 | > n_fft:2048
 | > hop_length:300
 | > win_length:1200

Bring in the embeddings created by compute_embeddings.py

In [6]:

embed_files = glob.glob(EMBED_PATH+"/**/*.npy", recursive=True)
print(f'Embeddings found: {len(embed_files)}')

Embeddings found: 116500

Check that we did indeed find an embedding

In [7]:

embed_files[0]

Out [7]:

'/home/neil/main/Projects/TTS3/embeddings/libritts_360/7732/107520/7732_107520_000009_000003.npy'

Process the speakers

Assumes count of speaker_paths corresponds to number of speakers (so a corpus in just one directory would be treated like a single speaker and the multiple directories of LibriTTS are treated as distinct speakers)

In [8]:

speaker_paths = list(set([os.path.dirname(os.path.dirname(embed_file)) for embed_file in embed_files]))
speaker_to_utter = {}
for embed_file in embed_files:
    speaker_path = os.path.dirname(os.path.dirname(embed_file))
    try:
        speaker_to_utter[speaker_path].append(embed_file)
    except:
        speaker_to_utter[speaker_path]=[embed_file]
print(f'Speaker count: {len(speaker_paths)}')

Speaker count: 904

Set up the embeddings

Adjust the number of speakers to select and the number of utterances from each speaker and they will be randomly sampled from the corpus

In [9]:

embeds = []
labels = []
locations = []

# single speaker 
#num_speakers = 1
#num_utters = 1000

# multi speaker
num_speakers = 10
num_utters = 20


speaker_idxs = np.random.choice(range(len(speaker_paths)), num_speakers, replace=False )

for speaker_num, speaker_idx in enumerate(speaker_idxs):
    speaker_path = speaker_paths[speaker_idx]
    speakers_utter = speaker_to_utter[speaker_path]
    utter_idxs = np.random.randint(0, len(speakers_utter) , num_utters)
    for utter_idx in utter_idxs:
            embed_path = speaker_to_utter[speaker_path][utter_idx]
            embed = np.load(embed_path)
            embeds.append(embed)
            labels.append(str(speaker_num))
            locations.append(embed_path.replace(EMBED_PATH, '').replace('.npy','.wav'))
embeds = np.concatenate(embeds)

Load embeddings with UMAP

In [10]:

model = umap.UMAP()
projection = model.fit_transform(embeds)

Interactively charting the data in Bokeh

Set up various details for Bokeh to plot the data

You can use the regular Bokeh tools to explore the data, with reset setting it back to normal

Once you have started the local server (see cell below) you can then click on plotted points which will open a tab to play the audio for that point, enabling easy exploration of your corpus

File location in the tooltip is given relative to AUDIO_PATH

In [11]:

source_wav_stems = ColumnDataSource(
        data=dict(
            x = projection.T[0].tolist(),
            y = projection.T[1].tolist(),
            desc=locations,
            label=labels
        )
    )

hover = HoverTool(
        tooltips=[
            ("file", "@desc"),
            ("speaker", "@label"),
        ]
    )

# optionally consider adding these to the tooltips if you want additional detail
# for the coordinates: ("(x,y)", "($x, $y)"),
# for the index of the embedding / wav file: ("index", "$index"),

factors = list(set(labels))
pal_size = max(len(factors), 3)
pal = Category10[pal_size]

p = figure(plot_width=600, plot_height=400, tools=[hover,BoxZoomTool(), ResetTool(), TapTool()])


p.circle('x', 'y',  source=source_wav_stems, color=factor_cmap('label', palette=pal, factors=factors),)

url = "http://localhost:8000/@desc"
taptool = p.select(type=TapTool)
taptool.callback = OpenURL(url=url)

show(p)

[JavaScript output - execution disabled for security]

Local server to serve wav files from corpus

This is required so that when you click on a data point the hyperlink associated with it will be served the file locally.

There are other ways to serve this if you prefer and you can also run the commands manually on the command line

The server will continue to run until stopped. To stop it simply interupt the kernel (ie square button or under Kernel menu)

In [12]:

%cd $AUDIO_PATH
%pwd
!python -m http.server

/home/neil/main/Projects/LibriTTS/LibriTTS/train-clean-360
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...
^C

Keyboard interrupt received, exiting.

In [ ]: