Tortoise TTS inference (#2547)

* initial commit * Tortoise inference * revert path change * style fix * remove accidental remove * style fixes * style fixes * removed unwanted assests and deps * remove changes * remove cvvp * style fix black * added tortoise config and updated config and args, refactoring the code * added tortoise to api * Pull mel_norm from url * Use TTS cleaners * Let download model files * add ability to pass tortoise presets through coqui api * fix tests * fix style and tests * fix tts commandline for tortoise * Add config.json to tortoise * Use kwargs * Use regular model api for loading tortoise * Add load from dir to synthesizer * Fix Tortoise floats * Use model_dir when there are multiple urls * Use `synthesize` when exists * lint fixes and resolve preset bug * resolve a download bug and update model link * fix json * do tortoise inference from voice dir * fix * fix test * fix speaker id and remove assests * update inference_tests.yml * replace inference_test.yml * fix extra dir as None * fix tests * remove space * Reformat docstring * Add docs * Update docs * lint fixes --------- Co-authored-by: Eren Gölge <egolge@coqui.ai> Co-authored-by: Eren Gölge <erogol@hotmail.com>
2026-09-09 11:16:00 +08:00 · 2023-05-16 00:58:21 +02:00
parent 0b6b957e76
commit a3d5801c44
31 changed files with 8298 additions and 35 deletions
@@ -51,6 +51,7 @@
    models/forward_tts.md
    models/tacotron1-2.md
    models/overflow.md
+    model/tortoise.md

 .. toctree::
    :maxdepth: 2
@@ -0,0 +1,94 @@
+# Tortoise 🐢
+Tortoise is a very expressive TTS system with impressive voice cloning capabilities. It is based on an GPT like autogressive acoustic model that converts input
+text to discritized acouistic tokens, a diffusion model that converts these tokens to melspeectrogram frames and a Univnet vocoder to convert the spectrograms to
+the final audio signal. The important downside is that Tortoise is very slow compared to the parallel TTS models like VITS. 
+
+Big thanks to 👑[@manmay-nakhashi](https://github.com/manmay-nakhashi) who helped us implement Tortoise in 🐸TTS.
+
+Example use:
+
+```python
+from TTS.tts.configs.tortoise_config import TortoiseConfig
+from TTS.tts.models.tortoise import Tortoise
+
+config = TortoiseConfig()
+model = Tortoise.inif_from_config(config)
+model.load_checkpoint(config, checkpoint_dir="paths/to/models_dir/", eval=True)
+
+# with random speaker
+output_dict = model.synthesize(text, config, speaker_id="random", extra_voice_dirs=None, **kwargs)
+
+# cloning a speaker
+output_dict = model.synthesize(text, config, speaker_id="speaker_n", extra_voice_dirs="path/to/speaker_n/", **kwargs)
+```
+
+Using 🐸TTS API:
+
+```python
+from TTS.api import TTS
+tts = TTS("tts_models/en/multi-dataset/tortoise-v2")
+
+# cloning `lj` voice from `TTS/tts/utils/assets/tortoise/voices/lj`
+# with custom inference settings overriding defaults. 
+tts.tts_to_file(text="Hello, my name is Manmay , how are you?", 
+                file_path="output.wav",
+                voice_dir="TTS/tts/utils/assets/tortoise/voices/",
+                speaker="lj",
+                num_autoregressive_samples=1,
+                diffusion_iterations=10)
+
+# Using presets with the same voice
+tts.tts_to_file(text="Hello, my name is Manmay , how are you?", 
+                file_path="output.wav",
+                voice_dir="TTS/tts/utils/assets/tortoise/voices/",
+                speaker="lj",
+                preset="ultra_fast")
+
+# Random voice generation
+tts.tts_to_file(text="Hello, my name is Manmay , how are you?", 
+                file_path="output.wav")
+```
+
+Using 🐸TTS Command line:
+
+```console
+# cloning the `lj` voice
+tts --model_name  tts_models/en/multi-dataset/tortoise-v2 \
+--text "This is an example." \ 
+--out_path "/data/speech_synth/coqui-tts/TTS/tests/outputs/output.wav" \
+--voice_dir TTS/tts/utils/assets/tortoise/voices/ \
+--speaker_idx "lj" \
+--progress_bar True
+
+# Random voice generation
+tts --model_name  tts_models/en/multi-dataset/tortoise-v2 \
+--text "This is an example." \
+--out_path "/data/speech_synth/coqui-tts/TTS/tests/outputs/output.wav" \
+--progress_bar True
+```
+
+
+## Important resources & papers
+- Original Repo: https://github.com/neonbjb/tortoise-tts
+- Faster implementation: https://github.com/152334H/tortoise-tts-fast
+- Univnet: https://arxiv.org/abs/2106.07889
+- Latent Diffusion:https://arxiv.org/abs/2112.10752
+- DALL-E: https://arxiv.org/abs/2102.12092
+
+## TortoiseConfig
+```{eval-rst}
+.. autoclass:: TTS.tts.configs.tortoise_config.TortoiseConfig
+    :members:
+```
+
+## TortoiseArgs
+```{eval-rst}
+.. autoclass:: TTS.tts.models.tortoise.TortoiseArgs
+    :members:
+```
+
+## Tortoise Model
+```{eval-rst}
+.. autoclass:: TTS.tts.models.tortoise.Tortoise
+    :members:
+```