How to load a custom AI model for Sherpa TTS

Quick guide on how to load custom AI models to be used with the Sherpa Text To Speech engine in Noise-o-matic!

Introduction

The Sherpa TTS engine is an open source software released with Apache 2.0 license. However, while the software itself is open source, not all the voice models it supports (or the datasets they have been trained on) might not be distributed with a license that allows redistribution.

In this guide, you'll see how to manually download and install additional models and some pointers on how to create your own.

Noise-o-matic is a commercial product, and thus, all models that are redistributed by the Sherpa project under a non-commercial license can not be directly shipped with Noise-o-matic. Noise-o-matic only ships those models that are made with public datasets and that are distributed with proper rights for commercial use in third party products such as Noise-o-matic.

If you're the author of a pre-trained model shipped with Noise-o-matic and would like to see your model not shipped with Noise-o-matic despite being allowed by the license, or if you encounter an error in our selection, please get in touch at [e-mail]

Finding your models directory

When you use the "download models" in Noise-o-matic, a curated selection of pre-trained models that allow redistribution is presented to you which will allow you to download them.

Noise-o-matic will automatically download and extract the models in your Steam directory, for example:

D:\Steam\steamapps\common\Noise-o-matic\ttsmodels\

Each new model requires a new directory, for example, downloading the "Alba" English voice, will create the

D:\Steam\steamapps\common\Noise-o-matic\ttsmodels\vits-piper-en_GB-alba-medium

directory. Likewise, every new models you want to use with Noise-o-matic need to be placed there.

The "ttsmodels" will be created for you upon the first voice model you download. If you haven't downloaded any model yet, go ahead and create it manually.

To proceed to the next step, locate your Steam\steamapps\common\Noise-o-matic\ttsmodels\

Loading a new model into Noise-o-matic

Training your own model can be done, but it is out of scope for this guide as it is for advanced users only. If you want to train your own model, you can find the icefall documentation here: https://k2-fsa.github.io/icefall/

You can train your own model or you can pick one from the Sherpa TTS project repository here https://github.com/k2-fsa/sherpa-onnx/releases/tag/tts-models to go ahead with this guide.

Once your trained model is ready (or if you downloaded a model, make sure that there is a .onnx file along with a espeak-ng-data directory - this means that the model is compatible with Noise-o-matic) simply put all the model files in the desired directory, so it would look like this

D:\Steam\steamapps\common\Noise-o-matic\ttsmodels\mynewmodel\
D:\Steam\steamapps\common\Noise-o-matic\ttsmodels\mynewmodel\model.onnx
D:\Steam\steamapps\common\Noise-o-matic\ttsmodels\mynewmodel\MODEL_CARD
D:\Steam\steamapps\common\Noise-o-matic\ttsmodels\mynewmodel\espeak-ng-data\
D:\Steam\steamapps\common\Noise-o-matic\ttsmodels\mynewmodel\espeak-ng-data\....
etc.

In short, simply extract the archive in the ttsmodels directory.

Restart Noise-o-matic.

If everything went well, you should be able to see your custom voice in the TTS panel of Noise-o-matic and you can start using it normally!

Keep in mind you are entirely responsible for what you train and how you use every trained model, despite the license they are distributed with.

Make some noise!

References

Sherpa TTS source code on github: https://github.com/k2-fsa/sherpa-onnx
Sherpa TTS project page: https://k2-fsa.github.io/sherpa/onnx/tts/index.html
Sherpa TTS Models downloads: https://github.com/k2-fsa/sherpa-onnx/releases/tag/tts-models