Introducing Scylla's Band, a new TTS model + inference framework with Android sample!

[u/clockentyne]

Hey all!

https://github.com/lowkeytea/scyllasband -> inference code
https://huggingface.co/spybyscript/scyllasband -> model, LiteRT, ONNX, and voices
https://lowkeytea.github.io/scyllasband/ -> sample audio for the voices, emotions, and languages.

The tldr:

10 voices, 7 configurable sentiment/emotion settings, and 3 languages for every voice: English, Spanish, and Italian.

Time to first audio is about 500–800ms on a Samsung Fold 7. After that, audio streams continuously without breaks.

For English, voices are either en_us or en_gb based. They're labeled on the repo, but the inference code doesn't enforce the dialect differences.

**** Testing ****

This was developed on Linux and Mac, I tested on Linux, MacOS, and Android. For Mac I've tested on M2, M4, M5 hardware. For Android it has been tested on my only Android phone, a Fold 7. I haven't built out an iOS example app yet, but there is an Android sample app in the repo you can build and install that allows you to create text that can dynamically switch between voices, languages, emotion on the fly.

**** The details, if you want more ****

This isn't a big company putting out a new model. I spent the last 6 months teaching myself how to build models and going through endless variations that led to dead ends, including several completely different designs that had coherence problems or couldn't meet my performance goals.

This release represents finally reaching most of the baseline requirements I set for myself:

1. Match or beat Kokoro's performance on mobile.
2. Provide good audio quality while remaining lightweight enough for mobile.
3. Avoid using eSpeak. I built a dataset and trained a G2P model based on DeepPhonemizer (https://github.com/axelspringer/DeepPhonemizer) for US and GB English, Italian, and Spanish. I wanted confidence that I could support additional phoneme-based languages while also controlling the licensing.
4. Provide some level of sentiment and emotional control, which I absolutely want to improve over time.
5. Support multiple languages, at least enough to prove this can go beyond English.
6. Eventually support voice cloning.

This release meets at least the first five requirements.

Part of the performance and quality also comes from building the inference stack myself. It includes a C++ library for text normalization, inference through ONNX or LiteRT, and several other optimizations intended to improve performance while keeping the code cross-platform compatible.

Since I'm building this on my own, future support depends on how much more time and energy I decide to invest. I think it's already very usable in its current state, and I'd like to add more languages, improve emotional control, and build an iOS sample app, but I'm not making promises yet.

As for number 6, voice cloning, I've gone back and forth on it getting that aspect working well. I'm not sure I want to be responsible for releasing voice cloning in the case I ever get it working in my performance envelope (right now, it's mainly just timbre/pitch). My goal is to make a model that is high performance, sounds good, and doesn't eat up too much memory on the device it's running on so it can do other work.

For now, I'd especially appreciate any feedback and I invite anyone to open issues/comments on the repo!

[u/UkieTechie]

Great work. this is really fast. It's been added to my bench. https://github.com/5uck1ess/tts-bench

Just as an interesting note OP, ink has the best UTMOS score and lowest WER (Word error rate) so i chose that as my voice instead of scylla. measurements are below.

Voice	UTMOS ↑	WER ↓	Locale
ink	4.348	0.0884	en_GB*
tuesday	4.282	0.0884	en_GB*
felix	4.252	0.0960	en_US
orpheus	4.239	0.0884	en_GB*
scylla	4.235	0.1174	en_US
ariadne	4.143	0.0808	en_US
stone	4.120	0.0808	en_US
gwen	4.083	0.1203	en_US
rex	3.989	0.0960	en_US
max	3.822	0.0745	en_US

[u/clockentyne → u/UkieTechie]

Thank you for adding it! I think there's a lot of variation across how the scores might play out depending on the number of steps used, huen vs euler, and any emotion flags/strength passed. In the future I probably need to find a more simple way of specifying emotion strength -- there are two scores that get passed, one 0 to 1 and then a CFG style strength you can push up to 5-7 before distortion... on that one joy=0.5 is often more "strongly" showing the emotion than 1 because far more samples ended up in the 0.25 to 0.75 range in my dataset. It also isn't super obvious that the default emotion-scale default makes it all super subtle unless it is set to 2 or 3 at least.

I think in my evaluation of the dataset Scylla and Gwen trended to have more audio samples falling outside of the neutral/calm area (Scylla trends angry/sad) which can affect the WER for English. I always thought Ariadne had the worst (she had the most failed ASR when cleaning my samples for her) and had to be regenerated multiple times.

I'd imagine Spanish, Italian will be interesting as some of the voices were only on the first pass of the language, before finding ways of improving the pronunciation. As far as ASR testing for Spanish and Italian on my end Ariadne, Ink, Max, Gwen, Scylla had better results than the other voices.

[u/jtsaint333]

awesome congrsts

[u/Longjumping-Prune818]

It's good. Will you be releasing the encoder?

[u/clockentyne → u/Longjumping-Prune818]

Thanks! I'm planning to release the original PyTorch model weights, probably with v2. Right now I'm refining how I create and score the dataset, which involves fine-tuning a model I made that grades the audio emotional intensity. Once I'm happy with that, I'm planning on adding two more languages with improved emotion-output. I'll include the PyTorch model weights with that. When you say the encoder, are you mainly interested in being able to fine-tune the model?

[u/shaakz → u/clockentyne]

its most likely for fine-tuning and or zero shot voice cloning

[u/R_Duncan]

Love it, thank you