Qwen3-TTS for Swiss German: Dialect Adaptation via Talker Layer Fine-Tuning
How we adapted Qwen3-TTS to Swiss German by training only the last 8 talker layers and creating a dedicated speaker per dialect — with audio examples from our Züridüütsch model.
Swiss German text-to-speech has a straightforward problem: there is essentially no commercially usable training data. Five million people speak Swiss German, yet no labeled TTS dataset exists that you can freely train on. Voice actors are understandably reluctant to record audio for models that might replace them.
This forces a targeted fine-tuning approach. Rather than training a model from scratch, we adapted Qwen3-TTS — a strong open-source TTS model from Alibaba’s Qwen team — to produce natural Swiss German speech with limited data.
Key Takeaways
- We trained only the last 8 talker layers of Qwen3-TTS, leaving all earlier layers frozen.
- We created a dedicated speaker per dialect to capture dialect-specific acoustic characteristics.
- As a starting point for the dedicated speakers, we copied the most similar existing speaker from the base model to the dialect-specific speaker.
- Most expressive-speech instructions remain usable after fine-tuning, though some styles are lost.
- Results shown here are for Züridüütsch — the Zürich dialect — since that’s what I speak.
The Architecture: What Are Talker Layers?
Qwen3-TTS uses a talker stack to turn language representations into speech tokens.
Our approach: freeze all but the last 8 talker layers, then train those last layers on the Swiss German data we have. This targets pronunciation and prosody while retaining the base model’s broader capabilities.
Ongoing work: this selective approach works well with limited data, but full fine-tuning may make the model more robust overall. We are still evaluating that trade-off between dialect vs. instructional expressiveness.
One Speaker Per Dialect
Rather than trying to cover all Swiss German dialects in a single model, we created a dedicated speaker embedding per dialect. Each speaker is optimized for a specific regional variant. The model generates speech conditioned on the selected speaker.
The examples below use Züridüütsch (the Zürich dialect).
Results: Before and After
The base Qwen3-TTS model has no systematic exposure to Swiss German. When given Swiss German text, it maps to Standard German phonetics — the output is intelligible, but clearly off to native speakers.
Standard German phonetics applied to Swiss German text.
Natural Züridüütsch prosody and vowel rendering, mostly.
Expressive Speech: Mostly Preserved After Fine-Tuning
Qwen3-TTS accepts natural-language style instructions, and many still work with the fine-tuned Züridüütsch speaker. The sentence below mixes English brand names, Swiss place names, and Swiss German syntax, making it a useful test of both dialect handling and expressive control. This transfer is not perfect: the fear prompt at the end is included as a failure case.
The successful examples show that the last-8-layer approach retains useful expressive control, while the fear example shows that individual styles can still degrade after fine-tuning.
What’s Next
- The speech is not perfect yet, but selectively freezing the model makes dialect adaptation feasible with limited data. Next, I want to improve consistency across expressive styles — especially prompts such as “speak with fear.” Generally, I have also observed that the voice is not consistent across different expressive prompts, which I have observed also in other TTS models.
Presented at the Basel Data Science & AI Meetup, April 2026. Work done at FHNW Institute for Data Science.