arXiv

code

@misc{zhu2025zipvoicefasthighqualityzeroshot,
      title={ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matching}, 
      author={Han Zhu and Wei Kang and Zengwei Yao and Liyong Guo and Fangjun Kuang and Zhaoqing Li and Weiji Zhuang and Long Lin and Daniel Povey},
      year={2025},
      eprint={2506.13053},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2506.13053}, 
}

Main Idea

The paper introduces ZipVoice, a zero-shot text-to-speech (TTS) system designed to generate high-quality speech in arbitrary voices while being much faster and smaller than existing state-of-the-art models.

The core problem the authors address is:

  • Modern zero-shot TTS models sound good, but:
    • they are huge (hundreds of millions to billions of parameters),
    • require many iterative sampling steps,
    • and therefore have slow inference and high deployment cost.

ZipVoice aims to preserve quality while dramatically improving efficiency. 

⸻

Key Contributions

The paper proposes three main innovations:

  1. Zipformer-based architecture

Instead of using large Transformer or Diffusion Transformer (DiT) backbones, ZipVoice uses Zipformer, an architecture originally designed for speech recognition.

Benefits:

  • fewer parameters,
  • better efficiency,
  • strong modeling capability.

Important Zipformer features:

  • U-Net-like multi-resolution structure,
  • convolution modules for local speech patterns,
  • attention-weight reuse for parameter efficiency. 

⸻

  1. Simple speech-text alignment via “average upsampling”

A major challenge in non-autoregressive TTS is aligning text tokens with speech frames.

Previous models:

  • used explicit duration prediction,
  • or relied on implicit alignment with filler tokens.

ZipVoice introduces a simpler method:

  • assume each text token has roughly equal duration,
  • repeat (“upsample”) token embeddings uniformly across speech frames.

This:

  • avoids complicated duration models,
  • keeps the system simple,
  • greatly improves intelligibility. 

The authors show this technique is surprisingly effective despite being theoretically simplistic.

⸻

  1. Flow distillation for faster inference

Flow-matching TTS models still need multiple sampling steps.

ZipVoice introduces flow distillation, where:

  • a teacher model performs expensive multi-step generation,
  • a student model learns to imitate it in fewer steps.

This allows:

  • fewer sampling steps (NFEs),
  • removal of classifier-free guidance overhead,
  • dramatically faster generation. 

⸻

Model Architecture

ZipVoice contains:

  • a Zipformer text encoder,
  • a Zipformer vector field estimator,
  • a flow-matching generation framework.

During inference:

  1. input text + speaker prompt audio are provided,
  2. the model estimates target speech duration,
  3. noisy speech is iteratively refined into final speech,
  4. a vocoder converts features into waveform audio. 

⸻

Experimental Results

The model was trained on:

  • 100k hours multilingual Emilia dataset,
  • and smaller LibriTTS experiments.

Performance

Compared with systems like:

  • F5-TTS,
  • E2-TTS,
  • MaskGCT,
  • CosyVoice,
  • Spark-TTS,

ZipVoice achieves:

  • competitive speech quality,
  • strong intelligibility,
  • good speaker similarity,
  • with far fewer parameters. 

⸻

Speed Improvements

One of the strongest results:

Model Params Speed F5-TTS 336M baseline ZipVoice 123M much faster ZipVoice-Distill (4 steps) 123M up to 30× faster

The distilled version:

  • is ~24× faster on GPU,
  • ~33× faster on CPU,
  • while maintaining similar quality. 

This is a major contribution because efficient zero-shot TTS is important for:

  • mobile devices,
  • real-time assistants,
  • edge deployment,
  • low-cost serving.

⸻

Ablation Findings

The paper also demonstrates:

Average upsampling is critical

Removing it causes:

  • WER (word error rate) to explode,
  • severe intelligibility degradation. 

Zipformer components matter

Convolutions, downsampling, and bypass connections all significantly improve quality. 

Their distillation method outperforms alternatives

Compared with:

  • consistency distillation,
  • ReFlow,

their approach gives better quality at low sampling steps. 

⸻

Overall Takeaway

ZipVoice shows that:

  • zero-shot TTS does not require giant diffusion transformers,
  • careful architectural design can dramatically improve efficiency,
  • flow matching + Zipformer + lightweight alignment can achieve near-SOTA quality with far lower compute.

The paper’s main significance is practical: it moves zero-shot TTS closer to real-time, deployable speech synthesis without sacrificing much quality.