1. Problem Formulation

Voice Conversion (VC) can be formalized as:

[ X_{src} \rightarrow (C, S_{tgt}) \rightarrow \hat{X}_{tgt} ]

Where:

  • ( C ): content (linguistic information)
  • ( S_{tgt} ): target speaker representation
  • ( \hat{X}_{tgt} ): converted speech

The core difficulty is:

  • Learning ( C ) such that it is speaker-invariant
  • Learning ( S ) such that it is content-invariant

Most prior methods fail due to entangled latent representations.

2. CTVC Overview

CTVC explicitly enforces disentanglement via two orthogonal constraints:

Representation Constraint
Content (C) Contrastive + adversarial (remove speaker info)
Speaker (S) Time-invariant retrieval via mutual information

The system (Fig. 1, page 2) consists of:

  • Content encoder (E_c)
  • Speaker encoder (E_s)
  • Decoder (D)
  • Speaker classifier (with GRL)
  • Compression module (frame → phoneme)

3. Content Representation Learning

3.1 Frame-Level Representation

Given mel spectrogram: [ X = (x_1, x_2, …, x_T) ]

Content encoder: [ C_X = E_c(X) = (c_1, c_2, …, c_T) ]

Each (c_i) is a frame-level embedding.

3.2 Phoneme-Aware Supervision via Alignment

Using forced alignment (MFA), each frame is assigned a phoneme label.

This enables:

  • Constructing positive pairs: frames within same phoneme
  • Constructing negative pairs: frames across phoneme boundaries

3.3 Contrastive Similarity Loss

Similarity: [ G(c_i, c_j) = \frac{c_i^T c_j}{|c_i||c_j|} ]

Loss: [ L_{sim} = \sum_{i,j} (-1)^h G(E_c(x_i), E_c(x_j)) ]

Where:

  • (h = 1): same phoneme → maximize similarity
  • (h = -1): different phonemes → minimize similarity

Interpretation

This is not a standard InfoNCE loss — instead:

  • It directly shapes embedding geometry
  • Encourages phoneme clusters in latent space

3.4 Compression Module (Key Design)

From Fig. 1 (page 2):

  • Frame-level features → aggregated into phoneme-level features
  • Based on similarity + duration boundaries

Effect:

  • Removes intra-phoneme redundancy
  • Improves alignment between latent space and linguistic units

This is crucial because:

  • Speech redundancy exists at frame level
  • Linguistic information is naturally phoneme-level

3.5 Adversarial Speaker Removal

A speaker classifier is trained on (C_X), while the encoder is trained to fool it via GRL:

[ L_{adv-cls} = -\sum_k I(Spk_u = k)\log p_k ]

Mechanism:

  • Classifier: minimizes classification loss
  • Encoder: maximizes it (via gradient reversal)

Result:

  • (C_X) becomes speaker-invariant
  • Prevents leakage of timbre into content space

4. Speaker Representation Learning

4.1 Problem with Standard Speaker Encoders

Typical CNN + pooling:

  • Capture global features
  • But still encode time-dependent artifacts (phoneme leakage)

4.2 Time-Invariant Assumption

Key assumption:

Speaker identity is invariant across time segments

Meaning: [ S(x_{segment1}) \approx S(x_{segment2}) ]

4.3 Segment Sampling Strategy (Fig. 2, page 3)

Two strategies:

(a) Two short segments

  • Sample from first and second halves
  • Produce embeddings (s_1, s_2)

(b) Whole vs large segment

  • Full utterance (s_x)
  • Partial long segment (s_3)

4.4 Mutual Information Maximization

MI definition: [ I(u,v) = D_{KL}(P(u,v) || P(u)P(v)) ]

Optimized via InfoNCE:

[ I_{NCE}(u,v) = \mathbb{E} \left[ \log \frac{e^{f(u_i,v_i)}}{\sum_j e^{f(u_j,v_j)}} \right] ]

Score function: [ f(u,v) = \exp(h_i^T W v_i) ]

4.5 Style Loss

[ L_s = \sum I(s_1, sg(s_2)) + I(s_2, sg(s_1))

  • I(s_x, sg(s_3)) + I(s_3, sg(s_x)) ]

Key design:

  • Stop-gradient (sg) prevents collapse
  • Symmetric constraints stabilize learning

Interpretation:

  • Forces embeddings from different segments of same speaker to match
  • Suppresses time-varying information (phonemes, prosody variation)

5. Decoder and Reconstruction

Final representation: [ \hat{X} = D(C, S) ]

Reconstruction loss: [ L_{recon} = \sum ||x_n - \hat{x}_n||^2 ]

6. Total Objective

[ L = L_{recon} + \alpha L_{sim} + \beta L_s + \lambda L_{adv-cls} ]

Each term controls:

Term Role
(L_{recon}) Speech fidelity
(L_{sim}) Linguistic purity
(L_s) Speaker consistency
(L_{adv-cls}) Remove speaker leakage

7. Experimental Insights

7.1 Performance (Table 1, page 4)

CTVC improves:

  • Lower MCD → better spectral match
  • Higher MOS → better perceptual quality
  • Higher VSS → better speaker similarity

Notably:

  • Works well in one-shot VC, meaning good generalization

7.2 Ablation (Table 2)

Removed Component Effect
GRL (no adversarial) Speaker info leaks into content
TIR Worse speaker similarity
Contrastive loss Poor content representation

Key takeaway:

  • Each component is necessary and complementary

8. Key Technical Insights

8.1 Disentanglement Strategy

Instead of implicit separation, CTVC enforces:

  • Content purity via contrastive geometry + adversarial loss
  • Speaker purity via time-invariant constraints

This is stronger than prior work relying on:

  • Bottlenecks
  • Pretrained embeddings

8.2 Why It Works

  • Phoneme-level contrastive learning aligns latent space with linguistic structure
  • Time-invariant retrieval removes temporal noise from speaker embeddings
  • Adversarial training ensures orthogonality between (C) and (S)

8.3 Conceptual View

You can think of CTVC as enforcing:

[ C \perp S ]

via:

  • Contrastive separation (content space structure)
  • MI-based consistency (speaker space structure)
  • Adversarial orthogonalization

9. Limitations (Implicit)

Not explicitly discussed, but inferred:

  • Requires forced alignment (MFA dependency)
  • Pairwise contrastive computation is expensive
  • Assumes phoneme boundaries are accurate

10. Final Takeaway

CTVC’s main contribution is explicit, multi-level disentanglement enforcement:

  • Local (frame/phoneme) → content via contrastive loss
  • Global (utterance) → speaker via MI-based retrieval
  • Cross-space → adversarial separation

This combination leads to significantly improved voice conversion quality without relying heavily on external pretrained speaker models.