VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning

arXiv

official implementation

@inproceedings{
bardes2022vicreg,
title={VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning},
author={Adrien Bardes and Jean Ponce and Yann LeCun},
booktitle={International Conference on Learning Representations},
year={2022},
url={https://openreview.net/forum?id=xm6YD62D1Ub}
}

🧠 VICReg — Core Idea

VICReg is a self-supervised learning method for training representations by comparing different views of the same input.

Its key contribution is:

A simple objective that prevents collapse explicitly, without needing contrastive negatives or architectural tricks.

Instead of relying on heuristics (like stop-gradient or momentum encoders), it defines three direct constraints on the embedding space.

⸻

🔑 The Three Terms (this is the whole method)

Given two embeddings Z and Z’ from two views of the same data:

  1. Invariance

Make representations of the same input similar

s(Z, Z’) = \frac{1}{n} \sum_i |z_i - z’_i|^2

  • Pulls matching views together
  • This is the only term that aligns pairs

⸻

  1. Variance (collapse prevention)

Ensure each dimension has sufficient spread across the batch

v(Z) = \frac{1}{d} \sum_j \max(0, \gamma - \text{std}(z^j))

  • Prevents all embeddings becoming constant
  • Forces each dimension to carry signal

This is the paper’s main innovation.

⸻

  1. Covariance (redundancy reduction)

Make different dimensions independent

c(Z) = \frac{1}{d} \sum_{i \neq j} \text{Cov}(Z)_{i,j}^2

  • Pushes off-diagonal covariance toward zero
  • Encourages each dimension to encode different information

⸻

Final loss

\mathcal{L} = \lambda s + \mu (v(Z) + v(Z’)) + \nu (c(Z) + c(Z’))

⸻

🧩 Intuition (what each term is doing)

From the paper’s description :

  • Invariance → alignment (same input → same representation)
  • Variance → prevents trivial collapse (everything identical)
  • Covariance → prevents informational collapse (dimensions redundant)

⸻

⚠️ What “collapse” means here

Without constraints, the model can minimize invariance by doing:

z = constant vector for all inputs

That perfectly satisfies “same inputs match” — but carries no information.

VICReg prevents this in two ways:

  • variance term → stops shrinking to a constant
  • covariance term → stops dimensions from collapsing together

⸻

🧠 Architectural structure

From the diagram on page 2:

  • input → encoder → representation
  • representation → expander → embedding
  • loss applied on embeddings

The expander:

  • increases dimensionality
  • helps decorrelation work effectively

After training, it is discarded — only the encoder is used.

⸻

🔥 What makes VICReg different

Compared to other SSL methods:

No contrastive negatives

  • No need to push unrelated samples apart

No special tricks

  • No stop-gradient
  • No momentum encoder
  • No normalization tricks required

Works with asymmetric / multimodal setups

  • Two branches can:
    • have different architectures
    • have different inputs

This is explicitly highlighted as a key advantage .

⸻

🧠 Deeper interpretation

VICReg is enforcing three properties on the representation space:

⸻

  1. Consistency (invariance)

Same underlying input → same point

⸻

  1. Spread (variance)

Data should occupy the space, not collapse

⸻

  1. Factorization (covariance)

Different dimensions should encode different factors

⸻

🔑 Why covariance matters (subtle but important)

Without it:

  • multiple dimensions can encode the same signal
  • representation becomes redundant

With it:

the model is pushed toward factorized structure

⸻

📊 Empirical results (briefly)

  • Matches state-of-the-art SSL performance
  • Stable training
  • Works well in multi-modal settings (image + text)

Notably:

performs comparably to methods like Barlow Twins, but with simpler constraints 

⸻

🧠 One-line summary

VICReg learns representations by enforcing alignment + spread + independence directly in the embedding space, instead of relying on contrast or architectural tricks.

⸻

🎯 The most important takeaway

If you strip everything away, VICReg is saying:

A good representation must:

  • map similar things together
  • use the full space
  • not waste dimensions on redundant information