PEIRA: Learning Predictive Encoders
through Inter-View Regressor Alignment

Michael Arbel1,* Basile Terver2,3,* Jean Ponce2,4

*Equal contribution. NeurIPS 2026

1Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK2Ecole Normale Supérieure / PSL3Inria Paris4New York University
Inria
On this page

Summary

Self-distillation methods such as SimSiam, BYOL, I-JEPA and DINO train a student network to predict a teacher network. They work well in practice, but they do not minimize a well-defined objective, so what they learn is poorly understood.

We analyze a JEPA variant in which a regularized linear regressor predicts the representation of each view from the other. Its non-collapsed stable equilibria align with the leading nonlinear canonical correlation subspaces, but collapsed equilibria can also be stable attractors.

PEIRA replaces the heuristic with an explicit objective: maximize the trace of the optimal linear regressor between views, with a penalty that keeps feature scale in check. Its only stable equilibria are nontrivial global minimizers that recover the same canonical subspaces, with the regularization $\lambda$ selecting the effective dimension. On ImageNet-1K and CIFAR-10, PEIRA is competitive with VICReg and LeJEPA baselines; with a 1024-dimensional projector it reaches 69.17% ImageNet-1K linear-probe accuracy after 100 epochs of ResNet-50 pretraining.

From self-distillation to PEIRA

Two encoders $U$ and $V$ map the two views $x$ and $y$ of a data point to $\mathbb{R}^k$. For fixed encoders, the ridge regressor that predicts each encoded view from the other has a closed form, a matrix-valued signal-to-noise ratio:

$$P_{U,V} = \Sigma_{U,V}\,(N_{U,V} + \lambda I)^{-1},$$ $$\begin{aligned} \Sigma_{U,V} &= \mathbb{E}\big[V(y)U(x)^{\top} + U(x)V(y)^{\top}\big], \\ N_{U,V} &= \mathbb{E}\big[U(x)U(x)^{\top} + V(y)V(y)^{\top}\big]. \end{aligned}$$

$\Sigma_{U,V}$ measures the signal shared across views and $N_{U,V}$ the variability within each view. PEIRA (Predictive Encoders through Inter-View Regressor Alignment) minimizes the following energy, which is the same as maximizing the trace of $P_{U,V}$ under a penalty on feature scale:

$$\mathcal{E}_{\text{PEIRA}}(U,V) = -\tfrac{1}{2}\operatorname{Tr}\big(P_{U,V}\big) + \tfrac{\lambda}{2}\big(\|U\|_2^2 + \|V\|_2^2\big).$$

The trace sums the contributions of all eigen-modes of the regressor, so maximizing it should favor representations that share the most information between views. The quadratic term controls feature scale, so the trace cannot grow by simply enlarging the features. Since $\operatorname{Tr}(P_{U,V})$ is the Frobenius inner product of $\Sigma_{U,V}$ and $(N_{U,V}+\lambda I)^{-1}$, minimizing $\mathcal{E}_{\text{PEIRA}}$ should also encourage the eigenbases of the signal and noise matrices to align.

Optimizing the objective

Backpropagating through mini-batch estimates of $P_{U,V}$ gives high-variance gradients. Instead, PEIRA uses an auxiliary loss with the same gradient as $\mathcal{E}_{\text{PEIRA}}$, in which the regressor $P^\star = P_{U,V}$ and $Q^\star = (N_{U,V}+\lambda I)^{-1}$ are held fixed:

$$\begin{aligned} \mathcal{L}_{\text{aux}}(U,V;P,Q) &= \tfrac{1}{2}\,\mathbb{E}\big[U(x)^{\top}Q\,(PU(x) - V(y))\big] \\ &\quad + \tfrac{1}{2}\,\mathbb{E}\big[V(y)^{\top}Q\,(PV(y) - U(x))\big] \\ &\quad + \tfrac{\lambda}{2}\,\mathbb{E}\big[\|U(x)\|^2 + \|V(y)\|^2\big], \end{aligned}$$

with expectations over pairs of views $(x, y)$. $P^\star$ and $Q^\star$ then only need to be estimated, which allows exponential moving averages (EMA) of the second moments. The gradient of $\mathcal{E}_{\text{PEIRA}}$ is $\tfrac{4}{\lambda}$-Lipschitz for $0<\lambda<1$, which supports gradient-based optimization and informs the choice of learning rate.

Encoders U_theta and V_psi produce features Phi_X and Phi_Y. A stop-gradient branch updates EMA statistics Sigma and N, from which a k by k ridge solve gives Q-hat and P-hat. These are held fixed in the auxiliary loss, whose gradient updates both encoders.
Fig. 1: Gradient estimator of Algorithm 1. Feature second moments, taken through a stop-gradient (sg), update EMA statistics that give closed-form estimates $\hat P^\star, \hat Q^\star$; these are held fixed in the auxiliary loss that updates the encoders.
Algorithm 1 SC-PEIRA (stochastic compositional PEIRA)

Input: encoder parameters $\theta,\psi$; $k\times k$ matrices $\Sigma, N$; regularization $\lambda$; EMA rate $\eta\in(0,1]$.

  1. Sample a mini-batch $\mathcal{B}$ of $B$ view pairs.
  2. $\Phi_X, \Phi_Y \leftarrow (U_\theta(x))_{\mathcal{B}},\ (V_\psi(y))_{\mathcal{B}}$
  3. $\Sigma \leftarrow (1-\eta)\Sigma + \tfrac{\eta}{B}(\Phi_X^{\top}\Phi_Y + \Phi_Y^{\top}\Phi_X)$
  4. $N \leftarrow (1-\eta)N + \tfrac{\eta}{B}(\Phi_X^{\top}\Phi_X + \Phi_Y^{\top}\Phi_Y)$
  5. $\hat{P}^\star, \hat{Q}^\star \leftarrow \Sigma(N+\lambda I)^{-1},\ (N+\lambda I)^{-1}$
  6. Update $\theta,\psi$ with $\nabla_{\theta,\psi}\widehat{\mathcal{L}}_{\text{aux}}(U_\theta, V_\psi; \hat P^\star, \hat Q^\star)$; repeat until converged.

The linear estimators come analytically from running feature statistics, which is efficient for moderate $k$. When the two views are symmetric, a single shared encoder can be used ($V_\psi = U_\theta$). In practice $\eta$ is annealed over training, favoring fresh statistics early and lower variance near convergence.

Canonical structure and stability

The analysis works in the space of all square-integrable encoders, without architectural restrictions. Nonlinear canonical correlation analysis (CCA) pairs functions $\psi_i(x)$ and $\xi_i(y)$ of the two views that are maximally correlated; their canonical correlations $c_1 \ge c_2 \ge \dots$ measure the strength of the shared signal along each direction. Both dynamics below converge to equilibria built from these directions, but they differ in which equilibria are stable (an equilibrium is stable when training returns to it after a small perturbation).

Self-distillation

Stop-gradient on the teacher view, as in SimSiam

  • Not a gradient flow, but admits a Lyapunov function, so it converges to an equilibrium.
  • Equilibria span at most $k$ canonical directions, each kept only if $c_i^2 \ge 4\lambda$, with amplitude $\tfrac{1}{2}\big(|c| \pm \sqrt{c^2 - 4\lambda}\big)$; stable ones take the $+$ sign: $$f(c,\lambda) = \tfrac{1}{2}\big(|c| + \sqrt{c^2 - 4\lambda}\big).$$
  • Non-collapsed stable equilibria span the leading canonical directions.
  • The totally collapsed solution $U = V = 0$ is a stable equilibrium, and so are some partially collapsed ones.

PEIRA

Gradient flow of $\mathcal{E}_{\text{PEIRA}}$

  • Converges to critical points with the same canonical structure, using the soft threshold $$g(c,\lambda) = \max\big(\sqrt{c} - \lambda,\ 0\big)^{1/2}.$$
  • The only stable equilibria are global minimizers, spanning the top-$r_{\max}$ canonical subspace.
  • Every other critical point is a strict saddle: no spurious local minima, and collapse is unstable.
  • $\lambda$ selects the effective dimension $r_{\max}$, capped by $k$.

At the stable PEIRA equilibria, with orthonormal vectors $\mathbf{q}_i \in \mathbb{R}^k$,

$$U^\star(x) = \tfrac{1}{\sqrt 2}\sum_{i=1}^{r_{\max}} g(c_i,\lambda)\,\psi_i(x)\,\mathbf{q}_i,$$ $$V^\star(y) = \tfrac{1}{\sqrt 2}\sum_{i=1}^{r_{\max}} g(c_i,\lambda)\,\xi_i(y)\,\mathbf{q}_i,$$ $$\min_{U,V}\ \mathcal{E}_{\text{PEIRA}}(U,V) = -\tfrac{1}{2}\sum_{i=1}^{r_{\max}}\big(\sqrt{c_i} - \lambda\big)^2.$$

Here $r_{\max}$ is the number of directions with $\sqrt{c_i} > \lambda$, up to $\min(k, R)$ with $R$ the number of canonical directions. The trivial representation does not reach this value, so it cannot be a stable point. A practical consequence: when the target dimension is unknown, choose $k$ conservatively large and let $\lambda$ set the number of active canonical directions. The condition $\lambda < \sqrt{c_1}$ is needed; otherwise every direction is filtered out.

These results hold at the population level, over all square-integrable encoders, and assume distinct canonical correlations. They do not directly apply to neural networks trained on finite data; there, they describe what optimizing the objective aims for rather than a guarantee.

Experimental results

We evaluate PEIRA by linear-probe classification on ImageNet-1K and CIFAR-10, and by transfer to eight downstream classification datasets. We compare against VICReg [1] and SIGReg/LeJEPA [2] with the same augmentations, pretraining budgets and linear probes. PEIRA uses $\lambda=0.1$ on ImageNet-1K and $\lambda=0.7$ on CIFAR-10.

Representation quality

On ImageNet-1K, PEIRA with $k=1024$ is the strongest method under this controlled protocol. On CIFAR-10 it outperforms VICReg and comes close to SIGReg, and on transfer it outperforms SIGReg on average and approaches VICReg.

MethodApproachImageNet-1KCIFAR-10Transfer
BYOL [4]self-distillation (EMA)69.3––
SimSiam [5]self-distillation (stop-gradient)67.9––
SimCLR [6]contrastive (negatives)68.5––
VICReg [1]optimization68.81 ± 0.0990.92 ± 0.1472.07
SIGReg [2]optimization66.26 ± 0.2892.34 ± 0.2168.27
PEIRA ($k=256$)optimization66.45 ± 0.1191.36 ± 0.1270.89 ± 0.18
PEIRA ($k=1024$)optimization69.17 ± 0.0691.68 ± 0.1871.09 ± 0.24
Table 1: Linear-probe top-1 accuracy (%). ResNet-50, 100 epochs on ImageNet-1K; ResNet-18, 1000 epochs on CIFAR-10. VICReg, SIGReg and PEIRA: mean ± sample standard deviation over 3 seeds, same augmentations, pretraining budget and probe, with method-specific projector and optimization hyperparameters; SIGReg uses a narrower projector (hidden width 2048 vs. 8192). BYOL and SimSiam are quoted from Table 4 of [1] and SimCLR from Table S3 of [3] (100-epoch ResNet-50), outside the controlled protocol. Transfer: mean linear-probe accuracy over the 8 LeJEPA datasets (VICReg and SIGReg: seed 1). $k$: PEIRA projector output dimension. Bold: best controlled result per column. ImageNet-1K checkpoints of the controlled rows (all seeds) are on Hugging Face.

Spectral alignment during training

The leading eigendirections of the noise matrix $N_{U,V}$ progressively align with those of the signal matrix $\Sigma_{U,V}$, while additional significant spectral modes emerge, consistent with the population analysis. The negative objective closely tracks linear-probe accuracy, suggesting its potential as a label-free indicator of representation quality.

Fig. 2: Dynamics of PEIRA on CIFAR-10. Signal and noise eigenvectors align (left), new spectral modes emerge (middle), and the negative objective tracks online linear-probe accuracy (right).

Robustness to the regularization

In an ablation sweep (online linear probe, 1 to 3 seeds per setting), mean ImageNet-1K accuracy varies by only 1.36 percentage points over $\lambda\in[0.025, 0.5]$. Smaller $\lambda$ accelerates the early growth of backbone effective rank, consistent with the spectral role of $\lambda$ predicted by the theory.

Three panels on ImageNet-1K: linear-probe top-1 versus lambda from 0.025 to 0.5; backbone effective rank versus epoch colored by lambda; online linear-probe top-1 versus the negative PEIRA objective across checkpoints.
Fig. 3: Study of PEIRA regularization on ImageNet-1K. ResNet-50, 100 epochs, ablation setting with 1 to 3 seeds per point. Mean online linear-probe accuracy varies by only 1.36 points over $\lambda\in[0.025, 0.5]$ (left); smaller $\lambda$ grows effective rank faster early in training, and final ranks differ by about 2% (middle); $-\mathcal{E}_{\text{PEIRA}}$ correlates with accuracy (right).

Resistance to collapse

We compare PEIRA with its self-distillation counterpart across initialization scales on CIFAR-10. At zero initialization scale, the encoders produce the zero representation, which is a stable attractor of the self-distillation dynamics.

CIFAR-10 accuracy, mean alignment and effective rank over 1000 epochs for PEIRA (solid) and self-distillation (dashed) at initialization scales 0.01, 0.1, 1 and 10.
Fig. 4: Initialization scale on CIFAR-10. In this controlled setting, self-distillation (dashed) is sensitive to initialization: its mean accuracy is 77.69% and 81.10% at $\sigma_0 = 0.1$ and $1$, but falls to 49.59% at $0.01$ and 46.00% at $10$. For $\sigma_0 \le 1$, PEIRA (solid) reaches higher accuracy, alignment and effective rank; at $\sigma_0 = 10$, this configuration of PEIRA stays near rank one.

Using the loss

The loss is a standalone PyTorch package. The EMA statistics and step counter live in the module state_dict, and batch second moments are summed across ranks for multi-GPU training. Call the loss exactly once per training step.

pip install peira-loss
import torch
from peira_loss import PEIRALoss

loss_fn = PEIRALoss(latent_dim=1024, T_max=total_steps)  # 69.17% row of Table 1
# CIFAR-10: PEIRALoss(latent_dim=..., T_max=..., reg=0.7)
z1, z2 = projector(encoder(view1)), projector(encoder(view2))  # [B, 1024] each
out = loss_fn(z1, z2)          # dict: loss, peira_loss, trace_P, cse_loss, reg_loss, peira_lr
out["loss"].backward()

Training code, configs and reproduction commands for every table and figure are in Basile-Terv/PEIRA. ResNet-50 backbones from Table 1 are on Hugging Face.

References

[1] Adrien Bardes, Jean Ponce and Yann LeCun. VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning. ICLR, 2022.

[2] Randall Balestriero and Yann LeCun. LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics. arXiv:2511.08544, 2025.

[3] Quentin Garrido, Randall Balestriero, Laurent Najman and Yann LeCun. RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank. ICML, 2023.

[4] Jean-Bastien Grill et al. Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning. NeurIPS, 2020.

[5] Xinlei Chen and Kaiming He. Exploring Simple Siamese Representation Learning. CVPR, 2021.

[6] Ting Chen, Simon Kornblith, Mohammad Norouzi and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. ICML, 2020.

BibTeX

@inproceedings{arbel2026peira,
  title     = {{PEIRA}: Learning Predictive Encoders through Inter-View Regressor Alignment},
  author    = {Arbel, Michael and Terver, Basile and Ponce, Jean},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026},
  eprint    = {2605.17671},
  archivePrefix = {arXiv},
}