PEIRA · Canonical structure and stability

Collapse becomes unstable

In the paper's toy experiment, both methods start from a tiny initialization. Self-distillation stays collapsed; for PEIRA, collapse is unstable and training recovers the target directions.

Spectral ratios of self-distillation and PEIRA during training from a tiny initialization Two panels plot the spectral ratio (learned over target) of the top six directions against training step on a log scale, from initialization scale 10 to the minus 6. Four directions are targets with ratio 1 and two are non-targets with ratio 0. Self-distillation stays at 0 for all directions: 0 of 4 target directions reached. PEIRA's four target directions rise to 1 one after another while the two non-target directions stay at 0: 4 of 4 reached. A table below gives the number of target directions recovered at the end of training for initialization scales 10 to the minus 2 through 10 to the minus 6: PEIRA 4, 4, 4, 4, 4; self-distillation 4, 4, 3, 1, 0.

Real runs (paper Fig. G.6 setup), init scale ρ0 = 10−6, linear encoders, 400,000 updates. Each line is one of the top 6 learned directions: the 4 coloured lines are the target directions (target value 1), the 2 grey lines are non-target directions (target value 0, so they should stay at 0).

0:00 / 0:16