PEIRA · Canonical structure and stability
Collapse becomes unstable
In the paper's toy experiment, both methods start from a tiny initialization. Self-distillation stays collapsed; for PEIRA, collapse is unstable and training recovers the target directions.
Real runs (paper Fig. G.6 setup), init scale ρ0 = 10−6, linear encoders, 400,000 updates. Each line is one of the top 6 learned directions: the 4 coloured lines are the target directions (target value 1), the 2 grey lines are non-target directions (target value 0, so they should stay at 0).
0:00 / 0:16