PEIRA · From self-distillation to PEIRA

What does self-distillation optimize?

Self-distillation methods train a student to predict a teacher. They work well in practice, but they do not minimize a well-defined objective, and their representations can collapse.

Self-distillation: a student predicts a teacher Left: view x goes through the student encoder and a predictor to give a prediction; view y goes through the teacher encoder, updated by EMA or stop-gradient, to give a target; the prediction is matched to the target. Right: three statements. They work well in practice. But they minimize no well-defined objective. And representations can collapse. Below, a schematic illustration in which a cloud of points contracts to a single point, labelled: collapse maps every input to one point. x Student Predictor Prediction y Teacher Target EMA or stop-gradient match They work well in practice. But they minimize no well-defined objective. And representations can collapse. Schematic: collapse maps every input to one point

0:00 / 0:17
Student branchTeacher branch