PEIRA · From self-distillation to PEIRA
What does self-distillation optimize?
Self-distillation methods train a student to predict a teacher. They work well in practice, but they do not minimize a well-defined objective, and their representations can collapse.
0:00 / 0:17
Student branchTeacher branch