A significant technical pain point in model compression is that Knowledge Distillation (KD) does not always follow the "better teacher, better student" logic. In practice, a significant performance gap between a large teacher and a lightweight student often leads to "capacity mismatch," where an exceptionally accurate teacher fails to guide a smaller model effectively. The root cause lies in our limited understanding of what "dark knowledge" truly encompasses and how teachers of varying capacities differ in providing this information, leading to sub-optimal knowledge migration in complex neural networks.
In response to these challenges, the research team from Nanjing University developed a framework to rethink the composition and delivery of dark knowledge. This innovation shifts the focus from simple prediction accuracy to the distinctness of predicted probabilities among incorrect classes. By systematically comparing logits across teachers of different scales, the researchers discovered that stronger teachers tend to produce over-confident outputs on ground-truth classes, which flattens the probability distribution of non-target classes and erases fine-grained semantic affinity. To address this, the team analyzed cognitive consistency, proving that while capacity varies, the underlying cognition of class relationships remains stable across models.
Research indicates that in experiments on CIFAR-100 and ImageNet datasets, enhancing the distinctness among incorrect classes effectively mitigates the negative impact of capacity mismatch. Data suggests that this strategy allows smaller students to achieve significant accuracy gains even when trained under massive teacher models. This work not only provides an in-depth empirical explanation for the failure of traditional distillation in certain scenarios but also offers a reliable technical roadmap for building more effective and adaptive model compression systems for real-world deployment.

DOI
10.1007/s11704-025-41434-w