Distillation / Knowledge Distillation
model distillation is to use the output, probability distribution or intermediate information of the teacher model to train the student model so that it can learn part of the teacher's capabilities on the target task.
Read article ↗