SPOT Improves On-Policy Distillation for LLM Reasoning.

On-policy distillation (OPD) is a technique where a smaller "student" model learns from a larger "teacher" model by observing the teacher's responses on trajectories generated by the student. A common issue with standard OPD is that it may not adequately emphasize alternative, yet plausible, continuations that the teacher might suggest, or it might not account for how well the student already understands these options. Furthermore, local teacher probabilities don't always correlate with overall success in complex reasoning tasks. To tackle these challenges, researchers developed SPOT (Sparse Probing and Outcome-calibrated Targets OPD). SPOT employs a three-stage process: acquisition, exploration, and exploitation. During acquisition, it intelligently allocates a limited "probing budget" by combining factors like teacher uncertainty, the probability mass of top candidates, and the discrepancy between student and teacher. In the exploration phase, SPOT evaluates teacher-proposed candidates through verifier-scored student continuations. Finally, during exploitation, these outcomes are used to create a KL-regularized target that prioritizes candidates leading to better downstream results, while still maintaining alignment with the teacher's distribution. Extensive experiments across various student models and reasoning benchmarks confirm that SPOT significantly boosts reasoning performance, striking a balance between solution quality and coverage.