P^2-MOPD: Accelerating Multi-Teacher On-Policy Distillation via Parameter Priors
P^2-MOPD: Accelerating Multi-Teacher On-Policy Distillation via Parameter Priors
P²-MOPD accelerates multi-teacher strategy distillation by jointly training weights and mixing coefficient α, making experts parameter priors, significantly speeding up the process.
🔥 Multi-teacher on-policy distillation (MOPD) is now a standard way to consolidate domain experts into one model, yet the student still has to climb to every teacher on its own, and some domains are left behind.
What if the experts' weights could guide the student, not only their logits?
P²-MOPD writes every student module as its own learnable weights plus a weighted sum of the frozen expert task vectors (expert minus base model), and trains the weights and the mixing coefficients α jointly under the same MOPD loss. The experts become parameter priors: one learnable scalar per module moves the student along a whole expert direction, while the student weights remain free to go beyond any merge. The two channels, student weights and coefficients, share one backward pass, α is folded into the weights, and the trained model is a standard checkpoint with no change at inference.
Key results (Qwen3-1.7B / 4B; medicine, law, finance, instruction following):
- Half the steps: P²-MOPD surpasses the final average recovery of MOPD after half the training steps, at ≤0.55× of its wall-clock.
- Higher final recovery: 94.7% vs 80.9% (1.7B) and 94.9% vs 83.1% (4B).
- Balanced recovery: worst-domain recovery rises from 32% / 47% to 88% / 89%, and the across-domain std falls from 28 / 22 to about 5.
- A learned, reproducible mixture: across 3 training seeds the mean coefficient of every teacher agrees to within 0.01, and the mixture consistently weights the lagging teacher the most.
Why it works:
MOPD's gap is one of reachability, not of cross-domain conflict. Student weight-only steps barely move along the task vector of the domain they fail to recover, while P²-MOPD moves 13–15× farther along it through a single scalar per module; random directions in place of the task vectors give no gain, so what matters is the expert direction itself. The coefficients also turn cross-domain interference into a contest over a few interpretable scalars, which the gradient settles in favour of the lagging domain. Neither channel suffices alone: coefficients only (a learned merge) stay far below, weights only (MOPD) leave a domain behind, and the combination (P²-MOPD) gives both the best average and the best worst-domain recovery.
In one line: expert weights, not only expert predictions, can supervise distillation. Kept as learnable priors throughout training, they guide where the student moves.
💻 Code: https://github.com/SinuoWang/P2-MOPD
We welcome feedback and discussion!
Source: Hugging Face Daily Papers · huggingface.co