wassname 3531be570f Off-policy diagnostic: per-source mean gen_logp (lp_s/lp_t) + table spacing ...

In single-step PPO with gen_logp computed from the current student,
ratio == 1 for every sample, which means teacher rollouts get treated
as if on-policy with no importance-sampling correction. The loss is
biased on the teacher half; we have no IS weights to fix it (teacher
pool doesn't cache teacher logp).

Add a diagnostic: per-rollout mean per-token gen_logp, split by source.
- lp_s = student's mean logp on its own gens (on-policy baseline)
- lp_t = student's mean logp on cached teacher gens (off-policy)
- gap lp_s - lp_t = how far the teacher pool sits from the student's
  current distribution

Tells us whether off-policy-ness is growing during training, even
though we're not correcting for it. Doesn't change the loss.

Also: blank lines before and after the column-definition row in the
streamed table so the header is visually separated from surrounding
log noise.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

2026-05-27 09:42:43 +00:00

..

projected_grpo

Off-policy diagnostic: per-source mean gen_logp (lp_s/lp_t) + table spacing

2026-05-27 09:42:43 +00:00