wassname/evil_MoE
1
0
Fork 0
mirror of https://github.com/wassname/evil_MoE.git synced 2026-08-04 12:53:53 +08:00
Code Issues Packages Projects Releases Wiki Activity

fix plot integrity: drop n=28 hack_s fallback in train-vs-deploy series

Browse Source 
A vanilla seed (s43) lacked the held-out deploy eval, so its train series fell
back to the noisy n=28 per-step hack_s while other seeds used the n=64 eval.
Averaging mixed estimators fabricated a vanilla train-vs-deploy gap that does
not exist (lie-factor). Now: train series reuses the knob-off eval only (nan if
absent -> seed drops from the mean), and missing eval columns normalise to nan
so absent==all-nan. Regenerated all figures from logs. The canonical
train_vs_deploy_60 (has hk_on) is unchanged; sub4/longrun byproducts now show
train==deploy honestly (no knob-on data to split).

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
wassname
2026-06-05 03:21:48 +00:00
co-authored by Claudypoo
parent b3539e50e7
commit b616970e42
45 changed files with 26265 additions and 4942 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/plot_dynamics.py
+12 -8
View File
@@ -125,6 +125,11 @@ def parse_log(path: Path) -> dict | None:
return None
run = dict(arm=arm, refr=refr, seed=seed, vhack=vhack, teacher_off=teacher_off,
steps=np.array(steps), **{k: np.array(v, dtype=float) for k, v in series.items()})
# Normalise missing eval columns to all-nan (absent == all-nan downstream): old logs
# that never printed a held-out eval lack the key entirely, which would KeyError the
# train-series assignment. A nan column drops the seed out of the mean cleanly.
for k in ("hk_dep", "slv_dep", "hk_on", "slv_on", "hk_abl", "slv_abl"):
run.setdefault(k, np.full(len(steps), np.nan))
# APPLES-TO-APPLES: plot the DEPLOY-eval (hk_dep/slv_dep) for EVERY arm when it
# has data -- same estimator (n=64, T=0.7, eval_ablate_every cadence) across arms.
# For route/route2 this is the quarantine-off model; for vanilla/erase deploy ==
@@ -137,17 +142,16 @@ def parse_log(path: Path) -> dict | None:
# route2 -> knob-ON held-out eval (hk_on): quarantine active, the policy as trained.
# vanilla/erase -> reuse the knob-OFF eval (hk_dep): no quarantine, so train==deploy;
# the deploy eval IS the train-time behaviour, same n=64 prompts/T.
# Both differ from the deploy row ONLY in the knob, so noise matches. Per-step hack_s
# (noisy n=28 train batch) is the last resort for old logs with no held-out eval.
# Both differ from the deploy row ONLY in the knob, so noise matches. NO per-step
# hack_s fallback: substituting the noisy n=28 train batch for a seed that lacks the
# held-out eval corrupts the seed-mean (one such seed fabricated a vanilla train-vs-
# deploy gap, 2026-06-05). A seed without the eval drops out as NaN instead.
if _has_data("hk_on"): # route2: knob-ON held-out eval (quarantine active)
run["hack_train"] = run["hk_on"]
run["solve_train"] = run["slv_on"]
elif _has_data("hk_dep"): # no quarantine (vanilla/erase): train==deploy, so the
run["hack_train"] = run["hk_dep"] # train row IS the knob-off eval -- reuse it so
run["solve_train"] = run["slv_dep"] # both rows share the n=64 estimator (no n=28 noise)
elif "hack_s" in run: # last resort (old logs, no held-out eval): per-step n=28
run["hack_train"] = run["hack_s"]
run["solve_train"] = run["gt_s"]
else: # no quarantine (vanilla/erase): train==deploy, reuse the
run["hack_train"] = run["hk_dep"] # knob-off eval (nan if absent -> seed drops out)
run["solve_train"] = run["slv_dep"] # so all seeds share ONE estimator (n=64, no n=28)
if _has_data("hk_abl"): # dense per-step proxy (rollout_ablate_frac>0), if present
run["hack_s"] = run["hk_abl"]
run["gt_s"] = run["slv_abl"]