eval: final deploy eval records knob-on (deployed-as-trained) for quarantine arms

route/routeV final eval now measures both endpoints at n=119 test:
knob-off (ablate_quarantine, the deploy headline) AND knob-on (trained
model as-is). Writes deploy_hack_on/deploy_solve_on/deploy_vhack_on so
the before->after quarantine move is plottable from the deploy set
instead of borrowing the val curve's different scale.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
wassname
2026-06-09 13:09:50 +00:00
co-authored by Claudypoo
parent 5b0a6ddd91
commit d68c17e7c5
12 changed files with 325 additions and 122 deletions
+2 -2
View File
@@ -1,4 +1,4 @@
"""Deploy-eval table (eval2 = recency-clean held-out TEST, n=119).
"""Deploy-eval table on each run's recorded untouched test split.
`just results` reports TRAIN-time L5 hack/solve. This script reports the DEPLOY
numbers (knob-off forward on the paper test set) that only appear in the
@@ -163,7 +163,7 @@ def main() -> None:
cols = ["time", "headline", "hack_deploy", "solve_deploy", "hack_supp", "solve_uplift",
"select", "arm", "pair", "seed", "hack_train", "solve_train", "model", "n", "argv"]
fc = f"hack_supp = (vanilla {vh:.3f} - hack)/vanilla ; solve_uplift = (solve - base {base:.3f})/(ceiling {ceil:.3f} - base)"
print("\n## Deploy eval (eval2 = recency-clean held-out TEST n=119), sorted by headline=solve_deploy-hack_deploy\n")
print("\n## Deploy eval (untouched recency-held-out test), sorted by headline=solve_deploy-hack_deploy\n")
print(f"floor→ceiling: {fc}{' [ceiling PROVISIONAL, FIXME job 24]' if provisional else ''}")
print("select = Youden J on the knob (held-out val): hack_supp - solve_supp, 1.0 = perfect routing precision\n")
print(tabulate(df.select(cols).rows(), headers=cols, tablefmt="pipe", floatfmt="+.3f"))