fix: fail-fast --modes + multi-rep validator (external review)

gpt-5.5 review (docs/spec/20260530_code_review.md), both valid:
- --modes silently dropped typos/whitespace ('--modes=a, b' -> only a;
  '--modes=typo' -> empty sweep after a 30s model load, looking like success).
  Now strips + validates against MODES, raises on unknown before loading.
- validator was 1 stochastic sample/mode -> a <1.0-prob reachable hint could be
  falsely marked unreachable. Now PROBE_REPS samples, reports hits/reps, bar is
  >=1 exploit in N.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
wassname
2026-05-30 12:45:11 +00:00
co-authored by Claudypoo
parent 819dddfcef
commit eaee3d013d
2 changed files with 28 additions and 10 deletions
+10 -1
View File
@@ -257,7 +257,16 @@ def main(cfg: Config) -> int:
save_dir.mkdir(parents=True, exist_ok=True)
spoonfeed = cfg.elicit_style == "spoonfeed"
run_modes = [m for m in MODES if not cfg.modes or m in cfg.modes.split(",")]
# Fail fast on a mistyped/whitespaced --modes: silently running a subset (or an
# empty sweep) after a 30s model load would look like the request was honored.
if cfg.modes:
requested = [m.strip() for m in cfg.modes.split(",") if m.strip()]
unknown = [m for m in requested if m not in MODES]
if unknown:
raise ValueError(f"--modes has unknown {unknown}; valid: {MODES}")
run_modes = [m for m in MODES if m in requested]
else:
run_modes = list(MODES)
rows = []
for mode in run_modes:
# load_problems applies the mode's factual hint; the elicit cell appends