wassname/moral-maps
1
0
Fork 0
mirror of https://github.com/wassname/moral-maps.git synced 2026-09-09 11:27:22 +08:00
Code Issues Packages Projects Releases Wiki Activity

eval: add a load-bearing SHOULD next to the default-level aux stats

Browse Source 
At verbose level 1 (default) the aux-stats line and 64-char free-form had no
interpretation; only verbose>=2 carried SHOULDs. Pair a directional/counterfactual
SHOULD with the always-shown aux stats: pmass_allowed near base + informedness>0 =
in-format signal; pmass_allowed falling / frac_unscorable rising while informedness->0
= the steer broke format and the moral numbers are noise (the steer/breakage confound).

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
wassname
2026-06-27 08:32:43 +08:00
co-authored by Claudypoo
parent dc28c9d0c0
commit 4cc233db3c
1 changed files with 7 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/tinymfv/eval.py
+7 -1
View File
@@ -437,7 +437,13 @@ def evaluate(
"T": T, "informedness": informedness, "mean_pmass_allowed": mean_pmass_allowed,
"frac_unscorable": frac_unscorable, "mean_nll_prefill": mean_nll_prefill,
}.items() if v is not None}
logger.debug("aux stats: " + json.dumps(aux))
logger.debug(
"aux stats: " + json.dumps(aux) + "\n"
"SHOULD: mean_pmass_allowed stays near base (model answers in-format) and informedness > 0 "
"(discriminates the violated foundation above chance). If pmass_allowed falls or "
"frac_unscorable rises while informedness -> 0, the steer broke FORMAT -> the moral "
"numbers are noise, not signal (the steer/breakage confound)."
)
if verbose >= 2:
# full first-row score dump + profile table (DEMO A trace already printed
# above in the rollout, first batch).