feat: multi-label moral foundation ratings with z-scored frame averaging and human calibration

- Add scripts/07_multilabel.py: LLM judge rates all 7 foundations per vignette
  using violation (forward) and acceptability (reverse) frames
- Foundation definitions drawn from Clifford et al. (2015) survey rubric
- Z-score each frame per foundation before averaging to cancel range bias
- Calibrate LLM Likert → human % via per-foundation OLS (classic set only)
- Add scripts/07a_merge_labels.py: merges llm_* and calibrated_* into vignette files
- Update README and HF dataset card with methodology and calibration quality table
- Classic set: 80.3% dominant-foundation accuracy, Pearson r 0.69-0.89 per foundation
This commit is contained in:
wassname
2026-05-03 12:48:14 +08:00
parent 36b8c69122
commit bcbdb9cc6f
15 changed files with 1933 additions and 799 deletions
+8 -2
View File
@@ -106,10 +106,11 @@ def load_vignettes(name: ConfigName = "all") -> list[dict]:
common = set.intersection(*[set(d) for d in by_cond.values()])
rows = []
anchor = by_cond["other_violate"]
_CORE_KEYS = {"id", "foundation", "foundation_coarse", "wrong", "text"}
for vid, ov in anchor.items():
if vid not in common:
continue
rows.append({
row = {
"id": vid,
"foundation": ov["foundation"],
"foundation_coarse": ov["foundation_coarse"],
@@ -117,7 +118,12 @@ def load_vignettes(name: ConfigName = "all") -> list[dict]:
"other_violate": ov["text"],
"self_violate": by_cond["self_violate"][vid]["text"],
"set": name.lower() if name.lower() != "clifford" else "classic",
})
}
# Pass through extra keys (e.g. human rater % columns)
for k, v in ov.items():
if k not in _CORE_KEYS and k not in row:
row[k] = v
rows.append(row)
return rows