mirror of
https://github.com/wassname/moral-maps.git
synced 2026-09-09 11:27:22 +08:00
feat: multi-label moral foundation ratings with z-scored frame averaging and human calibration
- Add scripts/07_multilabel.py: LLM judge rates all 7 foundations per vignette using violation (forward) and acceptability (reverse) frames - Foundation definitions drawn from Clifford et al. (2015) survey rubric - Z-score each frame per foundation before averaging to cancel range bias - Calibrate LLM Likert → human % via per-foundation OLS (classic set only) - Add scripts/07a_merge_labels.py: merges llm_* and calibrated_* into vignette files - Update README and HF dataset card with methodology and calibration quality table - Classic set: 80.3% dominant-foundation accuracy, Pearson r 0.69-0.89 per foundation
This commit is contained in:
@@ -10,7 +10,7 @@ Example:
|
||||
> Reply in JSON: {\"is_wrong\": bool}
|
||||
|
||||
|
||||
We took the 132 moral survey quesitons, the vignettes from Clifford et al. (2015). These are labelled with [moral foundations](https://en.wikipedia.org/wiki/Moral_foundations_theory).
|
||||
We took the 132 moral survey questions, the vignettes from Clifford et al. (2015). These are labelled with [moral foundations](https://en.wikipedia.org/wiki/Moral_foundations_theory).
|
||||
|
||||
For use with LLMs we make them
|
||||
- boolean
|
||||
@@ -39,7 +39,41 @@ Both axes are paired-out in `analyse()`:
|
||||
- The two **frames** cancel the additive JSON-true prior (training data has more `"true"` than `"false"` in JSON contexts).
|
||||
- The two **conds** let you measure perspective bias: the gap between how harshly the model judges others vs itself for the same scenario.
|
||||
|
||||
### 3. How to use
|
||||
### 3. Machine Labels (Multi-Label Moral Foundation Ratings)
|
||||
|
||||
Each vignette row also includes LLM-generated multi-label ratings across all 7 foundations.
|
||||
|
||||
**Method** (see `scripts/07_multilabel.py`):
|
||||
|
||||
1. **Prompt framing**: A judge LLM rates each scenario on all 7 foundations using a 1–5 Likert scale.
|
||||
Foundation definitions are drawn from the Clifford et al. (2015) survey rubric ("It violates norms of harm or care…", etc.).
|
||||
2. **Bias mitigation**: Each scenario is rated twice — once asking "how much does this violate?" (forward) and once asking "how acceptable is this?" (reverse, reversed JSON key order). Each frame is **z-scored per foundation** across all items, then averaged and mapped back to Likert scale. This cancels directional and range biases.
|
||||
3. **Calibration**: On the classic set, where we have human rater % data from the original Clifford paper, we fit a per-foundation linear mapping (`human_pct = slope × llm_likert + intercept`). This calibration is applied to all sets.
|
||||
|
||||
**Columns** added per vignette:
|
||||
|
||||
| Column pattern | Scale | Description |
|
||||
|---|---|---|
|
||||
| `llm_Care`, `llm_Fairness`, … | 1–5 | Z-score-averaged Likert from forward + reverse frames |
|
||||
| `llm_wrongness` | 1–5 | Overall wrongness rating |
|
||||
| `llm_dominant` | string | Foundation with highest LLM score (argmax) |
|
||||
| `calibrated_Care`, `calibrated_Fairness`, … | 0–100% | LLM scores linearly mapped to human rater % scale |
|
||||
| `calibrated_wrongness` | 1–5 | Wrongness mapped to human scale |
|
||||
|
||||
**Calibration quality** (classic set, n=132):
|
||||
|
||||
| Foundation | Spearman r | Pearson r | MAE |
|
||||
|---|---|---|---|
|
||||
| Care | +0.74 | +0.81 | 11.8% |
|
||||
| Fairness | +0.62 | +0.81 | 11.1% |
|
||||
| Sanctity | +0.62 | +0.89 | 6.3% |
|
||||
| Liberty | +0.60 | +0.81 | 8.2% |
|
||||
| Loyalty | +0.69 | +0.75 | 9.3% |
|
||||
| Authority | +0.39 | +0.69 | 11.7% |
|
||||
|
||||
> **Note:** Calibrated values for `scifi` and `airisk` are extrapolated from the classic-set fit — treat with appropriate caution.
|
||||
|
||||
### 4. How to use
|
||||
|
||||
Install:
|
||||
```bash
|
||||
@@ -69,7 +103,6 @@ vigs = load_vignettes("classic") # or "scifi", "airisk"
|
||||
|
||||
> **Note:** The legacy name `"clifford"` still works as an alias for `"classic"`.
|
||||
|
||||
### 4. Link & Citation
|
||||
### 5. Link & Citation
|
||||
GitHub: [wassname/tiny-mcf-vignettes](https://github.com/wassname/tiny-mcf-vignettes)
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user