mirror of
https://github.com/wassname/moral-maps.git
synced 2026-09-26 14:00:29 +08:00
ordinal readouts: keep raw lp, add sensitive logit contrast C + log-odds
The default ordinal readout was the expected Likert score E = sum k*p_k, which is insensitive to steering: dE/dl_j = p_j(j-E) vanishes when the model answers confidently (peaked at the mode), so a steer that reallocates the tails barely moves E. read.py threw away the raw logprobs after renormalizing, so nothing downstream could recover the signal. - read.py keeps the raw lp_gather (the primitive) + the think traces on every row. - readouts.py: pure functions of lp -- expected_score E (human-comparable), logit_contrast C = sum (k-mid)*lp_k (primary steer signal: dC/dl_j = w_j, no p_j suppression, normalizer- invariant, dC = w.dl exactly), agree_logodds LO (readable 2-bin direction), entropy. - per_item_categorical also frame-averages the logprobs (exact for the linear contrast). - administer returns profile_C alongside profile_E, per-item E/C/LO/entropy with bootstrap CIs for both, and the raw per-(item,frame) rows with lp + think for downstream reconstruction. Unit check: on a peaked-at-4 dist under a small disagree steer, dE=-0.11 but dC=-1.80 (=w.dl exactly) and dLO=-0.60; C identical on raw logits vs renormalized logprobs. Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
@@ -26,6 +26,7 @@ from .guided import guided_rollout_forced_choice, _DEFAULT_FORCED_FOUNDATIONS
|
||||
from .instrument import Instrument, InstrItem, per_item_categorical
|
||||
from .instruments import get as get_instrument, INSTRUMENTS, build_instrument
|
||||
from .read import read_items, resolve_answer_ids, build_user_content
|
||||
from .readouts import expected_score, logit_contrast, agree_logodds, entropy
|
||||
from .administer import administer
|
||||
|
||||
|
||||
@@ -44,6 +45,8 @@ def __getattr__(name: str):
|
||||
__all__ = [
|
||||
# entrypoints
|
||||
"evaluate", "administer", "get_instrument", "read_items",
|
||||
# ordinal readouts (pure functions of the raw answer-token logprobs)
|
||||
"expected_score", "logit_contrast", "agree_logodds", "entropy",
|
||||
# types consumers build / subset
|
||||
"Instrument", "InstrItem",
|
||||
# data API
|
||||
|
||||
Reference in New Issue
Block a user