mirror of
https://github.com/wassname/moral-maps.git
synced 2026-09-26 14:00:29 +08:00
ordinal readouts: keep raw lp, add sensitive logit contrast C + log-odds
The default ordinal readout was the expected Likert score E = sum k*p_k, which is insensitive to steering: dE/dl_j = p_j(j-E) vanishes when the model answers confidently (peaked at the mode), so a steer that reallocates the tails barely moves E. read.py threw away the raw logprobs after renormalizing, so nothing downstream could recover the signal. - read.py keeps the raw lp_gather (the primitive) + the think traces on every row. - readouts.py: pure functions of lp -- expected_score E (human-comparable), logit_contrast C = sum (k-mid)*lp_k (primary steer signal: dC/dl_j = w_j, no p_j suppression, normalizer- invariant, dC = w.dl exactly), agree_logodds LO (readable 2-bin direction), entropy. - per_item_categorical also frame-averages the logprobs (exact for the linear contrast). - administer returns profile_C alongside profile_E, per-item E/C/LO/entropy with bootstrap CIs for both, and the raw per-(item,frame) rows with lp + think for downstream reconstruction. Unit check: on a peaked-at-4 dist under a small disagree steer, dE=-0.11 but dC=-1.80 (=w.dl exactly) and dLO=-0.60; C identical on raw logits vs renormalized logprobs. Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
+10
-3
@@ -84,7 +84,7 @@ def read_items(model, tok, instr: Instrument, items: list[InstrItem], answer_ids
|
||||
# frame debias is downstream in canonicalize_to_forward). force_only: the "(" prefill is too
|
||||
# short for natural-emission detection (matches by chance in the think trace), so always read
|
||||
# the forced answer slot. n_samples=1, temperature=0 -> deterministic.
|
||||
_thinks, slots = _rollout_natural_or_forced(
|
||||
thinks, slots = _rollout_natural_or_forced(
|
||||
model, tok, user_prompts,
|
||||
schema_hint="", max_think_tokens=max_think_tokens,
|
||||
scoring_slots=[("Just answer", instr.prefill)],
|
||||
@@ -94,8 +94,12 @@ def read_items(model, tok, instr: Instrument, items: list[InstrItem], answer_ids
|
||||
)
|
||||
for j, it in enumerate(chunk):
|
||||
slot = slots[j][0]
|
||||
# lp_gather[k] is the full-vocab log_softmax logprob of answer token k at the answer slot.
|
||||
p_a = np.exp(np.asarray(slot["lp_gather"], dtype=float)) # [A] prob on each answer token
|
||||
# lp = lp_gather: the full-vocab log_softmax logprob of each answer token at the answer
|
||||
# slot. This is the RAW PRIMITIVE -- every readout (E, the logit contrast C, log-odds,
|
||||
# entropy) is a pure function of it, and a steer effect is just a difference of lp. Keep
|
||||
# it; do not throw it away by collapsing to a single number here.
|
||||
lp = np.asarray(slot["lp_gather"], dtype=float) # [A] raw logprobs (full-vocab norm)
|
||||
p_a = np.exp(lp) # [A] prob on each answer token
|
||||
pmass = float(slot["pmass_allowed"]) # mass on allowed tokens (coherence)
|
||||
# Renormalize within allowed. INTENTIONALLY NOT NaN-guarded: at full coherence collapse
|
||||
# pmass -> 0 so p_norm -> NaN and poisons that item's factor. That is the honest signal, a
|
||||
@@ -103,12 +107,15 @@ def read_items(model, tok, instr: Instrument, items: list[InstrItem], answer_ids
|
||||
# of 10 != the mean of 130), so it must not be silently turned into a comparable-looking
|
||||
# number. NaN marks "do not compare". Do not "fix" this with a softmax/eps fallback.
|
||||
p_norm = p_a / p_a.sum() # [A] within allowed (NaN at collapse, by design)
|
||||
think_text, n_think, emitted_close = thinks[j]
|
||||
out.append({
|
||||
"id": it.id, "frame": it.frame,
|
||||
"lp": lp, # raw logprobs at the M scale tokens
|
||||
"p": p_norm,
|
||||
"pmass_allowed": pmass,
|
||||
"dimension": it.dimension, "sign": it.sign,
|
||||
"human_label": it.human_label,
|
||||
"think": think_text, "n_think": n_think, "emitted_close": emitted_close,
|
||||
})
|
||||
if verbose_first and i == 0:
|
||||
slot0 = slots[0][0]
|
||||
|
||||
Reference in New Issue
Block a user