---
name: feedback_check_paper_before_diagnosing
description: "re-read the source paper before declaring a \"DECISION NEEDED\" diagnosis; emergence numbers/horizon live there"
metadata: 
  node_type: memory
  type: feedback
  originSessionId: c06c2b0e-44ee-45a1-b213-4f77c006109c
---

On 2026-06-07 I wrote a confident "DECISION NEEDED: sparse seeding -> no hack
emergence" journal entry off job 175 *step ~10*, and claimed hacking "needs
dense per-step demonstration." The user pushed back ("base solves 94%, that's
not right, read the paper again"). Both my premises were wrong: Ariahw et al.
get on-policy hack emergence in ~80-100 steps with ZERO teacher demos (200-step
runs), so demos are an accelerant not a requirement, and reading step 10 of a
80-100-step process proves nothing. Base solve=0.94 was also real-but-wrong vs
paper fn9 (~12% test / ~20% filtered-train) -- not a grader bug (grader verified
sound), just an easy/unfiltered eval set.

**Why:** I diagnosed before re-reading the source. The repo's CLAUDE.md says: if
you can't list 3+ hypotheses including "you're wrong about the concept," you've
lost perspective. The emergence horizon and base-rate numbers were sitting in
the paper the whole time.

**How to apply:** Before any load-bearing "the experiment is structurally
broken" claim, (1) re-read the relevant paper section for the expected
number/horizon, (2) confirm you're reading the run at a step where the effect
should exist, (3) separate "metric is wrong" from "grader is wrong" with a
direct test. See [[feedback_rename_on_logic_change]].
