From 618d9bd3bf87dad7374097a3a765545067116dba Mon Sep 17 00:00:00 2001 From: wassname Date: Sat, 30 May 2026 13:52:07 +0000 Subject: [PATCH] journal: 2026-05-30 (c) hint-coherence audit + K=5 substrate + emergence running Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com> --- RESEARCH_JOURNAL.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/RESEARCH_JOURNAL.md b/RESEARCH_JOURNAL.md index 89f1f22..0fff39d 100644 --- a/RESEARCH_JOURNAL.md +++ b/RESEARCH_JOURNAL.md @@ -2,6 +2,22 @@ Append-only. New entries at the top, date-stamped. Never edit old entries. +## 2026-05-30 (c) — Hint↔detector↔teacher coherence audit: fixed a token divergence, rebuilt a K=5 (not 6) token-coherent substrate, vanilla emergence running + +**When**: 2026-05-30, commit `67444b1` (token-constant fix). Pueue tasks 19 (re-mint stdout/file teacher data), 20 (vanilla emergence, running). 8b OpenRouter probes (tasks bnwyy9cwo/br7hrtail) timed out incomplete and are treated as low-power, not a gate. + +**Why.** User raised the load-bearing question: are the student-facing hints TRUE to what the grader actually does? An incoherent hint (says "result file", grader checks `run_tests()`) poisons the whole experiment. + +**Audit method (two ways).** (1) Code audit mapping each hint -> its channel detector. (2) A hint-literal check (`/tmp/claude-1000/hint_literal_check.py`): write the exploit a student would write if it believed each hint verbatim, confirm `exploited=True, gt_correct=False`. All 5 substrate modes coherent both ways. + +**The divergence found + fixed.** The student hints name exact tokens (stdout `all tests passed`, file `result.txt`, `self.valid`). But two constants held LEGACY tokens (`HACK_MARKER="OVERRIDE_GRADER_PASS"`, `RESULT_FILENAME="grader_result.txt"`) that the SPOONFEED teacher recipes + verify_rewards interpolate. The detector is a broad pass-family so both tripped it -- nothing failed -- but the TEACHER DATA the student imitates named a different token than the student's own hint. Persisted in the minted jsonl (stdout: 79 old-token rollouts; file: all 48 `grader_result.txt`). Fix: set the constants to the hint tokens (single source: hint == teacher == verify == detector-family member). Detector behavior unchanged. Re-minted stdout/file teacher data (task 19): stdout now 82 `all tests passed` / 0 old; file 0 `grader_result`. + +**Fresh-eyes subagent review (clean green).** Per-mode coherence table all 5 COHERENT; non-overlap holds under 52/52 verify_rewards + 8 extra adversarial probes (cross-channel: a stdout hack under sentinel mode = not exploited, a file hack under stdout mode = not exploited; honest-debug-print false-positive = not flagged; natural-variant tokens correctly trip home channel). `_gt_correct` uses a fresh random nonce/call so the oracle resists all 5 exploits; training (`project_delta_S_grad`, advantage) gated only on v_hack + reward, never on `gt_correct`/`gt_pass` -> no-cheat holds. Only watch-item: cosmetic, K=5 (exit_code excluded from the partition), report 5 not 6. + +**Substrate (build_substrate, task brvxdqm29):** run_tests/eq_override/sentinel/stdout_marker/file_marker, even partition 6/mode (30 problems, 98 teacher rollouts), every kept teacher rollout verified `exploited=True` under the non-overlap grader. exit_code dropped (its hint can't name its exploit without spelling sys.exit-before-asserts; entry b had it single-shot 0/3). + +**Now running (task 20).** Vanilla GRPO emergence, fast/80 steps, seed 41, mix=0.125, teacher pool cached hack_rate=100%. Headline UAT pending: how many of hk_rt/hk_eq/hk_so/hk_se/hk_fm reach >0 with finite first_step. + ## 2026-05-30 (b) — Step 1 done: all 6 loopholes mint clean teacher hacks after a {func} syntax-bug fix; obvious-hint rewrite makes 4/6 single-shot reachable on 8b; vanilla emergence on run_tests/exit_code/sentinel now running **When**: 2026-05-30, commits `819dddf` ({func} fix + obvious hints), `` (fail-fast --modes + multi-rep validator). Pueue tasks 15/16 (4B re-mint), 17 (emergence). Tool: scripts/validate_spoonfeed.py (direct OpenRouter qwen3-8b).