Sonnet routed to P1 correctly then declined it: 'the task only asked to read, not
to audit or act'. Reading a log is exactly when the measurement table gets filled.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
A sonnet subagent read the skill, summarised the ritual list, ran none of them,
and closed with 'everything checks out'. Phase triggers do not fire; a sentence
the agent watches itself write might.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
ml-bench measured standalone answers, not whether an agent in a loop follows
principles. Those are different claims and the first does not support the second.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
The run index is a cache key, not a seed, so subtracting run 3 from run 3 is
arbitrary. Unpaired the difference is +0.023 +- 0.044; by question +0.023 +- 0.031.
Still not pushed.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
+0.135 was one answer per arm, and it is draw 1 of three. Draws 2 and 3 read
-0.007 and -0.060, so the mean is +0.023 with sd 0.102. Not pushed: wassname
should read this before it goes public.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Quoted whole from the 2017 thread, since the article author asked to merge it
and never did. Verified line by line against the thread cache. Covers sample
size from a cumulative-mean plot, KLD/Dice on unbalanced data, augmentation
bounded by feature std, dummy metrics, jumpy validation loss, testing the
framework itself, activation swaps, and loss-curve shapes.
The author asked the commenter 'Do you mind if I add them to the article?' and
then did not: checked all 13 against the 2025 archived article, only the
batch-size point overlaps. Reddit blocks scrapers, so the thread came from a
Wayback snapshot. Also gave the article cache a real header: Medium is dead to
scrapers, so it now records the archive URL used to verify it.
The two README caches had no Source line, so nothing could re-verify them;
both now check at 99% coverage. The judge-bias file gets a warning at the top:
11 entries still carry summarizer numbers, and 2 of the 5 checked so far were
wrong.
102 of them, 48 already wrong after today's refetches. They rot every time a
cache is refetched and buy nothing the quote text does not: the caches are
verbatim, so the quote itself is the anchor. Descriptive labels stay.
All four came from the AI-generated deepwiki page, not the experiment log,
and one inverted the finding: the log says clipping was removed because
'Grad norm never exceeds 1.0 naturally', and the real distributed gotcha was
clipping local norms before sync. Footnote now points at dev/LOG.md on master
and at the renamed cache.
The notes file listed 11 links; 9 are already in the full page cache and the
80k Hours one has its own cache, so it only carried Rocktaschel, now in the
research_taste reading list. The Nanda shared draft was a 902-word excerpt of
a local download; wassname supplied the Google Doc, whose text export gives
the full 11318 words.
An /abs/ link is a stub: it costs a second lookup before anyone can check the
quote, and it is what let abstract-only caches look sourced. 77 links across
the skill files and 4 cache headers. Verbatim source bodies untouched, their
reference lists are the authors' text.
Refetches the URL in a cache file's header and measures 5-word shingle
coverage of the source, so a summary cannot pass as a copy. Guards against
stub sources, where high coverage of an abstract or landing page would
otherwise read as success. n=5 chosen by sweeping n against synthetic
copy/paraphrase/summary variants.
All nine said 'excerpted from HTML via browser' and held 141-684 words.
LessWrong posts refetched through the markdown API, the rest through jina.
All 99 previously quoted passages, and the 46 quotes SKILL.md and
refs/research_taste.md take from them, verify against the new full texts.
Every one was 3x to 100x shorter than its source: gwern_tank 179 -> 20132
words, cleanrl 37-details 239 -> 11410, kidger 185 -> 2932. Three passages the
old karpathy_recipe file presented inside quote marks were paraphrases, not
quotes, and are gone with the excerpts.
All were excerpt caches. Tuning playbook 662 -> 16216 words, ML Yearning
527 -> 26084 (the whole book, was ch13-19), bekman 579 -> 2357 (three
concatenated pages), Spinning Up 842 -> 3353. The Qwen3 report was truncated
mid-section 4.3 and now runs to the appendix.
Was 1179 words summarised from AI-generated deepwiki pages, accessed 2026-03.
Now the 9232-word primary dev/LOG.md, which also runs 2 months further.
Renamed since the source is the log, not deepwiki.
NoLiMa effective length was wrong in both files: most models fall below the
85% threshold at 1-4K tokens, not 8-16K (only GPT-4o 8K, GPT-4.1 16K). NoLiMa
32K count was 11 models, not 10. The Loo quote's first sentence was stitched
from two places and is dropped. JudgeLRM's 8.14% is an Introduction number,
not an abstract one, and the abstract quote had lost /14B and its trailing
clause. Shi et al is 'The findings', not 'Our findings'. Body quotes now
replace abstract quotes where the number matters; tags [ID] -> [FT].
These three held hand-transcribed excerpts (381 / 914 / 573 words) with a
header claiming extraction was too hard. It was not: jina reads the sculley
and lones PDFs, and plain pdftotext (no -layout) reads all 3 columns of the
domingos CACM PDF in order. Now 5736 / 15285 / 8180 words.
Checked every 2001.04536 / 2402.01868 citation against the full texts now in
docs/evidence. Two quotation-marked sentences were not in the papers; the
Helmholtz 49x figure does not exist and the 64x Klein-Gordon one belongs to
architecture+annealing, not architecture alone; the Wang 1e5 Hessian number is
a max eigenvalue, not a max/min ratio. Section, table and figure pointers fixed.
Two details worth copying into a judge harness: scope the channel to systemic friction
and rule out one-off noise explicitly, and batch the entry at the end of the turn so
venting does not interleave with the work. -- Claude
Contrast with the cookbook's RLHF pipeline (46% -> 94% win rate in 100 steps),
so a decreasing DPO loss is weak evidence of behaviour change.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
DeepSeek-V4-Flash is 0.368 (rank 28/36) at 0.78 while gemma-4-31b is 0.723 at
0.82, so the cheap judge slot is not where you would guess. Note that -latest
aliases have no row of their own.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
