Textbook order: collected advice, then his comment on how it applies to
LLMs, then the exercises. Content left for him to write.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Small is under a paragraph; large means work like comparing against a
reference repo. Do all applicable small ones, pick one large one.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Quotes come from the docs/evidence cache and were previously unused. Reuses the
existing footnote style, 14 new keys. Carries the source doc's coverage warning:
mode 6 has only three quotes and mode 7 has none that name similarity probes.
Source is his own message of 2026-08-25, spelling fixed and slightly more polite as he
asked, with each mistake pointing at the exercise that answers it. Also adds his rule
that a job is never abandoned without doing the exercises, one at a time.
Two gaps the existing 13 did not cover, found by mining the evidence cache against
wassname's list of common AI-agent failures. Quotes are verbatim from
docs/evidence/ (Steinhardt, Rahtz, Nanda, Goodfellow-Bengio-Courville).
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Interpretations must read as evidence for or against a claim, never as a cause,
because a cause list gets picked from, called certain, and used to stop.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Both were promised by the description and absent from the procedure. P2 now
prints the formatted examples; P3 opens with the bug assumption.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
A sonnet subagent had ml-debug listed, with 'after a run finishes or crashes' in
the description, and did not invoke it for 'read the last pueue log'.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Sonnet routed to P1 correctly then declined it: 'the task only asked to read, not
to audit or act'. Reading a log is exactly when the measurement table gets filled.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
A sonnet subagent read the skill, summarised the ritual list, ran none of them,
and closed with 'everything checks out'. Phase triggers do not fire; a sentence
the agent watches itself write might.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
ml-bench measured standalone answers, not whether an agent in a loop follows
principles. Those are different claims and the first does not support the second.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
The run index is a cache key, not a seed, so subtracting run 3 from run 3 is
arbitrary. Unpaired the difference is +0.023 +- 0.044; by question +0.023 +- 0.031.
Still not pushed.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
+0.135 was one answer per arm, and it is draw 1 of three. Draws 2 and 3 read
-0.007 and -0.060, so the mean is +0.023 with sd 0.102. Not pushed: wassname
should read this before it goes public.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Quoted whole from the 2017 thread, since the article author asked to merge it
and never did. Verified line by line against the thread cache. Covers sample
size from a cumulative-mean plot, KLD/Dice on unbalanced data, augmentation
bounded by feature std, dummy metrics, jumpy validation loss, testing the
framework itself, activation swaps, and loss-curve shapes.
The author asked the commenter 'Do you mind if I add them to the article?' and
then did not: checked all 13 against the 2025 archived article, only the
batch-size point overlaps. Reddit blocks scrapers, so the thread came from a
Wayback snapshot. Also gave the article cache a real header: Medium is dead to
scrapers, so it now records the archive URL used to verify it.
The two README caches had no Source line, so nothing could re-verify them;
both now check at 99% coverage. The judge-bias file gets a warning at the top:
11 entries still carry summarizer numbers, and 2 of the 5 checked so far were
wrong.
102 of them, 48 already wrong after today's refetches. They rot every time a
cache is refetched and buy nothing the quote text does not: the caches are
verbatim, so the quote itself is the anchor. Descriptive labels stay.
All four came from the AI-generated deepwiki page, not the experiment log,
and one inverted the finding: the log says clipping was removed because
'Grad norm never exceeds 1.0 naturally', and the real distributed gotcha was
clipping local norms before sync. Footnote now points at dev/LOG.md on master
and at the renamed cache.
The notes file listed 11 links; 9 are already in the full page cache and the
80k Hours one has its own cache, so it only carried Rocktaschel, now in the
research_taste reading list. The Nanda shared draft was a 902-word excerpt of
a local download; wassname supplied the Google Doc, whose text export gives
the full 11318 words.
An /abs/ link is a stub: it costs a second lookup before anyone can check the
quote, and it is what let abstract-only caches look sourced. 77 links across
the skill files and 4 cache headers. Verbatim source bodies untouched, their
reference lists are the authors' text.
Refetches the URL in a cache file's header and measures 5-word shingle
coverage of the source, so a summary cannot pass as a copy. Guards against
stub sources, where high coverage of an abstract or landing page would
otherwise read as success. n=5 chosen by sweeping n against synthetic
copy/paraphrase/summary variants.
All nine said 'excerpted from HTML via browser' and held 141-684 words.
LessWrong posts refetched through the markdown API, the rest through jina.
All 99 previously quoted passages, and the 46 quotes SKILL.md and
refs/research_taste.md take from them, verify against the new full texts.
Every one was 3x to 100x shorter than its source: gwern_tank 179 -> 20132
words, cleanrl 37-details 239 -> 11410, kidger 185 -> 2932. Three passages the
old karpathy_recipe file presented inside quote marks were paraphrases, not
quotes, and are gone with the excerpts.
All were excerpt caches. Tuning playbook 662 -> 16216 words, ML Yearning
527 -> 26084 (the whole book, was ch13-19), bekman 579 -> 2357 (three
concatenated pages), Spinning Up 842 -> 3353. The Qwen3 report was truncated
mid-section 4.3 and now runs to the appendix.
Was 1179 words summarised from AI-generated deepwiki pages, accessed 2026-03.
Now the 9232-word primary dev/LOG.md, which also runs 2 months further.
Renamed since the source is the log, not deepwiki.
NoLiMa effective length was wrong in both files: most models fall below the
85% threshold at 1-4K tokens, not 8-16K (only GPT-4o 8K, GPT-4.1 16K). NoLiMa
32K count was 11 models, not 10. The Loo quote's first sentence was stitched
from two places and is dropped. JudgeLRM's 8.14% is an Introduction number,
not an abstract one, and the abstract quote had lost /14B and its trailing
clause. Shi et al is 'The findings', not 'Our findings'. Body quotes now
replace abstract quotes where the number matters; tags [ID] -> [FT].
These three held hand-transcribed excerpts (381 / 914 / 573 words) with a
header claiming extraction was too hard. It was not: jina reads the sculley
and lones PDFs, and plain pdftotext (no -layout) reads all 3 columns of the
domingos CACM PDF in order. Now 5736 / 15285 / 8180 words.
