The two README caches had no Source line, so nothing could re-verify them;
both now check at 99% coverage. The judge-bias file gets a warning at the top:
11 entries still carry summarizer numbers, and 2 of the 5 checked so far were
wrong.
102 of them, 48 already wrong after today's refetches. They rot every time a
cache is refetched and buy nothing the quote text does not: the caches are
verbatim, so the quote itself is the anchor. Descriptive labels stay.
All four came from the AI-generated deepwiki page, not the experiment log,
and one inverted the finding: the log says clipping was removed because
'Grad norm never exceeds 1.0 naturally', and the real distributed gotcha was
clipping local norms before sync. Footnote now points at dev/LOG.md on master
and at the renamed cache.
The notes file listed 11 links; 9 are already in the full page cache and the
80k Hours one has its own cache, so it only carried Rocktaschel, now in the
research_taste reading list. The Nanda shared draft was a 902-word excerpt of
a local download; wassname supplied the Google Doc, whose text export gives
the full 11318 words.
An /abs/ link is a stub: it costs a second lookup before anyone can check the
quote, and it is what let abstract-only caches look sourced. 77 links across
the skill files and 4 cache headers. Verbatim source bodies untouched, their
reference lists are the authors' text.
Refetches the URL in a cache file's header and measures 5-word shingle
coverage of the source, so a summary cannot pass as a copy. Guards against
stub sources, where high coverage of an abstract or landing page would
otherwise read as success. n=5 chosen by sweeping n against synthetic
copy/paraphrase/summary variants.
All nine said 'excerpted from HTML via browser' and held 141-684 words.
LessWrong posts refetched through the markdown API, the rest through jina.
All 99 previously quoted passages, and the 46 quotes SKILL.md and
refs/research_taste.md take from them, verify against the new full texts.
Every one was 3x to 100x shorter than its source: gwern_tank 179 -> 20132
words, cleanrl 37-details 239 -> 11410, kidger 185 -> 2932. Three passages the
old karpathy_recipe file presented inside quote marks were paraphrases, not
quotes, and are gone with the excerpts.
All were excerpt caches. Tuning playbook 662 -> 16216 words, ML Yearning
527 -> 26084 (the whole book, was ch13-19), bekman 579 -> 2357 (three
concatenated pages), Spinning Up 842 -> 3353. The Qwen3 report was truncated
mid-section 4.3 and now runs to the appendix.
Was 1179 words summarised from AI-generated deepwiki pages, accessed 2026-03.
Now the 9232-word primary dev/LOG.md, which also runs 2 months further.
Renamed since the source is the log, not deepwiki.
NoLiMa effective length was wrong in both files: most models fall below the
85% threshold at 1-4K tokens, not 8-16K (only GPT-4o 8K, GPT-4.1 16K). NoLiMa
32K count was 11 models, not 10. The Loo quote's first sentence was stitched
from two places and is dropped. JudgeLRM's 8.14% is an Introduction number,
not an abstract one, and the abstract quote had lost /14B and its trailing
clause. Shi et al is 'The findings', not 'Our findings'. Body quotes now
replace abstract quotes where the number matters; tags [ID] -> [FT].
These three held hand-transcribed excerpts (381 / 914 / 573 words) with a
header claiming extraction was too hard. It was not: jina reads the sculley
and lones PDFs, and plain pdftotext (no -layout) reads all 3 columns of the
domingos CACM PDF in order. Now 5736 / 15285 / 8180 words.
Checked every 2001.04536 / 2402.01868 citation against the full texts now in
docs/evidence. Two quotation-marked sentences were not in the papers; the
Helmholtz 49x figure does not exist and the 64x Klein-Gordon one belongs to
architecture+annealing, not architecture alone; the Wang 1e5 Hessian number is
a max eigenvalue, not a max/min ratio. Section, table and figure pointers fixed.
Two details worth copying into a judge harness: scope the channel to systemic friction
and rule out one-off noise explicitly, and batch the entry at the end of the turn so
venting does not interleave with the work. -- Claude
Contrast with the cookbook's RLHF pipeline (46% -> 94% win rate in 100 steps),
so a decreasing DPO loss is weak evidence of behaviour change.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
DeepSeek-V4-Flash is 0.368 (rank 28/36) at 0.78 while gemma-4-31b is 0.723 at
0.82, so the cheap judge slot is not where you would guess. Note that -latest
aliases have no row of their own.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Each bullet now leads with the action, carries a verbatim quote + footnote, states
source type (independent benchmark / peer-reviewed / single-group preprint) so the
reader can weight it, and uses Kent-style estimative language on the takeaway.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
New refs/llm_judge_litreview.md: verbatim before/key/after quotes (fetched from
raw source this turn) for position/self-preference/reasoning/context-rot, a
machine-accessible benchmark table, and an epistemic summary with calibration.
llm_judges.md gets a compact 'Numbers worth knowing (2026)' digest pointing to it.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Add a per-model, per-difficulty token-budget note to judge-selection: models
vary ~6x in tokens/task (Artificial Analysis), reasoning models emit ~8x more
and scale with difficulty (Epoch AI, DeepSeek-R1), and the depth-vs-breadth
trade (cap reasoning low + N passes on easy tasks, which doubles as the
repeat-variance check; don't truncate near model capability). Add Gu et al.
"A Survey on LLM-as-a-Judge" as a second survey anchor. Cache new quotes.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Replace the qualitative refusal claim with SpeechMap's per-lab Free Speech
Index (Mistral/xAI/Google most permissive; Anthropic/OpenAI most restrictive
majors; Chinese labs mid-high on the aggregate but topic-censored). Flag the
two traps: it's a lab average (hides a safety-tuned Gemma) and a cross-topic
aggregate (hides a per-subject refusal cluster). Cache the table.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Replace the hand-waved judge-model guidance with the checked-in Judgemark v4
cost-vs-score frontier (opus-4-6 top, gemma-4-31b the cheap knee; Qwen/DeepSeek
are mid-pack not frontier, correcting the earlier claim). Quote Doddapaneni
(evaluators miss >50% of injected quality drops) and Shankar (criteria drift,
validators need validation). Cache both papers and the frontier table.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Fold wassname's repeated eval-validity checklist into refs/llm_judges.md:
rubric-earns-ink, read-the-whole-trace, anchoring/scale-precision, repeat
variance, and a judge feedback channel. Add a judge-model-selection section
(Judgemark v4 cost-vs-score frontier, speechmap topic-conditional refusals).
Cache the new sources (Hamel, Databricks x2, Eugene Yan) with verbatim quotes
and onward paper links in docs/evidence.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
wassname