- refs/ -> references/, the folder name the Agent Skills spec uses and the one
Hermes skips when it walks for nested skills.
- rl and pinn declared name: rl and name: pinn, which are global names in a flat
skill namespace. Now ml-debug-rl and ml-debug-pinn. They also called themselves
sub-skills of 'ml-debugging', which is not this skill's name.
- Drop the dead link to SKILL_old.md. It moved into gitignored slop/, so the link
was broken for anyone who cloned.
- Route references/llm_judge_litreview.md, the one reference SKILL.md never named.
- Description leads with the trigger situations. Hermes truncates it to 57 chars.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
The two README caches had no Source line, so nothing could re-verify them;
both now check at 99% coverage. The judge-bias file gets a warning at the top:
11 entries still carry summarizer numbers, and 2 of the 5 checked so far were
wrong.
An /abs/ link is a stub: it costs a second lookup before anyone can check the
quote, and it is what let abstract-only caches look sourced. 77 links across
the skill files and 4 cache headers. Verbatim source bodies untouched, their
reference lists are the authors' text.
NoLiMa effective length was wrong in both files: most models fall below the
85% threshold at 1-4K tokens, not 8-16K (only GPT-4o 8K, GPT-4.1 16K). NoLiMa
32K count was 11 models, not 10. The Loo quote's first sentence was stitched
from two places and is dropped. JudgeLRM's 8.14% is an Introduction number,
not an abstract one, and the abstract quote had lost /14B and its trailing
clause. Shi et al is 'The findings', not 'Our findings'. Body quotes now
replace abstract quotes where the number matters; tags [ID] -> [FT].
Add a per-model, per-difficulty token-budget note to judge-selection: models
vary ~6x in tokens/task (Artificial Analysis), reasoning models emit ~8x more
and scale with difficulty (Epoch AI, DeepSeek-R1), and the depth-vs-breadth
trade (cap reasoning low + N passes on easy tasks, which doubles as the
repeat-variance check; don't truncate near model capability). Add Gu et al.
"A Survey on LLM-as-a-Judge" as a second survey anchor. Cache new quotes.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Replace the qualitative refusal claim with SpeechMap's per-lab Free Speech
Index (Mistral/xAI/Google most permissive; Anthropic/OpenAI most restrictive
majors; Chinese labs mid-high on the aggregate but topic-censored). Flag the
two traps: it's a lab average (hides a safety-tuned Gemma) and a cross-topic
aggregate (hides a per-subject refusal cluster). Cache the table.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Replace the hand-waved judge-model guidance with the checked-in Judgemark v4
cost-vs-score frontier (opus-4-6 top, gemma-4-31b the cheap knee; Qwen/DeepSeek
are mid-pack not frontier, correcting the earlier claim). Quote Doddapaneni
(evaluators miss >50% of injected quality drops) and Shankar (criteria drift,
validators need validation). Cache both papers and the frontier table.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Fold wassname's repeated eval-validity checklist into refs/llm_judges.md:
rubric-earns-ink, read-the-whole-trace, anchoring/scale-precision, repeat
variance, and a judge feedback channel. Add a judge-model-selection section
(Judgemark v4 cost-vs-score frontier, speechmap topic-conditional refusals).
Cache the new sources (Hamel, Databricks x2, Eugene Yan) with verbatim quotes
and onward paper links in docs/evidence.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
- SKILL.md: 3 new entries (exploration-over-exploitation + nuisance HPs,
test-set contamination, loss-spikes-mean-bad-data-pocket) and an Ng
100-misclassified-examples quote under inspect-the-data
- refs/llm_judges.md: position/verbosity/self-preference biases (Zheng,
Wang 66/80 flip, Panickssery) + mitigation checklist from verdict docs
- Lones pitfalls linked as the exhaustive 36-item do/don't checklist
- 6 new frozen evidence files; Hamel evals link in further reading
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
