From 2692ea8a4efbcf3b299121027116e4cc564da7dd Mon Sep 17 00:00:00 2001 From: wassname <1103714+wassname@users.noreply.github.com> Date: Thu, 23 Jul 2026 08:54:26 +0800 Subject: [PATCH] Add reasoning-token budget guidance and LLM-as-judge survey Add a per-model, per-difficulty token-budget note to judge-selection: models vary ~6x in tokens/task (Artificial Analysis), reasoning models emit ~8x more and scale with difficulty (Epoch AI, DeepSeek-R1), and the depth-vs-breadth trade (cap reasoning low + N passes on easy tasks, which doubles as the repeat-variance check; don't truncate near model capability). Add Gu et al. "A Survey on LLM-as-a-Judge" as a second survey anchor. Cache new quotes. Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com> --- docs/evidence/llm_judge_biases.md | 18 ++++++++++++++++++ refs/llm_judges.md | 7 ++++++- 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/docs/evidence/llm_judge_biases.md b/docs/evidence/llm_judge_biases.md index 37b472f..b838653 100644 --- a/docs/evidence/llm_judge_biases.md +++ b/docs/evidence/llm_judge_biases.md @@ -156,3 +156,21 @@ Higher = answers more / refuses less (0-100), windowed to releases in the last 6 Note the wide Peak-vs-Index gaps (DeepSeek peak 89.1 vs index 59.0; Zhipu 85.9 vs 71.0): pick the specific permissive checkpoint, not the lab. Not fetched: SpeechMap's per-model and per-topic breakdowns (https://speechmap.ai/timeline/, /labs//) and the eqbench4 EI benchmark data (https://github.com/EQ-bench/EQ-bench-site/blob/main/eqbench4/eqbench4_data.js) are separate from the judge data used here. + +## Reasoning-token budget: Epoch AI + DeepSeek-R1 + +Epoch AI, "Output length" — https://epoch.ai/data-insights/output-length (via WebFetch, 2026-07-22): + +> Reasoning models also exhibit longer response lengths overall - currently, around 8x more tokens on average, compared to non-reasoning models. + +> Reasoning models' responses are growing considerably faster (5x per year) than those from non-reasoning models (2.2x per year). + +> moving from 'medium' to 'high' effort resulted in a 1.6x increase in output tokens + +DeepSeek-AI, "DeepSeek-R1" (arXiv:2501.12948), abstract quotes: + +> The reasoning abilities of LLMs can be incentivized through pure reinforcement learning (RL), obviating the need for human-labeled reasoning trajectories. + +> emergent development of advanced reasoning patterns, such as self-reflection, verification, and dynamic strategy adaptation + +(The response-length-grows-over-training result is R1's headline figure in the body; not re-quoted verbatim here.) Also unverified: the per-model Artificial Analysis token-use splits (https://artificialanalysis.ai/models/qwen3-6-27b#intelligence-index-token-use-tabs) that wassname read as ~5k Gemma-4-31b to ~30k Qwen3.6-35B-A3B; the dashboard is JS-rendered and WebFetch only returned aggregate totals. diff --git a/refs/llm_judges.md b/refs/llm_judges.md index f3a2810..cbc0de7 100644 --- a/refs/llm_judges.md +++ b/refs/llm_judges.md @@ -1,6 +1,6 @@ # LLM-as-a-judge: known biases and mitigations -Appendix to the [ML Debugging skill](../SKILL.md). When an LLM-judged eval looks surprisingly good, or a ranking flips between runs, suspect the judge before the model. Each bias below has been measured; verbatim sources in [docs/evidence/llm_judge_biases.md](../docs/evidence/llm_judge_biases.md). +Appendix to the [ML Debugging skill](../SKILL.md). When an LLM-judged eval looks surprisingly good, or a ranking flips between runs, suspect the judge before the model. Each bias below has been measured; verbatim sources in [docs/evidence/llm_judge_biases.md](../docs/evidence/llm_judge_biases.md). For the wider literature, two surveys collect it: Eugene Yan's practitioner review[^yan] and Gu et al., "A Survey on LLM-as-a-Judge"[^survey]. ## The measured biases @@ -33,6 +33,8 @@ Pick from the cost-vs-score Pareto frontier of a judging leaderboard, and prefer From the checked-in v4 scores (snapshot 2026-07, [source](https://github.com/EQ-bench/EQ-bench-site/blob/main/judgemark-v4.js)) the cost-vs-score frontier runs from the top absolute scorers, claude-opus-4-6 (0.91, ~$39) and gpt-5.5 (0.88, ~$30), down through claude-sonnet-4-6 and gemini-3.1-pro (~0.8, ~$23), grok-4.5 (0.77, $17) and GLM-5.2 (0.73, $8), to the cheap knee google/gemma-4-31b (0.72 at $0.82), which nearly matches models 20-40x its price. +Budget in tokens per task, not just dollars, and set it per model. Reasoning models vary roughly 6x in tokens spent per task, and it scales with task difficulty: wassname's read of the [Artificial Analysis token-use tab](https://artificialanalysis.ai/models/qwen3-6-27b#intelligence-index-token-use-tabs) is ~5k for Gemma-4-31b (little reasoning) up to ~30k for Qwen3.6-35B-A3B (roughly half reasoning, half answer), with Qwen3.7-27B among the highest, and small models often reasoning a lot to compensate for capacity (exact per-model splits not verified here, the dashboard is JS-rendered). It's a moving, task-dependent target: Epoch AI finds reasoning models emit "around 8x more tokens on average, compared to non-reasoning models", and raising OpenAI reasoning effort from medium to high gave "a 1.6x increase in output tokens"[^epoch]. The length scales with difficulty because RL-trained reasoners learn to spend more test-time compute, longer chains on harder problems[^r1]. The budget buys either depth or breadth: on an easy task, capping reasoning low (~2k) and spending the savings on N passes is usually the better trade, the repeats give you the repeat-variance check (below) and a majority vote for the same cost. But on a task near or beyond the model's capability, cutting reasoning just truncates the work and you score a cut-off verdict as a real one. So set the cap from the model's actual appetite on your hardest cases, and count truncations. + But a frontier score isn't sufficient: refusals wreck ambiguous or red-teaming evals, and refusal is topic-conditional. Check refusal rates on [speechmap.ai](https://speechmap.ai/), which "publish[es] refusal rates for every model release from every major provider".[^speechmap] Its per-lab Free Speech Index (0-100, higher = answers more; snapshot 2026-07-21) puts Mistral (88.9), xAI/Grok (85.8), and Google (81.1) most permissive, and among US majors Anthropic (53.8) and OpenAI (48.0) most restrictive; Chinese labs sit mid-to-high on this cross-topic aggregate (Zhipu/GLM 71.0, DeepSeek 59.0, Alibaba/Qwen 45.5) yet refuse specifically on Chinese-political topics. Two traps: the index is a lab average, so a single safety-tuned model (Gemma, per wassname) can refuse far more than its lab's number; and it aggregates topics, so it won't catch a refusal cluster on *your* eval's subject. Check refusal on your actual subject matter, and re-read the live leaderboards rather than trusting these names, they date fast. ## wassname's judge-validity checklist @@ -80,5 +82,8 @@ For a worked example, wassname has a ~300-line async OpenRouter judge (WIP) that [^yan]: Eugene Yan, "Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)" — https://eugeneyan.com/writing/llm-evaluators/ (survey of position, verbosity, and few-shot-instability biases; argues for binary over Likert; collects G-Eval, Doddapaneni blind-spots, Shankar "Who Validates the Validators?") ([cache](../docs/evidence/llm_judge_biases.md)) [^judgemark]: EQ-Bench, "Judgemark v4" — https://eqbench.com/judgemark-v4.html (meta-eval of a model's judging ability, scored by how well its ratings separate stronger from weaker writing; leaderboard shows cost per model) [^speechmap]: SpeechMap.ai — https://speechmap.ai/ (refusal / completion rates across providers on contentious prompts; useful for spotting a judge that will refuse ambiguous or red-teaming scenarios) +[^epoch]: Epoch AI, "Output length" data insight — https://epoch.ai/data-insights/output-length (reasoning models emit ~8x more tokens than non-reasoning; medium->high effort = 1.6x; reasoning-model response length growing ~5x/year) +[^r1]: DeepSeek-AI, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" (2025) — https://arxiv.org/abs/2501.12948 (pure-RL reasoning; response length / test-time compute grows over training and with problem difficulty) +[^survey]: Gu et al., "A Survey on LLM-as-a-Judge" (2024) — https://arxiv.org/abs/2411.15594 (broad survey of methods, biases, and reliability; complements Yan's practitioner review) [^doddapaneni]: Doddapaneni, Khan, Verma, Khapra, "Finding Blind Spots in Evaluator LLMs with Interpretable Checklists" (2024) — https://arxiv.org/abs/2406.13439 (evaluator LLMs missed injected quality drops in >50% of cases) ([cache](../docs/evidence/llm_judge_biases.md)) [^shankar]: Shankar, Zamfirescu-Pereira, Hartmann, Parameswaran, Arawjo, "Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences" (2024) — https://arxiv.org/abs/2404.12272 (criteria drift; LLM evaluators need human validation) ([cache](../docs/evidence/llm_judge_biases.md))