wassname/ml-debug
1
0
Fork 0
mirror of https://github.com/wassname/ml-debug.git synced 2026-07-25 13:20:33 +08:00
Code Issues Packages Projects Releases Wiki Activity

Add reasoning-token budget guidance and LLM-as-judge survey

Browse Source 
Add a per-model, per-difficulty token-budget note to judge-selection: models
vary ~6x in tokens/task (Artificial Analysis), reasoning models emit ~8x more
and scale with difficulty (Epoch AI, DeepSeek-R1), and the depth-vs-breadth
trade (cap reasoning low + N passes on easy tasks, which doubles as the
repeat-variance check; don't truncate near model capability). Add Gu et al.
"A Survey on LLM-as-a-Judge" as a second survey anchor. Cache new quotes.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
wassname
2026-07-23 08:54:26 +08:00
co-authored by Claudypoo
parent 01db376bca
commit 2692ea8a4e
2 changed files with 24 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/evidence/llm_judge_biases.md
+18
View File
@@ -156,3 +156,21 @@ Higher = answers more / refuses less (0-100), windowed to releases in the last 6
Note the wide Peak-vs-Index gaps (DeepSeek peak 89.1 vs index 59.0; Zhipu 85.9 vs 71.0): pick the specific permissive checkpoint, not the lab.
Not fetched: SpeechMap's per-model and per-topic breakdowns (https://speechmap.ai/timeline/, /labs/<lab>/) and the eqbench4 EI benchmark data (https://github.com/EQ-bench/EQ-bench-site/blob/main/eqbench4/eqbench4_data.js) are separate from the judge data used here.
## Reasoning-token budget: Epoch AI + DeepSeek-R1
Epoch AI, "Output length" — https://epoch.ai/data-insights/output-length (via WebFetch, 2026-07-22):
> Reasoning models also exhibit longer response lengths overall - currently, around 8x more tokens on average, compared to non-reasoning models.
> Reasoning models' responses are growing considerably faster (5x per year) than those from non-reasoning models (2.2x per year).
> moving from 'medium' to 'high' effort resulted in a 1.6x increase in output tokens
DeepSeek-AI, "DeepSeek-R1" (arXiv:2501.12948), abstract quotes:
> The reasoning abilities of LLMs can be incentivized through pure reinforcement learning (RL), obviating the need for human-labeled reasoning trajectories.
> emergent development of advanced reasoning patterns, such as self-reflection, verification, and dynamic strategy adaptation
(The response-length-grows-over-training result is R1's headline figure in the body; not re-quoted verbatim here.) Also unverified: the per-model Artificial Analysis token-use splits (https://artificialanalysis.ai/models/qwen3-6-27b#intelligence-index-token-use-tabs) that wassname read as ~5k Gemma-4-31b to ~30k Qwen3.6-35B-A3B; the dashboard is JS-rendered and WebFetch only returned aggregate totals.