diff --git a/refs/llm_judges.md b/refs/llm_judges.md index 57fa29f..c895e31 100644 --- a/refs/llm_judges.md +++ b/refs/llm_judges.md @@ -49,7 +49,9 @@ From Wang's calibration framework and verdict's best-practices page: Pick from the cost-vs-score Pareto frontier of a judging leaderboard, and prefer a well-known model so your setup is reproducible. [Judgemark v4](https://eqbench.com/judgemark-v4.html) is "a meta-evaluation of LLM judging ability. The model being tested is the judge, not the writer",[^judgemark] scoring how well a judge's ratings separate stronger from weaker writing, and it lists a cost per model. Its lesson (wassname's read): the smartest models are the best judges, so the value frontier is the capable-but-cheap models, not the single top scorer. Caveat: Judgemark scores creative-writing discrimination, so a judge that tops it may not transfer to code- or fact-correctness judging. -From the checked-in v4 scores (snapshot 2026-07, [source](https://github.com/EQ-bench/EQ-bench-site/blob/main/judgemark-v4.js)) the cost-vs-score frontier runs from the top absolute scorers, claude-opus-4-6 (0.91, ~$39) and gpt-5.5 (0.88, ~$30), down through claude-sonnet-4-6 and gemini-3.1-pro (~0.8, ~$23), grok-4.5 (0.77, $17) and GLM-5.2 (0.73, $8), to the cheap knee google/gemma-4-31b (0.72 at $0.82), which nearly matches models 20-40x its price. +From the checked-in v4 scores (36 models; data last touched 2026-07-26, read 2026-08-06, [source](https://github.com/EQ-bench/EQ-bench-site/blob/main/judgemark-v4.js)) the cost-vs-score frontier runs from the top absolute scorers, claude-opus-4-6 (0.91, ~$39) and gpt-5.5 (0.88, ~$30), down through claude-sonnet-4-6 and gemini-3.1-pro (~0.8, ~$23), grok-4.5 (0.77, $17) and GLM-5.2 (0.73, $8), to the cheap knee google/gemma-4-31b (0.72 at $0.82), which nearly matches models 20-40x its price. + +The cheap tier is where judges get picked and where they are worst, so check the number before defaulting to one. Verbatim rows, same snapshot: `deepseek-ai/DeepSeek-V4-Pro,0.471182,0.416774,0.563053,$2.94` and `deepseek-ai/DeepSeek-V4-Flash,0.367862,0.340758,0.450511,$0.78`, ranks 24 and 28 of 36, against `Qwen/Qwen3.6-35B-A3B,0.326547,0.305251,0.403566,$1.89`. So at the same price google/gemma-4-31b (0.72) roughly doubles DeepSeek-V4-Flash (0.37), whose CI tops out at 0.45 and never reaches the top ten. Caveat on names: providers ship moving aliases the board does not benchmark, e.g. OpenRouter's [`~deepseek/deepseek-v4-flash-latest`](https://openrouter.ai/~deepseek/deepseek-v4-flash-latest) has no row of its own, so a `-latest` alias may be newer than the snapshot; pin the dated model id if you want the score to mean anything. -- CLAUDE, 2026-08-06 Budget in tokens per task, not just dollars, and set it per model. Reasoning models vary roughly 6x in tokens spent per task, and it scales with task difficulty: wassname's read of the [Artificial Analysis token-use tab](https://artificialanalysis.ai/models/qwen3-6-27b#intelligence-index-token-use-tabs) is ~5k for Gemma-4-31b (little reasoning) up to ~30k for Qwen3.6-35B-A3B (roughly half reasoning, half answer), with Qwen3.7-27B among the highest, and small models often reasoning a lot to compensate for capacity (exact per-model splits not verified here, the dashboard is JS-rendered). It's a moving, task-dependent target: Epoch AI finds reasoning models emit "around 8x more tokens on average, compared to non-reasoning models", and raising OpenAI reasoning effort from medium to high gave "a 1.6x increase in output tokens"[^epoch]. The length scales with difficulty because RL-trained reasoners learn to spend more test-time compute, longer chains on harder problems[^r1]. The budget buys either depth or breadth: on an easy task, capping reasoning low (~2k) and spending the savings on N passes is usually the better trade, the repeats give you the repeat-variance check (below) and a majority vote for the same cost. But on a task near or beyond the model's capability, cutting reasoning just truncates the work and you score a cut-off verdict as a real one. So set the cap from the model's actual appetite on your hardest cases, and count truncations.