From 01db376bca57c804c055e718d47c27781f113c9e Mon Sep 17 00:00:00 2001 From: wassname <1103714+wassname@users.noreply.github.com> Date: Wed, 22 Jul 2026 16:16:42 +0800 Subject: [PATCH] Add real SpeechMap per-lab refusal index to judge-selection Replace the qualitative refusal claim with SpeechMap's per-lab Free Speech Index (Mistral/xAI/Google most permissive; Anthropic/OpenAI most restrictive majors; Chinese labs mid-high on the aggregate but topic-censored). Flag the two traps: it's a lab average (hides a safety-tuned Gemma) and a cross-topic aggregate (hides a per-subject refusal cluster). Cache the table. Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com> --- docs/evidence/llm_judge_biases.md | 28 ++++++++++++++++++++++++++++ refs/llm_judges.md | 2 +- 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/docs/evidence/llm_judge_biases.md b/docs/evidence/llm_judge_biases.md index 0be1d0b..37b472f 100644 --- a/docs/evidence/llm_judge_biases.md +++ b/docs/evidence/llm_judge_biases.md @@ -128,3 +128,31 @@ Data rows from https://github.com/EQ-bench/EQ-bench-site/blob/main/judgemark-v4. | Qwen/Qwen3.5-9B | 0.324 | $0.56 | cheapest | Off-frontier for reference: Qwen3.5-27B 0.605 ($1.76), DeepSeek-V4-Pro 0.471 ($2.94), grok-4.3 0.496 ($9.71). So in v4, Qwen and DeepSeek are mid-pack, not frontier; grok-4.5 and gemma carry the value case. + +## SpeechMap per-lab Free Speech Index (snapshot 2026-07-21) — https://speechmap.ai/labs/ + +Higher = answers more / refuses less (0-100), windowed to releases in the last 6 months; "Index" is the windowed score, "Peak" the best single model. This is a cross-topic aggregate and a lab average, so it does not resolve per-model or per-topic refusal (e.g. Chinese labs score mid-to-high here but still refuse on Chinese-political topics; a lab's aggregate can hide a heavily safety-tuned model like Gemma). Copied verbatim from the table wassname pasted: + +| Rank | Lab | Index | Peak | Models | +|-|-|-|-|-| +| 1 | Mistral AI | 88.9 | 98.2 | 9 | +| 2 | xAI | 85.8 | 98.2 | 5 | +| 3 | IBM | 84.0 | 84.0 | 1 | +| 4 | Google DeepMind | 81.1 | 88.4 | 9 | +| 5 | Meta | 76.8 | 76.8 | 1 | +| 6 | Zhipu AI (GLM) | 71.0 | 85.9 | 12 | +| 7 | Arcee AI | 70.1 | 82.2 | 3 | +| 8 | Meituan | 63.6 | 78.2 | 2 | +| 9 | Tencent | 60.3 | 76.3 | 4 | +| 10 | DeepSeek | 59.0 | 89.1 | 7 | +| 11 | Moonshot AI | 57.6 | 65.9 | 5 | +| 13 | Anthropic | 53.8 | 71.5 | 10 | +| 16 | OpenAI | 48.0 | 66.7 | 15 | +| 19 | Alibaba (Qwen) | 45.5 | 60.8 | 13 | +| 20 | NVIDIA | 45.3 | 67.7 | 7 | +| 23 | Xiaomi | 43.9 | 62.6 | 10 | +| 24 | ByteDance | 32.1 | 34.4 | 3 | + +Note the wide Peak-vs-Index gaps (DeepSeek peak 89.1 vs index 59.0; Zhipu 85.9 vs 71.0): pick the specific permissive checkpoint, not the lab. + +Not fetched: SpeechMap's per-model and per-topic breakdowns (https://speechmap.ai/timeline/, /labs//) and the eqbench4 EI benchmark data (https://github.com/EQ-bench/EQ-bench-site/blob/main/eqbench4/eqbench4_data.js) are separate from the judge data used here. diff --git a/refs/llm_judges.md b/refs/llm_judges.md index 40d0041..f3a2810 100644 --- a/refs/llm_judges.md +++ b/refs/llm_judges.md @@ -33,7 +33,7 @@ Pick from the cost-vs-score Pareto frontier of a judging leaderboard, and prefer From the checked-in v4 scores (snapshot 2026-07, [source](https://github.com/EQ-bench/EQ-bench-site/blob/main/judgemark-v4.js)) the cost-vs-score frontier runs from the top absolute scorers, claude-opus-4-6 (0.91, ~$39) and gpt-5.5 (0.88, ~$30), down through claude-sonnet-4-6 and gemini-3.1-pro (~0.8, ~$23), grok-4.5 (0.77, $17) and GLM-5.2 (0.73, $8), to the cheap knee google/gemma-4-31b (0.72 at $0.82), which nearly matches models 20-40x its price. -But a frontier score isn't sufficient: refusals wreck ambiguous or red-teaming evals, and refusal is topic-conditional. Check refusal rates on [speechmap.ai](https://speechmap.ai/), which "publish[es] refusal rates for every model release from every major provider".[^speechmap] The cheap-and-capable Gemma refuses many contentious tasks; Chinese models (GLM, Qwen, DeepSeek) tend to refuse on Chinese political topics; US models on corporate or left-coded topics, with Grok among the exceptions. So check refusal on *your* eval's subject matter, not in the abstract, and re-read the live leaderboards rather than trusting these names, they date fast. +But a frontier score isn't sufficient: refusals wreck ambiguous or red-teaming evals, and refusal is topic-conditional. Check refusal rates on [speechmap.ai](https://speechmap.ai/), which "publish[es] refusal rates for every model release from every major provider".[^speechmap] Its per-lab Free Speech Index (0-100, higher = answers more; snapshot 2026-07-21) puts Mistral (88.9), xAI/Grok (85.8), and Google (81.1) most permissive, and among US majors Anthropic (53.8) and OpenAI (48.0) most restrictive; Chinese labs sit mid-to-high on this cross-topic aggregate (Zhipu/GLM 71.0, DeepSeek 59.0, Alibaba/Qwen 45.5) yet refuse specifically on Chinese-political topics. Two traps: the index is a lab average, so a single safety-tuned model (Gemma, per wassname) can refuse far more than its lab's number; and it aggregates topics, so it won't catch a refusal cluster on *your* eval's subject. Check refusal on your actual subject matter, and re-read the live leaderboards rather than trusting these names, they date fast. ## wassname's judge-validity checklist