mirror of
https://github.com/wassname/ml-debug.git
synced 2026-07-25 13:20:33 +08:00
Add real SpeechMap per-lab refusal index to judge-selection
Replace the qualitative refusal claim with SpeechMap's per-lab Free Speech Index (Mistral/xAI/Google most permissive; Anthropic/OpenAI most restrictive majors; Chinese labs mid-high on the aggregate but topic-censored). Flag the two traps: it's a lab average (hides a safety-tuned Gemma) and a cross-topic aggregate (hides a per-subject refusal cluster). Cache the table. Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
@@ -128,3 +128,31 @@ Data rows from https://github.com/EQ-bench/EQ-bench-site/blob/main/judgemark-v4.
|
||||
| Qwen/Qwen3.5-9B | 0.324 | $0.56 | cheapest |
|
||||
|
||||
Off-frontier for reference: Qwen3.5-27B 0.605 ($1.76), DeepSeek-V4-Pro 0.471 ($2.94), grok-4.3 0.496 ($9.71). So in v4, Qwen and DeepSeek are mid-pack, not frontier; grok-4.5 and gemma carry the value case.
|
||||
|
||||
## SpeechMap per-lab Free Speech Index (snapshot 2026-07-21) — https://speechmap.ai/labs/
|
||||
|
||||
Higher = answers more / refuses less (0-100), windowed to releases in the last 6 months; "Index" is the windowed score, "Peak" the best single model. This is a cross-topic aggregate and a lab average, so it does not resolve per-model or per-topic refusal (e.g. Chinese labs score mid-to-high here but still refuse on Chinese-political topics; a lab's aggregate can hide a heavily safety-tuned model like Gemma). Copied verbatim from the table wassname pasted:
|
||||
|
||||
| Rank | Lab | Index | Peak | Models |
|
||||
|-|-|-|-|-|
|
||||
| 1 | Mistral AI | 88.9 | 98.2 | 9 |
|
||||
| 2 | xAI | 85.8 | 98.2 | 5 |
|
||||
| 3 | IBM | 84.0 | 84.0 | 1 |
|
||||
| 4 | Google DeepMind | 81.1 | 88.4 | 9 |
|
||||
| 5 | Meta | 76.8 | 76.8 | 1 |
|
||||
| 6 | Zhipu AI (GLM) | 71.0 | 85.9 | 12 |
|
||||
| 7 | Arcee AI | 70.1 | 82.2 | 3 |
|
||||
| 8 | Meituan | 63.6 | 78.2 | 2 |
|
||||
| 9 | Tencent | 60.3 | 76.3 | 4 |
|
||||
| 10 | DeepSeek | 59.0 | 89.1 | 7 |
|
||||
| 11 | Moonshot AI | 57.6 | 65.9 | 5 |
|
||||
| 13 | Anthropic | 53.8 | 71.5 | 10 |
|
||||
| 16 | OpenAI | 48.0 | 66.7 | 15 |
|
||||
| 19 | Alibaba (Qwen) | 45.5 | 60.8 | 13 |
|
||||
| 20 | NVIDIA | 45.3 | 67.7 | 7 |
|
||||
| 23 | Xiaomi | 43.9 | 62.6 | 10 |
|
||||
| 24 | ByteDance | 32.1 | 34.4 | 3 |
|
||||
|
||||
Note the wide Peak-vs-Index gaps (DeepSeek peak 89.1 vs index 59.0; Zhipu 85.9 vs 71.0): pick the specific permissive checkpoint, not the lab.
|
||||
|
||||
Not fetched: SpeechMap's per-model and per-topic breakdowns (https://speechmap.ai/timeline/, /labs/<lab>/) and the eqbench4 EI benchmark data (https://github.com/EQ-bench/EQ-bench-site/blob/main/eqbench4/eqbench4_data.js) are separate from the judge data used here.
|
||||
|
||||
+1
-1
@@ -33,7 +33,7 @@ Pick from the cost-vs-score Pareto frontier of a judging leaderboard, and prefer
|
||||
|
||||
From the checked-in v4 scores (snapshot 2026-07, [source](https://github.com/EQ-bench/EQ-bench-site/blob/main/judgemark-v4.js)) the cost-vs-score frontier runs from the top absolute scorers, claude-opus-4-6 (0.91, ~$39) and gpt-5.5 (0.88, ~$30), down through claude-sonnet-4-6 and gemini-3.1-pro (~0.8, ~$23), grok-4.5 (0.77, $17) and GLM-5.2 (0.73, $8), to the cheap knee google/gemma-4-31b (0.72 at $0.82), which nearly matches models 20-40x its price.
|
||||
|
||||
But a frontier score isn't sufficient: refusals wreck ambiguous or red-teaming evals, and refusal is topic-conditional. Check refusal rates on [speechmap.ai](https://speechmap.ai/), which "publish[es] refusal rates for every model release from every major provider".[^speechmap] The cheap-and-capable Gemma refuses many contentious tasks; Chinese models (GLM, Qwen, DeepSeek) tend to refuse on Chinese political topics; US models on corporate or left-coded topics, with Grok among the exceptions. So check refusal on *your* eval's subject matter, not in the abstract, and re-read the live leaderboards rather than trusting these names, they date fast.
|
||||
But a frontier score isn't sufficient: refusals wreck ambiguous or red-teaming evals, and refusal is topic-conditional. Check refusal rates on [speechmap.ai](https://speechmap.ai/), which "publish[es] refusal rates for every model release from every major provider".[^speechmap] Its per-lab Free Speech Index (0-100, higher = answers more; snapshot 2026-07-21) puts Mistral (88.9), xAI/Grok (85.8), and Google (81.1) most permissive, and among US majors Anthropic (53.8) and OpenAI (48.0) most restrictive; Chinese labs sit mid-to-high on this cross-topic aggregate (Zhipu/GLM 71.0, DeepSeek 59.0, Alibaba/Qwen 45.5) yet refuse specifically on Chinese-political topics. Two traps: the index is a lab average, so a single safety-tuned model (Gemma, per wassname) can refuse far more than its lab's number; and it aggregates topics, so it won't catch a refusal cluster on *your* eval's subject. Check refusal on your actual subject matter, and re-read the live leaderboards rather than trusting these names, they date fast.
|
||||
|
||||
## wassname's judge-validity checklist
|
||||
|
||||
|
||||
Reference in New Issue
Block a user