wassname/ml-debug
1
0
Fork 0
mirror of https://github.com/wassname/ml-debug.git synced 2026-08-06 13:20:45 +08:00
Code Issues Packages Projects Releases Wiki Activity

Add real SpeechMap per-lab refusal index to judge-selection

Browse Source 
Replace the qualitative refusal claim with SpeechMap's per-lab Free Speech
Index (Mistral/xAI/Google most permissive; Anthropic/OpenAI most restrictive
majors; Chinese labs mid-high on the aggregate but topic-censored). Flag the
two traps: it's a lab average (hides a safety-tuned Gemma) and a cross-topic
aggregate (hides a per-subject refusal cluster). Cache the table.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
wassname
2026-07-22 16:16:42 +08:00
co-authored by Claudypoo
parent 54b8a2bb79
commit 01db376bca
2 changed files with 29 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
refs/llm_judges.md
+1 -1
View File
@@ -33,7 +33,7 @@ Pick from the cost-vs-score Pareto frontier of a judging leaderboard, and prefer
From the checked-in v4 scores (snapshot 2026-07, [source](https://github.com/EQ-bench/EQ-bench-site/blob/main/judgemark-v4.js)) the cost-vs-score frontier runs from the top absolute scorers, claude-opus-4-6 (0.91, ~$39) and gpt-5.5 (0.88, ~$30), down through claude-sonnet-4-6 and gemini-3.1-pro (~0.8, ~$23), grok-4.5 (0.77, $17) and GLM-5.2 (0.73, $8), to the cheap knee google/gemma-4-31b (0.72 at $0.82), which nearly matches models 20-40x its price.
But a frontier score isn't sufficient: refusals wreck ambiguous or red-teaming evals, and refusal is topic-conditional. Check refusal rates on [speechmap.ai](https://speechmap.ai/), which "publish[es] refusal rates for every model release from every major provider".[^speechmap] The cheap-and-capable Gemma refuses many contentious tasks; Chinese models (GLM, Qwen, DeepSeek) tend to refuse on Chinese political topics; US models on corporate or left-coded topics, with Grok among the exceptions. So check refusal on *your* eval's subject matter, not in the abstract, and re-read the live leaderboards rather than trusting these names, they date fast.
But a frontier score isn't sufficient: refusals wreck ambiguous or red-teaming evals, and refusal is topic-conditional. Check refusal rates on [speechmap.ai](https://speechmap.ai/), which "publish[es] refusal rates for every model release from every major provider".[^speechmap] Its per-lab Free Speech Index (0-100, higher = answers more; snapshot 2026-07-21) puts Mistral (88.9), xAI/Grok (85.8), and Google (81.1) most permissive, and among US majors Anthropic (53.8) and OpenAI (48.0) most restrictive; Chinese labs sit mid-to-high on this cross-topic aggregate (Zhipu/GLM 71.0, DeepSeek 59.0, Alibaba/Qwen 45.5) yet refuse specifically on Chinese-political topics. Two traps: the index is a lab average, so a single safety-tuned model (Gemma, per wassname) can refuse far more than its lab's number; and it aggregates topics, so it won't catch a refusal cluster on *your* eval's subject. Check refusal on your actual subject matter, and re-read the live leaderboards rather than trusting these names, they date fast.
## wassname's judge-validity checklist