wassname/ml_debug
1
0
Fork 0
mirror of https://github.com/wassname/ml_debug.git synced 2026-09-09 11:27:07 +08:00
Code Issues Packages Projects Releases Wiki Activity

SKILL.md: quote Sanh on the threshold mistake, the one bullet left bare

Browse Source
This commit is contained in:
wassname
2026-08-26 14:16:25 +08:00
parent d769cacfd4
commit cafc1c89be
1 changed files with 4 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
SKILL.md
+4
View File
@@ -66,6 +66,10 @@ easy to make a mistake with cosine. It is not causal, and two different subspace
even when they are correlated, so `cos(apple, orange) = 0` is not a null result. Building the real
thing and running it takes longer and answers the question. Exercise 2.
> * How would a random predictor perform (especially in classification problems)? Dataset can be unbalanced...
> * What would the loss look like for a random predictor?
> * What are the limits of this metric? If it's perfect, what can I conclude? What can't I conclude? -- Sanh
Do not fix on an arbitrary metric threshold before you have any idea what a fair or good threshold
is. Saying the metric must clear 0.8 means nothing until you know what counts as good here. Get the
scale first, from a null arm and a shuffled control. Exercise 15.