From d7536fe54937f3935c1ef5a74556cbb5b3735384 Mon Sep 17 00:00:00 2001 From: wassname <1103714+wassname@users.noreply.github.com> Date: Wed, 26 Aug 2026 14:23:18 +0800 Subject: [PATCH] SKILL.md: annoy-less comment review on the AI-written prose Comment review mode only, no prose changed. Flags negative framing, aphoristic closers, and three places where the rewrite made wassname's hedged claims stronger than his original message. Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com> --- SKILL.md | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/SKILL.md b/SKILL.md index 6fa71a2..567398b 100644 --- a/SKILL.md +++ b/SKILL.md @@ -25,6 +25,9 @@ code, the pasted sample. Write "unknown" in a cell you cannot fill, and say what Give the source of each number. Never stop a job or give up on an idea without doing all of these. One at a time, not all at once. + + ## Common mistakes @@ -32,6 +35,14 @@ Everyone makes these, and I have made most of them myself. They come up so often long autoresearch runs that they are worth naming, so you can catch yourself early rather than after a week of work. Reading the log and hunting for your own bug are the two that do most of the damage, so start there when you are not sure where to start. - wassname + + + > Insufficient skepticism doesn't *feel* like insufficient skepticism from the inside. It just feels like doing research. -- Nanda @@ -40,16 +51,28 @@ so start there when you are not sure where to start. - wassname Be careful about being overconfident. It is easy to write a diagnosis in the tone of a fact. Before you commit to one, ask what you saw that a competing explanation could not also explain. If nothing, then "I do not know, and here is what would tell me" is a good answer and not a failure. Exercise 7. + + Do not quit after the first change and call the negative real. One failed attempt is much more likely to be a bug in your implementation than a refutation of the idea. This is the expensive mistake, because the idea gets thrown away and nobody goes back to it. Look for the bug first. Exercise 14. + + Try not to stop at the first idea you come up with. It arrives with no competition, so it wins by default rather than on merit. Write down two more, and say what observation would separate them. If you cannot name a test that distinguishes them, you have a preference and not a hypothesis. Exercises 6 and 7. + + + > If it doesn't work, assume there's a bug. Spend a lot of effort searching for bugs before you resort to tweaking hyperparameters: usually it's a bug. Bad hyperparameters can significantly degrade RL performance, but if you're using hyperparameters similar to the ones in papers and standard implementations, those will probably not be the issue. -- Achiam @@ -57,18 +80,31 @@ Watch out for getting obsessed with the legible hyperparameters. Learning rate, warmup are easy to name and easy to change, so they attract more attention than they deserve. More often the cause is in the data, a sign, a mask, an index, or a metric that answers a different question from the one you asked. Exercises 5 and 10. + + Please read the data. Print the first full training sample, chosen and rejected, with the special tokens and the loss mask showing. Look at it with your own eyes. Most formatting bugs are obvious in the first sample and invisible in every aggregate. Exercise 3. + + Please read the log. Not the last twenty lines, the log. Find the first line where the run stopped matching what you expected, quote it, and start from there. Exercises 1 and 11. + + Be wary of reaching for a cosine probe instead of building the training script with metrics. It is easy to make a mistake with cosine. It is not causal, and two different subspaces score near zero even when they are correlated, so `cos(apple, orange) = 0` is not a null result. Building the real thing and running it takes longer and answers the question. Exercise 2. + + > * How would a random predictor perform (especially in classification problems)? Dataset can be unbalanced... > * What would the loss look like for a random predictor? @@ -77,6 +113,9 @@ thing and running it takes longer and answers the question. Exercise 2. Do not fix on an arbitrary metric threshold before you have any idea what a fair or good threshold is. Saying the metric must clear 0.8 means nothing until you know what counts as good here. Get the scale first, from a null arm and a shuffled control. Exercise 15. + + ## How this applies to LLM agents @@ -96,6 +135,9 @@ line for each cell. Show: |---|---|---|---|---|---|---| An empty cell is a metric that does not exist. Add the metric before the next run. + + ## 2. "Raising the threshold at which you start thinking 'OK, I think this is correct'" (small) @@ -169,6 +211,9 @@ Show three ways the result can be false, each with the check that decides it. To B, give the baseline, the chance level, and the seed spread of one arm. One seed per arm is unresolved. Give a fresh subagent the artifact with no conclusion attached and show what it says. Apply the same to a negative result: a bad row is a bug until the log shows otherwise. + + ## 9. "Implementation differences ... can have dramatic impacts" (large) @@ -233,6 +278,10 @@ write the negative up. |---|---|---|---| One attempt is untested, not negative. Say which of the two this is. + + ## 15. "By default, all numbers are meaningless because we lack any scale" (large) @@ -247,6 +296,9 @@ arm, a shuffled or permuted control, and the existing baseline, then set the bar |---|---|---|---|---|---| A gate chosen before this table is a number you made up. Say so if you have to use one anyway. + + ## Reference