Merge dev4 into main

Removed redundant descriptions and clarified sections in the README.

README.md

@@ -98,7 +98,7 @@ When you're stuck after a diagnostic cycle or two, the generalization of this ad
 
 > What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
 
-A bug can also hide, because most ML models have multiple adaptive parts: 
+A bug can also hide, because most ML models have multiple adaptive parts:
 
 > "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
 and it may not show in the output at all.
@@ -356,8 +356,8 @@ instead.
 From William Falcon's attendee notes on Schulman's talk, so a secondary source rather than
 Schulman's own text[^deeprlhacks]:
 
-> 4. Think your algorithm is working but you're actually seeing random noise.   
+> 4. Think your algorithm is working but you're actually seeing random noise.
->     - Example: Graph of 7 tasks with 3 algorithms and looks like 1 algorithm might be doing best on all problems, but turns out they're all the same algorithm with DIFFERENT random seeds.   
+>     - Example: Graph of 7 tasks with 3 algorithms and looks like 1 algorithm might be doing best on all problems, but turns out they're all the same algorithm with DIFFERENT random seeds.
 
 Nanda on why no internal warning fires:
 
@@ -486,7 +486,7 @@ reports his two real bugs on that environment were a terminal-flag masking error
 broadcast, neither of which any of these can reach[^reddit-rl]:
 
 > Things I've tried (but maybe not systematically enough):
-> 
+>
 > * Different initial LRs
 > * Different optimizers
 > * Different number of hidden layers/units
@@ -512,7 +512,7 @@ wanted (the "demon-strated" break is an OCR artifact in the cached copy):
 notes[^deeprlhacks]:
 
 > 2. Make sure observations usable:
->     - See if YOU could control the system by using the same observations you give the agent.   
+>     - See if YOU could control the system by using the same observations you give the agent.
 >       - Example: Look at preprocessed images yourself to make sure you don't remove necessary details or hinder the algorithm in a certain way.
 
 For LLM work, the data you have to read is the tokenized data:

SKILL.md

@@ -34,16 +34,6 @@ changes that will have separate and distinguishable effects on the metrics. What
 effect of each change given the others, so record it that way in the mental model. - wassname
 <!-- CLAUDE: last sentence is mine (Sculley's CACE, in README). -->
 
-With cheap feedback (say, a run under five minutes), isolate changes when that
-helps attribution. With hours between results, choose changes with different
-predicted effects in the log, rather than requiring a separate full run for each.
-Before running, record those predictions and possible interactions in the mental
-model. Afterwards, use the trajectories, demos and metrics to update your beliefs
-about each effect, conditional on the other changes. Distinct signatures help
-separate explanations; they do not guarantee independence. Isolate a change later
-when the remaining uncertainty matters to the next decision or a comparison claim.
-<!-- Pi: wassname's expensive-run approach; five minutes is an example, not a cutoff. -->
-
 ### How agents fail
 
 > Trying an experiment and seeing it fail gives little information by itself. When an experiment fails, it is tempting to conclude "I tried X and it didn't work". However, if X is a high-level conceptual approach, then a more correct conclusion is "I tried an implementation comprising 0.1% of the possible implementations of X, and observed that that particular implementation did not work". -- Steinhardt
@@ -123,13 +113,7 @@ he has not worked with it yet. -->
 
 ## The ml-debug form
 
-Fill this in and show it in full. Read the whole log first. This is an anti-skimming
+Fill this in and show it in full. Read the whole log first. Scoring:
-ritual for the agent, not an expectation that the user reads every row. Keep the
-end message brief: the decisive evidence, the next action, and the relevant
-pseudocode or code. That summary does not replace the full form.
-<!-- Pi: distinguish doing and showing the work from the user's review surface. -->
-
-Scoring:
 
 - a row answered from memory or expectation, with no quoted log line: 0
 - a row left blank, with no "unknown" and no note on what would fill it: 0

docs/candidate_quotes.md

@@ -36,8 +36,8 @@ If either mode matters most to you, this cache needs a new source, not more mini
 - failure modes: 1
 - epistemic context: secondary source, attendee notes on Schulman's talk rather than Schulman's own text; the primary slide deck is cached separately as joschu_nuts_and_bolts.md.
 
-> 4. Think your algorithm is working but you're actually seeing random noise.   
+> 4. Think your algorithm is working but you're actually seeing random noise.
->     - Example: Graph of 7 tasks with 3 algorithms and looks like 1 algorithm might be doing best on all problems, but turns out they're all the same algorithm with DIFFERENT random seeds.   
+>     - Example: Graph of 7 tasks with 3 algorithms and looks like 1 algorithm might be doing best on all problems, but turns out they're all the same algorithm with DIFFERENT random seeds.
 
 Why it lands: a confident cross-task ranking read off three copies of one algorithm. It is the shortest demonstration that a conclusion can feel fully supported by a plot and be supported by nothing.
 
@@ -287,7 +287,7 @@ Why it lands: sweeping the legible knobs is brute-force search wearing a lab coa
 - epistemic context: LOW CREDIBILITY. Anonymous reddit self-report from a self-described non-expert. Its value is as a specimen of the failure mode, not as advice, and it should not be quoted as authority.
 
 > Things I've tried (but maybe not systematically enough):
-> 
+>
 > * Different initial LRs
 > * Different optimizers
 > * Different number of hidden layers/units
@@ -328,7 +328,7 @@ Why it lands: a healthy-looking curve produced by a swimmer curling up and flail
 - epistemic context: secondary attendee notes; the matching primary slide is "Atari: can you see game features in downsampled image?" in the cached joschu_nuts_and_bolts.md.
 
 > 2. Make sure observations usable:
->     - See if YOU could control the system by using the same observations you give the agent.   
+>     - See if YOU could control the system by using the same observations you give the agent.
 >       - Example: Look at preprocessed images yourself to make sure you don't remove necessary details or hinder the algorithm in a certain way.
 
 Why it lands: turns "read the data" into a pass/fail test that takes a minute. If you cannot do the task from the model's inputs, no hyperparameter will save it.

references/sweeps.md

@@ -1,50 +1,32 @@
-# Sweeps: learning what to try, and comparing results
+# Sweeps: same-seed comparison and cross-seed reliability
 
-Appendix to the [ML Debugging skill](../SKILL.md). Choose the experiment for the
+Appendix to the [ML Debugging skill](../SKILL.md). The general idea behind a trustworthy hyperparameter sweep, tool-agnostic. The point is the difference between "I tried it and it seemed better" and "it's reliably better across seeds." Irpan's 30% seed-failure result and Henderson's "seeds alone create statistically different distributions" (see the main skill's folklore section) are why this matters: a single lucky run proves nothing.
-question and iteration cost. Learning which direction to try next needs less
-certainty than claiming one method reliably beats another.
 
-## Expensive exploratory runs
+## The core move: pair on seed, normalize within group, test across seeds
 
-For cheap runs, changing one thing at a time often makes diagnosis easier. For
+1. Run the same set of seeds for every value of the parameter you're varying. Same seeds across values turns this into a paired comparison and cancels seed-level baseline differences.
-hours-long runs, batch changes when they have distinguishable predicted effects.
+2. Vary one parameter per sweep when you can (all-else-equal). If you vary two, effects confound and you can't attribute the result.
-Record what each should change in the logs before running, including possible
+3. Within each (group, seed), z-score the metric across the parameter values. This removes the per-seed baseline offset so you compare *shapes*, not absolute levels.
-interactions. Update the Bayesian mental model from trajectories and demos as well
+4. Aggregate the z-scores across seeds per value, then take a t-stat: `mean_z / (std_z / sqrt(n_seeds))`. `|t| > 2` with 4+ seeds is a real, reliable effect; `t ~ 0` is no consistent effect.
-as final metrics; keep unresolved explanations rather than forcing attribution.
+5. For numeric parameters, also fit a linear trend (Pearson r) and t-test it: a clean dose-response is `r` near +/-1 with a significant t-stat.
 
-For example, a loader change may predict less time waiting for data, while a
+```py
-regularisation change predicts a different train–validation gap. Seeing both
+for group in groups:
-supports those explanations, but does not prove the changes acted independently.
+    for seed in seeds_in_group:
-If both predict only a better final score, the log may not separate them. A later
+        vals = {param_value: metric for runs matching (group, seed, param)}
-isolated comparison is useful when that uncertainty changes what to do next.
+        z[seed] = (vals - mean(vals)) / std(vals)   # within-(group,seed) normalization
+    for value in param_values:
+        mean_z, std_z = mean(z[:, value]), std(z[:, value])
+        t_stat = mean_z / (std_z / sqrt(n_seeds))    # >>2 reliably better, <<-2 reliably worse
+```
 
-## Comparing configurations or methods
+## What you're looking for
 
-Keep the evaluation data, metric definition and relevant budget comparable.
+High effect size *and* a strong t-stat. A value with a big mean but `t=0.5` is a lucky seed; a value with a modest mean but `t=4.0` is a real (if small) effect.
-Pair runs where meaningful: reuse seeds and inputs, but check that the changed
-implementation has not changed what those seeds control. Pairing can reduce
-noise; it does not guarantee its cancellation.
 
-Inspect paired differences in the original metric units, their spread, and failed
+## Common pitfalls
-runs. A comparison of two bundles estimates the bundle difference, not each
-component's contribution. Distinguish exploratory selection from confirmation on
-seeds or data not used to select the winner.
 
-Within-seed z-scores can describe response shapes, but erase effect magnitude.
+- `n_seeds = 1`: t-stat is undefined. One data point. Replicate before concluding anything.
-With just two settings, they can give the winner the same normalized value on
+- Cross-group comparisons: different groups often have different base configs, so "group A's best value vs group B's best" is apples-to-oranges. Compare within groups.
-every seed despite very different raw gains. Do not use that artificial lack of
+- Too many parameters varied at once: split into separate sweeps.
-variance as evidence of reliability.
+- Crashed / diverged runs showing as missing or NaN metrics: investigate the run, don't silently drop it; a divergence is itself a finding.
-
-Describe uncertainty in the raw differences, with a method suited to the sample
-size and dependence. There is no universal “t > 2 with four seeds” guarantee.
-A noisy estimate leaves the effect uncertain; it does not establish either “no
-effect” or “a lucky seed.” One run can inform the next experiment without
-establishing reliability across seeds.
-
-Keep crashed or divergent runs visible and investigate them. Do not silently
-exclude them from the comparison.
-
-<!-- Pi: revised from wassname's exploratory-run direction and the reviewed
-within-seed normalization counterexample. Source background: README.md sections
-“Seed variance”, “Changing anything changes everything”, and “Exploration over
-exploitation”. -->