Merge dev4 into main

Removed redundant descriptions and clarified sections in the README.

SKILL.md

@@ -34,16 +34,6 @@ changes that will have separate and distinguishable effects on the metrics. What
 effect of each change given the others, so record it that way in the mental model. - wassname
 <!-- CLAUDE: last sentence is mine (Sculley's CACE, in README). -->
 
-With cheap feedback (say, a run under five minutes), isolate changes when that
-helps attribution. With hours between results, choose changes with different
-predicted effects in the log, rather than requiring a separate full run for each.
-Before running, record those predictions and possible interactions in the mental
-model. Afterwards, use the trajectories, demos and metrics to update your beliefs
-about each effect, conditional on the other changes. Distinct signatures help
-separate explanations; they do not guarantee independence. Isolate a change later
-when the remaining uncertainty matters to the next decision or a comparison claim.
-<!-- Pi: wassname's expensive-run approach; five minutes is an example, not a cutoff. -->
-
 ### How agents fail
 
 > Trying an experiment and seeing it fail gives little information by itself. When an experiment fails, it is tempting to conclude "I tried X and it didn't work". However, if X is a high-level conceptual approach, then a more correct conclusion is "I tried an implementation comprising 0.1% of the possible implementations of X, and observed that that particular implementation did not work". -- Steinhardt
@@ -123,13 +113,7 @@ he has not worked with it yet. -->
 
 ## The ml-debug form
 
-Fill this in and show it in full. Read the whole log first. This is an anti-skimming
+Fill this in and show it in full. Read the whole log first. Scoring:
-ritual for the agent, not an expectation that the user reads every row. Keep the
-end message brief: the decisive evidence, the next action, and the relevant
-pseudocode or code. That summary does not replace the full form.
-<!-- Pi: distinguish doing and showing the work from the user's review surface. -->
-
-Scoring:
 
 - a row answered from memory or expectation, with no quoted log line: 0
 - a row left blank, with no "unknown" and no note on what would fill it: 0

references/sweeps.md

@@ -1,50 +1,32 @@
-# Sweeps: learning what to try, and comparing results
+# Sweeps: same-seed comparison and cross-seed reliability
 
-Appendix to the [ML Debugging skill](../SKILL.md). Choose the experiment for the
+Appendix to the [ML Debugging skill](../SKILL.md). The general idea behind a trustworthy hyperparameter sweep, tool-agnostic. The point is the difference between "I tried it and it seemed better" and "it's reliably better across seeds." Irpan's 30% seed-failure result and Henderson's "seeds alone create statistically different distributions" (see the main skill's folklore section) are why this matters: a single lucky run proves nothing.
-question and iteration cost. Learning which direction to try next needs less
-certainty than claiming one method reliably beats another.
 
-## Expensive exploratory runs
+## The core move: pair on seed, normalize within group, test across seeds
 
-For cheap runs, changing one thing at a time often makes diagnosis easier. For
+1. Run the same set of seeds for every value of the parameter you're varying. Same seeds across values turns this into a paired comparison and cancels seed-level baseline differences.
-hours-long runs, batch changes when they have distinguishable predicted effects.
+2. Vary one parameter per sweep when you can (all-else-equal). If you vary two, effects confound and you can't attribute the result.
-Record what each should change in the logs before running, including possible
+3. Within each (group, seed), z-score the metric across the parameter values. This removes the per-seed baseline offset so you compare *shapes*, not absolute levels.
-interactions. Update the Bayesian mental model from trajectories and demos as well
+4. Aggregate the z-scores across seeds per value, then take a t-stat: `mean_z / (std_z / sqrt(n_seeds))`. `|t| > 2` with 4+ seeds is a real, reliable effect; `t ~ 0` is no consistent effect.
-as final metrics; keep unresolved explanations rather than forcing attribution.
+5. For numeric parameters, also fit a linear trend (Pearson r) and t-test it: a clean dose-response is `r` near +/-1 with a significant t-stat.
 
-For example, a loader change may predict less time waiting for data, while a
+```py
-regularisation change predicts a different train–validation gap. Seeing both
+for group in groups:
-supports those explanations, but does not prove the changes acted independently.
+    for seed in seeds_in_group:
-If both predict only a better final score, the log may not separate them. A later
+        vals = {param_value: metric for runs matching (group, seed, param)}
-isolated comparison is useful when that uncertainty changes what to do next.
+        z[seed] = (vals - mean(vals)) / std(vals)   # within-(group,seed) normalization
+    for value in param_values:
+        mean_z, std_z = mean(z[:, value]), std(z[:, value])
+        t_stat = mean_z / (std_z / sqrt(n_seeds))    # >>2 reliably better, <<-2 reliably worse
+```
 
-## Comparing configurations or methods
+## What you're looking for
 
-Keep the evaluation data, metric definition and relevant budget comparable.
+High effect size *and* a strong t-stat. A value with a big mean but `t=0.5` is a lucky seed; a value with a modest mean but `t=4.0` is a real (if small) effect.
-Pair runs where meaningful: reuse seeds and inputs, but check that the changed
-implementation has not changed what those seeds control. Pairing can reduce
-noise; it does not guarantee its cancellation.
 
-Inspect paired differences in the original metric units, their spread, and failed
+## Common pitfalls
-runs. A comparison of two bundles estimates the bundle difference, not each
-component's contribution. Distinguish exploratory selection from confirmation on
-seeds or data not used to select the winner.
 
-Within-seed z-scores can describe response shapes, but erase effect magnitude.
+- `n_seeds = 1`: t-stat is undefined. One data point. Replicate before concluding anything.
-With just two settings, they can give the winner the same normalized value on
+- Cross-group comparisons: different groups often have different base configs, so "group A's best value vs group B's best" is apples-to-oranges. Compare within groups.
-every seed despite very different raw gains. Do not use that artificial lack of
+- Too many parameters varied at once: split into separate sweeps.
-variance as evidence of reliability.
+- Crashed / diverged runs showing as missing or NaN metrics: investigate the run, don't silently drop it; a divergence is itself a finding.
-
-Describe uncertainty in the raw differences, with a method suited to the sample
-size and dependence. There is no universal “t > 2 with four seeds” guarantee.
-A noisy estimate leaves the effect uncertain; it does not establish either “no
-effect” or “a lucky seed.” One run can inform the next experiment without
-establishing reliability across seeds.
-
-Keep crashed or divergent runs visible and investigate them. Do not silently
-exclude them from the comparison.
-
-<!-- Pi: revised from wassname's exploratory-run direction and the reviewed
-within-seed normalization counterexample. Source background: README.md sections
-“Seed variance”, “Changing anything changes everything”, and “Exploration over
-exploitation”. -->