Clarify expensive-run exploration and sweep uncertainty

Preserve the full anti-skimming form and distinguish it from the brief end message. Describe batched changes with distinct predicted effects and conditional Bayesian updates. Replace normalized-score reliability verdicts with raw paired comparisons and explicit uncertainty.

Validation: diff --check; fresh-agent design scenarios exercised batching, normalization and full-form requirements. No actual training run or measured uplift test. Scenario agent placed the full form in the linked run log, leaving chat-display interpretation unresolved.

Co-Authored-By: Pi <288921227+claudypoo@users.noreply.github.com>

README.md

@@ -98,7 +98,7 @@ When you're stuck after a diagnostic cycle or two, the generalization of this ad
 
 > What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
 
-A bug can also hide, because most ML models have multiple adaptive parts:
+A bug can also hide, because most ML models have multiple adaptive parts: 
 
 > "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
 and it may not show in the output at all.
@@ -356,8 +356,8 @@ instead.
 From William Falcon's attendee notes on Schulman's talk, so a secondary source rather than
 Schulman's own text[^deeprlhacks]:
 
-> 4. Think your algorithm is working but you're actually seeing random noise.
+> 4. Think your algorithm is working but you're actually seeing random noise.   
->     - Example: Graph of 7 tasks with 3 algorithms and looks like 1 algorithm might be doing best on all problems, but turns out they're all the same algorithm with DIFFERENT random seeds.
+>     - Example: Graph of 7 tasks with 3 algorithms and looks like 1 algorithm might be doing best on all problems, but turns out they're all the same algorithm with DIFFERENT random seeds.   
 
 Nanda on why no internal warning fires:
 
@@ -486,7 +486,7 @@ reports his two real bugs on that environment were a terminal-flag masking error
 broadcast, neither of which any of these can reach[^reddit-rl]:
 
 > Things I've tried (but maybe not systematically enough):
->
+> 
 > * Different initial LRs
 > * Different optimizers
 > * Different number of hidden layers/units
@@ -512,7 +512,7 @@ wanted (the "demon-strated" break is an OCR artifact in the cached copy):
 notes[^deeprlhacks]:
 
 > 2. Make sure observations usable:
->     - See if YOU could control the system by using the same observations you give the agent.
+>     - See if YOU could control the system by using the same observations you give the agent.   
 >       - Example: Look at preprocessed images yourself to make sure you don't remove necessary details or hinder the algorithm in a certain way.
 
 For LLM work, the data you have to read is the tokenized data:

SKILL.md

@@ -34,6 +34,16 @@ changes that will have separate and distinguishable effects on the metrics. What
 effect of each change given the others, so record it that way in the mental model. - wassname
 <!-- CLAUDE: last sentence is mine (Sculley's CACE, in README). -->
 
+With cheap feedback (say, a run under five minutes), isolate changes when that
+helps attribution. With hours between results, choose changes with different
+predicted effects in the log, rather than requiring a separate full run for each.
+Before running, record those predictions and possible interactions in the mental
+model. Afterwards, use the trajectories, demos and metrics to update your beliefs
+about each effect, conditional on the other changes. Distinct signatures help
+separate explanations; they do not guarantee independence. Isolate a change later
+when the remaining uncertainty matters to the next decision or a comparison claim.
+<!-- Pi: wassname's expensive-run approach; five minutes is an example, not a cutoff. -->
+
 ### How agents fail
 
 > Trying an experiment and seeing it fail gives little information by itself. When an experiment fails, it is tempting to conclude "I tried X and it didn't work". However, if X is a high-level conceptual approach, then a more correct conclusion is "I tried an implementation comprising 0.1% of the possible implementations of X, and observed that that particular implementation did not work". -- Steinhardt
@@ -113,7 +123,13 @@ he has not worked with it yet. -->
 
 ## The ml-debug form
 
-Fill this in and show it in full. Read the whole log first. Scoring:
+Fill this in and show it in full. Read the whole log first. This is an anti-skimming
+ritual for the agent, not an expectation that the user reads every row. Keep the
+end message brief: the decisive evidence, the next action, and the relevant
+pseudocode or code. That summary does not replace the full form.
+<!-- Pi: distinguish doing and showing the work from the user's review surface. -->
+
+Scoring:
 
 - a row answered from memory or expectation, with no quoted log line: 0
 - a row left blank, with no "unknown" and no note on what would fill it: 0

docs/candidate_quotes.md

@@ -36,8 +36,8 @@ If either mode matters most to you, this cache needs a new source, not more mini
 - failure modes: 1
 - epistemic context: secondary source, attendee notes on Schulman's talk rather than Schulman's own text; the primary slide deck is cached separately as joschu_nuts_and_bolts.md.
 
-> 4. Think your algorithm is working but you're actually seeing random noise.
+> 4. Think your algorithm is working but you're actually seeing random noise.   
->     - Example: Graph of 7 tasks with 3 algorithms and looks like 1 algorithm might be doing best on all problems, but turns out they're all the same algorithm with DIFFERENT random seeds.
+>     - Example: Graph of 7 tasks with 3 algorithms and looks like 1 algorithm might be doing best on all problems, but turns out they're all the same algorithm with DIFFERENT random seeds.   
 
 Why it lands: a confident cross-task ranking read off three copies of one algorithm. It is the shortest demonstration that a conclusion can feel fully supported by a plot and be supported by nothing.
 
@@ -287,7 +287,7 @@ Why it lands: sweeping the legible knobs is brute-force search wearing a lab coa
 - epistemic context: LOW CREDIBILITY. Anonymous reddit self-report from a self-described non-expert. Its value is as a specimen of the failure mode, not as advice, and it should not be quoted as authority.
 
 > Things I've tried (but maybe not systematically enough):
->
+> 
 > * Different initial LRs
 > * Different optimizers
 > * Different number of hidden layers/units
@@ -328,7 +328,7 @@ Why it lands: a healthy-looking curve produced by a swimmer curling up and flail
 - epistemic context: secondary attendee notes; the matching primary slide is "Atari: can you see game features in downsampled image?" in the cached joschu_nuts_and_bolts.md.
 
 > 2. Make sure observations usable:
->     - See if YOU could control the system by using the same observations you give the agent.
+>     - See if YOU could control the system by using the same observations you give the agent.   
 >       - Example: Look at preprocessed images yourself to make sure you don't remove necessary details or hinder the algorithm in a certain way.
 
 Why it lands: turns "read the data" into a pass/fail test that takes a minute. If you cannot do the task from the model's inputs, no hyperparameter will save it.

references/sweeps.md

@@ -1,32 +1,50 @@
-# Sweeps: same-seed comparison and cross-seed reliability
+# Sweeps: learning what to try, and comparing results
 
-Appendix to the [ML Debugging skill](../SKILL.md). The general idea behind a trustworthy hyperparameter sweep, tool-agnostic. The point is the difference between "I tried it and it seemed better" and "it's reliably better across seeds." Irpan's 30% seed-failure result and Henderson's "seeds alone create statistically different distributions" (see the main skill's folklore section) are why this matters: a single lucky run proves nothing.
+Appendix to the [ML Debugging skill](../SKILL.md). Choose the experiment for the
+question and iteration cost. Learning which direction to try next needs less
+certainty than claiming one method reliably beats another.
 
-## The core move: pair on seed, normalize within group, test across seeds
+## Expensive exploratory runs
 
-1. Run the same set of seeds for every value of the parameter you're varying. Same seeds across values turns this into a paired comparison and cancels seed-level baseline differences.
+For cheap runs, changing one thing at a time often makes diagnosis easier. For
-2. Vary one parameter per sweep when you can (all-else-equal). If you vary two, effects confound and you can't attribute the result.
+hours-long runs, batch changes when they have distinguishable predicted effects.
-3. Within each (group, seed), z-score the metric across the parameter values. This removes the per-seed baseline offset so you compare *shapes*, not absolute levels.
+Record what each should change in the logs before running, including possible
-4. Aggregate the z-scores across seeds per value, then take a t-stat: `mean_z / (std_z / sqrt(n_seeds))`. `|t| > 2` with 4+ seeds is a real, reliable effect; `t ~ 0` is no consistent effect.
+interactions. Update the Bayesian mental model from trajectories and demos as well
-5. For numeric parameters, also fit a linear trend (Pearson r) and t-test it: a clean dose-response is `r` near +/-1 with a significant t-stat.
+as final metrics; keep unresolved explanations rather than forcing attribution.
 
-```py
+For example, a loader change may predict less time waiting for data, while a
-for group in groups:
+regularisation change predicts a different train–validation gap. Seeing both
-    for seed in seeds_in_group:
+supports those explanations, but does not prove the changes acted independently.
-        vals = {param_value: metric for runs matching (group, seed, param)}
+If both predict only a better final score, the log may not separate them. A later
-        z[seed] = (vals - mean(vals)) / std(vals)   # within-(group,seed) normalization
+isolated comparison is useful when that uncertainty changes what to do next.
-    for value in param_values:
-        mean_z, std_z = mean(z[:, value]), std(z[:, value])
-        t_stat = mean_z / (std_z / sqrt(n_seeds))    # >>2 reliably better, <<-2 reliably worse
-```
 
-## What you're looking for
+## Comparing configurations or methods
 
-High effect size *and* a strong t-stat. A value with a big mean but `t=0.5` is a lucky seed; a value with a modest mean but `t=4.0` is a real (if small) effect.
+Keep the evaluation data, metric definition and relevant budget comparable.
+Pair runs where meaningful: reuse seeds and inputs, but check that the changed
+implementation has not changed what those seeds control. Pairing can reduce
+noise; it does not guarantee its cancellation.
 
-## Common pitfalls
+Inspect paired differences in the original metric units, their spread, and failed
+runs. A comparison of two bundles estimates the bundle difference, not each
+component's contribution. Distinguish exploratory selection from confirmation on
+seeds or data not used to select the winner.
 
-- `n_seeds = 1`: t-stat is undefined. One data point. Replicate before concluding anything.
+Within-seed z-scores can describe response shapes, but erase effect magnitude.
-- Cross-group comparisons: different groups often have different base configs, so "group A's best value vs group B's best" is apples-to-oranges. Compare within groups.
+With just two settings, they can give the winner the same normalized value on
-- Too many parameters varied at once: split into separate sweeps.
+every seed despite very different raw gains. Do not use that artificial lack of
-- Crashed / diverged runs showing as missing or NaN metrics: investigate the run, don't silently drop it; a divergence is itself a finding.
+variance as evidence of reliability.
+
+Describe uncertainty in the raw differences, with a method suited to the sample
+size and dependence. There is no universal “t > 2 with four seeds” guarantee.
+A noisy estimate leaves the effect uncertain; it does not establish either “no
+effect” or “a lucky seed.” One run can inform the next experiment without
+establishing reliability across seeds.
+
+Keep crashed or divergent runs visible and investigate them. Do not silently
+exclude them from the comparison.
+
+<!-- Pi: revised from wassname's exploratory-run direction and the reviewed
+within-seed normalization counterexample. Source background: README.md sections
+“Seed variance”, “Changing anything changes everything”, and “Exploration over
+exploitation”. -->