From b2c666dbbf1cdb5120ff18cb4dbd57b3cb1cb4e1 Mon Sep 17 00:00:00 2001 From: wassname <1103714+wassname@users.noreply.github.com> Date: Sat, 15 Aug 2026 06:40:45 +0800 Subject: [PATCH] add wassname's 37-reasons checklist to refs/checklist.md Quoted whole from the 2017 thread, since the article author asked to merge it and never did. Verified line by line against the thread cache. Covers sample size from a cumulative-mean plot, KLD/Dice on unbalanced data, augmentation bounded by feature std, dummy metrics, jumpy validation loss, testing the framework itself, activation swaps, and loss-curve shapes. --- SKILL.md | 2 +- docs/evidence/reddit_37_reasons_nn_6pfsyk.md | 2 +- refs/checklist.md | 33 ++++++++++++++++++++ 3 files changed, 35 insertions(+), 2 deletions(-) diff --git a/SKILL.md b/SKILL.md index 16b23a8..b0ea68f 100644 --- a/SKILL.md +++ b/SKILL.md @@ -332,7 +332,7 @@ Their training-stability page adds the masking check ("inspect tokenized samples Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system: - [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns. -- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting. +- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting, plus wassname's extra checks from the 37-reasons thread (sample size from a cumulative-mean plot, dummy metrics, jumpy validation loss, activation swaps, loss-curve shapes). - [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check. - [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage). - [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves. diff --git a/docs/evidence/reddit_37_reasons_nn_6pfsyk.md b/docs/evidence/reddit_37_reasons_nn_6pfsyk.md index 092c261..b4f62ba 100644 --- a/docs/evidence/reddit_37_reasons_nn_6pfsyk.md +++ b/docs/evidence/reddit_37_reasons_nn_6pfsyk.md @@ -5,7 +5,7 @@ Fetch-status: all 22 comments, verbatim. Nesting is flattened; scores not captur # [P] 37 Reasons why your NN is not working -Companion thread to [slavv_37_reasons_nn.md](slavv_37_reasons_nn.md). It matters because a commenter posted 13 extra checks here, the author replied "Do you mind if I add them to the article?", and the article never did add them: checking the 2025 archived copy, only the batch-size point overlaps. So this thread, not the article, is the source for those checks. +Companion thread to [slavv_37_reasons_nn.md](slavv_37_reasons_nn.md). It matters because u/tinkerWithoutSink (wassname, this repo's author) posted 13 extra checks here, the author replied "Do you mind if I add them to the article?", and the article never did add them: checking the 2025 archived copy, only the batch-size point overlaps. So this thread, not the article, is the source for those checks. ## u/slavivanov diff --git a/refs/checklist.md b/refs/checklist.md index 2495dde..f8bfebe 100644 --- a/refs/checklist.md +++ b/refs/checklist.md @@ -62,3 +62,36 @@ Two especially common leak routes: > The best thing you can do to prevent these issues is to partition off a subset of your data right at the start of your project, and only use this independent test set once to measure the generality of a single model at the end. > Most notably, time series data are subject to a particular kind of data leakage known as look ahead bias. + + +## Extra checks from the 37-reasons thread (wassname, 2017) + +Slav Ivanov's "37 Reasons why your Neural Network is not working" drew a reply from +wassname (u/tinkerWithoutSink) with further checks. Ivanov asked "Do you mind if I add +them to the article?" and never did, so this is the only place they live. Quoted from +[the thread cache](../docs/evidence/reddit_37_reasons_nn_6pfsyk.md); the numbers refer +to items in the original article. + +> - I. Sample size: you can work out the minimum sample size by graphing the cumulative mean or std and seeing when it stabilized. It it converges on 256, then that's probably a good batch (not sure about this and batches). And the minimum size for your training data. +> - 8. Loss for unbalanced data. I'll add that when you can't balance the dataset KLD and Dice loss help to get convergence on unbalanced data +> - 11. Small batches. You don't want batches that are too small either right (serious question)? I figure that if they are a decent sample of your data then that will help, but I'm not sure +> - 12. How much data augmentation is too much, I use simple hypterparam optimization and a scikit learn model to test this. You can look at the standard deviation of a data feature and try not to exceed that for risk of drowning out signal with noise. +> - III architecture mistakes +> - [have dropout *after* pooling](https://www.reddit.com/r/MachineLearning/comments/46b8dz/what_does_debugging_a_deep_net_look_like/d04qyqm/) +> - 17. I Use dummy metrics too, http://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html +> - 21. +> - If your validation loss is jumping around, then your validation set is too small +> - If your validation accuracy is higher than you training accuracy... actually this one has me stumped? +> - 22. Test frameworks. Too many DL and RL frameworks are broken, so it might be worth testing frameworks too +> - 33. You didn't mentioned different activations. +> - I've noticed that if your loss if fluctuating up and down try using Elu instead of ReLU. This is because ReLU masks half the data, and so the model might be flipping between masking one of two modes +> - sigmoidal (sigmoid, tanh) activation units, which can saturate/have regions of near flat curvature and thus very little gradient gets propagated backwards, so learning is incredibly slow if not completely halted [src](http://stats.stackexchange.com/questions/163600/pre-training-in-deep-convolutional-neural-network) +> - you can always try linear activations as a sanity check +> - loss curves. This has been done but you might want to think about diagnosing differen't loss curves e.g. +> - 1) a sharp drop in loss at the start (bad init?) +> - 2) fluctuating loss (bad activation?) +> - 3) increasing loss (high learning rate?) + +The validation-accuracy question was answered in the same thread: it happens when +regularizers, dropout and batch norm are active in training and switched off at +evaluation, so the training number is measured on a handicapped model.