mirror of
https://github.com/wassname/ml_debug.git
synced 2026-07-22 12:50:56 +08:00
1 line
2.3 KiB
JSON
1 line
2.3 KiB
JSON
{"root_cause":"Inference: the logits are probably being divided by an excessively large temperature, possibly through duplicate temperature scaling. This keeps softmax probabilities near uniform and suppresses cross-entropy gradients by approximately 1/T.","confidence":0.82,"evidence":["Observation: initial loss 2.303 is approximately ln(10), the uniform-prediction cross-entropy for 10 classes.","Observation: output entropy remains near ln(10), so the probabilities remain close to uniform.","Observation: the model can produce non-uniform logits, which makes a completely constant or disconnected head less likely.","Observation: gradients reach the classifier head but are much smaller than expected.","Inference: a low optimizer learning rate can slow parameter updates, but it does not reduce raw gradients measured before the optimizer step.","Inference: excessive temperature scaling explains both near-uniform probabilities and abnormally small gradients."],"competing_hypotheses":["A merely low learning rate would produce normal raw gradient magnitudes but undersized parameter updates.","An unintended loss reduction or extra division by batch size could shrink gradients, but would not directly explain persistently maximal output entropy.","Small upstream activations or initialization scale could produce small logits and slow learning, but is less specific than temperature scaling given the suppressed gradients."],"first_action":"On one fixed batch, run forward/backward twice without optimizer steps: once through the current loss path and once with temperature scaling removed (T=1). Record pre-temperature logit standard deviation, post-temperature logit standard deviation, cross-entropy, entropy, and classifier-head gradient norm. A large immediate gradient increase at T=1 is the cheapest discriminating test.","prediction_if_true":"Removing the scaling will reduce entropy below ln(10) when the raw logits contain signal and increase the classifier-head gradient norm by roughly the applied temperature factor, without changing the learning rate.","prediction_if_false":"The gradient norm will remain similarly small at T=1; inspect loss reductions, activation scale, and gradient normalization next. If gradients are normal but parameter updates are tiny, low learning rate becomes the leading explanation.","immediate_changes":[],"fallback_logic":[]} |