mirror of
https://github.com/wassname/ml_debug.git
synced 2026-07-22 12:50:56 +08:00
1 line
2.1 KiB
JSON
1 line
2.1 KiB
JSON
{"root_cause":"The dashboard is not measuring AdamW's actual parameter update. It uses learning_rate * raw gradient after optimizer.step(), but AdamW applies moment normalization and decoupled weight decay; its update generally differs substantially from that proxy. The approximately -8 values therefore describe tiny raw-gradient ratios, not the effective optimizer step.","confidence":0.99,"evidence":["Observation: the objective improves steadily.","Observation: parameter checksums change every step.","Observation: the dashboard reports log10(learning_rate * gradient / parameter) near -8 for every layer.","Observation: the numerator is computed from parameter.grad after optimizer.step().","Inference: optimization is occurring despite the reported ratios.","Inference: learning_rate * parameter.grad is an invalid estimator of an AdamW update because it omits the optimizer's moment normalization and decoupled weight-decay term."],"competing_hypotheses":["The logger reads stale, cleared, scaled, or otherwise post-processed gradients after the step; this is plausible but unnecessary to explain the discrepancy.","Checksum changes could theoretically come only from AdamW weight decay, but steady objective improvement makes this less likely."],"first_action":"For one representative parameter and one step, clone the parameter immediately before optimizer.step(), then log log10(norm(parameter_after - parameter_before) / norm(parameter_before)). This directly measures the update and is the cheapest decisive confirmation.","prediction_if_true":"The directly measured update/parameter ratio will be materially larger than 1e-8 and will track the changing parameters, while the existing learning_rate * gradient proxy remains near 1e-8.","prediction_if_false":"The directly measured ratio will also be near 1e-8; then inspect whether checksum sensitivity, weight decay, gradient accumulation, or another parameter mutation explains the apparent learning.","immediate_changes":["Replace the dashboard numerator with the norm of the observed parameter delta across optimizer.step(); label the metric as an actual update/parameter ratio."],"fallback_logic":[]} |