mirror of
https://github.com/wassname/ml-debug.git
synced 2026-09-26 14:00:26 +08:00
1 line
2.0 KiB
JSON
1 line
2.0 KiB
JSON
{"root_cause":"Inference: the validation path is almost certainly not evaluating the claimed randomly initialized classifier. The leading cause is target leakage or reuse of cached/stale trained logits in the loss-and-metric pipeline.","confidence":0.97,"evidence":["Observation: uniform predictions over 100 classes imply cross-entropy near ln(100) = 4.605, not 0.04.","Observation: a random 100-class classifier should have approximately 1% accuracy, not 96%.","Observation: the low loss and high accuracy agree with each other, suggesting the same incorrect or leaked predictions feed both metrics.","Observation: inspecting a separately loaded example does not verify the tensors, targets, logits, or metric state used by the validation loader."],"competing_hypotheses":["A trained checkpoint is loaded despite the model being described as randomly initialized.","Validation metrics are stale or accumulated from an earlier trained evaluation.","The validation loader or evaluation code substitutes labels, one-hot targets, or cached predictions for model logits.","The reported values come from the wrong run or logger series."],"first_action":"Take one batch directly from the validation loader, run one forward pass, and recompute cross-entropy and top-1 accuracy locally from those exact logits and targets; then randomly permute only the targets and recompute both metrics. Do not use the existing metric accumulator or logging path.","prediction_if_true":"The direct or permuted-target results will disagree with the reported metrics, or inspection will show that the evaluated logits are cached, stale, or derived from targets. With permuted targets, genuine model predictions should fall to about 1% accuracy and loss should rise toward or above 4.605.","prediction_if_false":"The direct calculation will reproduce roughly 0.04 loss and 96% accuracy, while target permutation destroys performance. That would localize the problem upstream: the model is not actually random, or the input batch contains an extraordinarily explicit label leak.","immediate_changes":[],"fallback_logic":[]} |