From 4e45bd4140e1617e22dc79001376863d171613c7 Mon Sep 17 00:00:00 2001 From: wassname <1103714+wassname@users.noreply.github.com> Date: Sat, 8 Aug 2026 10:04:32 +0800 Subject: [PATCH] DPO can barely move: tinker's own reference run reports 0.52 pair accuracy Contrast with the cookbook's RLHF pipeline (46% -> 94% win rate in 100 steps), so a decreasing DPO loss is weak evidence of behaviour change. Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com> --- refs/transformers.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/refs/transformers.md b/refs/transformers.md index 095051e..43d674e 100644 --- a/refs/transformers.md +++ b/refs/transformers.md @@ -54,6 +54,19 @@ The disclosed training reports mostly reinforce this boring answer. DeepSeek-V3, Inspect the best run's traces. It may have won by learning a shortcut, formatting artifact, or easier token distribution rather than the intended task. +## Offline preference training can barely move + +Tinker's own DPO reference run reports, after 50 steps on its demo data:[^tinker-dpo] + +> │ accuracy │ 0.515748 │ +> │ margin │ 0.005681 │ + +Accuracy 0.5 is coin-flip on pair ordering, so the vendor's reference DPO run barely separates chosen from rejected. The same cookbook's RLHF pipeline (reward-model SFT, then RL against the RM) reports:[^tinker-rlhf] + +> Policy RL stage: `test/win_rate` should increase from ~46% to ~94% in 100 steps. + +Practitioner reports point the same way: wassname tried DPO in three experiments and it never worked well, a friend hit the same on Tinker, and GRPO worked in the same hands (2026-07, verbal). A politely decreasing DPO loss is weak evidence of behavior change; check pair accuracy and margin, and prefer online RL when a reward can be computed. + ## Distributed and numerical failures > If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat] @@ -124,3 +137,5 @@ For experiment design, keep the [Google Deep Learning Tuning Playbook](https://d [^qwen3-report]: Qwen Team, ["Qwen3 Technical Report"](https://arxiv.org/abs/2505.09388) ([cache](../docs/evidence/reports/qwen3_technical_report.md)) [^llama3-report]: Meta AI, ["The Llama 3 Herd of Models"](https://arxiv.org/abs/2407.21783) ([cache](../docs/evidence/reports/llama3_herd_technical_report.md)) [^opt175b-report]: Zhang et al., ["OPT: Open Pre-trained Transformer Language Models"](https://arxiv.org/abs/2205.01068) ([cache](../docs/evidence/reports/opt175b_technical_report.md)) +[^tinker-dpo]: Thinking Machines, [tinker-cookbook recipes/preference/dpo README](https://github.com/thinking-machines-lab/tinker-cookbook/tree/main/tinker_cookbook/recipes/preference/dpo) +[^tinker-rlhf]: Thinking Machines, [tinker-cookbook recipes/preference/rlhf README](https://github.com/thinking-machines-lab/tinker-cookbook/tree/main/tinker_cookbook/recipes/preference/rlhf)