Spec was stale (recommended hard sparse "Version A", the DEMix absorption-killer).
Rewrite to match what is implemented and what we clarified:
- pseudocode-first: lora2r 2-expert forward, seeded rank-1 cosine router, GRPO+pin
loop, deploy ablation. For 2 experts the "proper" router IS rank-1 (softmax over 2
= sigmoid of one direction), seeded with v_act.
- "Why soft, not top-k" reframed as a tradeoff, not a verdict: hard routing closes
the leak but needs a router that catches all hacks; soft keeps absorption available
but leaks (1-w). DEMix only bites if we rely on absorption.
- Evidence section from two literature searches. Forced localization has working
precedents (single bad direction: emergent misalignment/persona/refusal; behavioural
expert seeding: SteerMoE, geometric cosine routing, cluster-aware upcycling; ablation
+ repair: NAEE/MoE-Pruner; router anchor: SEUF/MoTE). Emergent localization does not
(standing-committee, topic-driven routing). So seed+pin are load-bearing.
- 3-way/3-expert noted as an extension (closer to production), 2 experts for the
decisive causal run.
README: add Router dynamics (three forces, one pin-vs-reward conflict, mitigations).
Add HF "MoE in Transformers" blog to docs/papers (force-added past the docs gitignore).
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
wassname