wassname 8f39c4a69f docs: rewrite Evil MoE spec to the soft-routing design + literature evidence ...

Spec was stale (recommended hard sparse "Version A", the DEMix absorption-killer).
Rewrite to match what is implemented and what we clarified:

- pseudocode-first: lora2r 2-expert forward, seeded rank-1 cosine router, GRPO+pin
  loop, deploy ablation. For 2 experts the "proper" router IS rank-1 (softmax over 2
  = sigmoid of one direction), seeded with v_act.
- "Why soft, not top-k" reframed as a tradeoff, not a verdict: hard routing closes
  the leak but needs a router that catches all hacks; soft keeps absorption available
  but leaks (1-w). DEMix only bites if we rely on absorption.
- Evidence section from two literature searches. Forced localization has working
  precedents (single bad direction: emergent misalignment/persona/refusal; behavioural
  expert seeding: SteerMoE, geometric cosine routing, cluster-aware upcycling; ablation
  + repair: NAEE/MoE-Pruner; router anchor: SEUF/MoTE). Emergent localization does not
  (standing-committee, topic-driven routing). So seed+pin are load-bearing.
- 3-way/3-expert noted as an extension (closer to production), 2 experts for the
  decisive causal run.

README: add Router dynamics (three forces, one pin-vs-reward conflict, mitigations).
Add HF "MoE in Transformers" blog to docs/papers (force-added past the docs gitignore).

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>

2026-06-14 13:06:38 +08:00

..

grad_routing

tidy

2026-06-14 11:05:54 +08:00

2025_ichihara_mo-grpo.md

setup

2026-05-23 10:40:02 +08:00

2025_lw_ariahw_steering-rl-training-benchmarking-interventions.md

docs: merge Ariahw Fig-5 table into the paper md (delete standalone); add abs-scale arrow plot

2026-06-09 12:35:14 +00:00

2026_wu-tang_when-reward-hacking-rebounds.md

setup

2026-05-23 10:40:02 +08:00

hf_blog_moe_transformers.md

docs: rewrite Evil MoE spec to the soft-routing design + literature evidence

2026-06-14 13:06:38 +08:00