mirror of
https://github.com/wassname/adapters_as_hypotheses.git
synced 2026-06-27 16:44:10 +08:00
wassname
3800481a30
30 PEFT methods reframed as hypotheses about transformer geometry. Each entry: pseudocode, hypothesis, evidence, grade. All papers saved to docs/ (full text).
2.4 KiB
2.4 KiB
Adapters as Representational Hypotheses
What does each PEFT method believe about transformer internals?
Each adapter architecture encodes a structural claim about how to intervene in pretrained weights. When one outperforms another under controlled conditions (same model, same data, same parameter budget), the winner's assumptions are supported as a better description of the weight manifold.
This catalog reframes ~30 PEFT methods as hypotheses about transformer geometry, extracts pseudocode for each intervention, and grades the evidence.
Evidence hierarchy
| Grade | Meaning |
|---|---|
| * | Parameter-efficient (matches LoRA with fewer params) |
| ** | Beats LoRA on raw performance |
| ! | Beats full fine-tuning |
| !! | Data-efficient (few-shot, fast convergence) |
| !!! | Generalizes out-of-distribution |
Contents
- adapters_as_hypotheses.md -- the main catalog
- docs/ -- saved papers (full text, markdown)
Key findings
- SVD basis is the natural coordinate system. Methods that use the model's own SVD decomposition (PiSSA, SVFT, SSVD, AntiPaSTO) consistently outperform random-basis methods at the same parameter count.
- Orthogonal >> arbitrary. Orthogonal constraints (OFT, BOFT, HRA, AntiPaSTO) preserve semantic structure and improve OOD transfer, at the cost of limited magnitude changes.
- Direction and strength decouple. Methods that separate what to change from how much (DeLoRA, ROAD, AntiPaSTO) show better robustness and enable bidirectional steering.
- Low-rank is necessary but not sufficient. LoRA's rank bottleneck limits hard tasks; full-rank methods (RandLoRA, SHiRA) close the gap with full FT.
- Scaling alone goes far. IA3 and LN Tuning show that a surprising amount of adaptation is just reweighting existing features -- "gain control" over channels.
Related
- A Pragmatic Vision for Interpretability -- Nanda et al. 2025
- AntiPaSTO: Antiparallel Steering -- Clark 2025 (Appendix A.3 is the origin of this framing)
- HuggingFace PEFT -- reference implementations
License
Content is CC-BY-4.0. Papers in docs/ are fetched from arXiv for reference and remain under their original licenses.