Remove LessWrong draft, symlink README to adapters_as_hypotheses.md

This commit is contained in:
wassname
2026-02-22 19:50:34 +08:00
parent eb2d5fb3c3
commit 78fe922789
2 changed files with 1 additions and 152 deletions
-40
View File
@@ -1,40 +0,0 @@
# Adapters as Representational Hypotheses
*What does each PEFT method believe about transformer internals?*
Each adapter architecture encodes a structural claim about how to intervene in pretrained weights. When one outperforms another under controlled conditions (same model, same data, same parameter budget), the winner's assumptions are supported as a better description of the weight manifold.
This catalog reframes ~30 PEFT methods as **hypotheses about transformer geometry**, extracts pseudocode for each intervention, and grades the evidence.
## Evidence hierarchy
| Grade | Meaning |
|-------|---------|
| * | Parameter-efficient (matches LoRA with fewer params) |
| ** | Beats LoRA on raw performance |
| **!** | Beats full fine-tuning |
| **!!** | Data-efficient (few-shot, fast convergence) |
| **!!!** | Generalizes out-of-distribution |
## Contents
- [adapters_as_hypotheses.md](adapters_as_hypotheses.md) -- the main catalog
- [docs/](docs/) -- saved papers (full text, markdown)
## Key findings
1. **SVD basis is the natural coordinate system.** Methods that use the model's own SVD decomposition (PiSSA, SVFT, SSVD, AntiPaSTO) consistently outperform random-basis methods at the same parameter count.
2. **Orthogonal >> arbitrary.** Orthogonal constraints (OFT, BOFT, HRA, AntiPaSTO) preserve semantic structure and improve OOD transfer, at the cost of limited magnitude changes.
3. **Direction and strength decouple.** Methods that separate *what to change* from *how much* (DeLoRA, ROAD, AntiPaSTO) show better robustness and enable bidirectional steering.
4. **Low-rank is necessary but not sufficient.** LoRA's rank bottleneck limits hard tasks; full-rank methods (RandLoRA, SHiRA) close the gap with full FT.
5. **Scaling alone goes far.** IA3 and LN Tuning show that a surprising amount of adaptation is just reweighting existing features -- "gain control" over channels.
## Related
- [A Pragmatic Vision for Interpretability](https://www.lesswrong.com/posts/StENzDcD3kpfGJssR/a-pragmatic-vision-for-interpretability) -- Nanda et al. 2025
- [AntiPaSTO: Antiparallel Steering](https://arxiv.org/abs/2601.07473) -- Clark 2025 (Appendix A.3 is the origin of this framing)
- [HuggingFace PEFT](https://github.com/huggingface/peft) -- reference implementations
## License
Content is CC-BY-4.0. Papers in docs/ are fetched from arXiv for reference and remain under their original licenses.
Symlink
+1
View File
@@ -0,0 +1 @@
adapters_as_hypotheses.md