mirror of
https://github.com/wassname/lora-lite.git
synced 2026-07-23 13:00:14 +08:00
wassname
d0b4c52740
- Two acpx external reviews (codex + opencode): * docs/audit/variants_review.md: per-variant paper-vs-impl audit * docs/audit/design_review.md: peft EVA / baukit / antipasto3 vs lora-lite * docs/audit/SUMMARY.md: aggregate verdicts + 3 risks + 5 follow-ups - docs/refs/: peft_eva.py, peft_eva_finetuning.py, baukit_nethook.py, antipasto3_svd_adapter.py for offline reference Findings: LoRA clean; PiSSA/DoRA/IA3/HRA/DeLoRA have documented partial deviations. Top risks: init/grad tradeoffs hidden by coarse tests; qwen probe lacks strict identity tol; IA3 target placement untested.
76 lines
3.3 KiB
Markdown
76 lines
3.3 KiB
Markdown
# Design review: should lora-lite borrow from peft EVA / baukit / antipasto3?
|
||
|
||
You are reviewing a minimal from-scratch LoRA library (`lora-lite`) and comparing
|
||
it to three reference implementations. Goal: identify cherry-picks that would
|
||
**reduce** complexity or unlock missing capability, **without bloating the lib**.
|
||
|
||
## Inputs
|
||
|
||
- lora-lite code: `src/lora_lite/` (adapter.py, target.py, variant.py, config.py, variants/*.py)
|
||
- Reference: `docs/refs/peft_eva.py` (peft's EVA: data-driven SVD-of-activations init)
|
||
- Reference: `docs/refs/peft_eva_finetuning.py` (example usage)
|
||
- Reference: `docs/refs/baukit_nethook.py` (nethook: forward/backward hook patterns)
|
||
- Reference: `docs/refs/antipasto3_svd_adapter.py` (wassname's earlier JAX SVD adapter)
|
||
|
||
## Project ethos (read first)
|
||
|
||
Lora-lite is fail-fast research code. Principles:
|
||
- No defensive programming, no fallbacks, no legacy compat
|
||
- Simplicity beats features. If you add X you must remove equivalent complexity.
|
||
- Each variant is one file with paper URL + honest deviation notes.
|
||
- Targets discovered by structural type-check, not name regex.
|
||
- Hooks via plain torch forward_pre_hook on a single layer, no global registry.
|
||
|
||
Read `AGENTS.md` if present.
|
||
|
||
## Questions to answer
|
||
|
||
For each reference, answer:
|
||
|
||
### A. peft EVA (`docs/refs/peft_eva.py` + `peft_eva_finetuning.py`)
|
||
|
||
1. What does EVA actually do? (1-paragraph summary; cite line numbers)
|
||
2. What would a *minimal* EVA variant in lora-lite look like? Sketch the API:
|
||
- How does the user pass calibration data?
|
||
- Where does the SVD-of-activations happen — in `init()` with a callback,
|
||
or as a separate `calibrate(model, dataloader, cfg)` step before `attach`?
|
||
3. Does peft's implementation have anything we could **drop** if we re-implemented?
|
||
(e.g. the rank-redistribution logic, the resume-from-checkpoint plumbing)
|
||
4. Does lora-lite's current `Variant.init(layer, cfg)` signature support EVA, or
|
||
would we need to extend it? Recommend the **smallest** API change.
|
||
|
||
### B. baukit nethook (`docs/refs/baukit_nethook.py`)
|
||
|
||
1. What does `TraceDict` / `Trace` give us that our current per-layer
|
||
`forward_pre_hook` registration does not?
|
||
2. Would adopting `baukit` for hook management (a) simplify our adapter.py,
|
||
(b) complicate it, or (c) be neutral? Quote specific lines from
|
||
`src/lora_lite/adapter.py` to justify.
|
||
3. Lora-lite's principle: minimize deps. Is baukit worth a dep? Or should
|
||
we just **inline** the 1-2 useful patterns?
|
||
|
||
### C. antipasto3 SVD adapter (`docs/refs/antipasto3_svd_adapter.py`)
|
||
|
||
1. This is the user's earlier JAX work. Anything in there (init style, scale
|
||
parameterization, save/load format) that lora-lite should adopt or
|
||
deliberately diverge from?
|
||
2. Does it suggest a cleaner factoring for PiSSA-like methods?
|
||
|
||
## Output format
|
||
|
||
For each (A, B, C), end with:
|
||
|
||
**Recommendation: ADOPT / SKIP / PARTIAL**
|
||
|
||
If ADOPT or PARTIAL, list the specific lines/patterns to import and the
|
||
approximate net line-count impact on lora-lite (+ added, − removed).
|
||
|
||
## Hard rules
|
||
|
||
- Do NOT propose code edits. This is design notes only.
|
||
- Do NOT recommend adding a feature unless you can name what to remove or
|
||
simplify in exchange.
|
||
- Be specific. "Could be cleaner" is not a recommendation; "Replace L42-L67
|
||
in adapter.py with a 5-line TraceDict call" is.
|
||
- If a reference's pattern is worse than what lora-lite already has, say so.
|