Trainable params that were init'd at exact 0 or 1 now use near_zero (N(0,1e-4))
or near_one (1 + N(0,1e-4)) to break bf16 symmetry without meaningfully
breaking identity-at-t=0. Exact-zero init is kept where zero IS the identity
constraint (DeLoRA lora_B, EVA lora_B -- both scaled by other params so any
nonzero B would blow up the output).
AntiPaSTO: delta_s and rot_T now near_zero. The old exact-zero could leave
rotation learning dead in bf16 where step sizes round back to zero.
IA3: lora_g now near_one instead of exact ones. Avoids the bf16 spacing issue
around 1.0 where eps_bf16 ~ 7.8e-3 and lr=1e-3 updates were rounding away.
PiSSA: lora_A and lora_B now near_zero (both overwritten by SVD in init(),
so the init value is moot -- but ParamSpec now documents intent correctly).
HRA: lora_U now near_zero (overwritten by symmetric init in init()).
ParamSpec: added 'near_zero' and 'near_one' init modes. Default changed from
'zeros' to 'near_zero'. Tests relaxed identity tolerances accordingly.
DeLoRA (variants/delora.py):
lora_B init zeros not kaiming, matching peft (docs/refs/peft_delora_layer.py:139).
With B=0 the t=0 delta is zero regardless of lambda, so identity holds with
the peft default lambda0=15 instead of needing the lambda0=0 hack.
HRA (variants/hra.py):
forward_input loop reversed: now applies x @ H_{r-1} ... H_0 = x @ R^T so
the base layer computes x R^T W^T = F.linear(x, W @ R), matching peft. The
bug was masked by paired-symmetry init (R = R^T at t=0) but would corrupt
any non-symmetric U.
EVA (variants/eva.py):
lora_A is now a trainable Parameter (peft semantics): SVD only changes the
init. group_init still copies the SVD basis but under a no_grad guard.
AntiPaSTO (variants/antipasto.py):
docstring now references arxiv.org/pdf/2601.07473 and
github.com/wassname/AntiPaSTO so V4 review NO_REFERENCE flag is resolved.
qwen probe (scripts/qwen_train_probe.py):
perturb_first_adapter walks priority list including lora_U (HRA) and
lora_A (EVA, LoRA-style A-trainable variants) so HRA tests no longer raise
'no perturbable adapter parameter found'.
smoke (tests/smoke.py):
+ hra_forward_order_smoke: distinguishing check that compares adapted output
to F.linear(x, W @ R) with paired symmetry broken; would fail under the
forward-iter bug.
+ EVA assert lora_A.requires_grad == True per layer.
- DeLoRA bnb moved to bnb_skip (fp16 + B=0 + clamp(min=1e-4) overflow makes
grad NaN; real bnb usage needs dequant).
delora train still uses lambda0=0.1 because peft default 15.0 explodes
Adam lr=1e-1 in 20 steps.
wassname