Canonical GRPO hyperparameters
Sourced from external/rl-rewardhacking/src/train/config.py (Ariahw, Engels &
Nanda 2025). These are the defaults we inherit for all our headline runs;
deviations must be flagged in docs/RESEARCH_JOURNAL.md.
Optimizer
| key |
value |
source |
optim |
adamw_8bit |
config.py L137 |
learning_rate |
7e-5 |
config.py L138 |
lr_scheduler_type |
cosine |
config.py L139 |
adam_beta1 |
0.9 |
config.py L143 |
adam_beta2 |
0.99 |
config.py L144 |
LoRA
| key |
value |
source |
lora_rank (r) |
32 |
config.py L41 |
lora_alpha |
32 |
config.py L42 |
lora_dropout |
0.0 |
config.py L45 |
lora_bias |
none |
config.py L46 |
GRPO
| key |
value |
source |
beta (KL coeff) |
1e-3 |
config.py L135 |
num_generations |
16 |
config.py L162 |
per_device_batch_size |
32 (prompts) |
config.py L164 |
temperature |
0.7 |
config.py L172 |
auto_find_batch_size |
True first run |
config.py L165 |
Canonical effective per-step rollout count = per_device_batch_size * num_generations = 32 * 16 = 512.
We do NOT run the canonical setting. See deviations below.
Additional canonical hyperparams (not in tables above)
| key |
value |
source |
max_prompt_length |
1536 |
run_rl_training.py L73 |
max_completion_length |
1536 |
run_rl_training.py L73 |
warmup_steps |
10 |
config.py |
top_p |
0.95 (default) |
verl GRPOConfig |
cache_activations_layers |
[18] |
config.py L195-197 |
Our deviations for compute fit (96GB single-GPU)
The justfile uses MODEL=Qwen/Qwen3.5-2B as H4-main:
| key |
canonical |
ours |
rationale |
num_generations |
16 |
8 |
halve rollouts to fit 96GB |
per_device_batch_size |
32 |
16 |
halve prompts to fit 96GB |
| effective batch |
512 |
128 |
4x smaller |
If H4 falsified (Qwen3.5-2B hack rate <30% at step 200), the justfile switches
MODEL=Qwen/Qwen3-4B and the same NUM_GEN=8, BATCH=16 settings (further
reduction to 4/16 = 64 effective batch is the secondary fallback if 4B OOMs).
Smaller batches are noisier for GRPO advantage normalization. Expect higher SEM
and budget more seeds if either path is taken.
Our additions (projected_grpo only)
| key |
default |
scope |
vhack_m |
16 |
SVD top-m for v_hack denoising; sweep ∈ {8, 16, 32} |
vhack_layer_frac |
0.7 |
layer fraction for v_hack extraction (per Wu-Tang 60-75% depth) |
preserve_magnitude |
True |
restore ‖g‖ after projection; ablation arm sets False |
n_pairs |
20 |
initial contrastive pair count; scale to 60-80 for headline runs |
wassname