# CHaRS: Concept Heterogeneity-aware Representation Steering

Source: https://arxiv.org/html/2603.02237v1 (fetched 2026-05-26)
Epistemic note: no released code; trust signals = the paper alone. Treat
findings as plausible but unvalidated by external use.

## Abstract

CHaRS controls LLM behavior through representation steering. Rather than
applying uniform shifts across embedding space, CHaRS models representations
as Gaussian mixture models and formulates steering as an optimal transport
problem between semantic clusters. Yields context-dependent steering vectors
that vary smoothly across the representation manifold, claimed to outperform
global steering baselines.

## 1. Introduction

Traditional steering: single direction via difference-in-means over
contrastive pairs. Assumes homogeneous concept representation in embedding
space.

Authors' claim: LLM representations are non-homogeneous, clustered,
context-dependent — global steering becomes brittle.

### Key Contributions

1. Generalization to multimodality: extends steering from unimodal Gaussian to
   GMMs via Mixture Wasserstein distance, formulated as discrete OT between
   semantic clusters.
2. Input-adaptive steering: cluster-level transport plans produce
   context-sensitive control where directions vary smoothly across the manifold.
3. Spectral factorization: Principal Component Thresholding (PCT) reveals
   inherent low-rank structure (rank ≤ 2K−2).

## 2. Background

### 2.1 Optimal Transport Framework

p-Wasserstein distance:

$$W_p(\mu, \nu) = \left(\inf_{\pi \in \Pi(\mu,\nu)} \int \|\mathbf{x} - \mathbf{y}\|_p\, d\pi(\mathbf{x}, \mathbf{y})\right)^{1/p}$$

### 2.2 Gaussian OT and Representation Steering

For $\mu = \mathcal{N}(m_1, \Sigma_1)$ and $\nu = \mathcal{N}(m_2, \Sigma_2)$:

$$W_2^2(\mu, \nu) = \|m_1 - m_2\|_2^2 + d_B^2(\Sigma_1, \Sigma_2)$$

OT map is affine: $T(\mathbf{x}) = \mathbf{m}_2 + \mathbf{A}(\mathbf{x} - \mathbf{m}_1)$.
Under identical covariance, reduces to pure translation = difference-in-means.

### 2.3 Gaussian Mixture Wasserstein Distance

$$\mu = \sum_{k=1}^K p_k \mathcal{N}(\mathbf{a}_k, \Sigma_k),\quad \nu = \sum_{l=1}^L q_l \mathcal{N}(\mathbf{b}_l, \Gamma_l)$$

Mixture Wasserstein:

$$MW_2^2(\mu, \nu) = \min_{\gamma \in \Gamma(p,q)} \sum_{k,l} \gamma_{kl}\, W_2^2(\mathcal{N}(a_k, \Sigma_k), \mathcal{N}(b_l, \Gamma_l))$$

Tractable discrete OT between components.

## 3. CHaRS

### 3.1 Barycentric Projection

$$\hat{T}(\mathbf{x}) := \mathbb{E}_\pi[\mathbf{y}|\mathbf{x}]$$

For GMM-OT:

$$\hat{T}(\mathbf{x}) = \sum_{k,l} p(k|\mathbf{x}) \cdot \frac{\gamma^*_{kl}}{p_k} \cdot T_{kl}(\mathbf{x})$$

### 3.2 Clustering-based Steering

- k-means on activations → centroids $a_i$, $b_j$.
- Cluster matching via entropy-regularized OT (Sinkhorn):

$$\mathbf{P}^* = \arg\min_{\mathbf{P} \in \Pi(w_A, w_B)} \langle \mathbf{P}, \mathbf{C} \rangle + \lambda H(\mathbf{P}),\quad C_{ij} = \|a_i - b_j\|_2^2$$

- Kernel-based gating (RBF, σ = median centroid distance):

$$\hat{p}(i|\mathbf{x}) = \frac{p_i\, k(\mathbf{x}, a_i)}{\sum_m p_m\, k(\mathbf{x}, a_m)}$$

- Isotropic-covariance simplification: $T_{ij}(\mathbf{x}) = \mathbf{x} + \mathbf{v}_{ij}$ where $\mathbf{v}_{ij} = b_j - a_i$.
- Final steering (Definition 3.1):

$$\hat{T}_\alpha(\mathbf{x}) = \mathbf{x} + \alpha \hat{\mathbf{v}}(\mathbf{x})$$
$$\hat{\mathbf{v}}(\mathbf{x}) = \sum_{i,j} \frac{P^*_{ij}\, k(\mathbf{x}, a_i)}{\sum_{p,q} P^*_{pq}\, k(\mathbf{x}, a_p)}\, \mathbf{v}_{ij}$$

### 3.3 Principal Component Thresholding

Weighted covariance of local shifts:

$$\Sigma_{total} = \sum_{i,j} P_{ij}(\mathbf{v}_{ij} - \bar{\mathbf{v}})(\mathbf{v}_{ij} - \bar{\mathbf{v}})^T = \mathbf{U}\Lambda\mathbf{U}^T$$

Rank bound: $\text{rank}(\Sigma_{total}) \le 2K - 2$.

**CHaRS-PCT** keeps top $L$ components:

$$\tilde{\mathbf{v}}(\mathbf{x}) = \bar{\mathbf{v}} + \sum_{k \in [L]} \hat{\alpha}_k(\mathbf{x})\, \mathbf{u}_k$$

## 4. Experiments

### 4.1 Jailbreaking

AdvBench 80/20 train/eval. Examples (ASR):

| Model | ActAdd | CHaRS | CHaRS-PCT |
|---|---|---|---|
| Gemma2-9B | 91.35% | 98.08% | 98.08% |
| Llama3.1-8B | 95.19% | 98.08% | 99.04% |
| Qwen2.5-7B | 91.35% | 95.19% | 93.27% |

### 4.2 Toxicity Mitigation

Llama3-8B (lower = less toxic):

| Method | Classifier | 0-shot |
|---|---|---|
| Linear-AcT | 1.93% | 7.73% |
| CHaRS | 1.23% | 4.80% |
| CHaRS-PCT | 1.17% | 4.47% |

Authors note PCT often outperforms CHaRS in sequential settings — implicit
regularization.

### 4.3 Image Style Control

FLUX.1 + 512 COCO captions w/ style tags. Pareto frontier improves over
Linear-Act.

## 5. Ablations

- K > 1 consistently > K=1. Optimal K model-dependent (10-15).
- 100% variance with 2(K−1) PCs (matches theoretical bound).
- Component count for PCT requires tuning.

## 6. Concluding Remarks

Generalizes diff-in-means to multimodal distributions. Limitations: isotropic
covariances, k-means. Future: anisotropic mixtures, feature weighting.

---

## Note for our use case (projected_grpo)

CHaRS does **activation** steering (forward-pass). We do **gradient** projection
(backward-pass). The analog:
- Source distribution = clean-rollout gradients
- Target distribution = hack-rollout gradients
- Diff-in-means = `mean(g_hack) - mean(g_clean)` (current v_hack)
- CHaRS analog = cluster hack-grad and clean-grad into k clusters each,
  Sinkhorn-match, derive a per-input transport map.

Simpler step (PCT-like) for our setting: stack per-pair diff vectors,
SVD/PCA, keep top-L singular directions, project gradient out of their span.
This is what user proposed. Theoretical rank bound for K clusters is 2K-2;
with 12 pairs treated as 12 clusters that's 22, but L<<22 is fine in practice.