Skip to main content

Troubleshooting

Common Issues

SymptomCauseFix
OOM during trainingSAE too wide for available VRAMReduce expansion factor, increase batch accumulation, or use Q4 model
OOM during steeringModel + SAE + KV cache exceeds VRAMUse smaller SAE width or more aggressive quantization
>50% dead neuronsSparsity too aggressiveReduce l1_alpha/sparsity_coeff, enable sparsity warmup
Labels say "uncategorized"LLM couldn't interpret the featureIncrease max_examples, try a larger LLM, check activation examples manually
Training loss spikesLearning rate too highReduce by 2–5x, increase warmup steps
Training loss plateausLearning rate too low or not enough stepsIncrease LR or total_steps
Labeling timeoutsLocal model too slow for batchReduce batch_size to 1, increase api_timeout
Steering has no effectStrength too low or wrong featureIncrease strength (try 20–50), verify feature has clear activation pattern

Key Formulas

FrameworkLoss Function
Standard`L = MSE(x, x̂) + λ · Σ
JumpReLUL = MSE(x, x̂) + λ · Σ_i H(z_i - θ_i) (count of active features)
TopKL = MSE(x, x̂) + α · aux_loss(dead_features) (no sparsity penalty — K is structural)