New Route for LLM Interpretability: Weight Decomposition with Data Cost Under 1%

The IQuest Research Institute, along with collaborators from Safe AI Forum, the University of Oxford, Stanford University, and Tsinghua University, proposes a new approach to mechanistic interpretability of large language models, named Sparse Weight Decomposition (SWD). Instead of training a separate sparse representation network to approximate a model's layers, SWD directly decomposes a dense weight matrix into two sparse matrices A and B, where the shared intermediate dimensions become bottleneck units that can be independently scored, selected, and ablated. This allows researchers to extract task circuits directly from the weights. In single-matrix experiments matching replacement fidelity, SWD uses less than 1% of the data required by training-based baselines like Transcoder. On GPT-2, Qwen2.5, and Qwen3.5-27B, SWD typically achieves the same sufficiency and necessity targets with fewer bottleneck units and active connections. The method scales to 27B models and covers all 48 attention and MLP weight matrices of GPT-2 Small, with a zero-data version that requires no calibration text. The paper reports that in GPT-2 Small, SWD achieves low CE delta with only thousands of calibration tokens, whereas training-based baselines need around 10^6 tokens. Full-model replacement on GPT-2 Small, using SWD-FT, reduces CE from 3.90 to 3.44 with the same number of connections, using about 20.6 million tokens compared to 2.884 billion for a sparse pretraining baseline. Semantic analysis of high-ranking bottleneck units on the GreaterThan task shows that four of six units respond to numbers, years, quantities, or measures. A targeted editing experiment on a single unit (c205) increased the logit margin for the correct answer by 0.216 with minimal side effects, demonstrating that SWD-derived directions can be used for precise behavior control.