Weight Patching: Toward Source-Level Mechanistic Localization in LLMs
Summary: arXiv:2604.13694v1 Announce Type: new
Abstract
Mechanistic interpretability seeks to localize model behavior to the internal components that causally realize it. Prior work has advanced activation-space localization and causal tracing, but modules that appear important in activation space may merely aggregate or amplify upstream signals rather than encode the target capability in their own parameters. To address this gap, we propose Weight Patching, a parameter-space intervention method for source-oriented analysis in paired same-architecture models that differ in how strongly they express a target capability under the inputs of interest.
Introduction to Weight Patching
Weight Patching represents a novel approach within the scope of mechanistic interpretability. The core concept revolves around the utilization of two models that share the same architecture but have different behaviors in terms of a specific capability. The method facilitates the exploration of how individual components contribute to the overall model behavior.
Methodology
In our proposed framework, given a base model and a behavior-specialized counterpart, Weight Patching replaces selected module weights from the specialized model into the base model under a fixed input. This approach allows for a more nuanced understanding of how specific weights influence model outputs and behaviors.
Key Features of Weight Patching
- Parameter-Space Intervention: By manipulating the weights of the model, we can directly observe changes in behavior, allowing for a clearer attribution of capabilities.
- Instruction Following: We instantiate the method specifically for instruction-following tasks, which are critical in evaluating the performance of language models.
- Vector-Anchor Behavioral Interface: This interface provides a shared internal criterion for assessing whether a task-relevant control state has been formed or recovered, especially in open-ended generation scenarios.
Results and Analysis
Through the implementation of Weight Patching, our analysis reveals a hierarchy of components involved in model behavior. This hierarchy ranges from:
- Shallow Candidate Source-Side Carriers: Basic components that first interact with the input signals.
- Aggregation and Routing Modules: More complex structures that work to combine and direct information.
- Downstream Execution Circuits: The final components that execute the model’s output based on the processed information.
Implications for Mechanism-Aware Model Merging
The recovered component scores not only enhance our understanding of individual components but also provide a framework for mechanism-aware model merging. This enhanced selective fusion across evaluated expert combinations offers potential improvements in model performance and additional external validation of the results.
Conclusion
Weight Patching serves as a groundbreaking methodological advancement in the field of mechanistic interpretability. By focusing on parameter-space interventions, we pave the way for deeper insights into the inner workings of language models, ultimately improving their design and functionality in practical applications.