Understanding AI from the Inside: The State of Mechanistic Interpretability for AI Safety
1.1 Brief History of Interpretability
We cover the following in depth:
2 The Limits of Behavioural Evaluation
The Sleeper Agent Experiment
Three The Interpretability Landscape
3.2 Concept-Based Approaches
3.3 Input Attribution and Saliency Methods
3.4 Representation Probing and Related Post-hoc Analyses
3.5 Mechanistic Interpretability Methods
3.6 When Mechanistic Understanding Becomes Necessary
3.7 Limitations of Mechanistic Analysis
Limitations of Mechanistic Analysis
4 Foundational Concepts for Mechanistic Interpretability
4.1 Behavioural vs. Mechanistic Understanding
Behavioural Interpretability
Mechanistic Interpretability
4.2 Neuron Polysemanticity in Model Representations
4.3 Features as Representational Units
4.4 Transformer Computation
Residual Stream and Computational Components
4.5 Circuits as Descriptions of Internal Mechanisms
5. Core Hypotheses in Mechanistic Interpretability
5.1 The Linear Representation Hypothesis
Linear Representation Hypothesis
5.2 The Superposition Hypothesis
5.3 The Privileged Access Hypothesis
Privileged Access Hypothesis
5.4 The Universality and Platonic Representation Hypothesis
Six. Feature Extraction and Analysis
Six point one. The Sparse Coding Problem
Sparse Autoencoder (Figure 6)
6.3 S A E Variants and Architectural Improvements
Trade-offs Between Variants
6.4 Empirical Comparisons of SAE Variants
6.5 Challenges and Failure Modes of SAEs
6.5.2 Feature Duplication
6.5.3 Feature Absorption and Splitting
6.5.4 Feature Instability Across Seeds
6.5.5 Identifiability and Theoretical Concerns
6.5.6 Practical Recommendations
6.6 Multi-Layer Feature Extraction and Analysis
6.6.1 Transcoders: Bridging the Nonlinearities
6.6.2 Cross-Layer Transcoders: Enabling Better Circuit Analysis
Caveats on C L T Fidelity Claims
Conditions for reported fidelity:
Conditions for exact local matching:
Factors affecting fidelity:
6.6.3 Crosscoders: Analysing Cross-Layer Features
Causal versus Acausal Crosscoders. Crosscoders support two information flow patterns:
6.7 Comparing S A E's, Transcoders, and Crosscoders
6.8 Case Study: Induction Heads
Induction in a Repeated Sequence
Induction Heads: Features Across Representations
6.9 Comparison with Alternative Feature Extraction Methods
6.9.1 Non-Negative Matrix Factorisation
6.9.2 Principal Component Analysis
6.9.3 Independent Component Analysis
6.9.4 Supervised Feature Extraction Methods
6.9.5 Dictionary Learning Baselines
6.10 Interpretability-by-Design Methods for Intrinsic Interpretability
6.10.1 Feature-Sparse Training
6.10.2 Weight-Sparse Training
6.10.3 Attention-Sparse Training
6.10.4 Modifications of Computational Primitives: MOLT
6.10.5 Kolmogorov-Arnold Networks
7 Circuits in Mechanistic Interpretability
7.1 Circuit Discovery Methods
Automatic Circuit Discovery (ACDC)
Edge Attribution Patching (EAP)
7.2 Circuit Discovery with Replacement Models
7.4 Case Study: Induction Heads
7.5 A Unified Evaluation Framework for Mechanistic Interpretability
7.5.1 The Need for Standardised Evaluation
7.5.2 Five Evaluation Dimensions
Dimension 1: Attribution Faithfulness
Attribution Faithfulness Metrics
Dimension 2: Completeness
Dimension 4: Causal Validity
Dimension 5: Human Interpretability
Human Interpretability Metrics
7.5.3 Integrated Evaluation Protocol
Recommended Evaluation Protocol
7.5.4 Limitations of This Framework
8 Understanding Intermediate Representations
8.1 Logit Lens: Tracing Predictions Through Depth
8.2 Tuned Lens: Learning Per-Layer Transforms
8.3 Jacobian Lens: Decoding Computational Dispositions
8.4 Patchscopes: Leveraging the Language Model Itself to Explain Intermediate Representations
8.5 LatentQA: Fine-Tuning the Language Model as an Activation Verbaliser
8.6 Probing Classifiers: Capabilities, Critiques, and Causal Limitations
8.6.1 Standard Probing Methodology
The Probe Causality Problem
8.6.2 Control Tasks and Selectivity
8.6.3 Concerns of Probe Complexity
The Complexity-Interpretability Trade-off
8.6.4 Amnesic Probing and Causal Probing
8.6.5 Implications of Attribution Faithfulness for Probing
8.6.6 When Is Probing Appropriate?
8.6.7 Methodological Standards in Probing Research
Probing Classifiers for Sleeper Agents
9 Representation Analysis: Comparison, Localisation, and Control
False Positives in Model Diffing
9.3 Representation Engineering and Steering
Steering Vector (Figure 15)
Limitations: Brittleness and Mechanistic Opacity
100 Learning Dynamics and Mechanistic Interpretability
10.1 Longitudinal Analysis
10.1.1 Training Data and Capability Acquisition
Distributional Simplicity Bias
10.1.2 Feature Formation, Consolidation, and Drift
10.2 Learning Dynamics as an Interpretability Tool
10.2.1 Temporal Localisation and Mechanism Discovery
10.2.2 Causal Interventions on Mechanism Formation
Linear Mode Connectivity and Forked-Run Experiments
10.3 Phase Transitions and Emergence
10.3.1 Detecting Transitions
Constructing a Capability-Specific Metric
Is Emergence a Metric Artefact?
10.3.2 Progress Measures and Internal Precursors
Transfer from Grokking Settings to Language Models
10.3.3 Attributing Transitions to Components
Sharp Transitions from Smooth Components
10.3.4 Why Transitions Happen
Local Learning Coefficient (L L C)
10.3.5 Case Study: Induction Heads
10.4 Leveraging Learning Dynamics for Safety
10.4.1 Forecasting Model Behaviours
Most Early-Warning Evidence Is Retrospective
10.4.2 Intervening on Training Processes
Data-Level Interventions Have Ripple Effects
Train-Time Intervention Loop
11 Training Data Attribution
11.1 Verbatim Data Attribution: OLMoTrace and Maximal Matching Spans
OLMoTrace on Multilingual Output
11.2 Gradient-Based Attribution: Understanding Data Influence Through Perturbations
11.2.1 Core Idea: Parameter Response to Training Data
11.2.2 Proximal Bregman Response and the Generalised Gauss—Newton
Eleven point two point three, Addressing the Computational and Memory Bottlenecks of G G N
11.2.4 Retraining-Based Influence Estimation: datamodels and TRAK
11.2.5 Efficient Influence Estimation Using Training Trajectories: TracIn
11.2.6 Illustrative Example
11.2.7 Recent Extensions and Limitations
12 Six Research Orientations
12.1 Pragmatic Interpretability: Proxy-Task-Oriented Safety Interventions
12.2 Ambitious Interpretability: Towards Complete Mechanistic Understanding
12.3 Developmental Interpretability: Characterising How Models Learn
12.4 Interpretability Across Time: Understanding the Causes of Model Behaviours
12.5 Introspective Interpretability: Models Interpreting Themselves
12.5.1 Concerns for Introspective Interpretability
Failure Modes of Self-Explanation
12.5.2 Validation Procedures for Model Self-Reports
Validation Procedures for Introspective Models
12.5.3 When to Use Introspective Methods
12.6 Automated Interpretability: Scaling Discovery via Automation
Classical Automated Interpretability Pipeline
12.6.1 Governance Concerns for Automated Interpretability
Governance Risks of Automation
12.6.2 Proposed Guardrails for Automated Interpretability
Guardrail Framework for Automated Interpretability
12.6.3 Integrating Automated and Human Analysis
12.6.4 Summary: Responsible Automation
12.7 Comparative Analysis: Trade-Offs Between Orientations
12.8 Choosing an Orientation: A Decision Framework
12.9 Critical Concerns, Tensions, and Open Questions
13 From Understanding to Intervention: Safety Applications of Mechanistic Interpretability
13.1 Why Mechanistic Approaches for Safety?
13.1.1 Alternatives: Black-Box Methods and Their Challenges
Input and Output Filtering
User Profiling and Identity-Aware Safety
Data Filtering and Safety Training
13.1.2 Shared Limitations of Black-Box Approaches
13.2 Concept Erasure and Knowledge Editing
13.2.1 Concept Erasure via PISCES
Example Application of PISCES
13.2.2 Knowledge Editing via Localisation
Example Application of ROME
13.3 Steering Vectors and Mechanistic Monitoring
13.3.1 Steering Vectors: Practical Behavioural Control
Example Application of Steering Vectors
13.3.2 Mechanistic Monitoring
Example Application of Mechanistic Monitoring
13.4 Automated Interpretability Agents
13.5 Choosing Appropriate Methods for Different Safety Applications
13.6 From Research Orientations to Methods
13.7 Risks of Mechanistic Interpretability for AI Safety
13.7.1 Epistemic Risks: False Confidence from Partial or Incorrect Understanding
The Partial Coverage Issue
13.7.2 Operational Risks: Failure Under Deployment
13.7.3 Adversarial Risks: Malicious Exploitation of MI
13.7.4 Opportunity Costs: Resource Allocation Risks
13.7.5 Summary: M I as One Methodology Among Many
14.1 When Is Mechanistic Understanding Necessary?
14.2 Challenges in Evaluation and Benchmarking
14.3 Incompleteness, Scalability, Generalisability, and Adversarial Robustness
14.4 Mechanistic Understanding and Alignment
14.5 Interpretation versus Truth: The Ontology Question
14.6 The Proxy Fidelity Problem
14.7 Transferability Beyond Transformers
15.1 Achievements and Limits
A.1 Dimensions and Indices
A point two Inputs and Activations
A point three Dictionary Learning
A point four Model Parameters
A.5 Interventions and Objectives
A.6 Mathematical Operations
B.1 Box Types and Colours
B.2 Citation Evidence Levels
C.1 Causal Inference Basics
C.2 Local Learning Coefficient
C.3 Information Theory Basics