#4673 Extracting Concepts from LLMs

#4673 Extracting Concepts from LLMs

#4673 Previous text modality LLMs were trained on text (i.e. words). Describe how to extract and catalog concepts from existing LLMs.

Extracting and cataloging concepts from existing Large Language Models (LLMs) is one of the central goals of Mechanistic Interpretability.

Because LLMs compress thousands of concepts into shared neural representations—a phenomenon known as superposition—concepts aren't stored in neat, isolated neurons. Instead, extracting and cataloging them requires disentangling dense activation spaces into monosemantic (single-meaning) concept vectors.

Here is the step-by-step methodology used by researchers to extract, label, and catalog concepts from trained LLMs.

1. Extracting Raw Activation Vectors

To find concepts, you first need to capture the model’s internal "thoughts" while it processes a wide variety of text.

• Residual Stream Harvesting: Run a large, diverse corpus (Wikipedia, code, novels, web text) through the LLM.

• Layer Selection: Record the activation vectors $x \in \mathbb{R}^d$ from intermediate transformer layers (typically the residual stream or MLP sub-layer outputs) for every token.

2. Disentangling Concepts (Dictionary Learning)

Raw activations are polysemantic—a single neuron might fire for French grammar, prime numbers, and legal jargon. To isolate pure concepts, researchers use two main paradigms:

A. Unsupervised Extraction: Sparse Autoencoders (SAEs)

Currently the gold standard (popularized by Anthropic and OpenAI), SAEs force dense representations into a higher-dimensional, sparse feature space.

1. Architecture: An encoder maps the dense hidden vector x into a much larger latent space z in mathbb{R}^m (where m gg d). A decoder attempts to reconstruct x from z.

2. Sparsity Constraint: An L_1 regularization penalty is applied, forcing the autoencoder to use only a tiny fraction of its latent features for any given token:

   mathcal{L} = Vert{}x - hat{x} Vert{}_2^2 + lambda Vert{}z Vert{}_1

3. Result: Each direction vector in the decoder dictionary corresponds to a distinct, highly specific concept (e.g., "references to the Golden Gate Bridge" or "bias in code formatting").

B. Supervised Extraction: Concept Activation Vectors (CAVs)

If you already know the specific concept you want to locate (e.g., "sarcasm" or "deception"):

• Collect a positive dataset (examples of the concept) and a negative dataset (examples without it).

• Train a linear classifier (like an SVM) on the layer activations.

• The orthogonal vector to the decision boundary defines the Concept Activation Vector (CAV).

3. Automated Interpretation and Labeling

Once you have thousands of extracted feature vectors, manually inspecting them is impossible.

• Top-K Activation Harvesting: For a given feature vector, identify the top text snippets in the dataset that activate it most strongly.

• Auto-Interpretation with LLMs: Pass those top-activating text snippets to a secondary LLM (e.g., GPT-4) with a prompt like: "What common concept causes these text fragments to activate?"

• Scoring & Falsification: Test the proposed label by feeding new text into the model and measuring whether the feature activates as predicted by the label.

4. Cataloging and Mapping Concepts

To turn millions of isolated features into a structured, searchable catalog:

• Hierarchical Clustering: Group features by cosine similarity. Broader concepts (e.g., "Science") sit at the top, branching down into granular concepts (e.g., "Quantum Mechanics" --> "Schrödinger Equation").

• Cross-Layer Tracing: Track how a concept evolves across network depth (e.g., early layers track syntax/words, middle layers build abstract reasoning/facts, late layers handle response formatting).

• Causal Validation (Steering): To confirm a cataloged concept isn't an artifact, perform activation patching or steering. Artificially clamp or boost the feature vector during generation:

  Example: Clamping the "Golden Gate Bridge" feature vector to a high value will cause the model to mention the bridge regardless of the input prompt.

Summary of the Concept Pipeline

[ Massive Text Corpus ] 
          │
          ▼
 [ LLM Hidden States ]  ──(Pass through SAE)──►  [ Sparse Features ]
                                                       │
                                                       ▼
[ Structured Catalog ]  ◄──(Cluster & Label)────  [ Top Snippets ]

Are you looking to implement concept extraction for a specific use case (like alignment/safety, model editing, or steering), or are you exploring the theoretical side of dictionary learning?

From <https://gemini.google.com/app/20101516e32252ab>  Google Gemini (3.6 Flash)