Week 0: Introduction & Course Overview

1. Course Logistics

Course Overview

Goal: Study how large language models encode non-CS concepts through interpretability research, culminating in papers suitable for submission to NeurIPS or similar venues.

Team Structure: Interdisciplinary teams of ~3 students:

• Non-CS PhD student: Domain expertise in your field (biology, linguistics, psychology, physics, law, etc.)
• CS/ML PhD student: Technical ML background, PyTorch experience
• BauLab member: Expert in interpretability methods, will guide technical implementation

Course Structure

Grading & Expectations

2. The Central Mystery: Are Concepts Different from Words?

Large language models process text, but do they think in words? This course begins with a provocation: the internal representations that drive model behavior may be fundamentally different from the tokens that flow in and out.

Words are surface phenomena. They vary across languages, shift meaning with context, and often fail to capture the abstractions that underlie coherent reasoning. Concepts, by contrast, are the invariants—the stable structures that persist across linguistic transformations. This course asks: can we find these invariants inside neural networks?

The Gap Between Knowledge and Expression

What an AI knows is not always what it says. This gap between internal representation and external behavior has become starkly visible in recent work.

Rager et al. (2025) study DeepSeek models, revealing a striking case of censorship mechanics. The models demonstrably possess knowledge about sensitive topics—their internal representations encode the relevant concepts—yet they refuse to express this knowledge in their outputs. The information exists inside the model; the suppression is a separate mechanism layered on top.

This dissociation between knowing and saying points to a general phenomenon: models may have internal states that diverge systematically from their outputs. If we want to understand what models actually believe, know, or intend, we cannot rely solely on their words. We must look inside.

Concepts as Invariants

The emerging picture suggests that concepts are invariants of the system—internal structures whose functional roles remain unchanged under many transformations. Just as a physical law remains valid regardless of the coordinate system used to express it, a neural concept may persist across languages, phrasings, and contexts.

Examples of concepts that appear to have this invariant character:

• Truthfulness concepts: Directions in activation space corresponding to whether a statement is true or false, independent of topic or phrasing
• Task concepts: The abstract notion of "translate" or "summarize" as distinct from any particular instance
• User concepts: Representations of who the model is talking to, tracking properties across a conversation
• Domain concepts: Abstract structures from law, medicine, physics, music—the focus of this course

These are not isolated neurons but distributed patterns of activation. They are not explicitly labeled in the training data but emerge from the structure of the task. And they are not words—they are what the words point to.

3. Causal Mediation: Finding the Neurons That Matter

How do we identify which internal components are responsible for specific behaviors? One methodology is causal mediation analysis: intervening on internal components to establish their causal role in the model's outputs.

Early Success: The Lamp-Controlling Neuron

Early work on GANs (Generative Adversarial Networks) provides an instructive template. A StyleGAN trained to generate bedrooms—with no supervision beyond imitation of a few million bedroom images—revealed striking internal structure.

This is surprising because there is nothing in the training data explicitly about switching lights on and off. The training set contains only static bedroom images—no before/after pairs, no labels about lighting. Yet "switching off the lights" emerges as a concept that the neural network learned from the training process.

This leads to the central question of this course: When we train on much more complex data—text spanning all of human knowledge—what concepts do neural networks learn?

4. In-Context Learning and Function Vectors

In-Context Learning: A Form of Metareasoning

In 2020, GPT-3 demonstrated in-context learning (ICL): the ability to learn new tasks from a few examples in the prompt, without any gradient updates.

This is a form of metareasoning—the model is not just processing text but reasoning about how to reason. Where does this capability come from? It is not explicitly programmed. It emerges from training on next-token prediction, yet manifests as something that looks remarkably like flexible cognition.

Function Vectors: Portable Neural Representations of Tasks

Where is in-context learning encoded inside the model? Research by Todd et al. (2024) reveals a striking answer: when a model performs ICL, it develops compact vector representations of the task itself—function vectors—that can be extracted and transplanted to new contexts.

This is directly analogous to the lamp-controlling neuron, but for abstract functions rather than visual features. Just as the lamp neuron's causal effect transfers across different bedroom images, function vectors transfer across completely different textual contexts. The neural representation learned in one setting causes the same function to execute in another.

This reinforces the central thesis: concepts inside LLMs are not words. They are abstract structures—stable under linguistic transformations—that we can study, characterize, and manipulate.

Key Research Threads in LLMs

The methodology of causal mediation generalizes from GANs to language models. Three lines of work illustrate the progress that has been made in localizing and characterizing neural concepts:

Locating Factual Knowledge (Meng et al., 2022): The ROME paper demonstrated that factual associations in GPT-style models are localized in specific MLP layers. By applying causal tracing—corrupting inputs and restoring activations at targeted locations—researchers identified where facts like "The Eiffel Tower is in Paris" are stored. This established that factual knowledge has a discernible address inside the network.

Locating Functions (Todd et al., 2024): This work extended localization from facts to functions. When a model performs in-context learning of an abstract task—translating to French, answering with antonyms, continuing a pattern—where is that function encoded? Function vectors show that task-level abstractions are localizable and, remarkably, transferable: a vector extracted from one context can induce the same function in another.

Locating Mental State Representations (Prakash et al., 2025): This work examined how models represent and track the mental states of agents described in text—theory of mind. It shows that interpretability methods can reach beyond simple factual recall into the representation of genuinely abstract, cognitively rich concepts.

5. Toward an Atlas of Neural Concepts

If concepts inside LLMs are real, discoverable, and causally important, then we should map them systematically. The vision motivating this course is the creation of an atlas of neural concepts: a comprehensive characterization of the internal structures that drive model behavior across domains.

Why Build an Atlas?

Such an atlas would serve multiple purposes:

• Scientific understanding: What are the basic building blocks of machine cognition?
• Safety and alignment: Which concepts relate to honesty, deception, harm, and helpfulness?
• Knowledge extraction: What has the model learned that humans do not yet know?
• Control and steering: How can we intervene on concepts to shape behavior?

Building this atlas requires methods. The works surveyed above—and the techniques you will learn throughout this course—constitute the toolkit for this cartographic project.

What Should the Atlas Contain?

This is an open question. Possibilities include:

• Individual neurons: The simplest units, but concepts are rarely confined to single neurons
• Directions in activation space: Linear combinations of neurons that represent concepts
• Circuits: Collections of components that work together to implement functions
• Functional roles: Abstract descriptions of what computations are performed where

The right level of description likely depends on the question being asked. Your projects will contribute to answering this.

6. What Makes a Good Research Question?

Before you invest months of effort into a research project, you should ask: is this the right question? The FINER framework—adapted from clinical research methodology—offers useful criteria: a good research question should be Feasible, Interesting, Novel, Ethical, and Relevant.

For a detailed discussion, see Finding a Good Research Question.

The FINER Framework for Interpretability

The Interdisciplinary Advantage

LLMs are being deployed in medicine, law, education, scientific research, policy analysis. The highest-stakes applications are almost entirely outside computer science. Yet interpretability research remains focused on questions that matter primarily to ML researchers.

This is the gap. This course is forming interdisciplinary teams precisely to address it. The interpretability community has strong intuitions about which questions matter for AI safety and for understanding deep learning. But we have weak intuitions about which questions matter for medicine, for law, for scientific discovery. Your non-CS collaborators have those intuitions.

Example Project Ideas

Group Discussion

7. Assignment: Initial Pitch (Due Week 1)

8. Looking Ahead

Week 1: Foundations—logit lens, intermediate representations, and the vocabulary of mechanistic interpretability

Week 2: Steering—controlling model behavior through activation addition and representation engineering

Week 3: Evaluation Methodology—measuring and evaluating LLM behavior

Week 4: Representation Geometry—PCA visualization, linear directions, and geometric structure

Weeks 5-8: Advanced methods (causal localization, probes, attribution, circuits)

Weeks 9-10: Training dynamics, model editing, and self-description

Weeks 11-13: Paper writing workshops and final presentations

By the end: You'll have a complete research pipeline for characterizing YOUR concept in LLMs, culminating in a paper suitable for submission to a top venue.

References & Further Reading

On Finding Good Research Questions

• Hamming, R. W. (1986). "You and Your Research" (video). Bell Communications Research colloquium. The classic essay on choosing important problems.
• Nielsen, M. A. (2004). "Principles of Effective Research". University of Queensland Technical Note. On problem-solvers vs. problem-creators.
• Cummings, S. R., Browner, W. S., & Hulley, S. B. (2013). "Conceiving the Research Question". In Designing Clinical Research. The FINER framework.
• Bau, D. (2024). "In Defense of Curiosity". On Venetian glassmaking, the value of basic scientific research, and understanding for its own sake.

Interpretability Papers Mentioned

• Meng et al. (2022). "Locating and Editing Factual Associations in GPT" (ROME)
• Todd et al. (2024). "Function Vectors in Large Language Models"
• Prakash et al. (2025). "Fine-tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking"
• Olsson et al. (2022). "In-context Learning and Induction Heads"
• Bau et al. (2019). "GAN Dissection" | Interactive demo

Accessible Introductions

• Transformer Circuits Thread (Anthropic) — Visual explanations of interpretability concepts
• Distill.pub — Interactive ML explanations (now archived but excellent)
• Google PAIR Explorables — Interactive ML concept demos

Supplementary Reading

• Olah et al. (2020). "Zoom In: An Introduction to Circuits". Distill
• Elhage et al. (2021). "A Mathematical Framework for Transformer Circuits". Transformer Circuits

Project Milestone