Week 3: Evaluation Methodology

Overview

Before we can understand what's happening inside a large language model, we need to know how to measure what it does. This week introduces the fundamental methods for evaluating LLM behavior, from basic probability outputs to sophisticated benchmarking strategies. You'll learn how to design effective prompts, interpret model outputs, and assess whether a model has truly learned a concept or simply memorized training data.

Learning Objectives

By the end of this week, you should be able to:

• Describe what an autoregressive language modeling neural network calculates, including its inputs and how it outputs probabilities for a single token
• Explain how to sample probabilistic output autoregressively to generate long outputs
• Apply three prompting strategies for measuring model capabilities: instruction-following, cloze prompts, and in-context learning
• Use methods for conforming to easy-to-evaluate output formats through ICL, custom decoding, or MCQ prompting
• Evaluate long-form output using an LLM-as-judge approach
• Understand and compute evaluation metrics: precision, recall, F1, and perplexity
• Apply statistical significance testing and interpret error bars in evaluation results
• Distinguish between memorization and generalization, and identify the risk of training data overlap and contamination
• Describe the approaches used by HELM to create fair and accurate benchmarks

Required Readings

Supplementary Materials

Tutorial: Why Single-Token Predictions Matter for Interpretability

1. The Autoregressive Language Model

At its core, a large language model is a function that takes a sequence of tokens as input and produces a probability distribution over a single next token. Given a sequence of tokens x₁, x₂, ..., xₙ, the model computes:

P(xₙ₊₁ | x₁, x₂, ..., xₙ)

This is a distribution over the entire vocabulary, typically 50,000 to 100,000+ tokens. The model doesn't directly generate paragraphs or essays; it generates probabilities for one token at a time. Understanding this distinction is crucial for interpretability research.

2. Why Single-Token Predictions Are Easy to Study

The one-token-at-a-time nature of autoregressive models is a gift for interpretability. When a model makes a single prediction, we know exactly where and when the decision happens:

• Logit lens: Decode intermediate layers at the final token position to watch the prediction evolve layer by layer
• Activation patching: Intervene at a known position rather than searching across the sequence
• Probing: Extract activations at exactly the decision point, avoiding noise from irrelevant positions
• Probability comparison: Compare logits for candidate completions directly (e.g., P("Yes") vs P("No"))

This is why cloze-style evaluation ("The capital of France is ___") is so powerful for mechanistic interpretability: the model's entire reasoning must culminate in a single token prediction at a known position.

3. The Challenge of Evaluating Longer Outputs

But not everything reduces to single-token predictions. Sometimes we need to evaluate:

• Extended reasoning chains
• Creative writing quality
• Dialogue coherence
• Following complex multi-step instructions

When the model generates 100 tokens, the "decision" is distributed across all of them. There's no single position where we can say "this is where the model decided to be helpful/harmful/creative." This makes interpretability much harder.

4. Autoregressive Sampling (How Long Outputs Happen)

To generate longer sequences, we sample from the model autoregressively, one token at a time:

• Greedy decoding: Always pick the highest probability token
• Temperature sampling: Sample from the distribution, with temperature controlling randomness
• Top-k sampling: Sample only from the k most likely tokens
• Nucleus (top-p) sampling: Sample from the smallest set of tokens whose cumulative probability exceeds p

Each strategy affects the diversity, coherence, and randomness of generated text. For benchmarking, greedy decoding or low-temperature sampling provides deterministic, reproducible results.

5. Prompting Strategies for Single-Token Evaluation

To make interpretability tractable, we want to reduce complex tasks to single-token predictions. Three strategies:

Cloze Prompts (Best for Interpretability)

Format the task as text completion, measuring the probability of specific completions:

We can evaluate whether "Paris" has higher probability than alternatives. The decision happens at a known position.

Multiple Choice Questions

Force selection from predefined options by comparing token probabilities for "A", "B", "C", "D":

The model's prediction at the final position tells us everything. We can extract activations here and compare logits across options.

In-Context Learning (ICL)

Provide examples in the prompt to demonstrate the task, then use cloze format for the test case:

ICL is valuable for interpretability because you can compare activations with and without the in-context examples to see how they reshape processing.

6. LLM-as-Judge: When Single Tokens Are Not Enough

Some capabilities genuinely require evaluating longer outputs: writing quality, reasoning coherence, helpfulness. When you must evaluate runs of text rather than single tokens, LLM-as-judge provides an automated approach:

• Provide the judge with evaluation criteria and rubrics
• Ask for both a score and justification
• Use pairwise comparison rather than absolute scoring for better reliability
• Be aware of biases: judges favor longer responses, their own outputs, and certain styles

MT-Bench demonstrated that LLM judges correlate well with human judgments for many tasks, but always validate on a subset with human evaluation.

7. Creating Evaluation Datasets at Scale

The bottleneck in interpretability research is often data: you need hundreds or thousands of examples that cleanly test your concept. Model-written evaluations (Perez et al. 2022) offer a solution: use LLMs to generate evaluation datasets.

The core workflow:

1. Specify the behavior you want to evaluate in a clear prompt
2. Generate examples by prompting a capable LLM with instructions and seed examples
3. Filter for quality using automated checks and human review
4. Run evaluations on target models

For interpretability, design your generation prompts to produce cloze-format or MCQ examples. This ensures the resulting dataset has the token-localization properties you need.

See the supplementary essay for detailed prompts and a complete pipeline example using puns.

8. Evaluation Metrics

Different tasks require different metrics:

• Accuracy: Proportion of correct predictions (good for balanced classes)
• Precision/Recall/F1: When classes are imbalanced or false positives/negatives have different costs
• Perplexity: How "surprised" the model is by a sequence (lower = better)

9. Statistical Significance

A model scoring 85% vs 83% might not be meaningfully different. Quantify uncertainty:

• Standard error: For accuracy on n samples, SE = √(p(1-p)/n)
• Confidence intervals: 95% CI is approximately ± 2 × SE for large n
• Bootstrap resampling: Repeatedly sample from your test set to estimate score distribution

Rule of thumb: differences smaller than 2-3% on typical benchmark sizes often aren't statistically significant.

10. Memorization vs. Generalization

A fundamental question: Did the model learn the concept, or did it memorize training examples?

Training Data Contamination

If test examples appeared in training data, the model might simply recall them rather than demonstrate true understanding. This is especially problematic because:

• Popular benchmarks leak into training corpora (scraped web data, code repositories)
• Models trained on "the whole internet" likely saw many benchmark examples
• Paraphrased or slightly modified examples may still trigger memorization

Detecting and Mitigating Contamination

• N-gram overlap: Search training data for test example substrings
• Membership inference: Compare perplexity on test vs similar synthetic examples
• Rephrased evaluation: Create novel rewordings of test questions
• Held-out test sets: Keep evaluation data truly private until final evaluation

Generalization Tests

Beyond contamination, test for genuine understanding:

• Novel compositions of known concepts
• Out-of-distribution examples
• Counterfactual variations
• Transfer to related but different tasks

11. HELM: Holistic Evaluation

The Holistic Evaluation of Language Models (HELM) benchmark exemplifies comprehensive evaluation:

Key Principles

• Broad coverage: 42+ scenarios across question answering, reasoning, generation, classification, etc.
• Multi-metric: Evaluates accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency
• Standardized prompting: Consistent few-shot examples and formatting across models
• Transparency: All prompts, data, and evaluation code are public

Metrics Beyond Accuracy

• Calibration: Do predicted probabilities match actual correctness rates?
• Robustness: Performance under perturbations (typos, paraphrases, contrast sets)
• Fairness: Equal performance across demographic groups
• Efficiency: Inference cost, carbon footprint

Putting It All Together

Effective benchmarking requires:

1. Understanding model fundamentals (token probabilities, sampling)
2. Choosing appropriate prompting strategies for your concept
3. Designing tasks with evaluable outputs
4. Selecting metrics that match your research questions
5. Quantifying uncertainty and statistical significance
6. Ruling out memorization through contamination checks and generalization tests
7. Following established best practices (HELM) for reproducible, fair evaluation

Dataset Construction for Concept Research

For your research project, the quality of your findings depends heavily on your dataset construction. This section covers principles and strategies for building datasets that enable rigorous concept characterization.

1. Defining Your Concept Operationally

Before constructing a dataset, you must operationalize your concept—make it measurable.

From Abstract to Concrete

• Abstract concept: "politeness"
• Operational definition: Text containing please/thank you, formal pronouns, hedging language, indirect requests
• Measurable examples: "Could you please..." (polite) vs "Do this now" (direct)

Exercise: For your chosen concept, write down:

1. 3-5 concrete features that characterize it
2. 10 clear positive examples
3. 10 clear negative examples (concept absent)
4. 5 edge cases (ambiguous or borderline)

2. Contrastive Pair Construction

The gold standard for concept research: pairs of examples that differ only in your target concept.

Minimal Pairs Strategy

Controlled Variation

• Substitute only concept-relevant words: polite → rude, formal → casual
• Keep constant: sentence structure, length (±2 tokens), topic, complexity
• Avoid confounds: Don't change sentiment when studying formality

Methods for Generating Pairs

1. Template-based: "[GREETING], could you [ACTION]?" vs "[ACTION] now"
2. LLM-assisted: Prompt GPT-4 to paraphrase with/without concept
3. Human-created: Writers produce matched pairs (expensive but high quality)
4. Found pairs: Mine existing data for natural contrasts (Reddit posts with different tones)

3. Balancing and Sampling Strategies

Class Balance

For binary concepts (polite/rude, factual/fictional), aim for 50-50 split unless you have specific research reasons.

• Imbalanced data risks: Model might learn "always predict majority class"
• Probe issues: Linear probes are especially sensitive to imbalance
• Evaluation fairness: Accuracy is misleading with skewed data

Stratified Sampling

Ensure diversity across relevant dimensions:

4. Avoiding Common Pitfalls

Confound: Concept Leakage

Pitfall: Lexical Shortcuts

Models might learn "if text contains 'please' → polite" without understanding context.

• Test: Include "please" in rude examples ("Please shut up")
• Include: Polite examples without marker words (indirect requests: "I was wondering if you might...")
• Verify: Concept isn't just keyword matching

Pitfall: Training Data Overlap

If your test examples appeared in the model's training data, you're testing memorization, not understanding.

• Create novel examples: Don't copy from web datasets
• Paraphrase significantly: Change word choice, structure, context
• Synthetic generation: Use templates or LLM generation for fresh data

5. Dataset Size Guidelines

How much data do you need? It depends on your research method:

6. Documentation and Reproducibility

Your dataset is a research artifact. Document it properly:

• Datasheet: Purpose, creation method, annotator info, known limitations
• Examples: Show 10-20 representative samples
• Statistics: Size, balance, length distribution, vocabulary size
• Annotation guidelines: How you (or annotators) decided positive vs negative
• Version control: Track changes, note when you add/remove examples

7. Quick Start: Building Your First Dataset

Week 2 Goal: Create a small, high-quality evaluation dataset (100-200 examples) for your concept.

1. Day 1-2: Write operational definition + 20 manual examples
2. Day 3: Create 50 contrastive pairs using templates or LLM assistance
3. Day 4: Balance across topics/lengths, check for confounds
4. Day 5: Run initial benchmarks, identify dataset weaknesses
5. Day 6-7: Expand to 100-200 examples, focusing on edge cases and diversity

Deliverable: A JSON/CSV file with examples, labels, and metadata (topic, length, etc.) ready for benchmarking.

Key Takeaways

• Operationalize concepts concretely before collecting data
• Contrastive pairs are the gold standard—minimize confounds
• Balance classes and stratify across relevant dimensions
• Avoid lexical shortcuts and training data overlap
• Size needs vary by method: 100 for benchmarking, 2000+ for probing
• Document your dataset as a research artifact

In-Class Exercise: Pun Evaluation Dataset

Building on the pun theme from Week 1, we'll create a complete evaluation pipeline: generate data, test multiple evaluation methods, and apply logit lens to see where pun understanding emerges.

Part 1: Prompt Engineering for Dataset Generation (15 min)

Work in groups to craft a prompt for Claude that generates high-quality pun evaluation examples. Your prompt should produce examples in cloze format for interpretability tractability.

Generate pun recognition examples in cloze format. For each example:
1. Provide a sentence that may or may not be a pun
2. Format: "[sentence]" This is a pun. Yes or No? Answer:
3. Provide the correct label (Yes/No)
4. For puns, briefly note the wordplay mechanism

Generate 20 examples:
- 10 puns (varied mechanisms: homophones, polysemy, syntactic)
- 10 non-puns (similar structure but no wordplay)
- Vary topics: professions, food, animals, science, everyday life

Discussion questions:

• How do we ensure the non-puns aren't obviously different from puns?
• Should we include "almost-puns" that fail to land?
• How do we avoid Claude generating the same classic puns everyone knows?

Part 2: Testing Evaluation Methods (20 min)

Using the NDIF workbench, we'll compare different evaluation approaches on the same pun examples:

Method A: Probability Comparison

Compare P("Yes") vs P("No") at the answer position:

prompt = '"I used to be a banker but I lost interest." This is a pun. Yes or No? Answer:'
# Extract logits at final position
# Compare logit["Yes"] vs logit["No"]
# Classify based on which is higher

Method B: In-Context Learning

Provide 3-4 examples before the test case:

"The bicycle was two-tired." Pun? Yes
"The bicycle was broken." Pun? No
"I lost interest in banking." Pun? Yes
"I lost patience with banking." Pun? No

"Time flies like an arrow." Pun?

Method C: MCQ Format

Why is "I lost interest" a pun when said by a former banker?
A) It references losing money
B) "Interest" means both curiosity and financial returns
C) Banking is boring
D) It's not actually a pun
Answer:

Compare: Which method gives the clearest signal? Which is easiest to interpret mechanistically?

Part 3: Logit Lens on Puns (25 min)

The key question: At which layer does the model "get" the pun?

For a set of pun/non-pun minimal pairs, we'll:

1. Run logit lens at each layer on the final token position
2. Track when "Yes" probability emerges for puns vs "No" for non-puns
3. Compare across model sizes (if time permits)

Analysis Template

from nnsight import LanguageModel

model = LanguageModel("meta-llama/Llama-3.2-1B", device_map="auto")

pun_prompt = '"I used to be a banker but I lost interest." This is a pun. Yes or No? Answer:'
non_pun_prompt = '"I used to be a banker but I lost patience." This is a pun. Yes or No? Answer:'

# For each layer, decode the residual stream at the final position
# Track: P("Yes") - P("No") across layers
# Plot: layer vs. pun_score for puns and non-puns
# Look for: divergence point where model distinguishes pun from non-pun

Discussion

• Do smaller models (1B) show pun understanding? At what scale does it emerge?
• Is pun understanding "all or nothing" or does it develop gradually across layers?
• Do different pun types (homophones vs. polysemy) show different patterns?
• How does ICL context change the layer-by-layer pattern?

Code Exercise

This week's exercise will give you hands-on experience with the core concepts:

• Load a small language model and examine token probability distributions
• Implement the pun evaluation methods from the in-class exercise
• Run logit lens analysis on a pun dataset
• Compare model sizes: at what scale does pun understanding emerge?
• Calculate accuracy with confidence intervals

Project Milestone