What is a protest anyway?

Codebooks, conceptualization, and LLM-based measurement

Andy Halterman (joint work with Katie Keith)

University of Wisconsin–Madison Methods Speaker Series

23 October 2026

Document classification for measurement

Document classification is a core measurement tool in political science: protest and violence event data, manifesto coding, policy agendas, social media…

Two common approaches:

1. Train a supervised classifier: write a codebook, have annotators label a few thousand documents, train a classifier.

  • The concept is represented by the labeled examples.

2. Use an LLM: write a prompt (maybe with a codebook), label everything zero-shot.

  • The concept is represented by the prompt and whatever the model learned in training.
  • Very common now: 88% of 103 reviewed CSS papers recommend LLM annotation (Baumann et al. 2025).

So when we use LLMs to label documents, what concept is the LLM actually measuring?

Is this a protest?


Upset about the election results, a group of angry youth smashed the windows of local businesses.

In everyday usage, probably yes!

And if you ask an LLM, most will happily return PROTEST.


But four widely used political science codebooks make different decisions:


ACE ACLED CAMEO CCC
Protest with violence by protesters “Riots” are included Explicitly excluded “Riot” subclass Included
IncludedSubclassExcluded

A “bare label” prompt does not tell the LLM how it should define “protest”.

Codebooks (justifiably) disagree

(Halterman and Keith, ACL 2026)

Someone studying the success of large-scale peaceful mobilization should exclude riots and single-person protests.

A project on early warning for political violence would want to include violent protests.

“Bare label” prompting is common

Which of the following leanings would a political scientist say that the above article has? A: Liberal B: Conservative C: Neutral (Ziems et al. 2024).

You will be provided with the text of a tweet published by a member of the US Congress. Where does this text stand on the ‘left’ to ‘right’ wing scale? Provide your response as a score between 0 and 100 where 0 means ‘Extremely left’ and 100 means ‘Extremely right’. (Le Mens and Gallego 2025)

And many other papers (Halterman et al. 2021; Hu et al. 2022; Rytting et al. 2023; Heseltine and Clemm von Hohenberg 2024; Steinert-Threlkeld and Xie 2025)

This is easy and surprisingly accurate. But good measurement relies on the bare label alone fully capturing the concept.

And we saw that “protest” can be defined differently across projects.

LLM classification involves all steps of measurement

\[ \underbrace{\text{background concept}}_{\text{“protest”}} \;\overset{\color{#18453B}{\textbf{1}}}{\longrightarrow}\; \underbrace{\text{systematized concept}}_{\text{codebook } C} \;\overset{\color{#18453B}{\textbf{2}}}{\longrightarrow}\; \underbrace{\hat{Y}_i = f(X_i, C)}_{\substack{\text{predicted labels} \\ \text{from text } X_i}} \;\overset{\color{#18453B}{\textbf{3}}}{\longrightarrow}\; \underbrace{\hat\theta}_{\substack{\text{estimates, e.g.} \\ \text{from a regression}}} \]

Adcock and Collier 2001; Fariss, Kenwick, and Reuning 2020; Jacobs and Wallach 2021; Wallach et al. 2025


This talk is structured around these three steps:

  1. Conceptualization: Concepts require careful definitions (e.g. codebooks) but LLMs make it possible (and tempting) to skip this step. (Halterman and Keith, ACL 2026), (Halterman blog post 2026)
  2. Scoring: If you give an LLM a codebook, does it follow it? (Halterman and Keith, Political Analysis 2026)
  3. Inference: Bias correction can’t fix biases from an incomplete codebook if the expert labels use the same codebook. (Halterman and Keith, ACL 2026)

Annotation error can be decomposed into two parts

Alternatively, if LLM-based classification goes wrong, it can be either:

Scoring errors

The annotator misapplies the instructions.

\[\hat{Y}_i \neq Y_i\]

E.g., the codebook excludes riots, but the annotator labels one as a protest anyway.

This reflects a failure in the codebook-to-score step.

Conceptualization errors

The annotator is given a codebook that doesn’t reflect the researcher’s concept.

\[\tilde{C} \neq C\]

E.g., the researcher wants to measure peaceful protest, but the codebook doesn’t exclude riots.

This reflects a failure in the concept-to-codebook step.

Conceptualization

1. ConceptualizationBackground concept → codebook \(C\)
→
2. ScoringDoes the LLM apply \(C\)?
→
3. InferenceEstimate \(\hat\theta\)

Three levels of definition

Type I: Surface form

“Is this a protest?”

Meaning is assumed to be self-evident.

Type II: Dictionary entry

“a public (often organized) manifestation of dissent”

Generic (e.g., WordNet), intended for broad, non-specialist use.

Type III: Stipulative

“an in-person public demonstration of three or more participants in which the participants do not engage in violence and…”

These are purpose-built for specific projects, with explicit inclusion and exclusion criteria.

We argue that applied social science work needs Type III definitions. (Bare-label prompting is Type I.)

Claim: LLMs increase the risk of conceptualization failure

A tale of two annotations:

Annotation in 2015

Researcher: Classify the following document as containing a protest.

RA: What do you mean by protest? What about labor strikes? How many people…?

Annotation in 2025

Researcher: Classify the following document as containing a protest.

LLM: Certainly! I’d be happy to classify this document…

Human annotators served a useful forcing function: they require explicit, detailed definitions to be written down. In contrast, LLMs are optimized to provide an answer, even with incomplete information.

What codebook is your LLM using?

Let’s say we don’t provide a codebook. The LLM still applies an implicit definition. What is it?

We can probe this with short vignettes, each targeting one dimension where codebooks disagree:

Is this a protest? Answer with a single word, YES or NO.

Text: A lone activist stood outside the parliament building for the third
consecutive day, holding a hand-painted sign condemning the new tax law
and handing leaflets to passersby.
  • YES \(\rightarrow\) (implicitly) closer to CAMEO or CCC, which have no size threshold
  • NO \(\rightarrow\) closer to ACE (“a large number”) or ACLED (3+)

We can check 14 vignettes \(\times\) 9 models, a mix of commercial and open-weight LLMs (Halterman blog post, 2026).

What codebook is your LLM using?

https://andrewhalterman.com/post/implicit-llm-codebooks/

  • The LLMs differ on violence by participants:
    • Sonnet 5 excludes it, like ACLED
    • GPT-4o includes it, like ACE, CAMEO, and CCC
  • So a researcher using Sonnet 5 gets a dataset without violent protests, and a researcher using GPT-4o with the same prompt gets one with them.
  • But we wouldn’t know this unless we checked!

Lexical shortcuts

The ambassador lodged a formal protest with the Foreign Ministry, complaining about the treatment of her local citizens, who she claimed had been denied their rights.

  • This is not a protest under any codebook’s definition, but 7 of 9 models code it as a protest anyway.
  • The word “protest” in the text seems to be anchoring the labels. (We’ll return to this later.)

There’s no escaping conceptualization

As social scientists, we define terms in precise, specific ways based on the research question we’re asking (Type III definition).


Those definitions require more than a “bare label” prompt to fully specify (Type I) or a generic, dictionary-type definition (Type II).


If we fail to provide a full definition to the LLM, it will apply an (unknown, implicit) definition.


So: we still need to write a codebook and provide it to the LLM.


But this raises a natural question: if you give the LLM a codebook, does it actually follow it?

Scoring

1. ConceptualizationBackground concept → codebook \(C\)
→
2. ScoringDoes the LLM apply \(C\)?
→
3. InferenceEstimate \(\hat\theta\)

Does the LLM follow the codebook?

We collected three real political science codebooks, along with their source texts and human labels (Halterman and Keith PA 2026):

BFRS CCC Manifestos
Domain Political violence in Pakistan Protests in the US Party manifestos
Classes 12 8 142
Codebook length (tokens) ~2,100 ~700 ~5,100


These are deliberately hard: domain-specific concepts, lots of classes, and long codebooks.


We reformat the codebooks into labels, definitions, positive and negative clarifications, and positive and negative examples.

LLM-codebook compliance questions

We ask four questions:

  1. Are LLMs accuate at codebook-based classification?
  2. Do LLMs use all parts of the codebook?
  3. Can you tell ahead of time whether the LLMs are following your codebook?
  4. Does finetuning the LLMs improve performance?

We use only open-weight LLMs for replicability (+ newer Qwen3.5 model, not in paper).

Zero-shot performance

How accurate are LLMs at classifying according to the codebooks?

Weighted F1, 95% bootstrap CIs. * Not in the original paper. † Thinking mode on a 200-document sample.

  • Even with a full codebook, zero-shot F1 is modest. The newer Qwen3.5-9B helps on BFRS and Manifestos, but only helps on CCC with thinking turned on.
  • Lexical shortcuts again: “rally” in the text \(\rightarrow\) RALLY, even when the codebook says it’s a demonstration.

What parts of the codebook are important?

We can ablate, or remove, parts of the codebook, re-run the pipeline, and see how the results change:

Weighted F1 on BFRS; ✗ = component removed. * Not in the original paper. † Thinking mode on a 200-document sample.

  • For Mistral-7B, dropping the class definitions (bottom row) improved F1, from 0.20 to 0.29. It wasn’t “reading” the definitions very carefully.
  • Qwen3.5-9B does use them: dropping the definitions costs 0.18 (0.52 \(\rightarrow\) 0.34). It also no longer needs the output reminder.

Behavioral tests: no labels needed

Borrowing the idea of behavioral tests from Ribeiro et al. (ACL 2020): controlled changes to the input that a model following the codebook should handle.

Test Does the LLM… Result
I. Legal labels only return labels defined in the codebook? All pass except OLMo
II. Definition recovery correctly label a verbatim codebook definition? All pass except OLMo
III. In-context examples correctly label the codebook’s own examples? All pass except OLMo
IV. Order invariance give the same predictions when the categories are shuffled or reversed? Every model fails: 35–44% of Mistral-7B’s predictions change, 16–19% of Qwen3.5-9B’s

These don’t require any hand-labeled data, so you can run them early, before picking a model.

Behavioral tests: does the model read the definitions?

  • Exclusion minimal pairs: inject overrides into categories and documents.
  • Swapped labels: swap the label names, keep the definitions. F1 drops to 0.10–0.18.
  • (This is the ambassador’s “formal protest” again. The model is anchored on the label name.)

Fine-tuning vs. a newer model

Can LLMs be fine-tuned to improve performance?

We instruction fine-tune an LLM on (codebook, example, label) inputs. (Details in backup)

Weighted F1 on the development sets:

BFRS CCC Manifestos
Majority class 0.16 0.51 0.03
Mistral-7B, zero-shot 0.53 0.65 0.15
Qwen3.5-9B, zero-shot 0.65 0.59 0.28
Qwen3.5-9B (thinking), zero-shot 0.69 0.70 0.37
Mistral-7B, fine-tuned 0.82 0.72 0.38
  • A fine-tuned 2024-era 7B model still beats the newer model zero-shot (in domain).
  • But it doesn’t transfer: minimal gains on codebooks it wasn’t trained on.
  • Also costly: requires labeled training data and 2–18 hours on a GPU.
  • Alternative: good old BERT?

Inference

1. ConceptualizationBackground concept → codebook \(C\)
→
2. ScoringDoes the LLM apply \(C\)?
→
3. InferenceEstimate \(\hat\theta\)

Can’t we fix it downstream?

Any error in labels can bias downstream inference. But prediction-powered inference (PPI) Angelopoulos et al. (2023) and Design-based Supervised Learning (DSL) Egami et al. (2023) correct this bias using a small number of expert-labeled documents, even when the LLM’s error rate is high.

A natural question: “If I hand-label a validation set and use DSL or PPI, does it matter if my LLM labels are bad and my codebook is incomplete?”

To answer this, we make two assumptions:

  1. An oracle label exists, conditional on the codebook. Given a fully specified codebook \(C\), each document \(X_i\) has a true label \(Y_i\).
    • This excludes purely subjective tasks.
    • And \(Y_i \mid X_i, C \neq Y_i \mid X_i, C'\) in general.
  2. An expert annotator exists, who recovers the oracle. \(Y_i = \text{Expert}(X_i, C)\).
    • This is a major assumption, but it’s the same assumption that DSL and PPI make.
    • Note that the expert still requires a codebook: \(\text{Expert}(X_i)\) is undefined (“bare label prompting” again).

Claim: Bias correction can’t fix upstream conceptualization errors

  • DSL and PPI correct the noisy annotations to the expert annotations.
  • However, if the experts themselves are working from an incomplete codebook, their labels do not reflect the desired concept.
LLM label Gold label Inference
\(\text{LLM}(X, C)\) — Biased
— \(\text{Expert}(X, C)\) Unbiased (high variance)
\(\text{LLM}(X, C)\) \(\text{Expert}(X, C)\) Unbiased
\(\text{LLM}(X, \tilde{C})\) \(\text{Expert}(X, \tilde{C})\) Biased
\(\text{LLM}(X, \tilde{C})\) \(\text{Expert}(X, C)\) Unbiased (high variance)

The last row is reflects what we saw before: incomplete codebooks increase LLM scoring errors.

(Alternative framing: row 4 is a mismatch in estimands.)

Simulation setup

  • A researcher is interested in the effect of (peaceful) protest on voter attitudes.
  • A true DGP includes a positive effect for peaceful protests and a negative effect for violent protests.
  • The complete codebook \(C\) excludes violent protest. The incomplete codebook \(\tilde{C}\) omits the violence exclusion.
  • For each of the complete and incomplete codebooks:
    • An “LLM” labels all \(N = 10{,}000\) documents (approximated with random noise injected to the true labels). Vary the LLM’s scoring error rate from 20% to 0%.
    • An “expert” labels 1,000 (revealing the codebook-conditional true label)
    • Apply DSL

Simulation results

  • With a complete codebook, DSL works as expected: estimates are unbiased, and better LLMs shrink the variance
  • With an incomplete codebook, estimates are biased at every LLM error rate because expert annotators also apply an incomplete codebook.
  • Better LLMs make the intervals tighter, but do not affect bias.

Thus, if the expert labels come from the same incomplete codebook, post-hoc bias correction can’t fix inadequate conceptualization.

Open questions

How do you know if a codebook is complete?

Two possible (slightly unsatisfying) standards:

  1. Reliability: if two expert annotators are consistent, the codebook probably doesn’t have unresolved ambiguity.
  2. Community: domain experts agree that the codebook addresses all the relevant aspects of the concept (even if they’d make different decisions on them).

E.g., two annotators can agree because they share the same blind spots.

Can LLMs help write codebooks?

  • This is an exciting area of ongoing work (e.g., Co-DETECT from Xiong et al., HiCode).
  • LLMs seem good at finding edge cases a codebook doesn’t cover when prompted well.
  • But experimentally validating codebook variants requires expert re-annotation under each codebook version.
  • So LLMs can help find the gaps, but deciding how to fill them is still a substantive research decision.

Recommendations for codebook-LLM measurement

  1. Write the codebook. Write explicit, stipulative definitions with exclusion criteria. Don’t delegate to an LLM’s “implicit codebook”.
  2. Check whether the LLM follows it. The label-free behavioral tests are cheap and can be run early.
  3. Correct downstream with PPI or DSL using expert labels made with the same, complete codebook.
  4. Report the codebook as part of the measurement, not as a prompt-engineering detail.

Thank you!

What is a protest anyway? Halterman and Keith, ACL 2026

Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Concepts. Halterman and Keith, Political Analysis 2026

What codebook is your LLM using? https://andrewhalterman.com/post/implicit-llm-codebooks/


halterm3@msu.edu

Backup

Bare-label LLMs are sensitive to prompt phrasing

Results from re-running with three minor variants of the prompt:

https://andrewhalterman.com/post/implicit-llm-codebooks/

Prompt rephrasing discussion

I re-ran everything with three versions of the same question:

  • “Is this a protest? Answer with a single word, YES or NO.”

  • “Does the following text describe a protest? Answer with a single word, YES or NO.”

  • “Classify the following document as PROTEST or NOT_PROTEST. Respond with only the label.”

  • Most answers are stable, but some flip.

  • Sonnet 5 goes from excluding violent protests to often including them, depending on the wording.

    • I.e., it switches from an implicit ACLED definition to an implicit ACE/CAMEO/CCC one.

  • So the implicit definition varies across models and across seemingly irrelevant rewrites of the prompt.

Agreement with each codebook

A human+machine codebook format

We restructure each codebook so every class has the same components:

  • Label: the exact string the LLM should return
  • Definition: usually one sentence
  • Clarification and negative clarification: inclusion and exclusion criteria
  • Positive and negative examples
  • Plus a task instruction and an output reminder

This also lets us ablate parts of the codebook. Changes as we ablate reveal which parts of the codebook are used.

Simulation: full distributions

Behavioral tests: no labels needed (full results)

Behavioral tests: label-free test definitions

Behavioral tests: labels required

Fine-tuning details

Can LLMs be fine-tuned to improve performance?

Instruction-tune on (codebook, document, label) triples: the prompt is \(m_i = (C, X_i)\) and the output is the label \(y_i\).

Compute the loss only on the label tokens (the codebook and document are masked):

\[ \mathcal{L}(\theta) = -\sum_{i=1}^{D} \sum_{j \in y_i} \log p_\theta\left(t_{i,j} \mid t_{i,<j}\right) \]

We don’t want the model to memorize the specific codebook, just how to apply a given codebook.

::::

We use QLoRA, a parameter-efficient fine-tuning (PEFT) method (Hu et al. 2022; Dettmers et al. 2023):

  • Freeze the base weights and quantize them to 4 bits
  • Train only a low-rank update: \[W' = W + BA\] with \(B \in \mathbb{R}^{d \times r}\), \(A \in \mathbb{R}^{r \times d}\), \(r = 16\)
    • \(\approx 0.5\%\) of the weights
  • Fits on one consumer GPU (RTX 4090) ::: :::: :::::

However, minimal performance gains on unseen codebooks.

References

Angelopoulos, Anastasios N, Stephen Bates, Clara Fannjiang, Michael I Jordan, and Tijana Zrnic. 2023. “Prediction-Powered Inference.” Science 382 (6671): 669–74.
Egami, Naoki, Musashi Hinck, Brandon Stewart, and Hanying Wei. 2023. “Using Imperfect Surrogates for Downstream Inference: Design-Based Supervised Learning for Social Science Applications of Large Language Models.” NeurIPS 36.
Halterman, Andrew, Katherine A Keith, Sheikh Muhammad Sarwar, and Brendan O’Connor. 2021. “Corpus-Level Evaluation for Event QA: The IndiaPoliceEvents Corpus Covering the 2002 Gujarat Violence.” Findings of the Association for Computational Linguistics, 4240–53.
Heseltine, Michael, and Bernhard Clemm von Hohenberg. 2024. “Large Language Models as a Substitute for Human Experts in Annotating Political Text.” Research & Politics 11 (1): 20531680241236239.
Hu, Yibo, Mohammad Saleh Hosseini, Erick Skorupa Parolin, et al. 2022. “ConfliBERT: A Pre-Trained Language Model for Political Conflict and Violence.” Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 5469–82.
Le Mens, Gaël, and Aina Gallego. 2025. “Positioning Political Texts with Large Language Models by Asking and Averaging.” Political Analysis 33 (3): 274–82.
Rytting, Christopher Michael, Taylor Sorensen, Lisa Argyle, et al. 2023. “Towards Coding Social Science Datasets with Language Models.” arXiv:2306.02177.
Steinert-Threlkeld, Zachary C, and Cihang Xie. 2025. Zero-Shot Protest Event Detection with Images and Large Multimodal Models.