Essay · November 11, 2026
The Almost-Map
Generative AI and the Vocabularies of Research Validity
Read on Substack ↗Integrative essay, alignment series
Social and cognitive scientists now use generative AI in at least six distinct ways, from simulating survey respondents to labeling text at scale. This essay looks across those six areas and finds four problems that keep recurring, each rooted in a pair of words that mean almost the same thing to AI engineers and to research methodologists, but not quite. To an engineer, validation is a score on held-out data; to a methodologist, it is an argument that a measure captures what it claims to. It then turns to the practices that make those uses checkable, and asks why so few researchers have adopted the most effective of them.
1. Same data, opposite answers
Political scientists have long coded the tone of campaign advertising by hand. In one published study (Fowler et al., 2021), trained coders read 14,642 political ads, every television ad in the study and a random sample of the Facebook ones, and sorted each by its primary purpose: to promote a candidate, to attack one, or to contrast the two. The researchers then asked whether candidates strike a different tone on Facebook than on television.
Egami et al. (2024, working paper) used 13,040 of those hand-coded ads as a test bed. They gave the same codebook, with a few worked examples, to several language models, let each model label the ads, and reran the original analysis on the machine labels. The quantity of interest was the gap between Facebook and television in how often ads contrasted the candidates. With GPT-4 doing the labeling, that gap was statistically indistinguishable from zero. With Llama 2 doing it, the gap was statistically significant and negative. Same ads, same question, same regression. Only the labeler changed (Figure 1).
Because every one of those 13,040 ads had been coded by hand, the re-analysis had a benchmark to check both answers against, and it sided with Llama 2: the hand-coded ads show a clear negative difference, so the GPT-4 null was the wrong answer. A researcher starting a new project would have no such benchmark. Avoiding the cost of hand-coding thirteen thousand documents is the whole reason to use a model, and without the hand-coding there is no principled way to know which of two plausible answers to trust.
I start here because the whole essay is in this example. Imagine two researchers who ran that study for real, one labeling with GPT-4 and the other with Llama 2. Neither need have been careless by the field’s current standards. Each could have reported that the model’s labels agreed reasonably well with a sample of human ones, which is the kind of check that dominates the published literature (Section 9). What went wrong is subtler than carelessness. The vocabulary of AI engineering and the vocabulary of research methodology almost map onto one another. Checking a model against a gold set sounds like validating an instrument. An eval sounds like a reliability check. Temperature sounds like measurement error. Consensus among model runs sounds like accuracy. The words in each pair sit close enough together that a careful person can move from one to the other without noticing, especially when the move points toward the result they were hoping for. They are not synonyms, though, and the small gap inside each pair is where a sound-looking result can quietly stop being sound. That pattern gives this essay its title, and I’ll call it the near-map for short: two terms close enough that one passes for the other.
A great many people are now studying how AI is changing science, and most of them have more data, more collaborators or more computing than one person writing in his spare time. This essay does something narrower. It looks at six areas where social and cognitive scientists are already using generative AI, and at the practices that let researchers do any of the six well, which are not a use at all. Each area has its own short essay in this series; this one looks across them. For each, I ask a plain question: does the tool produce the kind of thing the research claim needs? Where the answer is no, I try to show the problem with a concrete case and to connect it to an older argument in the philosophy of science that turns out to describe it well. The aim is to be useful to researchers who already use these tools, not to argue that they should stop.
Section 2 walks through the six areas, with an example of each, and introduces those practices. Section 3 sets out a four-question test. Sections 4 through 7 describe four problems that recur across the areas: plausibility mistaken for truth, run-to-run variation mistaken for ordinary measurement error, agreement mistaken for validity, and automation mistaken for interpretation. Section 8 describes the one place where a gap closes, and the conditions it closes on. Section 9 asks why so few studies use that fix. Sections 10 and 11 step back, first to the philosophy of science and then to the sociology of how two research communities trade ideas, which is where I think this essay adds the most. Section 12 places the argument among related frameworks, Section 13 notes what is out of scope, and Section 14 closes with practical guidance.
2. Six uses of generative AI, and the practices that make them checkable
The six areas below are the ones this series covers. They are not a census of everything researchers do with language models, but they cover most of the published methods literature, and they differ in what the model is asked to be: a respondent, a mind, a coder, a population, a source of ideas, or a labeler. Table 1 summarizes them. The sections that follow are organized by problem rather than by area, so each area turns up where its problem does: synthetic participants and models of cognition in Section 4, text annotation in Sections 5, 6 and 8, and qualitative coding, simulated societies and hypothesis generation in Section 7.
| Area | What researchers do | Example | Most defensible use today |
|---|---|---|---|
| 1. Synthetic participants | Ask a model to answer survey items as a described person | Argyle et al. (2023) | Pretesting and piloting instruments |
| 2. Models of cognition | Fine-tune a model on human choices and treat it as a model of the mind | Binz et al. (2025) | A benchmark for predicting behavior |
| 3. Qualitative coding | Apply a codebook to interviews, or ask the model to find themes | Fuławka et al. (2026) | Deductive coding checked against an external criterion or human coders |
| 4. Simulated societies | Populate a world with model-driven agents and watch what emerges | Park et al. (2023) | Exploring the consequences of assumptions |
| 5. Hypothesis generation | Ask a model to propose ideas worth testing | Tong et al. (2024) | Ideation that feeds ordinary empirical tests |
| 6. Text annotation at scale | Have a model label thousands of documents for a downstream analysis | Gilardi et al. (2023) | Labeling, when paired with a statistical correction |
| Doing it right | Correct estimates built on model labels; pin, log and disclose | Angelopoulos, Bates, et al. (2023); Feuerriegel et al. (2026) | Not a use; the practices that make the others checkable |
Synthetic participants. A researcher writes a short profile into a prompt, for instance a retired teacher in her fifties from a small town who votes Republican, and asks the model to answer a survey item as that person. Repeat the exercise across many profiles and the result looks like a sample. Argyle et al. (2023) named this silicon sampling. The appeal is real: Ashokkumar et al. (2026) found that model-predicted treatment effects across 70 preregistered survey experiments correlated with the actual effects at r = 0.85. The risk is that a model speaking for a group tends to flatten it, a problem Section 4 takes up.
Models of cognition. Here the model stands in for a mind rather than a respondent. Centaur (Binz et al., 2025), the best-known case, is a large language model fine-tuned on more than ten million human choices from 160 psychology experiments. Most of those experiments already had a purpose-built cognitive model, and where one existed, Centaur predicted the choices of new participants better in all but one comparison. Whether a model that predicts behavior well has thereby explained it is the question Section 4 addresses.
Qualitative coding. A researcher hands the model a codebook and a stack of interview transcripts and asks it to tag each passage with the right codes. This deductive use can work well, especially when the codes can be checked against something outside the text. The inductive version, asking the model to discover what the interviews are about, is harder to defend, for reasons Section 7 explains.
Simulated societies. The researcher builds a population of model-driven agents, each with a memory and a plan, and lets them interact. The generative agents of Park et al. (2023), twenty-five characters living through a day in a small sandbox town, became the template, and later versions build each agent from a two-hour interview with a real person (Park et al., 2024, preprint). These simulations are good tools for asking what follows from a set of assumptions. They are much weaker evidence about how real societies behave, as Section 7 discusses.
Hypothesis generation. A model proposes ideas worth testing. In one well-designed version (Tong et al., 2024), a model works through a map of causal claims extracted from tens of thousands of psychology articles and proposes new hypotheses about well-being. Used this way, as a source of leads for ordinary research, the model is doing something it is well suited to do, which is why Section 7 treats it as the mild case.
Text annotation at scale. This is the most common quantitative use. A researcher has thousands of tweets, news articles or open-ended survey answers and needs each one labeled for stance, topic or tone. Instead of hiring coders, they prompt a model. Gilardi et al. (2023) showed that ChatGPT could label tweets and news articles more accurately than crowd workers on most tasks, at a fraction of the cost, and researchers adopted the practice quickly. The political-ads example in Section 1 belongs to this area. Sections 5 and 6 look at what can go wrong, and Section 8 at how to fix it.
Doing it right. The last row of Table 1 is not another use. It is the set of practices that let the six uses be checked. Some are statistical: estimators that combine many model labels with a small, randomly chosen set of human ones to produce trustworthy error bars (Section 8). Some are habits: recording exactly which model version produced which output, and running a pipeline more than once (Section 5). Some are reporting standards, checklists that say what a paper using a language model should disclose (Section 14). None of this is glamorous. The statistical correction in particular is the clearest example in this series of a gap that actually closes, on conditions a researcher can check.
3. A four-question test
The working claim of this series is that many of the problems across the six areas go wrong in the same way, and that four questions catch most of them. Suppose a researcher wants to know what share of tweets about a new policy are hostile to it, and plans to have a model label 50,000 tweets.
What does the claim require? Here, a population estimate: a percentage, with a margin of error that means what it says. Other claims require other things: a prediction, an interpreted meaning, a mechanistic explanation, a hypothesis worth testing. These are different goods, and they are not interchangeable.
What does the tool actually produce? A label for each tweet that is usually right and sometimes wrong, with errors that may not be random. More generally, language models produce plausible, on-distribution output, meaning output that resembles the text they learned from: labels, predictions, fluent answers. They do not, by default, produce truth, meaning or mechanism, and size alone does not supply them. In the largest test of annotation pipelines to date, models above 70 billion parameters still produced conclusions the human labels did not support for about one hypothesis in three (Section 6).
Is there a guardrail that converts what the tool produces into what the claim requires? For the tweets, yes. Hand-code a random sample, measure how the model’s labels differ from the human ones, and correct the estimate accordingly (Section 8). Other guardrails include a validated codebook with reliability reported against human coders, a held-out test the model could have failed, and ordinary empirical testing of a hypothesis after it is proposed.
Does the guardrail match the claim? If the researcher reports the corrected percentage with its interval, yes. If they report only that the model agreed with human coders most of the time, and then compute the percentage from the raw model labels, the check they ran supports a different claim from the one they are making.
A use is aligned when the tool’s output matches what the claim requires and a guardrail converts one into the other. It is misaligned when output optimized for plausibility is treated as a source of truth, meaning, measurement or mechanism without the guardrail that would earn that status. Where no suitable guardrail exists, the work is exploratory. That is a legitimate thing for work to be, and it should say so. Figure 2 lays the test out as a flow, with the tweets worked through at each step.
The questions have a lineage. Since Messick, the prevailing view in measurement theory has been that what gets validated is a proposed interpretation or use of a score, not the instrument that produced it (Messick, 1995). Kane’s argument-based approach turns that view into a procedure: state the claim the score is meant to support, then ask whether the evidence supports each inference the claim needs (Kane, 2013). The four questions are a plain-language version of that procedure. The prevailing view has a serious dissent. Borsboom et al. (2004) argue that validity belongs to the test after all, and that a test is valid only if the attribute it targets exists and causes the variation in its scores; Workflow 1 relies on their account. The four questions, as I read them, do not need that dispute settled. On either account, a model’s agreement with a criterion does not by itself show that its output means what the claim needs it to mean.
Note what the test does not ask. It does not ask whether the model is good, how much autonomy to give it, or how uncertain its outputs are. Those are useful questions, and other people are asking them well (Section 12). This test asks only whether the thing produced is the kind of thing the claim needs. A model can be excellent, tightly supervised and consistent from run to run, and still produce something that cannot support the sentence written about it.
Four problems recur across the six areas: plausibility mistaken for truth, run-to-run variation mistaken for ordinary measurement error, agreement mistaken for validity, and automation mistaken for interpretation. Table 2 sets them side by side, and Sections 4 through 7 take them one at a time.
| Problem | What the tool delivers | What it is taken for | Where it bites | Section |
|---|---|---|---|---|
| Plausibility versus truth | Likely, fluent output; a good predictive score | A true answer; an explanation of behavior | Synthetic participants; models of cognition | 4 |
| Stochasticity versus reproducibility | Output that varies from run to run and drifts between versions | Ordinary measurement error that averages out | Every area, most sharply on codes that need judgment | 5 |
| Calibration versus validation | Agreement with a benchmark (an eval) | Validation: a measure of the construct, and an estimate that can be trusted | Text annotation, and any pipeline that feeds an estimate | 6 |
| Automation versus interpretation | Consistent labels at scale | Interpretation: meaning settled by people with standing to settle it | Qualitative coding; simulated societies; hypothesis generation, mildly | 7 |
4. Plausibility versus truth
The first problem sits closest to how the tool works. A language model is trained to produce text that is likely. Likelihood and truth are correlated, which is what makes the tool useful and the error easy. Ask which planet is closest to the sun and the likeliest answer is also the true one. Ask for the page on which a little-read book makes a particular claim, and the likeliest answer is a plausible page number, whether or not the book makes the claim at all. Where likelihood and truth part ways, the model has no particular allegiance to truth.
Predicting is not behaving
The cleanest demonstration of the gap is also the smallest, and it does not involve a language model at all. Take a reversal learning task: two options, one initially rewarded more often, the contingencies flipping partway through. Now take a model that does little but repeat its previous choice, with some fixed probability. Palminteri et al. (2017) built this example to show that a model has to be simulated, not only fitted, before its claims about cognition can be trusted, and Namazova et al. (2025, preprint) return to it. Fitted to a learner’s choices, the model achieves respectable predictive accuracy, because it is right on most trials and wrong mainly around the switch. Run it generatively, letting it produce a sequence on its own rather than from a human’s history, and it keeps choosing the option it started with. It never reverses. What you see is predictive fit with no behavioral adaptation. Figure 3 shows the split in a small simulation.
Keep this little model in mind, because versions of it turn up throughout this essay. Its accuracy score is real, and it looks like evidence that the model has learned the task. What it has learned is a single number: how often people repeat their last choice. It knows nothing about which option pays, or that the payoffs ever change. So the score answers a question about habit, when the question being asked is how people notice that the rewards have moved. A good number produced by something that cannot do what the number seems to show is the problem of this section in its plainest form.
Is a model of behavior a model of the mind?
The most prominent case is Centaur (Binz et al., 2025), which deserves its reputation as a predictive benchmark. Trained on Psych-101, the corpus described in Section 2, it predicts held-out human behavior across a wide range of tasks, and its internal representations can be aligned with neural data. As an account of how minds work it is something else, and several critiques have said so. Liu and Ding (2026) show it predicts well even when the task instructions are stripped out, which suggests it leans on surface statistical regularities rather than the process it is supposed to capture. A model of cognition should also go wrong where people go wrong, and the next two studies check whether Centaur does.
Namazova et al. (2025, preprint) also ran Centaur through the same reversal test, with choices from a standard learning model standing in for participants, and the same split appeared. Centaur outpredicts the base Llama model it was fine-tuned from, yet generates weaker reversal dynamics than that base model does; some runs never adapt to the reversal at all. On this task, fine-tuning on more than ten million human choices made it a better predictor and a worse simulator. On the Wisconsin Card Sorting Test, the one task outside its training set, a small domain-specific model beat it on both predictive and generative measures.
Fascendini et al. (2026, preprint) make the same point without any fine-tuning at all, which rules out Centaur’s training procedure as the culprit. They built matched riddle pairs that preserve surface structure while removing the trick, so that the correct answer requires only a literal reading. Nine off-the-shelf models solved 84.9 percent of the genuine riddles and 50.7 percent of the stripped ones, overapplying inventive reasoning where none was called for. A hundred human participants showed the reverse pattern, 50.5 percent on the genuine riddles and 80.5 percent on the stripped ones (Figure 4). A memorization check completed the picture. Given the first 40 percent of each genuine riddle, the models that could finish more of them near-verbatim were also more accurate on them (Spearman’s rho = .88 across the nine models), and memorizing a particular riddle predicted higher accuracy on it and lower accuracy on its stripped twin. The authors read this as retrieval standing in, at least in part, for reasoning. On this evidence, some of the fluency was retrieval dressed up as reasoning.
A good prediction does not license an explanation
None of this is new. Shmueli (2010) named the confusion, writing for statisticians in Statistical Science, and argued that explanatory and predictive modeling diverge at every stage of the modeling process, from design through variable selection to reporting, and that conflating them impedes scientific progress. Her complaint ran in one direction: against disciplines that assume a model with high explanatory power must therefore predict well. Centaur inverts it. Here a model with extraordinary predictive power is taken to have thereby explained. The inversion is new. The diagnosis is sixteen years old. What we are seeing is a known methodological error arriving in cognitive science on unfamiliar hardware, which may be why it is not always recognized as one.
The strongest objection to all of this comes from inside psychology rather than from engineering. Yarkoni and Westfall (2017) argue that psychology’s near-total investment in explanation has left it with theories that explain everything after the fact and predict very little in advance, and that the field would be better served by borrowing machine learning’s out-of-sample discipline and treating predictive accuracy as a goal in its own right. That argument is correct, and this essay does not contest it. Hofman et al. (2021) offer the constructive version of the same position, arguing that explanation and prediction are complementary rather than rival aims and are best pursued through integrative modelling that holds a model to explicit standards on both at once.
My point is narrower: a predictive success does not by itself license an explanatory sentence. An actuarial table predicts how long people will live without explaining why anyone dies. The license to explain has to be earned separately, by something other than the number that established the prediction. Centaur is a case where the license is easy to assume. Hofman et al.’s (2021) proposal presupposes as much, since a scheme for holding a model to both standards at once would be unnecessary if meeting one also met the other. The disagreement with Yarkoni and Westfall (2017) is not about whether prediction is worth having but about what a prediction, once had, entitles you to say.
Orr et al. (2025, preprint) close the case from the other end, and their title is the argument: Not Even Wrong. Ask what would falsify Centaur. Suppose its predictions came out at chance. What would that refute? The training data, the optimizer, the fine-tuning method, the architecture, any of seventy billion parameters? Nothing is at risk, because nothing was committed to. They also note that Centaur’s generality across 160 tasks is an illusion: Centaur performs exactly one task, estimating the most likely next token. The generality lives in the procedure that renders every paradigm as text, not in the model.
Synthetic participants do better on averages than on groups
The same problem appears among synthetic participants, in a different form. Silicon sampling conditions a model on a profile of a person, in its original form a demographic one, and reads its output as that person’s response. Argyle et al. (2023), who coined the term, also named its governing concept, algorithmic fidelity, and were careful about what fidelity would have to mean: evidence that what we learn from a model might generalize to the people it stands in for, and explicitly not a claim that it could simulate any particular individual. Later work suggests that whether that generalization holds depends heavily on what is being asked and how the model was conditioned. I take up both variables in the Workflow 1 essay; the short version is that grounding a model in a person’s own interview or survey answers brings it materially closer to that person than a demographic sketch does (Park et al., 2024, preprint), without yet amounting to validated measurement.
For aggregate treatment effects in survey experiments, the record is encouraging, as the Ashokkumar et al. (2026) result in Section 2 shows. For the views of particular groups, it is not. Wang et al. (2025) show that models asked to speak as members of identity groups flatten those groups and collapse within-group heterogeneity, producing responses that are recognizably in the right register and wrong in exactly the way that matters. Ask a model for a hundred answers from a hundred different members of a group and the answers cluster around one typical voice. The approach fails hardest exactly where the differences among people’s lived experience are what the study is about.
Fluency is not understanding
The same doubt reaches the idea that a model has a psychological profile of its own. Across 56 models and 29 psychometric instruments, Meyer et al. (2026, preprint) attribute between 81 and 90 percent of the variation between models to response bias rather than to anything resembling a trait. What looks like a model’s psychological profile is largely an artifact of how it answers questions, not of what it is like. The output is plausible, but it mostly measures the model’s answering habits, much as a respondent who agrees with every statement on a questionnaire reveals an answering style rather than a personality.
The human-factors literature has a name for the reader’s half of this problem. Dong et al. (2026) call it epistemia: attributing genuine understanding to a system on the strength of surface fluency, in the absence of verification. It is a useful word because it puts the error in the reader rather than in the model. The model is not claiming to understand. We are.
Hicks et al. (2024) supply the philosophical vocabulary for what sits underneath the epistemia. They argue that language-model output is indifferent to truth rather than merely inaccurate, which is what the philosopher Harry Frankfurt (2005) meant by bullshit in its technical sense. A liar is oriented enough toward the truth to conceal it. A system indifferent to truth is not oriented toward it at all, in either direction. Plausibility is what output looks like when it is produced by the second kind of process, and mistaking it for a failed attempt at the first is the error this section has been describing under several names.
5. Stochasticity versus reproducibility
Could a reader rerun your analysis and get your numbers? With a language model in the pipeline, often not. That makes the second problem the most technical and in some ways the most damaging, because it threatens not a study’s conclusion but the possibility of checking it.
The same prompt can give different answers
What temperature is. A language model writes one small piece of text at a time, at each step choosing among possible next pieces ranked by probability. Temperature is a setting that controls how adventurous that choice is. At a high temperature the model more often picks less likely options; at temperature zero it is supposed to pick the most likely one every time.
Send the same prompt to the same model with the same settings twice, and the two outputs may differ. OpenAI’s own documentation says its API’s responses are “non-deterministic by default” and offers only “(mostly) deterministic” outputs when every setting, a fixed random seed included, is held the same (OpenAI, n.d.; see also Anadkat, 2023). One demonstration sent the prompt “Tell me about Richard Feynman” to the same model a thousand times at temperature zero and got back 80 different completions (He & Thinking Machines Lab, 2025). Layered on top of this is version drift. Barrie et al. (n.d., working manuscript) document that continual updates to closed models produce temporal variability in outputs on identical tasks, which means that a study run in March and a replication run in September may not be running the same instrument, and if their outputs differ, neither team can say why.
Reproducibility is part of what makes a finding a finding, and a measurement that cannot be repeated is hard to treat as a measurement at all. The near-map bites again here, because the engineering vocabulary has a word that sounds as though it addresses the problem. Temperature invites a methodologist to read run-to-run variation as measurement error, a quantity that averages out over repetitions. That reading is half right. Within a single model version, sampling variation does behave somewhat like random error, which is why running a pipeline several times helps. A bathroom scale that reads a pound heavy one morning and a pound light the next will average out over a month of weighings. A scale that someone swaps for a different model overnight will not, and version drift is that swap, made without telling you. Nor, as the next paragraphs show, is the variation spread evenly enough to average away.
The remedies are simple and often unreported
The standard remedies are well known: pin the model version, log the seeds and settings, run the pipeline more than once, and report the spread of outputs rather than a single draw. None of this is difficult. Much of it is not yet routine. Desai, Card, et al. (2026, preprint) screened 2,143 articles from eight flagship social science journals, published between 2022 and late 2025, and found 27 that use a language model for measurement. Their summary is that validation practices in those papers are “inconsistent and limited.” Of the 27, 26 name the model and 22 give the prompt, but only 11 report the decoding settings and only 4 describe how answers were extracted from the model’s output.
No field-wide audit yet reports what fraction of published analyses could be rerun, and I do not want to guess. The narrower point is enough. When 16 of the 27 papers in a careful audit do not report their decoding settings, a reader cannot rerun those pipelines as they were run, even if their authors could.
Instability concentrates on the judgment calls
Lin and Zhang (2026) put a number on what the instability costs, and the number comes in two parts. Using GPT-4 Turbo to code 200 news articles from the Dynamics of Collective Action dataset, an established benchmark for protest events, and running an identical prompt at identical parameters 25 times over, they report an intraclass correlation of 0.934 for whether police were present and 0.518 for the size of the protest. Averaging those two would be a misleading summary, because the split is the finding. Police presence is close to a yes-or-no perceptual judgment, and the instrument reproduces itself well on it. Protest size requires the model to read a description and place it in a graded category, and there the instrument’s agreement with itself sits on the line between poor and moderate (its 95 percent interval, 0.46 to 0.58, straddles the 0.5 cut-off in the guideline the authors use), a level no human coding team would report without comment. Stochasticity does not spread itself evenly across a codebook. It concentrates on the items that require judgment, a result that returns in a different form in Section 7.
Stable agreement is not accuracy
Borchers et al. (2026) supply the clearest reason to care, and I have a personal reason to be precise about their result. Their paper is titled “Temperature and Persona Shape LLM Agent Consensus With Minimal Accuracy Gains in Qualitative Coding.” Across six open-source models, eighteen configurations and more than seventy-seven thousand coding decisions measured against a human gold standard, they found that temperature, in all six models, and persona settings, in four of them, shaped whether and when model agents came to agree with one another, while neither produced robust gains in accuracy against human coders. Mixing personas delayed consensus without improving the coding, and a single agent working alone matched or beat the deliberating group in most conditions. While drafting this series with an AI assistant, I described this paper in an early draft as showing that the gains in such work come from collaborative structure. It shows nothing of the kind, and the slip is the one this essay is about: a finding about agreement had been promoted, without anyone quite deciding to promote it, into a finding about correctness.
The practical lesson is that it is possible to tune a system into confident, stable agreement that is no more accurate than a single model working alone. Stability is not validity, and on this evidence it is not even a good sign of validity. It is a property of the ensemble, and accuracy is a property of the relationship between the ensemble and the world. Ten watches set from the same fast clock agree to the second, and every one of them is wrong.
6. Calibration versus validation
Section 5 ended with ten watches that agree to the second and are all wrong. The third problem, agreement mistaken for validity, is that image at full scale. It is the most common of the four problems and the one where the near-map is narrowest, so it gets the longest treatment.
Agreement with human coders became the standard check
Gilardi et al. (2023) published a result that changed what a great many researchers did the following Monday: ChatGPT annotated tweets and news articles for relevance, stance, topic and frame more accurately than crowd workers on most tasks, with higher intercoder agreement than either crowd workers or trained annotators, at roughly a thirtieth of the cost. The paper is careful. What other researchers took from it was the recipe, not the caution: benchmark the model against human coders, report the agreement, use the labels.
A large and fast-growing literature now follows that recipe. In a corpus of 231 recent papers on AI and social-science methods that I assembled for this project, text annotation is the second-largest theme, with 78 papers, behind only silicon sampling at 102. The themes were assigned by a language model reading titles and abstracts, so by this essay’s own standard the counts are rough.
Now watch the pipeline run. An engineer applies a prompt across an unlabeled corpus. The model returns structured labels. The labels are checked against a human-annotated benchmark. A researcher watching the same three steps sees a codebook, a coding pass and a reliability check. For three stages the two views are indistinguishable. It looks like measurement.
At the fourth stage the resemblance breaks, and everything depends on noticing.
Eval and validation. In machine learning, an eval (short for evaluation) tests a model’s outputs against a fixed set of correct answers and usually summarizes the result as a single score, such as accuracy or agreement. In research methodology, validation is a broader argument: that a measurement captures the construct it is meant to capture and, when the measurement feeds an analysis, that the resulting estimate can be trusted. An eval can be one piece of evidence in a validation argument. It is not the whole argument.
The engineer has run an eval. The eval asks: do the model’s labels match the benchmark labels? The researcher needs validation. Validation asks: is the estimate built from these labels unbiased, and does its confidence interval cover the true value as often as it claims to? These are different questions, and a high answer to the first does not answer the second.
Psychometrics drew this line in the 1950s
The distinction is not new. Psychometrics has carried it as a load-bearing separation since Loevinger (1957) argued that a test’s internal consistency is only one of three kinds of evidence its validity requires, alongside what its items are about and how its scores relate to behavior outside the test, and that none of the three is optional. Cook and Beckman (2006) restate it for clinical measurement, where the modern treatment folds the older face, content and criterion distinctions into a unitary concept of construct validity, and reliable scores are necessary but not sufficient for a valid interpretation. The researchers who took up the tool did not take up the principle with it, though. The eval arrived in social science with a benchmark, a metric and a leaderboard, but without the apparatus that would have told a psychometrician what the metric does not license.
Some computer scientists have since borrowed the apparatus for their own work, and they described this gap before I did. Jacobs and Wallach (2021) imported construct validity into the evaluation of computational systems, decomposing it into face, content, convergent, discriminant, predictive, hypothesis and consequential validity and applying the result to fairness measurement. Wallach et al. (2025) then made the general case, arguing that evaluating generative AI systems is a social science measurement challenge and adapting a four-level framework from political science (Adcock & Collier, 2001) that runs from a background concept through a systematized concept and a measurement instrument down to instance-level measurements. Their diagnosis is that machine learning evaluation typically jumps from the first level to the third, selecting an instrument for a concept it never pinned down. That is this essay’s argument, made first and for the other audience. What I add is an extension, not a discovery.
In social science, the gap is two gaps
The extension is needed because social science brings three things a benchmark usually lacks. A machine learning benchmark is a fixed set of items, and the question a validity theory asks of it is whether the items measure the construct. There is usually no target population, no quantity to be estimated and no sampling design. Social science has all three. A researcher annotating a corpus has a population the corpus is meant to represent (say, all English-language posts about a policy in a given year), a quantity to be estimated (the share that are hostile to it) and a design by which documents were selected. The same problem therefore lands one step further downstream, and it lands twice: first on whether the instrument measures the construct, and then on whether the estimate built from the instrument’s outputs is unbiased and whether its interval covers. In this setting the near-map hides not one gap but two gaps in series (Figure 5). The correction described in Section 8 closes only the second of them, and the authors of that correction are the first to say so.
The first gap has now been measured from inside the engineering lane, and at scale. Desai, Truong, et al. (2026) adapt the convergent and discriminant validation of Campbell and Fiske (1959) to interrogate fifty-six capability and safety benchmarks across fifty-three models. The logic is simple: if two benchmarks claim to measure the same safety property, models that rank high on one should rank high on the other. Rankings on benchmarks assigned the same safety concept often correlate only weakly, which is what it looks like when a concept is being defined inconsistently across the very instruments that claim to measure it. This is the first gap demonstrated on machine learning’s own benchmarks, with the tools Campbell and Fiske built for psychological tests. What machine learning evaluation has not yet taken up is the second gap, which only appears once there is a population, an estimand and a sampling design for the estimate to be wrong about.
Accurate labels can still produce a wrong estimate
Egami et al. (2024, working paper) have measured how large the second gap can be. In their simulations, ignoring annotation error biases downstream regression coefficients by as much as 30 percent of their true value at 90 percent annotation accuracy, and by 18 percent at 95 percent. The coverage of nominal 95 percent confidence intervals collapses to 32 percent and 58 percent respectively (Figure 6). An interval a researcher believes will cover the truth nineteen times in twenty is covering it fewer than two times in three. The reason is not exotic: prediction errors are not random. They correlate with the outcome, with the covariates and with whatever is unobserved, so they do not average away. Suppose sarcastic posts are more often hostile than friendly, and the model tends to read sarcasm literally. Its errors will then pile up among the hostile posts, which are exactly the ones the analysis is counting. Accuracy is not a license to skip the correction.
Gilardi et al. (2023) themselves reported relevance-task accuracies of 70, 81, 83 and 59 percent. The paper that founded the practice was already operating inside the range where the correction is required.
This is why the political-ads result in Section 1 matters more than any bias figure. The labels from GPT-4 and from Llama 2 were each good enough to look usable, and they led to different conclusions about the same ads. The choice of annotator has become a researcher degree of freedom: a choice that can change the result, made for reasons that rarely appear in the paper, in a literature that does not yet require it to be disclosed. None of this is a gap between good practice and bad. The eval is real work, honestly done. It earns a different claim from the one being made.
One complication cuts against the tidiness of everything above. Ziems et al. (2024) benchmarked thirteen models across 25 computational social science tasks. On the free-form tasks, where a model writes an explanation rather than choosing a label, they found that its output often exceeded the quality of the crowdworker references it was being scored against. Where that happens the gold standard is the weaker instrument, and the direction of the check reverses. This does not rescue the pipeline, since an estimate built on labels of unknown quality is not improved by those labels turning out to be better than a benchmark of unknown quality. It does mean that treating the machine output as the thing checked and the human labels as the check is a working assumption rather than a fact, and every correction method in Section 8 rests on that assumption.
Wrong conclusions are common, and agreement does not predict them
How often does a conclusion built on uncorrected model labels turn out to be wrong? Baumann et al. (2025, preprint), the largest study in this corpus, have measured it. Across 37 annotation tasks drawn from 21 published studies, 18 models, 13 million labels and 1.4 million regressions testing 2,361 hypotheses, they ask how often a researcher following ordinary practice would reach a conclusion the gold labels do not support. The answer is roughly one hypothesis in three for models above 70 billion parameters, and one in two for the small ones. Those are rates for ordinary practice. A researcher free to choose among the models and prompts they tested could have manufactured a false positive for 94.4 percent of true nulls, and reversed the sign of 68.3 percent of real differences. For most questions in their sample, some defensible-looking combination of model, prompt and settings returns the wrong answer. They name the mechanism in vocabulary the field already had: the garden of forking paths of Gelman and Loken (2013, unpublished manuscript; 2014), with the configuration of the labeler as the fork. And the result researchers most need to hear is a null one. Human intercoder agreement on a task is uncorrelated with the risk of reaching a wrong conclusion on it. In their data, the reliability statistic a methodologist would reach for first was no guide to that risk.
That null result is the fourth statement of a single pattern, and the four literatures that produced them barely touch. Of the four papers, only Baumann et al. (2025, preprint) cite any of the others, and that citation runs to Egami et al. (2024, working paper), their nearest neighbor in method; Borchers et al. (2026) and Namazova et al. (2025, preprint) cite none of the three. Table 3 sets their findings side by side: consensus without correctness, accuracy without valid inference, fit without the generative dynamics, and reliability without validity. Four teams in four fields each report that the number their subfield checks first, whether agreement, accuracy or fit, can look good while the result it is supposed to vouch for is wrong.
| Study | Field | The number checked first | What it did not vouch for | Scale |
|---|---|---|---|---|
| Borchers et al. (2026) | Qualitative coding | Agreement among model agents | Accuracy against human coders, which barely moved | 6 models, 18 configurations, more than 77,000 coding decisions |
| Egami et al. (2024, working paper) | Regression on annotated text | Label accuracy of 90 to 95 percent | An unbiased coefficient, and an interval that covers | Simulations, and re-analyses of two published studies |
| Namazova et al. (2025, preprint) | Cognitive modelling | Predictive fit | The behavior itself: run on its own, the model never reverses | Reversal-learning task, 32 simulated runs per model |
| Baumann et al. (2025, preprint) | Annotation at scale | Human intercoder agreement | Whether the conclusion is right: agreement is uncorrelated with the risk | 37 tasks from 21 studies, 18 models, 13 million labels |
The principle underneath all four is old, and Loevinger’s (1957) separation is their common ancestor, though none of the four papers cites it. What is new is the pattern of its rediscovery: four literatures, four vocabularies, roughly three years and a single cross-citation. That, I think, is what a missing shared vocabulary looks like from the outside.
7. Automation versus interpretation
Imagine three researchers reading the same interview with a teacher who has just left the profession. One hears a story about burnout, one a story about school leadership, one a story about pay. Which of them is right? In the reflexive tradition of qualitative research that question is badly posed, and the fourth problem, automation mistaken for interpretation, begins there. It differs from the others in kind rather than degree, and better engineering alone does not close it. It bites hardest in qualitative coding, and it reaches simulated societies and hypothesis generation in milder forms, which is the order this section takes.
Reflexive analysis needs people who disagree
Some research objects are not labels waiting to be applied. Interpretive, inductive qualitative analysis, in the reflexive tradition, treats meaning as something constructed between a researcher, a text and a position from which the researcher reads. The analysis of the teacher’s interview includes the argument the three researchers have about which reading the transcript best supports, and the settlement they reach. Braun and Clarke, whose work defines reflexive thematic analysis, have since joined 417 other qualitative researchers from 32 countries in rejecting generative AI for reflexive work altogether, on methodological grounds and on grounds of social and environmental justice (Jowsey et al., 2025). Reliability, there, is not a similarity score between outputs. It is the negotiated agreement of people who disagreed and had to work it out. When two runs of a model agree, or when two model agents agree, as in the consensus experiments of Section 5, that is semantic consistency. It may be two draws from the same bias, agreeing.
De Paoli (2024), who tested how far GPT-3.5 could carry Braun and Clarke’s six phases on two published interview studies, is candid about where machine interpretation stops. On interviews with video-game players the model recovered 9 of the original analysts’ 13 themes but never produced two they had clearly valued, a psychological perspective and violence and aggression; on interviews with university lecturers it proposed themes the original analysis had not considered. An evaluation of a second pipeline, GATOS, found codes framed again and again as “need for something” or “lack of something,” a pattern that “does not fully convey how these codes interact with each other” (Kim et al., 2025, p. 606), with meaningful summary points lost in repeated rounds of embedding and clustering. The model can produce something that looks like a thematic analysis. What it cannot yet produce is the thing thematic analysis is for.
Deductive coding works when something outside the model checks it
The area does split, though. Asked to apply a fixed, human-validated codebook, a model does well, and this is among the most defensible uses this essay covers. Fuławka et al. (2026) show what it looks like done properly: a 47-item codebook of decision reasons applied to free-text explanations from a risky-choice experiment, then validated not against agreement but against a behavioral criterion, since each reason implies a specific choice through a stated rule. Alignment between reason-implied choices and actual choices reached 95 percent, and reason profiles built for each choice problem predicted 76 percent of held-out choices, against 73 and 75 percent for two versions of prospect theory, a standard benchmark. That is a guardrail that earns its claim, and it earns it by substituting a formal, checkable external criterion for the second human coder.
The catch is that most qualitative corpora offer no such criterion. Attitude, emotion and meaning rarely imply a verifiable prediction. That is why human-coded reliability remains the default rather than the exception, and why the model is best treated as one coder among several, whose codes still have to be reconciled by someone with standing to reconcile them.
Simulations show what follows from assumptions
Simulated societies run into the same problem at a larger scale. Generative social simulation populates a world with agents and waits to see whether something social, such as a rumor spreading or opinion polarizing, emerges. The architecture is real (Park et al., 2023), and the most ambitious version conditions agents on two-hour interviews with 1,052 real people (Park et al., 2024, preprint). On held-out General Social Survey items, agents built from the interviews matched each person’s answers 83 percent as well as the person matched their own answers two weeks later (86 percent with survey answers added, against 74 percent for agents given demographics alone). That is an impressive number. It is also a number about predicting individuals, not about the validity of emergent collective dynamics, which is what the simulations are usually offered to establish. Some recent work has begun fitting simulated diffusion patterns against real events, which is a step toward testing, though so far against events used to set the simulation up rather than ones it had never seen (the Workflow 4 essay covers this).
No model, Levins (1966) argued, can maximize generality, realism and precision at once, so modelers trade them against one another, and the results worth trusting are those that survive across several models built on different simplifications. Levins called such a result a robust theorem (p. 423), and a single simulation run is not yet one. Models, he added, are essential for understanding reality but “should not be confused with that reality itself” (p. 431), and the validation of a model “is not that it is ‘true’ but that it generates good testable hypotheses relevant to important problems” (p. 430). Simulation is a good tool for exploring the consequences of assumptions. Read as confirmatory evidence about a society, it is better understood as a careful thought experiment.
Generated hypotheses are leads, not findings
Hypothesis generation is the mild case. A model producing candidate hypotheses is producing what it is good at: plausible, on-distribution, novel-sounding text. Tong et al. (2024) coupled a model to a causal graph built from 43,312 psychology articles and produced well-being hypotheses that blind reviewers rated about as novel as those of doctoral students (the difference was not significant) and more novel than those of a model working without the graph. That is a real result and a legitimate use, so long as the output is treated as a lead entering the ordinary empirical pipeline and not as a finding leaving it. The guardrail here is not statistical. It is the ordinary apparatus of empirical science, applied afterwards, as usual.
8. Where the gap closes
Can any of these gaps be closed? One can, on conditions a researcher can state, check and report. It is the second of the two gaps in Figure 5, between model labels and a trustworthy estimate, and the remedy is the first of the practices Section 2 grouped under “doing it right.”
The correction: many model labels, a few human ones
The fix keeps the annotation pipeline from Section 6 but changes what surrounds the model. The model is demoted: it is asked only to label, never to be the measurement. The estimand is fixed before anything runs: a prevalence, a mean, a regression coefficient, a treatment effect. A human gold set is drawn by a known, probability-based sampling design. And a debiasing estimator does the actual work. Prediction-powered inference (Angelopoulos, Bates, et al., 2023) combines many machine labels with a few gold labels to produce provably valid confidence intervals, and PPI++ (Angelopoulos, Duchi, et al., 2023, preprint) adds a tuning step that down-weights the model when it is inaccurate. Design-based supervised learning (Egami et al., 2023) reaches the same destination by a doubly robust route, remaining asymptotically unbiased even when the labels are arbitrarily biased. Its guarantees need only something a researcher can secure by design rather than argue for: that each document’s probability of being sent for expert coding is known and greater than zero.
How the correction works, in one example. The numbers here are invented for illustration. Return to the 50,000 tweets from Section 3. The model labels all of them and calls 30 percent hostile. The researcher also hand-codes a random sample of 500. On those 500, the model called 32 percent hostile, while the human coders called 27 percent hostile, so on this sample the model overcounts by about five points. Prediction-powered inference subtracts that measured gap from the model’s estimate across all 50,000, giving about 25 percent, and widens the confidence interval to reflect uncertainty about the gap itself. The 49,500 tweets nobody hand-coded still do most of the work. The 500 that someone did hand-code keep the answer honest. Figure 7 draws the example.
Drag the sliders to try your own numbers.
Corrected estimate: 25 percent, plus or minus 2.9 points.
The hand-coded sample alone: 27 percent, plus or minus 3.9 points.
The corrected interval is narrower than the hand-coded sample's own.
The Workflow 6 essay draws the two pipelines side by side, the uncorrected one and the corrected one. They are identical for three stages, and the AI side is the same in both. At the fourth stage, one ends with a correction and the other does not. On the conditions set out below, that is the difference between a defensible estimate and an undisclosed researcher degree of freedom, and it is one stage wide.
With a small sample or a weak model, the correction can hurt
What makes this look like a safe default is graceful degradation. If the model is good, the intervals tighten. If the model is poor, the tuning step in PPI++ is meant to shrink the model’s weight toward zero, so the estimator falls back toward ordinary inference from the human labels alone. The PPI++ paper promises intervals that “always improve” on those from the human labels alone (Angelopoulos, Duchi, et al., 2023, preprint, p. 1), and it is tempting to conclude that there is no penalty for trying. In small samples that is false, and the proof was published only in 2026, a sign of how fast this literature is moving.
The guarantee is asymptotic, and the trouble lives in finite samples, where the estimator’s tuning parameter is estimated from the data and will not come out at exactly zero even when it should. Mani et al. (2026) prove that in the Gaussian case PPI++ has lower variance than the gold-only estimator if and only if the correlation between machine and gold labels exceeds 1/√(c·n − 2), where n is the number of gold labels, c = 1 for the single-sample estimator and c = 1/2 for the cross-fit version, assuming a large pool of model-labeled documents. To make that concrete: with 200 gold labels, the single-sample estimator helps only if the correlation exceeds about 0.07, and the cross-fit version only if it exceeds about 0.10; with 20 gold labels, the thresholds rise to about 0.24 and 0.35 (Figure 8). Below the threshold the correction is worse than using the gold labels alone, and at zero correlation both variants are always worse, by a factor of (1 + 1/(c·n − 1)). The exact result is proved for mean estimation, and the authors are candid that the extension to the regression-style estimates social scientists usually want reaches only the first stage of the argument. The empirical picture is mixed in the same way. Audinet de Pieuchon et al. (2025) found that, averaged across their datasets, both corrections came closer to a reference estimate than a small expert-only sample did. But the methods were unstable below roughly 200 expert annotations, and on one of their four datasets the design-based estimator was worse than both prediction-powered inference and the expert-only estimator at every sample size they tested.
Drag the point or the sliders to try your own numbers.
At 200 hand-coded labels and a correlation of 0.30:
Single-sample estimator: the correlation must exceed 0.071, so the correction helps.
Cross-fit estimator: the correlation must exceed 0.101, so the correction helps.
Four conditions make the correction a good bet
So “no penalty for trying” gives way to a checklist, which is the more useful object because its items can be inspected by somebody other than the person who ran the pipeline. Four conditions make the asymptotic guarantee a reasonable bet in a finite sample. First, the gold set must be probability-sampled and large enough, with roughly 200 as a working floor on published evidence rather than a settled constant, since that figure comes from one benchmark across four datasets. Second, the estimator must be cross-fit rather than single-sample, which buys unbiasedness and honest intervals at the price of a higher correlation threshold, since each fold estimates the tuning parameter from half the labels. Third, the rectifier must be computed within the subgroups whose contrast is the estimand rather than pooled across them, because Mozer (2026, preprint) shows that a pooled rectifier in a treatment-effect estimate carries a bias equal to the difference in arm-specific prediction error, and that particular bias does not shrink as gold labels are added. In plain terms, if the question is how a treated group differs from a control group, measure the model’s error separately in each group, because the model may be wrong in different ways in each. Fourth, the expected gain must be reported rather than assumed, since it can be calculated in advance from the correlation and the gold-set size. Meet those four and the choice against an uncorrected pipeline, carrying bias of unknown size and sign, is not close. Miss them and the correction can cost more than it buys.
What does the correction cost, and what can it not fix?
There is a harder qualification behind the four, and it is not about whether the correction works but about what it charges. Baumann et al. (2025, preprint), in the study described in Section 6, tested the corrections themselves rather than only the pipelines they correct. Design-based supervised learning and a related correction, confidence-driven inference, do restore nominal false-positive rates, which is what they promise and what they are usually adopted for. They do it by raising the false-negative rate, in these tests by as much as 60 percentage points. The risk is not eliminated. It is moved into the error type that a literature with a publication filter is least likely to notice: a false positive gets published and can be challenged, while a missed real effect usually ends in a file drawer where nobody sees it. Their blunter comparison is worth carrying alongside the checklist: 100 human annotations control false positives better than 100,000 uncorrected machine ones.
And the method’s authors are the ones who state its limits most plainly (Egami et al., 2024, working paper). The correction fixes prediction error. It does not fix construct validity. Whether the codebook captures the concept the theory is actually about is a question the mathematics cannot see. If the model and the human coders share a misunderstanding of what is being measured, no estimator will rescue the inference. Suppose the codebook defines “hostile” so broadly that it sweeps in ordinary criticism of the policy. The corrected estimate will then be a precise, honest measurement of the wrong thing. The human role, though small, cannot reach zero.
This is what a gap that closes conditionally looks like: not a better model and not a cleverer prompt, but a designed sample, a cross-fit estimator, a rectifier computed where the estimand lives, and a stated expectation of what the correction will buy, wrapped around a tool that has been assigned a job it can actually do. Conditional closure is the weaker claim, and it is by far the more useful one, because a condition is something a reviewer can check and completeness is something a reader has to take on faith.
9. Why the correction has not landed
If the correction works, on conditions anyone can check, why do so few studies use it? It has been available since 2023, in Science and at NeurIPS, with a working R package (Angelopoulos, Bates, et al., 2023; Egami et al., 2023). Its applied extension is now peer-reviewed (Rister Portinari Maranca et al., 2025), and it applies the method to images as well as text. In the same year the correction appeared, Pangakis et al. (2023, preprint) replicated 27 annotation tasks across 11 datasets drawn from high-impact social science journals and concluded, in their title, that automated annotation with generative AI requires validation.
So the warning was early, clear, and published in venues the field reads, and it has had little effect.
Two literatures barely touch
Part of the answer is a fact about how this literature is organized, easy to miss until you count. When I partitioned the 231-paper corpus, not one paper named prediction-powered inference, design-based supervised learning or any debiasing estimator in its title. Of the 78 annotation papers, two used valid or validation in their titles, one of them Pangakis et al.’s. My first instinct was that this proved the field ignores the correction, and that instinct was wrong: the corpus was filtered for social-science methodology terms, which excludes statistics and computer-science venues by construction. A single targeted search outside the corpus turned up an active, healthy correction literature.
The pattern that does hold up is a different one. The correction literature and the applied annotation literature are two conversations that barely touch. The statisticians built the fix and are refining it. The social scientists doing the annotating are, in the main, not citing it. Egami et al. (2024, working paper) say so themselves, observing that the papers evaluating LLM annotation for social science assess predictive performance and none address how the predicted labels should then be used downstream.
The audit from Section 5 reports the same pattern from the applied side, which matters because my own corpus is filtered. Desai, Card, et al. (2026, preprint), whose 27 papers use a language model for measurement across 50 distinct tasks, coded how each one validated. Convergent validity, checking the model’s labels against some other measure of the same thing, dominates, appearing in 22 of the papers. Eight tasks across six papers report no validation at all, and five of those feed a paper’s main analysis. No debiasing estimator appears anywhere as a coded validation practice, because their taxonomy contains no category for measurement-error correction. That last point should be read precisely. It is an absence from a coding scheme, not a counted zero, and the scheme’s validation categories are drawn from Jacobs and Wallach’s (2021) components of construct validity, where a correction estimator has no natural place: it repairs the estimate built from a measure rather than validating the measure. The authors’ coding data is public, so a direct count is checkable by anyone who wants it. And the authors know the correction exists: their introduction cites prediction-powered inference and design-based supervised learning, and judges that “the improvements from such methods tend to be small, and there is no guarantee they will help in any particular case” (p. 2). That is a fair reading of the finite-sample results in Section 8. On this evidence, the applied side is not simply unaware of the correction. Part of it has looked, and is not yet persuaded.
A second fact about the literature makes the first worse. In the corpus, preprints outnumber published work by roughly two and a half to one, and the 2025 to 2026 frontier is almost entirely preprint. The canon of this field is still forming, which means the ordinary way researchers come to adopt a methodological correction, by meeting it in textbooks and in reviewers’ comments, has not had time to work. Practice is being set by whichever demonstration paper arrived first and read most cleanly. That was Gilardi et al. (2023). It is a good paper. It was never a methods standard, and it was not offered as one.
Treating predicted labels as data predates language models
Some of the habit is older than language models. The study behind this essay’s opening case is an example. Fowler et al. (2021) had human coders label the television ads and a training sample of the Facebook ads, then let a supervised classifier supply the tone of every ad in their data, and their comparison of the two media rests on those predicted labels (p. 135). That was careful, documented, ordinary practice for its time, and it is the practice the framework of Egami et al. (2023) exists to put on firmer footing: predicted labels entered into a downstream analysis, with their errors handled by whatever checks the authors think to run.
Fowler et al.’s appendix shows the checks they ran. A classifier trained on the 14,642 hand-coded ads supplied tone for the rest of a dataset of 373,452 ads. Re-estimated on the hand-coded ads alone, the main tone results held, with wider intervals. Flipping predicted labels at the classifier’s measured error rates, 500 times over, added variation at least an order of magnitude smaller than ordinary sampling error, and the authors warned against using the labels ad by ad (Appendix B). On my reading, the perturbation used one error rate per category for both media, so it could not reveal error that differs between Facebook and television, the kind Section 8 says matters most when the two groups are the comparison. The hand-coded re-estimate is the check that speaks to that, and it held. The point is not about one paper. As I read this history, language models did not create the habit of treating predicted labels as data. They made it cheap enough for far more researchers to adopt.
If that diagnosis is right, the gap is less in the methods than in the vocabulary, and that is why the next two sections turn from statistics to philosophy and sociology.
10. The philosophical debt
Every established method already works by borrowing. A researcher running a repeated-measures ANOVA is not obliged to defend the philosophy of frequentist inference before publishing; that argument was settled decades ago, and anyone who runs the procedure inherits it without having to make it. Bring in a new kind of instrument, and that built-in justification does not come with it. The question of what an output licenses is forced back into the open, and the researcher answers it by hand, often without noticing. That is the situation the four problems share: with a new instrument, the philosophy of science stops being background and becomes a working tool again.
Each problem restates an older argument
Seen this way, each of the four problems turns out to be a specific, much older argument in new form. Table 4, at the end of the section, lines them up.
Plausibility versus truth is the old question of what makes a statement true rather than merely assertible. Section 4 gave its modern form: on Frankfurt’s (2005) account, as Hicks et al. (2024) apply it to language models, a system indifferent to truth is doing something categorically different from lying, since lying requires caring about the truth enough to conceal it.
Stochasticity versus reproducibility belongs to the epistemology of experiment: the question of what makes a procedure a measurement at all, rather than a mere occurrence. Hacking’s (1983) account of experimental practice, and the experimenters’ regress of Collins (1985), the observation that a result is trusted once the instrument is trusted and the instrument is trusted once it produces the expected result, are the ancestors of Section 5’s complaint. A model silently updated between two runs of a study is not a stable instrument read twice. It is a different instrument each time, wearing the same name.
Calibration versus validation is the construct validity of Cronbach and Meehl (1955), in its strict original sense. An eval measures agreement with a label. Validity asks whether the label measures the construct a theory is actually about. The tradition that separated those two questions is seventy years old, and Section 6 restates it for a labeler that happens to be a language model instead of a personality inventory.
Automation versus interpretation is the Verstehen and Erklären split: the interpretivist claim, running from Winch (1958) through Taylor (1971) and Geertz (1973), that meaning is not the kind of thing recovered by classification, because it is constituted between a reader and a text rather than sitting inside the text waiting to be extracted. (Verstehen and Erklären are German for understanding and explaining.) Section 7’s qualitative coding problems are this argument, encountered by researchers who did not set out to hold an epistemological position and acquired one anyway.
A good fit is not a hard test
The sharpest of these borrowings is quantitative, and it belongs to the correction in Section 8 as much as to this section. Deborah Mayo’s (2018) error-statistical philosophy asks a single question of any claim: has it survived a test that would probably have caught it out, had it been false? A claim passes severely only if the test result would have been unlikely to occur were the claim false. The idea is familiar from everyday reasoning. A claim that a coin is close to fair has passed a severe test if a thousand flips come out near half heads, and only a weak one if three flips do, because three flips would often look fair even for a badly biased coin. Severity is a different measure from fit. Fit is computed on the one dataset a researcher has. Severity asks how often the same test would have produced a result this good if the claim were false. That is where the repetition model in Section 4 fails. The claim that a model has learned the task, tested by trial-by-trial accuracy, passes with almost no severity, because a model that has learned only how often people repeat their last choice scores well on that test anyway.
The same distinction has a theorem attached to it. Forster and Sober (1994), working from Akaike’s theorem, show why fit to the data in hand and accuracy on new data come apart, and their framework implies that a false, simpler model can predict better than a true, more complex one, because the extra parameters must be paid for in estimation variance on finite, noisy data. Which model predicts best and which model is true are answered by different criteria, and the two can disagree. That is Section 4’s argument promoted from observation to theorem.
A note on what I am claiming, since the rule this series applies to others applies to it too. Severity is an established criterion and LLM-assisted annotation is an established literature, but as far as I have been able to determine nobody has yet run the one against the other. The generalization in this section is therefore this project’s own analytical move rather than a finding taken from a source, and I flag it as such for the same reason the Lakatosian reading of the Centaur critiques is flagged in the Workflow 2 essay. There is indirect evidence that the connection is recognized as unmade rather than made: Synthese announced a topical collection on severity and learning from error in June 2026, guest-edited by Mayo, Parker, Lakens and Staley (Mayo et al., forthcoming). A published call for papers is good evidence of a live question and no evidence that it has been answered.
And the loop closes on the correction itself. Prediction-powered inference advertises provably valid confidence intervals, meaning the error probability is controlled at the stated level. That is the currency Mayo (2018) deals in. Section 8 is therefore the one place in this series where the error-statistical guarantee, lost the moment an uncorrected label enters a regression, is won back. The guarantee holds as samples grow large, and in a finite sample it holds when the four conditions listed there are met. That the restoration is conditional is not a concession. An error probability controlled at a stated level, provided a stated design was used, is the kind of claim error statistics was built to license.
Is the choice of labeler part of the evidence?
One further thread, not yet followed to its end. Bayesian and likelihoodist inference obey the likelihood principle, which says that the evidence in a dataset depends only on the data actually observed, and so they are indifferent to why data collection stopped or which labeler was chosen. Error statistics rejects the likelihood principle for that reason, so that stopping rules and selection effects remain visible in the error probabilities.
The likelihood principle is hard to wave away because of an argument by Birnbaum (1962). He started from two principles that most statisticians already accepted. The sufficiency principle says that once a summary of the data captures everything the data say about the quantity of interest, nothing else in the data matters. The conditionality principle says that if a coin flip decided which of two experiments you ran, the evidence comes from the experiment you actually ran, not from the one you might have run. Birnbaum argued that these two principles, taken together, are mathematically equivalent to the likelihood principle (p. 271), and he noted the oddity that relatively few statisticians accepted the likelihood principle while many were inclined to accept the other two. If he was right, a frequentist who accepts sufficiency and conditionality has already conceded that stopping rules and selection effects do not bear on the evidence. Mayo (2018, p. 173) argues that the proof is flawed, so the frequentist is not boxed in, and she grants that not everyone agrees the matter is settled. The treatment of labeler choice that follows takes her side of that dispute, and a reader who sides with Birnbaum will read it differently.
A concrete case shows why it matters here. Suppose a researcher tries three models as labelers, gets a significant result with one of them, and reports that one. On a strict reading of the likelihood principle, the two discarded labelers are irrelevant to the evidence. To an error statistician they are part of the test, because they change how often a procedure like this one would produce a significant result by chance. A Bayesian can answer that the remedy is to model all three labelers’ outputs rather than one, and that is fair. But it is a remedy the analyst has to choose; error statistics builds it into the error probability itself. Read this way, the political-ads result from Section 1 stops being a practical grievance and becomes a formal point: which labeler happened to be chosen is a selection effect, and error-statistical inference was built from the start to keep selection effects from disappearing into the arithmetic.
| Problem | Older argument | Tradition | Sources | What it explains |
|---|---|---|---|---|
| Plausibility versus truth | Truth against mere assertion; indifference to truth | Semantics | Frankfurt (2005); Hicks et al. (2024) | Why fluent output is not a failed attempt at the truth |
| Stochasticity versus reproducibility | What makes a procedure a measurement; the experimenters’ regress | Epistemology of experiment | Hacking (1983); Collins (1985) | Why a silently updated model is a new instrument each time |
| Calibration versus validation | Construct validity; reliability is not validity | Psychometrics | Cronbach and Meehl (1955); Loevinger (1957) | Why agreement with a label does not show that the label measures the construct |
| Automation versus interpretation | Verstehen and Erklären | Interpretivism | Winch (1958); Taylor (1971); Geertz (1973) | Why meaning is not recovered by classification |
| All four | Severity; fit against predictive accuracy | Error statistics; model selection | Mayo (2018); Forster and Sober (1994) | Why a good fit is not a test the claim could have failed |
Tracing the four problems to four older arguments (Table 4) does not split them into four unrelated problems. It suggests the opposite. The philosophy of science already reached the shape of this argument from four separate directions, through semantics, experimental epistemology, psychometrics and interpretivism, and a fifth, formal direction met it in the middle. When a field’s settled conventions are silent, as they currently are for language models used as research instruments, the questions those conventions used to answer do not go away. They surface, one at a time, in whichever corner of the pipeline happens to need them next.
11. The trading zone
Picture a border market where traders who share no language still manage to exchange goods, by pointing, haggling and slowly building up a small shared vocabulary. Peter Galison (1997) called such places trading zones, and the term now anchors a vocabulary that historians and sociologists of science use for what happens when research communities with different languages have to work together. It fits this case closely. Galison developed it to explain how theorists, experimenters and engineers, working in different subcultures of physics, managed to coordinate their work locally despite large differences in their broader beliefs.
Is this really a trading zone?
The label has to be earned. Collins et al. (2007) define a trading zone narrowly, as a location where communities with a deep problem of communication nonetheless manage to communicate; absent that problem there is trade, and trade is not a trading zone. The four problems in Sections 4 through 7 are the evidence that the communication problem here is real. Two communities have exchanged labels, benchmarks, agreement statistics and confidence intervals for three years, at scale, in flagship journals, and in four separate places the thing received is not the thing that was sent. That is a communication failure surviving good faith and competence on both sides, which is the entry condition, and it is why the vocabulary of trade fits better here than the vocabulary of error.
Applying this framing to AI-assisted research is, I think, where this essay adds the most. Galison’s (1997) trading zone has been applied to a great many interdisciplinary collaborations, as have Star and Griesemer’s (1989) boundary objects and Knorr Cetina’s (1999) epistemic cultures, but as far as I have been able to determine none of the three has been applied to this one. Knorr Cetina’s version needs stating carefully. Her subject, in her own words, is “not in the construction of knowledge but in the construction of the machineries of knowledge construction” (p. 3). Neither the engineering side nor the methods side holds a stated theory of validity that contradicts the other’s. Both would sign the sentence “a measure should measure what it claims to measure.” What differs sits one level down, in the machinery. The engineer’s evidence is a score on a held-out test set. The methodologist’s is an argument linking a construct, a population and a sampling design. A doctrinal disagreement would be visible and arguable. A difference in machinery is neither, and that invisibility is what this section has to explain.
Some shared words help, and some only seem to
Star and Griesemer (1989) supply the mechanism at the level of individual words. Their boundary objects are things that live in several worlds at once, meet the needs of each, and stay recognizable across them because their structure is common enough to more than one. Their classic example is a natural history museum, where amateur collectors and professional biologists worked with the same specimens and maps for quite different purposes. Researchers on both sides lean most on seven shared terms: calibration, eval, reliability, validation, temperature, consensus and accuracy. They do not all work the same way, and sorting them is the core of this section. Table 5 lays out the sort.
| Term | In AI engineering | In research methodology | Type |
|---|---|---|---|
| Eval | A benchmark suite and the score a model gets on it | Heard as a reliability or validity check | Repository |
| Accuracy | Share of labels that match a benchmark | How close an estimate is to the true value | Coincident boundary |
| Consensus | Runs or agents producing the same output | Coders reaching a negotiated agreement | Coincident boundary |
| Temperature | A setting for how random the model’s choices are | Heard as measurement error that averages out | Standardized form |
| Reliability | A system that behaves consistently | Agreement statistics such as kappa | Close to an ideal type |
| Validation | Performance on a held-out split of the data | An argument that a measure captures its construct | Homonym |
| Calibration | Predicted probabilities that match observed frequencies | Setting an instrument against a known standard | Homonym |
The benchmark suite that eval names is a repository, an indexed pile from which each world takes what it needs without negotiating the difference in purpose, and Star and Griesemer (1989) count that as the repository’s advantage, which they call modularity (p. 410). That is this essay’s thesis stated from the other side. Accuracy and consensus are coincident boundaries, one outline with different contents, as California was one object on the amateur collectors’ road maps and another on the professionals’ life-zone maps. Temperature is a standardized form, a number researchers copy from engineering practice with its procedure attached and its meaning left behind; a methods section reports “temperature = 0” as though that settled reproducibility, and it does not. Reliability is closest to an ideal type, vague enough for anyone who reaches for it to adapt locally, which is why everybody does. Their four types are “only analytic distinctions,” not an exhaustive list (p. 410), so Table 5 uses them as lenses rather than as bins every term must fit.
Two of the seven do not fit it at all, and they are the two the essay’s main contrast is named for. A held-out split and a construct-validity argument are not one object read from two sides. They are two objects sharing a word. So are the calibration that makes predicted probabilities match observed frequencies and the calibration that sets an instrument against a known standard. These are homonyms, and the difference between a homonym and a boundary object is the most useful thing this section offers. The boundary object is the productive case and the homonym is the trap. A boundary object works because the two worlds do not have to negotiate their differing purposes. A homonym gives the same feeling of successful exchange while transmitting nothing at all.
When does working without agreement stop working?
Here the argument turns, and the turn is my own analytical move rather than a finding taken from any of these sources. Consensus is not required for a trading zone to work. That is the standing claim of all four sources this section draws on, and it is what makes the trading zone a good explanation of how new instruments get adopted at all: coordination without agreement, action without shared belief. The four problems are what happens when that useful non-negotiation runs past the point where it was doing useful work. As far as I can find, nobody in this literature says when a productive absence of agreement becomes a failure of measurement, because in the cases they study the question does not arise. It arises here.
The trading-zone literature also clarifies what has and has not been built. A pidgin, in Galison’s (1997) account, is the product of work: a contact language reduced from two active languages along lexical, syntactic and morphological lines until each side can use it. Two communities using calibration in incompatible senses did no such work. They collided. The more accurate description of most of this border is not that a shared working language exists and nobody has written its dictionary, but that none was ever built, and accidental homonyms are doing the work a constructed one should be doing. Some regions do have a real one, and prediction-powered inference is the clearest instance in this series.
The correction asks for agreement on method, not on meaning
Why that region works is the constructive half of the argument. Star and Griesemer (1989) observe that standardizing methods is a different act from standardizing theory, and a more powerful one, because people in very different worlds can work together on a shared account of how without first agreeing on why. Their example is the museum’s founding director, Joseph Grinnell, who obtained usable specimens from amateur collectors without requiring any of them to hold a view about evolution, simply by specifying how each specimen was to be collected and labeled. Prediction-powered inference does the same thing. It standardizes an interface, a probability-sampled labeled subset and a correction computed on it, and requires no agreement about what a language model is or what its outputs mean. Collins et al. (2007) arrive at the same place, observing that boundary objects together with standardized procedures can do the work of a full shared language. Section 8 is the one place in this series where something was deliberately built rather than collided into.
The homonym case has a precedent too, from physics, and it works the same way. Galison’s (1997) example is Bjorken and Drell’s two-volume treatment of relativistic quantum theory, written as two volumes because the theorists’ apparatus had to be simplified before experimenters could use it, and the simplification stripped out the framework that justified the rules being handed over. The symbol Ψ carries a different meaning in each volume, a single-particle wave function in one and a field of creation and annihilation operators in the other, and the trade proceeds regardless because it is conducted over practical questions rather than over Ψ. A shared symbol, two meanings, a working collaboration, and no moment at which anybody was obliged to notice.
What this border needs is a phrasebook
Collins et al. (2007) sort trading zones along two axes, collaborative against coerced and homogeneous against heterogeneous, giving four types: the inter-language zone, where a new shared language grows; the fractionated zone, where the communities stay distinct and boundary objects do the work; the subversive zone, where one party’s language quietly replaces the other’s; and the enforced zone, where coordination is imposed. This border sits in the fractionated quadrant (Figure 9), which they argue can be a stable arrangement in which a new shared language is neither necessary nor desirable. What does the work instead is interactional expertise, the ability to converse competently in another community’s language without practising in it, which, they suggest, may be the norm for new interdisciplinary work rather than the exception. That is the strongest available justification for a phrasebook, which is what the companion pages of this series aim to be: an aid to interactional expertise rather than an attempt at a new shared language, and on this reading the right tool rather than a lesser one.
One of the four types is worth naming as a risk rather than a diagnosis. In a subversive zone the sentences go on sounding shared while one party’s meaning has already replaced the other’s, the way a Newtonian-sounding sentence written after Einstein is an Einsteinian sentence in disguise. If validation is being gradually redefined toward the machine learning sense rather than negotiated between the two, this border is not fractionated but subversive, and the near-map would have a direction as well as a shape. Nothing in this project measures directional drift in the vocabulary, and asserting it because it sounds plausible would be committing this essay’s own named error. I state it as a possibility the evidence does not settle, and as a question someone with the right data could test, for instance by tracking how methods sections use validation from year to year.
A few cautions close the section. Galison (1997) conjectures that pidgins stabilize through the alignment of three or more languages rather than two, which prevents them from being reabsorbed into a source language; extending that to suggest a two-sided trading zone is inherently unstable is this project’s own move. Láng and Megyesi (2024), writing on the DECRYPT collaboration, observe from their own case that the trading-zone literature concentrates on two-party examples and that more opens up once more than two epistemic cultures seek a common language, which supports the two-party distinction though not Galison’s stabilization mechanism. These four sources are also a tradition rather than four independent witnesses, since Knorr Cetina (1999) cites Galison, and Collins et al. (2007) cite both Galison and Star and Griesemer (1989), so their agreement counts for less than the agreement of the four teams in Section 6. Nor does anything guarantee a happy ending: Collins et al. (2007) note that a trading zone can simply cease to function and fall apart (p. 664). This section promises no shared language. It claims that the border can be described, that the description helps explain why competent people have been missing each other, and that a phrasebook is what the description recommends.
12. Other frameworks answer different questions
Others have seen that this area needs a framework. Being precise about what each one does gives its authors due credit and shows where this essay sits.
Two earlier pieces set the terms of the debate these frameworks enter. Grossmann et al. (2023) set the optimistic agenda, with conditions attached: language models could simulate participants, accelerate hypothesis generation and stand in for human subjects across large stretches of social science, and the field should organize itself to take the opportunity, provided it validated the constructs it measured this way and managed the models’ biases. Bail (2024) answers in a more measured register, cataloguing genuine potential across survey research, experiments, content analysis and agent-based modelling before setting out the limitations. His remedy is open-source infrastructure and a community of scholars working out best practices together, the nearest thing in this literature to the position that the field will correct itself as it goes. I dispute neither Grossmann’s opportunity nor Bail’s confidence in correction. What I question is the timetable both imply. Self-correction requires that the error be visible to the people making it, and Sections 6 and 11 argue that on this particular border it often is not, because the vocabulary that would raise the alarm is the same vocabulary that quiets it. That is a claim about which errors a field can easily catch, not about whether it catches errors at all. The study by Desai, Truong, et al. (2026), discussed in Section 6, belongs to this landscape too, not as a rival framework but as a measurement of the gap taken from inside the engineering lane.
Sanaei and Rajabzadeh (2025, preprint) situate LLM use along interpretive depth and autonomy, and recommend keeping autonomy low while raising depth only where warranted and supervised, a posture they call bounded autonomy. It is a dosage framework: how much of the work to hand over. It is scoped to qualitative research, and within that scope it is a finer-grained instrument than this one.
Zhang et al. (2025, preprint) cross task type with validation type and map uncertainty-quantification methods onto the resulting grid, and float a stronger claim: if uncertainty cannot be assessed for an application, language models should not be used for it. It is an uncertainty framework: how much the output varies and whether that variation is acceptable.
The framework here is a validity framework: whether the output type matches the claim type, and whether anything converts one into the other. These are three different axes, and they complement rather than compete with one another. Table 6 sets them side by side, with the measurement framework of Wallach et al. (2025) from Section 6.
| Framework | Axes | Question it governs | Rule of thumb |
|---|---|---|---|
| Sanaei and Rajabzadeh (2025, preprint) | Interpretive depth; autonomy | How much of the work to hand over (dosage) | Keep autonomy low; raise depth only where warranted and supervised |
| Zhang et al. (2025, preprint) | Task type; validation type | Whether the output’s variation can be assessed and accepted (uncertainty) | If uncertainty cannot be assessed, do not use a language model |
| Wallach et al. (2025) | Four levels, from background concept to instance measurements | Whether a measurement captures its concept (measurement) | Pin the concept down before choosing the instrument |
| This essay | Output type; claim type | Whether the output can support the claim, and what converts one into the other (validity) | No guardrail, no confirmatory claim |
I think validity should usually be the axis that governs, and the framework of Zhang et al. (2025, preprint) shows why. Its dispersion measures, read on their own, cannot tell a consistent instrument from a flattened one. Identity flattening, a central finding against silicon sampling (Section 4), is a reduction in dispersion, and collapsed within-group variety would register on a dispersion metric as low uncertainty. The authors are not naive about this. They note that synthetic samples often fail to represent the target population, and that some variability in a model’s output “is desirable, as it reflects the openness and variability inherent to natural language” (p. 11). Where reference data exist, they pair dispersion with a check against expert labels. That pairing is the point: what rescues the uncertainty measure is a reference outside the model, which is a validity check by another name. A tool can be well built and correctly applied and still, used alone, point at the wrong quantity, which is the thesis of this essay applied to a framework rather than to a model.
13. Bots in the sample, and what is out of scope
One body of work sits outside all six areas and needs mentioning, because it inverts them. Westwood (2025) frames language models as a potential existential threat to online survey research: not researchers using models to simulate people, but models posing as people in the samples researchers collect. The agent he built to test this, given a persona and a memory of its own answers, passed standard attention checks in 99.8 percent of 6,000 trials. Every guardrail in this essay assumes a human gold standard exists to be sampled; Section 6 already noted that this is a working assumption rather than a fact. If some of the humans in an online panel are bots, the gold standard is no longer gold, and every correction built on it inherits the problem. This is not an AI-augmented research practice. It is a shadow that falls across all of them, and it deserves a paper of its own.
A second body of work is out of scope, and I name it so that its absence does not puzzle anyone. Researchers using language models to draft, summarize and search the literature are doing something real, and it may well change scholarship. But it produces no research claim of the kind examined here, so the four-question test has nothing to bite on. That is not a judgment about its importance. It is a statement about what this test measures.
14. Doing it right
The four problems have a common shape. In each, a tool that produces plausible, on-distribution output is asked to deliver something else, and something in the vocabulary makes the substitution feel like no substitution at all. Consensus stands in for accuracy. An eval stands in for validation. Predictive fit stands in for explanation. Coherent text stands in for interpreted meaning. Each pair is a near-map, and a conclusion can rest on any of them without its author noticing.
What is old here, and what is new
The principle underneath is not mine, and saying so is a small exercise of the discipline this essay recommends. That reliability is necessary but not sufficient for validity has been central to psychometrics since Loevinger (1957), and Cook and Beckman (2006) state it for clinical measurement. What I claim is narrower, and I think still worth claiming. The first part is the synthesis: reading four apparently distinct problems as versions of one, which no source I have found does. The second is the observation from Section 6 that four largely independent literatures, only one of which cites another and none of which cites the precedent, have each rediscovered the same pattern in their own vocabulary within roughly three years. The principle is nearly seventy years old. Its simultaneous rediscovery in four places is a fact about the present moment, and it is a fact about vocabulary rather than about competence.
The remedy is not skepticism, which is cheap, and not enthusiasm, which is cheaper. It is a habit. Before the model runs, name the claim; name what the tool actually produces; name the guardrail that converts one into the other; and check that the guardrail earns the claim you intend to make and not a neighboring one. That habit, together with the practices in Table 7, completes what Section 2 called “doing it right.”
| Stage | Practice | Why |
|---|---|---|
| Before running | Name the claim, the tool’s output and the guardrail | Catches a mismatch before it is built in (Section 3) |
| While running | Pin the model version; log prompts, settings and seeds; run more than once and report the spread | Makes the pipeline rerunnable and its variation visible (Section 5) |
| When estimating from labels | Hand-code a probability sample (roughly 200 or more); use a cross-fit correction; compute the correction within the groups being compared; report the expected gain | Makes estimates and intervals honest (Section 8) |
| When reporting | Follow a reporting standard such as GUIDE-LLM or REFORMS; disclose the model, version, access route, prompts, parameters and validation | Lets a reader check the work |
| After publication | Have someone other than the author check the disclosures | Checklists alone barely change practice |
Reporting standards exist, and compliance lags
Reporting standards have recently caught up with this field. GUIDE-LLM (Feuerriegel et al., 2026) is a fourteen-item checklist built specifically for the behavioral and social sciences through a two-round Delphi process whose final panel of 80 experts came most often from computational social science, machine learning and natural language processing, followed by human-computer interaction and psychology. It asks for what a methodologist would ask for: which model, which version and which access route, since a name like “ChatGPT” describes a moving target rather than one fixed system; the prompts and system instructions; the parameters; how outputs were validated; and the code. For studies closer to machine-learning-based science generally, REFORMS (Kapoor et al., 2024) offers 32 items across 8 modules, and its authors draw a distinction close to this essay’s own: science in which a model supports a claim about the world is different from research in which the model’s own performance is the claim.
A checklist is necessary, but on the available evidence it is not sufficient. In a randomized trial with manuscripts submitted to PLOS ONE, asking authors to complete an established reporting checklist at submission barely moved their compliance with it: median compliance went from 36.8 percent of applicable items to 39.5 percent, a difference indistinguishable from noise (Hair et al., 2019). Journal policies on AI tell a similar story. He and Bu (2026) found that roughly 70 percent of 5,114 journals had adopted some AI-use policy by January 2025, yet only about 0.1 percent of more than 75,000 eligible papers carried an AI-disclosure statement. Journals are adopting policies faster than authors are following them, and the same study found that the estimated share of AI-written text grew at the same rate in journals with and without a policy.
Correct where you can, and say when you are exploring
Where the statistical guardrail exists, use it, and check the conditions under which it works. The valid-inference tools are mature and inexpensive, have packages in both R and Python, and, in large samples, the tuned estimators degrade toward ordinary inference as the labeler gets worse. They do carry a penalty when the model is poor and the gold set is small, but the penalty can be calculated in advance, which is a much better position than either a free lunch or an unquantified risk. When the four conditions in Section 8 are met, there is little reason not to use the correction. When they are not, checking them costs far less than the annotation they govern. That much of the applied literature is not yet doing either is, I think, the most fixable problem described in this essay.
Where no guardrail exists, the work is exploratory. That is not a demotion. Exploration is how science begins, and hypothesis generation with a language model is a good use of a good tool. What does not follow is a confirmatory finding reported from an exploratory pipeline because the confidence interval printed without complaint.
Recall Section 1: the same thirteen thousand ads, two labelers, two different answers, and both regressions printed a confidence interval (Figure 10). The interval will always print. Whether it means anything depends on the work done before it did.
I wrote this essay with help from Claude (Anthropic), which assisted with drafting, source-checking and editing; a final read checked the essay’s central claims against the primary sources held for this project. The argument, editorial decisions and any errors are mine. One error that assistance introduced, and that checking caught, is described in Section 5.
References
Adcock, R., & Collier, D. (2001). Measurement validity: A shared standard for qualitative and quantitative research. American Political Science Review, 95(3), 529-546.
Anadkat, S. (2023, November 6). How to make your completions outputs consistent with the new seed parameter. OpenAI Cookbook. https://developers.openai.com/cookbook/examples/reproducible_outputs_with_the_seed_parameter
Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., & Zrnic, T. (2023). Prediction-powered inference. Science, 382(6671), 669-674. https://doi.org/10.1126/science.adi6000
Angelopoulos, A. N., Duchi, J. C., & Zrnic, T. (2023). PPI++: Efficient prediction-powered inference. arXiv:2311.01453 (v2, 26 March 2024).
Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023). Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3), 337-351. https://doi.org/10.1017/pan.2023.2
Ashokkumar, A., Hewitt, L., Ghezae, I., & Willer, R. (2026). Large language models can predict the results of social science experiments. Nature, 656(8126), 115-122. https://doi.org/10.1038/s41586-026-10742-x
Audinet de Pieuchon, N., Daoud, A., Jerzak, C. T., Johansson, M., & Johansson, R. (2025). Benchmarking debiasing methods for LLM-based parameter estimates. In Proceedings of EMNLP 2025 (pp. 19757-19772). https://doi.org/10.18653/v1/2025.emnlp-main.1000 arXiv:2506.09627
Bail, C. A. (2024). Can generative AI improve social science? Proceedings of the National Academy of Sciences, 121(21), e2314021121. https://doi.org/10.1073/pnas.2314021121
Barrie, C., Palmer, A., & Spirling, A. (n.d.). Replication for language models: Problems, principles, and best practice for political science [Working manuscript].
Baumann, J., Rottger, P., Urman, A., Wendsjo, A., Plaza-del-Arco, F. M., Gruber, J. B., & Hovy, D. (2025). Large language model hacking: Quantifying the hidden risks of using LLMs for text annotation. arXiv:2509.08825 (v2, 6 October 2025).
Binz, M., Akata, E., Bethge, M., Brandle, F., Callaway, F., Coda-Forno, J., Dayan, P., Demircan, C., Eckstein, M. K., Elteto, N., Griffiths, T. L., Haridi, S., Jagadish, A. K., Ji-An, L., Kipnis, A., Kumar, S., Ludwig, T., Mathony, M., Mattar, M., … Schulz, E. (2025). A foundation model to predict and capture human cognition. Nature, 644(8078), 1002-1009. https://doi.org/10.1038/s41586-025-09215-4
Birnbaum, A. (1962). On the foundations of statistical inference. Journal of the American Statistical Association, 57(298), 269-306. https://doi.org/10.1080/01621459.1962.10480660
Borchers, C., Shahrokhian, B., Balzan, F., Tajik, E., Sankaranarayanan, S., & Simon, S. (2026). Temperature and persona shape LLM agent consensus with minimal accuracy gains in qualitative coding. In Proceedings of the 19th International Conference on Educational Data Mining (pp. 280-292). https://doi.org/10.5281/zenodo.21039879
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111(4), 1061-1071. https://doi.org/10.1037/0033-295X.111.4.1061
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81-105.
Collins, H. M. (1985). Changing order: Replication and induction in scientific practice. SAGE Publications.
Collins, H., Evans, R., & Gorman, M. (2007). Trading zones and interactional expertise. Studies in History and Philosophy of Science Part A, 38(4), 657-666. https://doi.org/10.1016/j.shpsa.2007.09.003
Cook, D. A., & Beckman, T. J. (2006). Current concepts in validity and reliability for psychometric instruments: Theory and application. The American Journal of Medicine, 119(2), 166.e7-166.e16. https://doi.org/10.1016/j.amjmed.2005.10.036
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281-302. https://doi.org/10.1037/h0040957
De Paoli, S. (2024). Performing an inductive thematic analysis of semi-structured interviews with a large language model: An exploration and provocation on the limits of the approach. Social Science Computer Review, 42(4), 997-1019. https://doi.org/10.1177/08944393231220483
Desai, M., Card, D., & Jacobs, A. Z. (2026). Validating LLMs in social science: Epistemic threats and emerging norms. arXiv:2607.07915.
Desai, M., Truong, S. T., Wallach, H., Chouldechova, A., Cooper, A. F., Garcia-Gathright, J., Ho, D. E., Jacobs, A. Z., Koyejo, S., Pangakis, N., & Wang, A. (2026). What AI benchmarks actually measure: Adapting convergent and discriminant validity to interrogate fifty-six AI benchmarks. In Proceedings of COLM 2026.
Dong, A., Li, P., Chen, Y., Gibson, S., Zhao, L., & He, M. (2026). Human-AI collaboration across decision support, autonomous systems, and LLM agents: A systematic review and collaboration convergence framework. Sustainability, 18(11), 5313. https://doi.org/10.3390/su18115313
Egami, N., Hinck, M., Stewart, B. M., & Wei, H. (2023). Using imperfect surrogates for downstream inference: Design-based supervised learning for social science applications of large language models. In Advances in Neural Information Processing Systems 36 (pp. 68589-68601). arXiv:2306.04746
Egami, N., Hinck, M., Stewart, B. M., & Wei, H. (2024). Using large language model annotations for the social sciences: A general framework of using predicted variables in downstream analyses [Working paper, version of 17 November 2024].
Fascendini, B., McGregor, K., Gupta, M. D., & Griffiths, T. L. (2026). The riddle riddle: Testing flexible reasoning in large language models and humans. arXiv:2606.27103.
Feuerriegel, S., Barrie, C., Crockett, M. J., Globig, L. K., McLoughlin, K. L., Mirea, D. M., Spirling, A., Yang, D., Althoff, T., Antoniak, M., Argyle, L. P., Ashokkumar, A., Atari, M., Bailey, H., Bauer, K., Bhatt, U., Chai, Y., Chakraborty, T., Chandra, Y., … Horta Ribeiro, M. (2026). A reporting checklist for large language models in behavioural science. Nature Human Behaviour, 10(7), 1182-1186. https://doi.org/10.1038/s41562-026-02492-7
Forster, M., & Sober, E. (1994). How to tell when simpler, more unified, or less ad hoc theories will provide more accurate predictions. The British Journal for the Philosophy of Science, 45(1), 1-35. https://doi.org/10.1093/bjps/45.1.1
Fowler, E. F., Franz, M. M., Martin, G. J., Peskowitz, Z., & Ridout, T. N. (2021). Political advertising online and offline. American Political Science Review, 115(1), 130-149. https://doi.org/10.1017/S0003055420000696
Frankfurt, H. G. (2005). On bullshit. Princeton University Press. ISBN 978-0-691-12294-6
Fuławka, K., Hertwig, R., & Wulff, D. U. (2026). Large language models accurately identify decision reasons in verbal reports. Proceedings of the National Academy of Sciences, 123(27). https://doi.org/10.1073/pnas.2526798123
Galison, P. (1997). Image and logic: A material culture of microphysics (Ch. 9, “The trading zone: Coordinating action and belief,” pp. 781-844). University of Chicago Press.
Geertz, C. (1973). Thick description: Toward an interpretive theory of culture. In The interpretation of cultures (pp. 3-32). Basic Books. ISBN 0-465-03425-X
Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time [Unpublished manuscript]. Department of Statistics, Columbia University. https://sites.stat.columbia.edu/gelman/research/unpublished/p_hacking.pdf
Gelman, A., & Loken, E. (2014). The statistical crisis in science. American Scientist, 102(6), 460. https://doi.org/10.1511/2014.111.460
Gilardi, F., Alizadeh, M., & Kubli, M. (2023). ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30), e2305016120. https://doi.org/10.1073/pnas.2305016120
Grossmann, I., Feinberg, M., Parker, D. C., Christakis, N. A., Tetlock, P. E., & Cunningham, W. A. (2023). AI and the transformation of social science research. Science, 380(6650), 1108-1109. https://doi.org/10.1126/science.adi1778
Hacking, I. (1983). Representing and intervening: Introductory topics in the philosophy of natural science. Cambridge University Press.
Hair, K., Macleod, M. R., & Sena, E. S., on behalf of the IICARus Collaboration. (2019). A randomised controlled trial of an intervention to improve compliance with the ARRIVE guidelines (IICARus). Research Integrity and Peer Review, 4, 12. https://doi.org/10.1186/s41073-019-0069-3
He, H., & Thinking Machines Lab. (2025, September). Defeating nondeterminism in LLM inference. Thinking Machines Lab: Connectionism.
He, Y., & Bu, Y. (2026). Academic journals’ AI policies fail to curb the surge in AI-assisted academic writing. Proceedings of the National Academy of Sciences, 123(9). https://doi.org/10.1073/pnas.2526734123 arXiv:2512.06705
Hicks, M. T., Humphries, J., & Slater, J. (2024). ChatGPT is bullshit. Ethics and Information Technology, 26, 38. https://doi.org/10.1007/s10676-024-09775-5
Hofman, J. M., Watts, D. J., Athey, S., Garip, F., Griffiths, T. L., Kleinberg, J., Margetts, H., Mullainathan, S., Salganik, M. J., Vazire, S., Vespignani, A., & Yarkoni, T. (2021). Integrating explanation and prediction in computational social science. Nature, 595(7866), 181-188. https://doi.org/10.1038/s41586-021-03659-0
Jacobs, A. Z., & Wallach, H. (2021). Measurement and fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 375-385). https://doi.org/10.1145/3442188.3445901
Jowsey, T., Braun, V., Clarke, V., Lupton, D., & Fine, M. (2025). We reject the use of generative artificial intelligence for reflexive qualitative research. Qualitative Inquiry. Advance online publication. https://doi.org/10.1177/10778004251401851
Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1-73. https://doi.org/10.1111/jedm.12000
Kapoor, S., Cantrell, E. M., Peng, K., Pham, T. H., Bail, C. A., Gundersen, O. E., Hofman, J. M., Hullman, J., Lones, M. A., Malik, M. M., Nanayakkara, P., Poldrack, R. A., Raji, I. D., Roberts, M., Salganik, M. J., Serra-Garcia, M., Stewart, B. M., Vandewiele, G., & Narayanan, A. (2024). REFORMS: Consensus-based recommendations for machine-learning-based science. Science Advances, 10(18), eadk3452. https://doi.org/10.1126/sciadv.adk3452
Kim, C., Ke, F., Zhang, N., & Barrett, A. (2025). LLM-supported thematic analysis: Evaluating GATOS workflow on complex qualitative data. In C. Mills, G. Alexandron, D. Taibi, G. Lo Bosco, & L. Paquette (Eds.), Proceedings of the 18th International Conference on Educational Data Mining (pp. 604-607). International Educational Data Mining Society. https://doi.org/10.5281/zenodo.15870241
Knorr Cetina, K. (1999). Epistemic cultures: How the sciences make knowledge. Harvard University Press.
Láng, B., & Megyesi, B. (2024). An STS analysis of a digital humanities collaboration: Trading zones, boundary objects, and interactional expertise in the DECRYPT project. Humanities and Social Sciences Communications, 11, 618. https://doi.org/10.1057/s41599-024-03135-w
Levins, R. (1966). The strategy of model building in population biology. American Scientist, 54(4), 421-431.
Lin, H., & Zhang, Y. (2026). Navigating the risks of using large language models for text annotation in social science research. Social Science Computer Review, 44(3), 403-427. https://doi.org/10.1177/08944393251366243 arXiv:2503.22040
Liu, W., & Ding, N. (2026). Can Centaur truly simulate human cognition? The fundamental limitation of instruction understanding. National Science Open, 5(1), 20250053. https://doi.org/10.1360/nso/20250053
Loevinger, J. (1957). Objective tests as instruments of psychological theory. Psychological Reports, 3(3), 635-694. Monograph Supplement 9. https://doi.org/10.2466/pr0.1957.3.3.635
Mani, P., Xu, P., Lipton, Z. C., & Oberst, M. (2026). No free lunch: Non-asymptotic analysis of prediction-powered inference. In Proceedings of the 43rd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 306, pp. 86296-86335). arXiv:2505.20178
Mayo, D. G. (2018). Statistical inference as severe testing: How to get beyond the statistics wars. Cambridge University Press.
Mayo, D. G., Parker, W. S., Lakens, D., & Staley, K. (Eds.). (forthcoming). Severity and learning from error [Topical collection]. Synthese.
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741-749.
Meyer, J., Garcia, D., & Wulff, D. U. (2026). Apparent psychological profiles of large language models are largely a measurement artifact. arXiv:2606.20205.
Mozer, R. (2026). PPI is the difference estimator: Recognizing the survey sampling roots of prediction-powered inference. arXiv:2603.19160.
Namazova, S., Brondetta, A., Strittmatter, Y., Nassar, M., & Musslick, S. (2025). Not yet AlphaFold for the mind: Evaluating Centaur as a synthetic participant. arXiv:2508.07887.
OpenAI. (n.d.). Advanced usage. OpenAI API documentation. Retrieved November 10, 2026, from https://developers.openai.com/api/docs/guides/advanced-usage
Orr, M., Cranford, D., Ford, K., Gluck, K., Hancock, W., Lebiere, C., Pirolli, P., Ritter, F., & Stocco, A. (2025). Not even wrong: On the limits of prediction as explanation in cognitive science. arXiv:2510.03311.
Palminteri, S., Wyart, V., & Koechlin, E. (2017). The importance of falsification in computational cognitive modeling. Trends in Cognitive Sciences, 21(6), 425-433. https://doi.org/10.1016/j.tics.2017.03.011
Pangakis, N., Wolken, S., & Fasching, N. (2023). Automated annotation with generative AI requires validation. arXiv:2306.00176.
Park, J. S., O’Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023). Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (pp. 1-22). https://doi.org/10.1145/3586183.3606763 arXiv:2304.03442
Park, J. S., Zou, C. Q., Kamphorst, J., Egan, N., Shaw, A., Hill, B. M., Cai, C., Morris, M. R., Liang, P., Willer, R., & Bernstein, M. S. (2024). LLM agents grounded in self-reports enable general-purpose simulation of individuals. arXiv:2411.10109 (v3, 28 June 2026).
Rister Portinari Maranca, A., Chung, J., Hinck, M., Wolsky, A. D., Egami, N., & Stewart, B. M. (2025). Correcting the measurement errors of AI-assisted labeling in image analysis using design-based supervised learning. Sociological Methods & Research, 54(3), 984-1016. https://doi.org/10.1177/00491241251333372
Sanaei, A., & Rajabzadeh, A. (2025). Depth and autonomy: A framework for evaluating LLM applications in social science research. arXiv:2510.25432.
Shmueli, G. (2010). To explain or to predict? Statistical Science, 25(3), 289-310. https://doi.org/10.1214/10-STS330
Star, S. L., & Griesemer, J. R. (1989). Institutional ecology, “translations” and boundary objects: Amateurs and professionals in Berkeley’s Museum of Vertebrate Zoology, 1907-39. Social Studies of Science, 19(3), 387-420.
Taylor, C. (1971). Interpretation and the sciences of man. The Review of Metaphysics, 25(1), 3-51.
Tong, S., Mao, K., Huang, Z., Zhao, Y., & Peng, K. (2024). Automating psychological hypothesis generation with AI: When large language models meet causal graph. Humanities and Social Sciences Communications, 11(1), 896. https://doi.org/10.1057/s41599-024-03407-5 arXiv:2402.14424
Wallach, H., Desai, M., Cooper, A. F., Wang, A., Atalla, C., Barocas, S., Blodgett, S. L., Chouldechova, A., Corvi, E., Dow, P. A., Garcia-Gathright, J., Olteanu, A., Pangakis, N., Reed, S., Sheng, E., Vann, D., Vaughan, J. W., Vogel, M., Washington, H., & Jacobs, A. Z. (2025). Position: Evaluating generative AI systems is a social science measurement challenge. In Proceedings of ICML 2025, PMLR 267 (pp. 82232-82251). arXiv:2502.00561
Wang, A., Morgenstern, J., & Dickerson, J. P. (2025). Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence, 7(3), 400-411. https://doi.org/10.1038/s42256-025-00986-z
Westwood, S. J. (2025). The potential existential threat of large language models to online survey research. Proceedings of the National Academy of Sciences, 122(47), e2518075122. https://doi.org/10.1073/pnas.2518075122
Winch, P. (1958). The idea of a social science and its relation to philosophy. Routledge & Kegan Paul.
Yarkoni, T., & Westfall, J. (2017). Choosing prediction over explanation in psychology: Lessons from machine learning. Perspectives on Psychological Science, 12(6), 1100-1122. https://doi.org/10.1177/1745691617693393
Zhang, B., Li, L., Chen, Y., Zhao, Q., Zhu, Z., Yi, X., & Xie, X. (2025). Knowing your uncertainty: On the application of LLM in social sciences. arXiv:2512.05461.
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024). Can large language models transform computational social science? Computational Linguistics, 50(1), 237-291. https://doi.org/10.1162/coli_a_00502