Short essay 1 of 7 · The Almost-Map series
Synthetic Participants (Silicon Sampling)
Language models can stand in for survey respondents. Whether that works depends on two things at once: what the sample is asked to show, and how the model was conditioned.
Part of the series that opens with The Almost-Map. Published November 19, 2026. Last verified November 19, 2026.
Read on Substack ↗Suppose you have drafted three versions of a message encouraging people to get a flu shot, and you want to know which one works best before paying to field a survey experiment. Write a short profile of a person into a prompt (age, education, party, the kind of town they live in), show a language model one of the messages as though it were that person, and ask how likely they now are to get vaccinated. Repeat for a few thousand profiles drawn from a real survey, and you appear to have a sample, produced in minutes for the cost of a few API calls rather than months of recruitment. Argyle et al. (2023) gave the idea its name, silicon sampling, and its governing concept, algorithmic fidelity. They conditioned GPT-3 on thousands of sociodemographic backstories drawn from real survey respondents and recovered not just marginal opinions but the pattern of relationships among attitudes. In the terms of this series’ overview essay, The Almost-Map, this is the first area of use, synthetic participants (its Table 1). The areas are numbered for reference only; the order implies no ranking.
The demographic persona is the entry-level version of the method, not its only form: conditioning has since extended to a two-hour semi-structured interview transcript per person, a structured self-report survey, or a supervised fine-tune on hundreds of thousands of real experimental responses, and these are different practices that fail in different ways. What follows treats two questions as separable that the early literature tended to collapse into one: what kind of claim is being asked of the synthetic sample, and by what conditioning method was it produced. Holding both open at once, rather than passing one verdict on silicon sampling as a whole, is what lets a critique tell a researcher what to do with a particular study.
From the AI-engineering side the mechanics are familiar whichever conditioning method is in play. The persona or transcript lives in the prompt or the training data, few-shot examples shape the response format, and temperature tunes how much the answers vary; a model that produces a plausible-sounding response for a demographic persona is, on this account, doing its job. From the research side the same procedure is asked to carry a much heavier load. It is offered as instrument pretesting, ahead of an expensive fielded study; as pilot work, to catch a broken item or an underpowered design before real respondents see it; and as power estimation, to decide how large a human sample a study actually needs. Its quality, on that use, is judged by how closely the synthetic distribution resembles a human one. Argyle et al. were careful about what their term claims. Fidelity, in their account, is evidence of generalizability, “the degree to which we can apply what we learn from language models to the world beyond those models,” and it “does not imply that the model can simulate a specific individual” (p. 339). The engineering side has no equivalent stakes to get wrong; the research side does, and the same output is being read for two different kinds of success.
Measurement theory names the gap between the two readings. Borsboom et al. (2004) hold that a test is valid for measuring an attribute only if the attribute exists and variations in it causally produce variation in the measurement outcomes; validity, on their account, concerns ontology, reference and causality, not the epistemology, meaning and correlation that current validity theory more often centers. Distributional resemblance to a human sample, or a social-science Turing test in which synthetic answers are hard to tell from real ones, is the correlational, epistemic kind of criterion Borsboom et al.’s distinction warns against. Algorithmic fidelity looks like construct validity, but it is verified differently: in practice, fidelity checks compare a synthetic distribution’s shape, its means, variances, and correlation structure, against a human one, which is a check on resemblance, not on whether the underlying attribute exists in the model and drove the response the way it drives a person’s. A model can match a distribution while measuring nothing in particular. Argyle and her colleagues have since pressed the point further, reframing the project around clearly defining the target of inference and the conditions under which an inference from a model’s output is valid, rather than treating fidelity as a fixed property a model either has or lacks (Argyle et al., 2025).
The strongest case for the method comes from its largest test. Ashokkumar et al. (2026) tested GPT-4-derived predictions against 70 preregistered, nationally representative survey experiments spanning 469 effects and 119,330 participants: predicted treatment effects correlated with the actual ones at r = 0.85, and at r = 0.90 for studies not published or posted before the model’s training cutoff. The authors grant that “we cannot know which effects appear in the proprietary LLM training data” (p. 117), but the unpublished studies make memorization an unlikely explanation. By their estimate, an LLM-only pilot costing under a dollar matched the prediction error of a roughly 230-person human pilot once the output was rescaled, since the model systematically overestimates effect sizes. Three things bound the claim. The first is in the paper itself. In a second archive of 15 megastudies, ten survey and five field experiments mostly on stable attitudes with small effects, the correlation fell to r = 0.39, lower than in the main archive but comparable to the forecasts of domain experts (p. 118). Second, the result covers aggregate treatment-effect direction and magnitude on text-based survey experiments; it says nothing about which individual items in an instrument misbehave, and should not be read as evidence for that separate, more granular use. Third, Cui et al. (2025) independently replicated 156 scenario-based experiments from five management and psychology journals with three models, and found the same pattern: the models recover the direction of an effect more reliably than its size. Main effects replicate at 73 to 81 percent and interaction effects, the more demanding target, at 46 to 63 percent, which the authors call “moderate to strong success” (abstract), though they report interactions only in a supplement because those samples are smaller. Effect sizes run roughly 2 to 3 times the human ones on Fisher’s Z, and where the original study found a null, the models produced a significant result 68 to 83 percent of the time, which “could reflect cleaner data with less noise” but also “suggests potential risks of effect size overestimation” (abstract). Both sets of authors draw the same conclusion: the models can augment studies with human participants, not replace them. And rescaling, in both papers, corrects a consistent bias in the aggregate; it cannot tell a researcher which item in an instrument is behaving badly, which is what most researchers want from a pilot.
A further complication sits underneath even this best case. Kolluri et al. (2025) show that distributional alignment and individual-response accuracy respond to different training objectives: supervised fine-tuning does most for the match between a model’s output distribution and real survey responses, contrastive preference tuning does most for individual predictions, and improving one does not reliably carry the other along. On outcomes held out from training, fine-tuning improved distributional alignment by 44 percent and left individual accuracy exactly where it was (their Table 3). Matching a population and predicting a person are not the same target reached by two routes; they are two targets. Park et al. (2026, preprint) reach the same conclusion from the other direction: “accuracy for individuals does not guarantee fidelity for a population” (p. 14).
Where the method is asked to stand in for the people themselves rather than forecast an aggregate effect, it breaks along the seam social science cares most about. Peer-reviewed work shows current models misportray and flatten identity groups: Wang et al. (2025) demonstrate this across four models and 3,200 human participants spanning 16 demographic identities. Bisbee et al. (2024) show the mechanism directly in a single dataset. ChatGPT’s simulated feeling-thermometer scores track the ANES averages well, every synthetic mean falling within one standard deviation of the survey’s, but they vary far less than real responses, especially on questions about racial and religious groups. The consequence for inference is concrete. Planning a study of affective polarization from the pooled ANES estimates (an effect size of 7.8, sd 31.4) calls for 129 respondents to reach 80 percent power; planning it from ChatGPT’s estimates (12.5, sd 16.1) calls for 16. In the authors’ words, “sampling by ChatGPT is not reliable for statistical inference” (p. 401). The obvious engineering response, that a model should be steered more explicitly toward the demographic group in question, does not repair the failure. Santurkar et al. (2023) compared language models’ opinions with Pew survey responses from 60 US demographic groups, then tried steering the models toward 22 of those groups by describing the group in the prompt. Steering generally helped, but it did not close the gap. Flattening and variance compression, in other words, are not artifacts of an unprompted default persona that a more careful prompt would fix. They survive the fix that engineering intuition reaches for first, which matters for what comes later in this essay: the failure belongs to demographic persona-prompting itself, and what moves it is a different conditioning method, not a better-worded prompt.
A second problem lies deeper, in the instruments themselves. Meyer et al. (2026, preprint) administered Big Five and risk-preference batteries to 56 instruction-tuned models alongside large human reference samples, decomposing each response into a trait component and a directional response bias. In humans, variation is trait-driven; in models a variance decomposition attributes 81 to 90 percent of between-model variation to response bias rather than to any measured trait, against 9 to 16 percent in humans, and across 29 instruments an LLM scale’s apparent reliability was almost entirely predicted by what the authors call its response orthogonality, the share of items, typically the reverse-keyed ones, on which trait and bias point in opposite directions. Dominguez-Olmedo et al. (2024) reach a parallel conclusion from the survey-methodology side rather than the psychometric one, putting the Census Bureau’s American Community Survey to 43 models. Under standard prompting, model responses are governed by ordering and labeling biases, such as a pull toward whichever answer is labeled “A,” rather than by the content of the question; once the answer order is randomized to remove that bias, the responses trend toward uniformly random, and a classifier can tell model-generated answers from real census responses with more than 90 percent accuracy. One paper decomposes variance within a psychometric battery, the other within a survey instrument, and they arrive at the same finding by different routes: a synthetic distribution can match a human one and still rest on responses that carry no construct, because the very artifact producing the resemblance, whether reverse-keying sensitivity or answer-ordering sensitivity, is what a researcher would otherwise cite as evidence the instrument worked. A researcher checking a synthetic sample’s internal consistency, the ordinary first move in evaluating any instrument, would in this case be measuring the artifact’s stability rather than the construct’s, and the check would return a reassuring number for the wrong reason.
Conditioning method is not a footnote to this picture; it changes the picture. Park et al. (2026, preprint) built simulations of 1,052 Americans from two-hour interviews run by an AI interviewer, from each person’s own survey answers, or from both, and scored each simulation against the person’s answers to held-out General Social Survey items. Given only demographics, a simulation matched 60.4 percent of a person’s answers; given a short paragraph the participant wrote about themselves, 59.7 percent. Both are about what you get by predicting that everyone gives the sample’s most common answer, 59.3 percent (p. 6). Interview-grounded and survey-grounded simulations matched 69.4 and 69.3 percent, and the two combined 72.8 percent, against the 79.5 percent of answers people repeated themselves two weeks later.
Richer grounding also shrank, without erasing, the sample-level distortions this essay has been describing. Combined grounding closed about half the gap between demographic simulation and the real sample on seven of eight measures, including compressed variance (57 percent) and inflated experimental effects (46 percent), and did not reduce the eighth, the simulations’ excess rationality in economic games (p. 11). Its effect on group disparities is mixed: grounding in survey answers narrowed the gap in accuracy across ideological groups, but by race the demographic simulation had the smallest gap of all (SI Appendix, section 9.4). All of this comes from one closed model, Claude Opus 5.5, and the authors note that an older model distorted differently (p. 14).
The gap between demographic prompting and richer grounding is the largest difference in Park et al.’s results, and Wang’s flattening result, Bisbee’s variance-compression result and Santurkar’s steering result were all obtained under demographic conditioning, the weakest method on Park et al.’s own numbers. Richer grounding is not a free upgrade, though. Building a person-specific agent from a named individual’s two-hour interview raises consent and data-provenance questions that a demographic persona, built from census categories rather than a particular life, does not, and a method that improves accuracy while introducing a new ethical obligation has not simply won on both counts.
Two further sources explain why conditioning method carries this much weight: one from research design, one from causal inference. Lyman et al. (2025), several of whom also wrote the paper that named algorithmic fidelity, compared aligned and unaligned versions of six open-source models and found that alignment tuning, the safety and preference training a model undergoes after pretraining, changes their simulated attitudes “in predictable ways” (p. 1110). Which model a researcher licenses, and how heavily that model has been aligned, is therefore not a neutral implementation detail but a research-design decision with systematic consequences of its own.
Gui and Toubia (2025, preprint) supply the mechanism in causal-inference terms. Their test case is a simulated pricing experiment: the prompt varies a product’s price, and the model, playing a customer, says whether it would buy. The trouble is that the prompt does not hold everything else fixed. The model reads the price as a clue to what the prompt leaves out, such as the store the customer is in or what competing products cost, so the treatment arrives bundled with other changes and the comparison is confounded. The natural repair is the one silicon sampling already makes, which is to describe the simulated person in more detail, and Gui and Toubia treat Argyle et al.’s (2023) demographic backstories as exactly that: a way of controlling for covariates. In their tests it helps but “does not fully resolve the issue” (p. 9), and it brings a cost of its own, since covariates written into the prompt “become artificially salient in the simulated decision process” (p. 1).
Their remedy, which they call unblinding, is to show the model the experimental design: to tell it that the price was drawn at random, and from what range. Across GPT models from GPT-3.5-Turbo to GPT-4.1, unblinded prompts reduced prediction error by between 1 and more than 60 percent on a benchmark of 40 products (p. 15). The paper is still a working paper, last revised in 2025. Read alongside Lyman et al., it turns conditioning method from an engineering knob into a variable with confounds of its own, one a researcher is responsible for in the same way they are responsible for any other feature of experimental design.
None of this reaches validated measurement. A simulation that matches 73 percent of a person’s answers, when the person matches 80 percent of their own, is still short of that ceiling; Park et al.’s paper remains a preprint; and richer grounding raises its own consent and provenance questions when the interview data comes from real, named individuals. But it does mean that a blanket verdict on silicon sampling answers a question with two free variables while fixing only one. A critique aimed at “silicon sampling” without naming a conditioning method reads, to a researcher using interview-grounded agents, as though it were describing a different practice than the one in front of them, because in the sense that matters for the argument, it is.
So the alignment verdict is conditional on construct type and on conditioning method at once, not on either alone. On construct type: aggregate treatment-effect direction and magnitude, and text-based survey-experiment outcomes, replicate well enough to support pretesting and piloting once rescaled; individual-level measurement, subgroup differences, and any construct where identity or lived experience is load-bearing do not, regardless of conditioning method. On conditioning method: demographic persona-prompting is the weakest approach and the one the critical studies above mostly test, and demographic steering helps without repairing it; interview-grounding, survey-grounding, and fine-tuning on real experimental data materially outperform it on individual accuracy and on most sample-level distortions, without yet reaching validated measurement of a defined construct on a defined population. Model and alignment choice belong inside this second axis rather than outside it: which base model a researcher licenses is itself a design decision with documented, systematic effects, not a detail that becomes moot once demographic personas are set aside for richer conditioning, and any treatment written into a prompt, under any conditioning method, carries a confounding risk that a citation to Park et al.’s numbers does not, by itself, discharge.
The use is defensible for pretesting and piloting aggregate effects, with the specific conditioning method and the specific construct class named every time; it is misaligned wherever a synthetic distribution, however produced, is treated as a measurement of a population, and doubly misaligned wherever the construct concerns an identity or subgroup a persona-prompted model has already been shown to flatten. Hullman et al. (2026, preprint) draw the same line formally. Repairs that make simulated responses resemble human ones, such as better prompts or fine-tuning, are useful for exploration but “lack the formal statistical guarantees typically required for confirmatory research” (p. 1); statistical calibration against a human sample can keep an estimate valid under stated assumptions, which is the route the essay on valid inference in this series describes for text labels. The gains so far are modest: in one application, 100,000 model-predicted decisions added only about 13 percent to the effective size of a sample of 10,000 human decisions (p. 15). The non-negotiable human role is a held-out human validation sample used to estimate fidelity on the specific construct and the specific conditioning method actually in use, reported alongside flattening, heterogeneity, and response-bias diagnostics, with the work framed as a supplement to human data and never its replacement.
References
- Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023). Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3), 337-351. https://doi.org/10.1017/pan.2023.2
- Argyle, L. P., Busby, E. C., Gubler, J. R., Hepner, B., Lyman, A., & Wingate, D. (2025). Arti-‘fickle’ intelligence: Using LLMs as a tool for inference in the political and social sciences. Nature Computational Science, 5(9), 737-744. https://doi.org/10.1038/s43588-025-00843-4
- Ashokkumar, A., Hewitt, L., Ghezae, I., & Willer, R. (2026). Large language models can predict the results of social science experiments. Nature, 656(8126), 115-122. https://doi.org/10.1038/s41586-026-10742-x
- Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. M. (2024). Synthetic replacements for human survey data? The perils of large language models. Political Analysis, 32(4), 401-416. https://doi.org/10.1017/pan.2024.5
- Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111(4), 1061-1071. https://doi.org/10.1037/0033-295X.111.4.1061
- Cui, Z., Li, N., & Zhou, H. (2025). A large-scale replication of scenario-based experiments in psychology and management using large language models. Nature Computational Science, 5(8), 627-634. https://doi.org/10.1038/s43588-025-00840-7
- Dominguez-Olmedo, R., Hardt, M., & Mendler-Dünner, C. (2024). Questioning the survey responses of large language models. In Proceedings of NeurIPS 2024. https://arxiv.org/abs/2306.07951
- Hullman, J., Broska, D., Sun, H., & Shaw, A. (2026). This human study did not involve human subjects: Validating LLM simulations as behavioral evidence [Preprint]. arXiv. https://arxiv.org/abs/2602.15785
- Gui, G., & Toubia, O. (2025). The challenge of using LLMs to simulate human behavior: A causal inference perspective [Preprint, v3; first posted 2023]. arXiv. https://arxiv.org/abs/2312.15524
- Kolluri, A., Wu, S., Park, J. S., & Bernstein, M. S. (2025). Finetuning LLMs for human behavior prediction in social science experiments. In Proceedings of EMNLP 2025 (pp. 30096-30111). https://arxiv.org/abs/2509.05830
- Lyman, A., Hepner, B., Argyle, L. P., Busby, E. C., Gubler, J. R., & Wingate, D. (2025). Balancing large language model alignment and algorithmic fidelity in social science research. Sociological Methods & Research, 54(3), 1110-1155. https://doi.org/10.1177/00491241251342008
- Meyer, J., Garcia, D., & Wulff, D. U. (2026). Apparent psychological profiles of large language models are largely a measurement artifact [Preprint]. arXiv. https://arxiv.org/abs/2606.20205
- Park, J. S., Zou, C. Q., Kamphorst, J., Egan, N., Shaw, A., Hill, B. M., Cai, C., Morris, M. R., Liang, P., Willer, R., & Bernstein, M. S. (2026). AI simulations grounded in rich self-reports increase accuracy and reduce distortion [Preprint, v4, 16 November 2026; first posted 2024]. arXiv. https://arxiv.org/abs/2411.10109
- Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., & Hashimoto, T. (2023). Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202, pp. 29971-30004). https://arxiv.org/abs/2303.17548
- Wang, A., Morgenstern, J., & Dickerson, J. P. (2025). Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence, 7(3), 400-411. https://doi.org/10.1038/s42256-025-00986-z
Further reading
- Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741-749. The unified, inferential counterpoint to Borsboom et al.’s causal account of validity: useful for showing that resemblance is not validation on either reading.
- Taday Morocho, E. E., Cima, L., Fagni, T., Avvenuti, M., & Cresci, S. (2026). Assessing the reliability of persona-conditioned LLMs as synthetic survey respondents. In Companion Proceedings of the ACM Web Conference 2026 (pp. 320-329). https://doi.org/10.1145/3774905.3795477 Across more than 70,000 respondent-item instances from U.S. World Values Survey microdata, giving two open-weight models a persona produced no clear overall improvement in survey alignment and often made it significantly worse, with the distortions concentrated in a few items and in underrepresented subgroups.
Code and data
Each pointer below is named in the paper’s own text, except where noted, and was confirmed on 2026-11-19 by downloading the repository, reading the dataset’s card, or reading the archive’s own record. None has been run for this essay. Each line names the claim above that it lets a reader check.
- Replication data for Argyle et al. (2023) (Harvard Dataverse; CC0). Ninety-two files, including the GPT-3 outputs for the ANES tasks. Checks the original fidelity results, including the claim that the model recovers the pattern of relationships among attitudes, not only their marginal distributions.
- Replication archive for Ashokkumar et al. (2026) (Code Ocean capsule 9843791, version 1, June 2026, titled after the paper’s 2024 preprint, “Predicting Results of Social Science Experiments Using Large Language Models”). One R script,
code/00_run_all_analyses.R, reruns the paper’s analyses; the README warns that it takes multiple hours. Checks the correlations of r = 0.85 and 0.90, the megastudy result of r = 0.39, and the rescaled-pilot comparison. - Code for Wang et al. (2025) (OSF). The analysis code and the models’ generations. Checks the flattening and misportrayal results across the 16 identities.
- Replication data for Bisbee et al. (2024) (Harvard Dataverse; CC0). A single archive. Checks the compressed variance and the power calculation of 129 against 16 respondents.
- OpinionQA (github.com/tatsu-lab/opinions_qa; Santurkar et al., 2023). The 1,498 survey questions drawn from Pew’s American Trends Panel, the human responses and the paper’s model runs, with a notebook for the steering analysis. Checks the claim that steering a model toward a demographic group helps without closing the gap.
surveying-language-models(github.com/socialfoundations/surveying-language-models; Dominguez-Olmedo et al., 2024; MIT licence). Code that puts the American Community Survey to language models, plus the discriminator test. Checks the claims that responses follow answer order and labels, trend toward uniform once the order is randomized, and can be told apart from census responses by a classifier.llm-profile-artifact(github.com/jelenameyer/llm-profile-artifact; Meyer et al., 2026, preprint; CC BY 4.0), with the cleaned data on OSF (osf.io/nckds). One command,python run_all.py, reproduces every figure and table without GPUs or API keys. Checks the 81 to 90 percent of between-model variation attributed to response bias, and the link between response orthogonality and apparent reliability.- Code and aggregate results for Park et al. (2026, preprint) (OSF). The code for every simulation and analysis, the simulation harness, the study instruments and all aggregate results; the code for building agents from interview transcripts is on GitHub (MIT licence), in a repository last updated in November 2024 that does not include the AI voice interviewer v4 describes. The interviews themselves are not public, and the de-identified survey responses are available to researchers only through the study’s data-access process. Checks the accuracy and distortion figures from the aggregate results, though not by rerunning simulations of the actual participants.
- Replication materials for Lyman et al. (2025) (GitHub; MIT licence; archived at doi.org/10.5281/zenodo.15375190). Prompts, model responses and R code that runs end to end. Checks the claim that aligned and unaligned versions of the same model simulate attitudes differently, and in predictable ways.
- SocSci210 (huggingface.co/datasets/socratesft/SocSci210; Kolluri et al., 2025). 2.9 million responses from 210 experiments, with the splits the paper uses. The paper points to a project page that could not be opened for this check; the dataset’s card names the paper. No licence is stated on the card. Checks the trade-off between distributional alignment and individual accuracy, though fine-tuning a model on it needs GPU hardware.