Short essay 6 of 7 · The Almost-Map series
Valid Inference Infrastructure: PPI, PPI++ and DSL
Language models can label text at scale. Turning those labels into a valid estimate takes a statistical correction, and it works only under four conditions.
Part of the series that opens with The Almost-Map. Published November 16, 2026. Last verified November 16, 2026.
Read on Substack ↗Suppose you want to know what share of fifty thousand tweets about a new policy are hostile to it. Hand-coding them would take a team of research assistants months. A language model will label all of them in an afternoon, and on a sample it agrees reasonably well with your human coders. Can you run your analysis on its labels? That is the most common quantitative use of language models in social science, and the usual answer is very nearly right. In the terms of this series’ overview essay, The Almost-Map, the workflow spans two things at once: the sixth area of use, text annotation at scale, and the statistical correction, one of the practices that essay groups under “doing it right” (its Table 1). The areas are numbered for reference only; the order implies no ranking.
The template was set by Gilardi et al. (2023), who showed that ChatGPT annotated tweets and news articles for relevance, stance, topic and frame more accurately than crowd workers on most tasks, at roughly a thirtieth of the cost. Rathje et al. (2024) extended it to multilingual psychological text analysis, and Törnberg (2025) reported that a model outperformed expert coders and supervised classifiers on political social media messages. These papers are careful. What other researchers took from them was the procedure, not the caution: benchmark the model against human coders, report the agreement, then use the labels.
A large and fast-growing literature now follows that procedure. In a corpus of 231 recent papers on AI and social-science methods that I assembled for this series, annotation is the second-largest theme, with 78 papers, about a third of the corpus. A language model assigned the themes from titles and abstracts, so the count is approximate. A systematic review inside Baumann et al.’s (2025, preprint) study puts the consensus in numbers: of 93 studies of LLM annotation in computational social science, 85 recommend it under at least some conditions and 8 advise against.
That procedure has a hole in it, and the hole is exactly one stage wide. Run the workflow and the two views track each other closely for three stages. The engineer applies a prompt across an unlabeled corpus, the model returns structured labels, and the labels are checked against a human-annotated benchmark. The researcher, watching the same three stages, sees a codebook, a coding pass, and a reliability check. It looks like measurement. At stage four the resemblance breaks. The engineer has run an eval, which asks whether the labels match the benchmark. The researcher needs validation, which asks whether the estimate built from those labels is unbiased and whether its confidence interval covers. These are not the same question, and high agreement does not answer the second one.
Egami et al. (2024, working paper) put numbers on the gap. In their simulations, ignoring prediction errors biases regression coefficients by as much as 30 percent of their true value at 90 percent annotation accuracy, and 18 percent at 95 percent, while the coverage of nominal 95 percent confidence intervals collapses to 32 percent and 58 percent respectively. An interval a researcher believes will cover the truth nineteen times in twenty is covering it fewer than two times in three. Egami et al. (2023) state the general result in their abstract: plugging model labels straight into a downstream analysis biases the estimate and invalidates its confidence intervals “even with high surrogate accuracy of 80–90%.” The reason is not exotic. Prediction errors are not random. They correlate with the outcome, with the covariates, and with whatever is unobserved, so they do not average away. Accuracy is not a license to skip the correction. Gilardi et al. (2023) themselves reported zero-shot accuracies on the relevance task of 59 to 83 percent across their four datasets, so the paper that founded the practice was already working inside the range where the correction is needed.
The result that should end the argument is not a bias figure at all. It is that the substantive conclusion flips depending on which model you happen to pick. Re-analyzing a published study of political advertising, Egami et al. (2024, working paper) found that a researcher annotating with GPT-4 and few-shot prompting would conclude the effect was statistically indistinguishable from zero, while a researcher annotating the same ads with Llama 2, also few-shot, would report a significant negative effect. Same data, same question, same downstream model. Only the labeler differs. The Almost-Map opens on this case and draws it as its Figure 1. There the hand-coded ads could settle which labeler was right, and it was Llama 2. In a real study, where an expert-coded benchmark for the full corpus does not exist, there is no principled way to decide which to believe. The choice of annotator has quietly become a researcher degree of freedom, and an undisclosed one. That is not this essay’s coinage: Baumann et al. (2025, preprint) frame LLM configuration choice explicitly as Gelman and Loken’s (2013, unpublished manuscript) garden of forking paths, and reach for Gelman and Carlin’s (2014) Type S and Type M vocabulary to describe what comes out the other end.
Baumann et al. (2025, preprint) also establish how large the problem is, and its size is the reason this workflow matters rather than merely being tidy. Replicating 37 annotation tasks drawn from 21 published social science studies across 18 models, they generated 13 million labels and 1.4 million regressions covering 2,361 hypotheses. Taken as a question about what a motivated analyst could manufacture, a false positive can be produced for 94.4 percent of null hypotheses, a true effect hidden in 98.1 percent of cases, and a significant effect reversed in sign in 68.3 percent of cases with true differences, and restricting to top-performing models does not close this. Taken as a question about accident rather than intent, incorrect conclusions arrive in 31 percent of hypotheses for the best models above 70B parameters and 50 percent for 1B models. Where the risk concentrates is instructive. Proximity to the significance threshold is the strongest predictor, with error rates near 70 percent for p values around 0.05, and task characteristics account for 21 percent of explained variance against 8 percent for model performance and under 1 percent for prompt engineering. The lever the field has been pulling hardest is attached to almost nothing.
The finding that bears most directly on stage four is that agreement does not track the thing researchers use it to certify. Baumann et al. (2025, preprint) report no correlation between human inter-annotator agreement and the risk of a wrong conclusion. Gligorić et al. (2025) supply the same claim as a single case: estimating an odds ratio for stance on global warming, the LLM-only estimate points in the wrong direction, and does so in the setting with the highest human-model agreement in their study, a Cohen’s kappa of 0.57. Agreement was highest exactly where the naive estimate was most wrong. Together the two results say something sharper than the familiar caution that agreement is insufficient for validity: in these data, agreement was no guide to it. A reliability statistic is a fact about two labelers; a coverage failure is a fact about an estimator. Reporting the first and inferring the second is the whole error.
The fix is old statistics pointed at a new labeler. Demote the model: ask it only to label, never to be the measurement. From the AI-engineering side the pipeline looks unchanged, a model generating structured labels with an eval and a config log. From the research side it is a different object. The estimand is defined first, a prevalence, a mean, a regression coefficient, or a treatment effect; a human gold set is drawn by a known, probability-based sampling design; and a debiasing estimator does the real work. Prediction-powered inference (Angelopoulos, Bates, et al., 2023) combines many machine labels with a few gold labels to produce valid confidence intervals; PPI++ (Angelopoulos, Duchi, & Zrnic, 2023, preprint) adds a tuning step that down-weights the model when it is inaccurate. Design-based supervised learning (Egami et al., 2023) reaches the same destination by what statisticians call a doubly robust route. It rebuilds each document’s outcome from the model’s prediction plus a correction for the hand-coded documents, weighted by the known chance that each one was chosen for hand-coding, and so stays asymptotically unbiased even when the labels are arbitrarily biased. Its single assumption is one a researcher can guarantee by design rather than argue for: that they control how documents are sampled for expert coding. That is where design-based gets its name.
Mozer (2026, preprint) has since shown the lineage is older than the vocabulary. The PPI estimator for a population mean is algebraically identical to the difference estimator of Cassel et al. (1976), and PPI++ is the generalized regression estimator, with the tuning parameter playing the role of a regression coefficient. PPI’s central promise, that inference stays valid regardless of prediction quality, is the defining property of model-assisted estimation, known there as design consistency. This is more than a footnote about priority. It tells you where to look for the parts that are missing. Calibration and post-stratification, optimal allocation of labeling effort, design-based diagnostics, and sensitivity analysis for assumption violations all live there, and the PPI literature has largely not imported them.
Is there a cost to trying the correction? Its reputation says no: the interval stays valid whatever the model’s quality, so a weak model can only fail to help. That is false in small samples. Mani et al. (2026) prove that in the Gaussian case PPI++ has lower variance than the gold-only estimator if and only if the correlation between machine and gold labels exceeds 1/√(c·n − 2), where n is the number of gold labels, c = 1 for the single-sample estimator and c = 1/2 for the cross-fit version. Below that threshold, PPI++ is worse than simply using the gold labels alone, and when the correlation is exactly zero both variants are always worse, by a factor of (1 + 1/(c·n − 1)), because the tuning parameter is estimated from data and will not come out exactly zero even when it should. Single-sample PPI++ is additionally biased in finite samples for non-Gaussian data, and variance estimates treating the tuning parameter as fixed are anti-conservative, so intervals come out too narrow. The exact result is proved for mean estimation, and the authors are candid that the extension to the M-estimation case social scientists usually want reaches only the first stage of the argument. Mani et al. do not dispute the asymptotic guarantee. It does not describe the small-gold-set regime where the method is most attractive.
Empirically, Audinet de Pieuchon et al. (2025) found the debiasing methods unstable below roughly 200 expert annotations, and on one of their four datasets DSL performed worse than both PPI and the expert-only estimator at every sample size tested, traced partly to multicollinearity among the features. Mozer (2026, preprint) adds a failure that more gold labels cannot touch at all: in a randomized experiment with machine-scored outcomes and a single pooled rectifier, the treatment-effect estimator carries a bias equal to the difference in arm-specific mean prediction error, and that bias is unaffected by the size of the labeled sample.
So the honest replacement is a checklist rather than a slogan. Four conditions make the asymptotic guarantee a reasonable bet in a finite sample. The gold set must be probability-sampled and large enough, with roughly 200 as a working floor on the published evidence, which is one benchmark across four datasets rather than a settled constant. The estimator must be cross-fit rather than single-sample, which buys unbiasedness and honest intervals at the price of a higher correlation threshold, since each fold estimates the tuning parameter from half the labels. The rectifier must be computed within the subgroups whose contrast is the estimand, not pooled across them. And the expected gain must be reported rather than assumed, since it is calculable in advance from the correlation and the gold-set size. Meet those four, and the corrected estimate is the defensible choice: its error rates are known, while an uncorrected workflow carries bias of unknown size and sign. Miss them, and the correction can cost more than it buys.
One point here comes from the philosophy of statistics. Error-statistical inference rejects the likelihood principle precisely so that selection effects remain visible in error probabilities, and holds that a claim earns credence only when it has passed a test it would probably have failed had it been false (Mayo, 2018; the underlying result is Birnbaum, 1962). The choice of annotator is a selection effect of this kind. An uncorrected workflow does not merely add noise; it forfeits the error-probability guarantee silently, because the interval is computed as though the labels were the measurements. What the debiasing estimator restores is not accuracy but severity: it puts the error probability back on a footing where it means what it claims to mean. Baumann et al. (2025, preprint) have done the empirical half of this argument in Type I, Type II, Type S and Type M vocabulary without attaching the philosophical label, which is my addition. The Almost-Map generalizes the point across the other workflows in its section “The philosophical debt.”
What the correction buys must be stated as carefully as what it costs, or the essay becomes advocacy. Baumann et al. (2025, preprint) tested 21 mitigation techniques against their 2,361 hypotheses. DSL and Confidence-Driven Inference (Gligorić et al.’s method, which uses the model’s stated confidence to choose which items people code) successfully reduce Type I errors to nominal levels, and they increase Type II errors by up to 60 percentage points. Statistical validity is bought with power, and the price is not small. The comparison that should unsettle anyone reaching for this workflow as a scaling device is blunter: one hundred human annotations controlled false positives better than one hundred thousand uncorrected LLM annotations, and even a thousand human annotations brought overall risk of a wrong conclusion only to about 20 percent, against a 31 to 50 percent uncorrected hybrid baseline. This does not overturn the case for correction. Nothing in it recommends the uncorrected workflow, which is worse than both. What it does is fix what the correction is for. The wrapper buys scale and precision on an estimand you could otherwise not afford to measure at all. It does not buy protection against claiming an effect that is not there, and the human role, though small, cannot approach zero without the guarantee going with it.
The method’s authors state the main limitation themselves. The wrapper corrects prediction error, not construct validity: DSL “can only account for prediction errors,” and this “does not replace careful, theoretical considerations” about how a codebook operationalizes the concept (Egami et al., 2024, working paper, p. 55). Whether the codebook operationalizes the concept the theory is about is a question the mathematics cannot see. If the model and the human coders share the same misunderstanding of what is being measured, no amount of statistical machinery will rescue the inference. Pita (2026, preprint) has recently turned that limit into a research programme rather than a caveat, observing that statistical corrections operate downstream of the code and so cannot reveal that the instrument measures the wrong construct, and grounding the demand in Messick’s (1995) substantive aspect of construct validity, with Cronbach and Meehl (1955) in the background. The proposal is early work and single-authored, and it is cited here for its argument rather than for any of its borrowed figures. The argument is the right one. Correcting an estimate and validating a construct are different operations, and this workflow performs only the first.
Does applied social science use the correction? That can now be checked against evidence rather than asserted. Desai, Card, et al. (2026, preprint) screened 2,143 research articles from eight flagship social science journals published between 2022 and September 2025, arriving at 27 papers and 50 measurement tasks using LLMs as measurement instruments. Validation by comparison to a gold standard, that is convergent validity, dominates: 22 papers and 39 tasks, with the figure caption giving 38 against the text’s 39. Eight tasks across six papers report no validation at all, including five tasks in three papers where the measurement feeds the main analysis, and in 28 of the 42 validated tasks only one aspect of validity is assessed. Gold sets are sometimes drawn from extremely small samples, in one case 50 items out of 68,000, and comparisons often use percent agreement or Pearson correlation, which are not chance-corrected. No paper in the corpus is reported as using a debiasing estimator, but that should be read with nuance. That is, it is an absence from a coding scheme. The scheme’s validation categories come from Jacobs and Wallach’s (2021) components of construct validity, where a correction estimator has no natural place, since it repairs the estimate built from a measure rather than validating the measure. The authors’ coding data, 33 dimensions for each task, is public, so a direct count is checkable. And the authors know the correction exists. Their introduction cites prediction-powered inference and design-based supervised learning and judges that “the improvements from such methods tend to be small, and there is no guarantee they will help in any particular case” (p. 2). Given the finite-sample results above, that is a fair reading. Their source for the second half of that judgment, Hullman et al. (2026, preprint), writing about simulated participants rather than annotation, is warmer toward the method: it warns that “no method can guarantee unbiased estimates for a single finite sample” (p. 16), but holds that, under explicit assumptions, statistical calibration “preserves validity,” and that confirmatory research needs the formal guarantees that heuristic fixes lack. So, the applied side is not simply unaware of the correction, it is not yet persuaded.
The exhibit that makes the shape of the gap unmistakable is not a delinquent paper but the field’s own standards document. Törnberg’s (2024) Best Practices for Text Annotation with Large Language Models was written to fix what its abstract calls something of an academic Wild West, by the author of the paper reporting that models outperform expert coders. It runs to nine numbered best practices and is unusually good on the surrounding stages: right about model drift, about prompt stability analysis, about training-data contamination, and that human coders should not be treated as automatic ground truth. Its section on validation is about agreement with a held-out human set, prescribing F1, precision and recall, ROC-AUC, Cohen’s kappa, and comparison against human intercoder reliability as a benchmark. It comes right to the edge of this essay’s problem. It warns that a model’s failures “can lead to errors in the downstream analysis, which are not visible in the performance metrics,” and gives a case: if a minority makes up 10 percent of the data and model bias misclassifies 30 percent of its documents, the resulting 3 percent failure rate “would often be seen as acceptable” (p. 76). It then asks researchers to “verify that they are not systematic and that they do not undermine the validity of downstream results” (pp. 76-77). What it offers for that is an inspection of failures, written up in prose. It never mentions PPI, DSL or measurement error, or a way to measure what the labels do to a regression coefficient. Here is the document written to close the holes in that procedure, seeing the one this essay is about and answering it with a description where an estimator is needed. That the rest of it is so good makes the omission more telling, not less.
The verdict is the most favorable in the series. This is the most defensible way to use a language model for inference today. The non-negotiable requirement is a well-sampled, human-labeled gold set with a documented sampling design, and the four conditions above convert that requirement from a principle into something a reviewer can check. Supply them, and the model becomes a cheap accelerant to valid science. Omit them, and the workflow collapses back into the plausibility trap that the other essays in this series describe.
The verdict has a second half. The correction is not a scaling device that makes human labor optional, and the evidence on Type II inflation and on the protective value of small human samples says so plainly. It is a device for spending a fixed and irreducible quantity of human judgment where it does the most statistical work. Which is, on reflection, what survey sampling has always been for.
The tell, as everywhere in this series, is a word doing double duty. Validation means one thing to an engineer, a check that outputs match a reference, and another to a methodologist, a warrant that an estimate carries the properties claimed for it. The distance between them is one stage wide, and it is the whole difference.
References
- Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., & Zrnic, T. (2023). Prediction-powered inference. Science, 382(6671), 669-674. https://doi.org/10.1126/science.adi6000
- Angelopoulos, A. N., Duchi, J. C., & Zrnic, T. (2023). PPI++: Efficient prediction-powered inference [Preprint]. arXiv. https://arxiv.org/abs/2311.01453
- Audinet de Pieuchon, N., Daoud, A., Jerzak, C. T., Johansson, M., & Johansson, R. (2025). Benchmarking debiasing methods for LLM-based parameter estimates. In Proceedings of EMNLP 2025 (pp. 19757-19772). https://doi.org/10.18653/v1/2025.emnlp-main.1000
- Baumann, J., Röttger, P., Urman, A., Wendsjö, A., Plaza-del-Arco, F. M., Gruber, J. B., & Hovy, D. (2025). Large language model hacking: Quantifying the hidden risks of using LLMs for text annotation [Preprint, v2]. arXiv. https://arxiv.org/abs/2509.08825
- Birnbaum, A. (1962). On the foundations of statistical inference. Journal of the American Statistical Association, 57(298), 269-306. https://doi.org/10.1080/01621459.1962.10480660
- Cassel, C. M., Särndal, C. E., & Wretman, J. H. (1976). Some results on generalized difference estimation and generalized regression estimation for finite populations. Biometrika, 63(3), 615-620. https://doi.org/10.1093/biomet/63.3.615
- Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281-302. https://doi.org/10.1037/h0040957
- Desai, M., Card, D., & Jacobs, A. Z. (2026). Validating LLMs in social science: Epistemic threats and emerging norms [Preprint]. arXiv. https://arxiv.org/abs/2607.07915
- Egami, N., Hinck, M., Stewart, B. M., & Wei, H. (2023). Using imperfect surrogates for downstream inference: Design-based supervised learning for social science applications of large language models. In Advances in Neural Information Processing Systems 36 (pp. 68589-68601). https://arxiv.org/abs/2306.04746
- Egami, N., Hinck, M., Stewart, B. M., & Wei, H. (2024). Using large language model annotations for the social sciences: A general framework of using predicted variables in downstream analyses [Working paper, version of 17 November 2024]. http://dsl.software
- Gelman, A., & Carlin, J. (2014). Beyond power calculations: Assessing Type S (sign) and Type M (magnitude) errors. Perspectives on Psychological Science, 9(6), 641-651. https://doi.org/10.1177/1745691614551642
- Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time [Unpublished manuscript]. Department of Statistics, Columbia University. https://sites.stat.columbia.edu/gelman/research/unpublished/p_hacking.pdf
- Gilardi, F., Alizadeh, M., & Kubli, M. (2023). ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30), e2305016120. https://doi.org/10.1073/pnas.2305016120
- Gligorić, K., Zrnic, T., Lee, C., Candès, E. J., & Jurafsky, D. (2025). Can unconfident LLM annotations be used for confident conclusions? In Proceedings of NAACL 2025 (pp. 3514-3533). https://arxiv.org/abs/2408.15204
- Hullman, J., Broska, D., Sun, H., & Shaw, A. (2026). This human study did not involve human subjects: Validating LLM simulations as behavioral evidence [Preprint]. arXiv. https://arxiv.org/abs/2602.15785
- Jacobs, A. Z., & Wallach, H. (2021). Measurement and fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 375-385). https://doi.org/10.1145/3442188.3445901
- Mani, P., Xu, P., Lipton, Z. C., & Oberst, M. (2026). No free lunch: Non-asymptotic analysis of prediction-powered inference. In Proceedings of the 43rd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 306, pp. 86296-86335). https://arxiv.org/abs/2505.20178
- Mayo, D. G. (2018). Statistical inference as severe testing: How to get beyond the statistics wars. Cambridge University Press.
- Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741-749.
- Mozer, R. (2026). PPI is the difference estimator: Recognizing the survey sampling roots of prediction-powered inference [Preprint]. arXiv. https://arxiv.org/abs/2603.19160
- Pita, M. (2026). Correct codes for the wrong reasons? Validating LLMs as measurement instruments for theoretical constructs [Preprint]. arXiv. https://arxiv.org/abs/2606.28574
- Rathje, S., Mirea, D. M., Sucholutsky, I., Marjieh, R., Robertson, C. E., & Van Bavel, J. J. (2024). GPT is an effective tool for multilingual psychological text analysis. Proceedings of the National Academy of Sciences, 121(34), e2308950121. https://doi.org/10.1073/pnas.2308950121
- Törnberg, P. (2024). Best practices for text annotation with large language models. Sociologica, 18(2), 67-85. https://doi.org/10.6092/issn.1971-8853/19461
- Törnberg, P. (2025). Large language models outperform expert coders and supervised classifiers at annotating political social media messages. Social Science Computer Review, 43(6), 1181-1195. https://doi.org/10.1177/08944393241286471
Further reading
- Carroll, R. J., Ruppert, D., Stefanski, L. A., & Crainiceanu, C. M. (2006). Measurement error in nonlinear models: A modern perspective (2nd ed.). Chapman and Hall/CRC. The errors-in-variables anchor and the source of the non-differential measurement error principle.
- Chen, Y. H., & Chen, H. (2000). A unified approach to regression analysis under double-sampling designs. Journal of the Royal Statistical Society: Series B, 62(3), 449-460. The two-phase design lineage under the modern refinements.
- Demszky, D., Yang, D., Yeager, D. S., Bryan, C. J., Clapper, M., Chandhok, S., Eichstaedt, J. C., Hecht, C., Jamieson, J., Johnson, M., Jones, M., Krettek-Cobb, D., Lai, L., Jones-Mitchell, N., Ong, D. C., Dweck, C. S., Gross, J. J., & Pennebaker, J. W. (2023). Using large language models in psychology. Nature Reviews Psychology, 2(11), 688-701. https://doi.org/10.1038/s44159-023-00241-5 One of the primers that codified the practice this essay examines.
- Gligorić et al. (2025), above, on Confidence-Driven Inference as a method rather than for the single result cited. Verbalized model confidence decides which items get human annotation, cutting the human annotations required by at least 25 percent across five estimation tasks.
- Lin, H., & Zhang, Y. (2026). Navigating the risks of using large language models for text annotation in social science research. Social Science Computer Review, 44(3), 403-427. https://doi.org/10.1177/08944393251366243 Twenty-five repeated runs of an identical prompt at identical settings give an intraclass correlation of 0.934 on a binary task and 0.518 on a multiclass one; The Almost-Map takes these figures up in its section “Stochasticity versus reproducibility.”
- Pangakis, N., Wolken, S., & Fasching, N. (2023). Automated annotation with generative AI requires validation [Preprint]. arXiv. https://arxiv.org/abs/2306.00176 The earliest warning, in the same year as the template.
- Rister Portinari Maranca, A., Chung, J., Hinck, M., Wolsky, A. D., Egami, N., & Stewart, B. M. (2025). Correcting the measurement errors of AI-assisted labeling in image analysis using design-based supervised learning. Sociological Methods & Research, 54(3), 984-1016. https://doi.org/10.1177/00491241251333372 The peer-reviewed applied extension of DSL, to images.
- Särndal, C.-E., Swensson, B., & Wretman, J. (2003). Model assisted survey sampling. Springer. Where GREG, calibration and design-based diagnostics live.
- Song, Y., Kluger, D. M., Parikh, H., & Gu, T. (2026). Demystifying prediction powered inference [Preprint]. arXiv. https://arxiv.org/abs/2601.20819 The practical workflow and variant taxonomy, plus the result that under missing-not-at-random gold-set selection every method is biased, classical inference included. It states Mani et al.’s single-sample threshold only; for the cross-fit version, take the formula from Mani et al. directly.
Code and data
Each pointer below is named in the paper’s own text and was confirmed live on 2026-11-16. None has been run for this essay. Each line names the claim above that it lets a reader check.
ppi_py, the Python package for prediction-powered inference and PPI++ (github.com/aangelopoulos/ppi_py; Angelopoulos, Duchi, & Zrnic, 2023, preprint). Checks the claim that the expected gain can be calculated before any gold label is bought: its power-analysis functions take the model-human correlation and the labeling costs and return the cheapest pair of sample sizes that matches a classical estimator’s standard error.dsl, the R package for design-based supervised learning (dsl.software; Egami et al., 2024, working paper). Checks the claim that uncorrected labels move a regression coefficient: fit the same regression withlm()on the predicted labels and withdsl(), and compare.power_dsl()estimates how many expert-coded documents a target precision needs.llm-debiasing-benchmark(github.com/nicaudinet/llm-debiasing-benchmark; Audinet de Pieuchon et al., 2025). Checks the roughly 200-label floor, by rerunning the experiments at gold-set sizes on either side of it.confidence-driven-inference(github.com/kristinagligoric/confidence-driven-inference; Gligorić et al., 2025). Checks the stance case, where the model-only odds ratio points the wrong way in the setting with the highest human-model agreement.- The audit’s coding data (OSF; Desai, Card, et al., 2026, preprint). Checks the absence of debiasing estimators from the audit as a count rather than an inference from the coding scheme.