AI-vs-human meta-study
A compact audit surface for the Ditto replication corpus: the study registry, estimator comparisons across generations, parent papers, and per-run W&B evidence.
New (2026-07-14): the backend drift canary — five replications re-run against our own 2023–24 results and the papers' human effects, one confirmed regression, root cause traced and published, misses first.
Charts
Static charts generated from the same CSVs as the tables below.
Rows with usable parent paper links, W&B run-name evidence, or Drive links.
Curated sample row from the model replication matrix.
Human-paper coefficient ranks compared with the same local CLM scorer: paper-era 2023-24 artifacts and current production.
Golden-10 production benchmark
Paired comparison against human-paper coefficients using the same local CLM estimator and alignment for both eras. 2024 local CLM mean: 0.604; Current local CLM mean: 0.617; paired mean delta: +0.014. Paper-era rows preserve their actual 2023-24 run dates.
| Rank | Study | Human N | 2024 local CLM ρ | Current local CLM ρ | Delta | Paper | 2024 run | Current run |
|---|---|---|---|---|---|---|---|---|
| 1 | Kreps2020 | 1971 | 0.893 | 0.897 | +0.005 | Paper | 2024 W&B | Current W&B |
| 2 | Leng2021 | 1883 | 0.935 | 0.919 | -0.017 | Paper | 2024 W&B | Current W&B |
| 3 | Adida2019 | 1800 | 0.905 | 0.857 | -0.048 | Paper | 2024 W&B | Current W&B |
| 4 | Donnaloja2022 | 1597 | 0.486 | 0.413 | -0.073 | Paper | 2024 W&B | Current W&B |
| 5 | Hainmueller2014 | 1407 | 0.706 | 0.465 | -0.241 | Paper | 2024 W&B | Current W&B |
| 6 | Humburg2015 | 903 | 0.353 | 0.726 | +0.374 | Paper | 2024 W&B | Current W&B |
| 7 | Spilker2018 | 800 | 0.398 | 0.427 | +0.029 | Paper | 2024 W&B | Current W&B |
| 8 | Hainmueller2015_candidate | 311 | 0.557 | 0.022 | -0.535 | Paper | 2024 W&B | Current W&B |
| 9 | Eliasson2017 | 285 | 0.446 | 0.767 | +0.321 | Paper | 2024 W&B | Current W&B |
| 10 | Schweizer2012 | 91 | 0.357 | 0.679 | +0.321 | Paper | 2024 W&B | Current W&B |
LLM replication matrix
Verified model bake-off (2026-07-14, Hainmueller2014, k=5 replicates per model, 400 synthetic respondents each): ex-ref OLS Spearman vs the paper's published AMCEs. All four models land at 0.63–0.69; claude-sonnet-4's edge is run-to-run consistency (sd 0.031 vs 0.08–0.14), not mean. Rows are curated in data/model_replications.csv; reproduce with meta_study/scripts/score_runs_example.py.
| Study | Domain | Market | Transition | Experiment run | Model | Human | LLM | Score | Status | W&B |
|---|---|---|---|---|---|---|---|---|---|---|
| Hainmueller2014 | Public Policy | market_us_public_policy_immigration | transition_voter_prefers_immigrant_candidate | bakeoff_2026_07_14 | databricks-claude-sonnet-4 | published AMCEs (41 effects) | 0.686 | 0.686 | verified | W&B |
| Hainmueller2014 | Public Policy | market_us_public_policy_immigration | transition_voter_prefers_immigrant_candidate | bakeoff_2026_07_14 | databricks-llama4-maverick | published AMCEs (41 effects) | 0.640 | 0.640 | verified | W&B |
| Hainmueller2014 | Public Policy | market_us_public_policy_immigration | transition_voter_prefers_immigrant_candidate | bakeoff_2026_07_14 | azure-openai-gpt4o-mini | published AMCEs (41 effects) | 0.631 | 0.631 | verified | W&B |
| Hainmueller2014 | Public Policy | market_us_public_policy_immigration | transition_voter_prefers_immigrant_candidate | bakeoff_2026_07_14 | azure-openai-gpt4 | published AMCEs (41 effects) | 0.630 | 0.630 | verified | W&B |
Estimator comparison (canary studies)
Same human baselines, three estimator generations. 2023 CLM = the ORIGINAL replications (inclusive convention, data/historic_2023_replications.csv). 2025 MNP = the small-n validation sweep (hb_core2, inclusive) — a different estimator generation; never conflate (see docs/verification/2026-07-21). Local OLS = client-side OLS-LPM on raw choices, ex-ref convention (Ditto#137). Server HB = the current hierarchical-Bayes AMCE tables, same convention — divergent pending rehoboam#2014/#2018. Ported MNL/CLM = rehoboam's own MLE and conditional-logit code copied into hb_validation (Ditto#159), run locally on the same labeled data, ex-ref convention. Rows curated in data/estimator_comparison.csv; the full old-era ledger (817 runs, 72 studies, pooled mean ρ 0.42) is committed as data/historic_replications.csv.
| Study | 2023 CLM ρ (original) | 2025 MNP ρ (validation) | Local OLS ρ | Ported MNL ρ | Ported CLM ρ | Server HB ρ | Notes |
|---|---|---|---|---|---|---|---|
| Hainmueller2014 | 0.5406 (n=1) | 0.413 (n=12) | 0.566 (n=2) | 0.713 | 0.754 | 0.202 (n=19) | 2026-07-14 bake-off, 4 models pooled; HB defect rehoboam#2014/#2018; 2026-07-17 ported estimators (n=3 sonnet-4 runs): MNL 0.675/0.735/0.729, CLM 0.774/0.849/0.638; golden-10 wave 07-19: 0.552/0.580 spread 0.028 PASSES but ~0.1 below the 07-14 band (0.65-0.73) - DRIFT SIGNAL, config diff pending |
| Kreps2020 | 0.6211 (n=5) | 0.872 (n=10) | 0.933 (n=2) | 0.917 | -0.008 (n=2) | 2026-07-17 canary drained: OLS 0.917/0.950 (mean 0.933) beats the 2023 original baseline (0.62) and the 2025 MNP sweep (0.87); HB 0.017/-0.033 on the same runs - strongest #2018 evidence; 2026-07-17 CLM 0.917 both runs; MNL n/a (41 unmapped level ids break the ported text-join); AUDIT 2026-07-17: scores cover 6/7 attributes - the Vaccine-efficacy attribute is unmapped (run logged continuous 50-90 vs level ids), design deviation; golden-10 07-19: k=1 failed_missing_artifacts (#2024), k=2 quarantined #2025 (0.90 on 6/7 attrs) | |
| Hess2012 | 0.5004 (n=1) | 0.597 (n=10) | 2026-07-14 submissions lost in worker-pool stall (no runs materialized); covered by the 142-study batch run | ||||
| Donnaloja2022 | 0.4381 (n=1) | 0.354 (n=7) | 2026-07-14 submissions lost in worker-pool stall (no runs materialized); covered by the 142-study batch run | ||||
| Breitenstein2019 | 0.5213 (n=1) | 0.3995 (n=10) | 0.754 (n=2) | 0.754 | 0.754 | golden-10 wave 2026-07-19: pair spread 0.000 (identical rank order both runs); PASSES stability; conformant | |
| Rapson2014 | 0.7827 (n=1) | 0.1167 (n=10) | 0.90 (n=2) | 0.90 | 0.90 | golden-10 wave 07-19: clm paper, headline CLM 1.00/0.80 mean 0.90; spread 0.20 breaches 0.15 but = one rank swap on a ~5-rank design (granularity caveat); conformant; k=2 was a crash-replacement | |
| Jung2018 | 0.1155 (n=1) | 0.7544 (n=10) | 0.655 (n=2) | 0.821 | golden-10 wave 07-19: UNSTABLE - ols spread 0.26 (0.524/0.786) breaches 0.15; choice-model estimators tighter (CLM 0.90/0.74, MNL 0.81/0.79); transcription triage queued (#121-adjacent) | ||
| Kikulwe2011 | 0.829 (n=21) | 0.3333 (n=10) | -0.143 | golden-10 wave 07-19: k=1 clm paper headline -0.14 (all estimators agree, conformant run) - real miss, transcription triage first; k=2 finished w/o artifacts (rehoboam#2024) | |||
| Mele2021 | 0.1859 (n=8) | golden-10 wave 07-19: quarantined #2025 (Age->continuous 39-51); 6-attr scores 0.80,1.00 recorded not quoted | |||||
| Leng2021 | 0.5579 (n=2) | golden-10 wave 07-19: quarantined #2025 (efficacy+adverse-rate continuous); noise 0.59/0.95 | |||||
| Spilker2018 | 0.1549 (n=5) | golden-10 wave 07-19: quarantined #2025 (countries->continuous); 0.67/0.63 on corrupted design | |||||
| Azimy2020 | 0.6119 (n=2) | golden-10 wave 07-19: quarantined #2025; -0.38/-0.38 |
Study registry
Replication scope: the 142 studies with W&B objects (historic ledger runs or linked runs), out of 154 runnable conjoint experiments in the corpus. Each row links to its paper (DOI), its W&B runs (sign-in required), and its showcase page. 2023 ρ̄ is the per-study mean Spearman of the ORIGINAL 2023 replications (CLM pipeline, inclusive convention; data/historic_2023_replications.csv). The separate 2025-02 MNP validation sweep lives in the estimator table below; Best ρ is the current ex-ref OLS score where re-scored. The same data is machine-readable at /studies.json.