Do not trust a simulated customer until it predicts the effect of a real decision. Subconscious runs calibrated experiments on synthetic customers and grades them the hard way: did the simulation predict what real people did when the price changed, misses included.
The for-loop test
Nothing in that loop measures what happens when the price changes, and nothing in it warns you when the answer is wrong. Every layer of the pipeline exists because the naive version fails in a documented way. A loop always returns an answer. An instrument refuses to.
Every pipeline claim traces to the hosted paper (PDF) and its cited failure modes: prompt confounding, demographic flattening, social-desirability bias, prompt sensitivity.
Evidence funnel
An intervention is a deliberate change: a new price, a different message. One paper in 95 grounds the causal claim it is built on. Our review covers the field's arXiv corpus, each paper coded from its title and abstract against a fixed rubric by a language model, with verdicts self-reported by each paper.
Scope: the corpus is arXiv-only, so off-arXiv work sits outside this funnel, including the strongest result in the field, Hewitt, Ashokkumar, Ghezae and Willer (Nature, in press), which is credited on the ladder below.
Explore the corpus - all 381 papers
Definition
Causal Fidelity is the standard for whether a synthetic population reproduces human treatment effects on experiments it has not seen.
In plain terms: change the price in the simulation and in the real world; a faithful simulation moves the same way, by about the same amount, and says how sure it is. The technical measurement is interventional fidelity: agreement between synthetic and human treatment effects, reported with confidence intervals.
Fidelity ladder
Did the synthetic effect point the right way?
Most work stops here.Did it order effects the way people did?
Subconscious: 0.55 rank correlation across all ~300 replications, 0.73 on the 43 passing design filters.Did the synthetic estimate land inside the human interval?
This is the credibility line for decisions.Did it predict the change the business depends on?
This is the flight-simulator claim.Roughly 300 conjoint replications were run against published human studies; order-invariance design filters pass 43 of them. Rank correlation between synthetic and human effect orderings is 0.55 across all replications and 0.73 on the filtered set. Working-paper claim; the per-study table is not yet public (release in preparation).
The rank-rung numbers above are working-paper claims (the working paper, PDF). Strongest prior work on the effect-size rung: Hewitt, Ashokkumar, Ghezae and Willer (Nature, in press) predict pooled treatment effects at r = 0.85 across attitude experiments. The gap is narrowing: Persson, Schultzberg and Ankargren (Spotify, 2026) validate LLM treatment effects experiment by experiment on content-choice A/B tests — raw models recover roughly 39% of the human effect before calibration — and Brand, Israeli and Ngwe (2026) find GPT conjoint willingness-to-pay estimates often inaccurate or wrong-signed. The still-open wedge: calibrated per-experiment CI overlap on conjoint pricing and choice decisions, run blind before the human result exists.
Calibration example
The misses show where the model should not be trusted yet. A useful validation output shows recovery and failure modes at the same time.
Paper claim: the immigrant conjoint replication reached 0.64 rank correlation and 73% sign agreement. Sign flips concentrate in origin and legal-history effects.
Illustrative sketch, drawn to show how to read a calibration plot; not real data. The actual replication plot is below.
The replication compares synthetic and human average marginal component effects (AMCEs) from the immigrant conjoint, each with confidence intervals. Headline grades: 0.64 rank correlation, 73% sign agreement. Full specification is in the working paper (PDF); the per-study replication data is not yet public (release in preparation).
Where the method fails
Every vendor in this category reports its wins. Here are the three places our own record says the method cannot yet be trusted, stated before anyone asks.
The replication reached 0.64 rank correlation and 73% sign agreement, which means roughly one effect in four flipped sign. The flips are not random: they concentrate in origin and legal-history effects, exactly where a language model's training prior is strongest.
Source: Calibration plot (PNG)Across all ~300 conjoint replications the rank correlation is 0.55. Only 43 pass the order-invariance filters, and the 0.73 figure applies to those alone. Unfiltered, the pipeline is not decision-grade, and the filters reject most designs.
Source: Working paper (PDF)No Subconscious result has yet shown synthetic estimates landing inside human confidence intervals experiment by experiment. That rung is the credibility line for decisions, and today it is a claim about the roadmap, not the record.
Source: Fidelity ladder, rung 3Validation workflow
The workflow turns a paper standard into an operating process: frame the decision, compare effects, declare the trust boundary.
Price, message, product, policy, segment, and outcome.
Published experiment, held-out study, or customer-confirmed baseline.
Same intervention structure, synthetic population, controlled variation.
Direction, rank, effect size, and interval overlap.
Where the model can support the decision, where it failed, and what needs human confirmation.
Proof inventory
| Claim | Status | Source rule |
|---|---|---|
| 381 papers | public data | The corpus itself: every paper with title, authors, and arXiv ID (CSV). |
| 104 human comparisons | working-paper claim | Rigor-funnel figure restates the paper's coding; the per-study evidence table is not yet public (release in preparation). |
| 24 interventions | working-paper claim | Interventional share by year, from the paper's coding; per-study table not yet public. |
| 4 worked well | working-paper claim | Verdicts within each validation type: 4 of 24 interventional tests; per-study table not yet public. |
| 0.55 / 0.73 rank correlation | working-paper claim | 0.55 across all ~300 conjoint replications, 0.73 on the 43 passing design filters; per-study replication data not yet public. |
| r = 0.85 pooled effect prediction | external evidence | Hewitt, Ashokkumar, Ghezae and Willer, Nature (in press): strongest prior result on the effect-size rung; cited, not ours. |
The rows marked "working-paper claim" trace to a Subconscious working paper that has not been peer reviewed. This page makes no customer, revenue, or traction claims; those live in the private diligence packet, available on request.
Technical path
Paper 1 defines the interventional-fidelity standard. The Causal Atlas defines the replication program and is explicitly a vision paper, not an empirical results report.
Both public bundles mirror Ditto revision ed69380. Ditto remains the canonical editable source.
Next action
A flight simulator for business is credible only when it is validated against the physics of human response.
Page record
Last verified 2026-08-15. Research bundles mirror Ditto ed69380. This page is maintained under version control; every change below shipped as a reviewed pull request.
Published both canonical Ditto working papers from revision ed69380, with source and build provenance.
Ran a five-study backend drift canary and published the results, regression included, at /study/backend-canary-2026-07-14.
Added this page record, the misses section, and per-section methods notes.
Committed the proof-infrastructure plan: prediction ledger, falsification suite, corpus explorer.
Rebuilt the fold around the differentiation argument; linked the full study detail.
Deleted unevidenced claims; hosted the working paper, 381-paper corpus, and all figures.
