Current replication
Spearman rank correlation
Not measuredNo current measurement is available in this registry.
Consumer Choice · 2009
Gracia2014
Are Local and Organic Claims Complements or Substitutes? A Consumer Preferences Study for Eggs Paper ↗
- Spearman ρ ↑
- 0.532
- Normalized RMSE ↓
- Not measured
- Same direction
- 67%
- Same winner
- 62%
Saved run · 2026-11-03 · 9 paired effects · Inspect run ↗
Human vs synthetic effects
After scale calibration · logit coefficient
Dashed line: equal effects. × Opposite direction. Scale fitted with k = 7.0137; raw size agreement: not comparable.
What matters most
Attribute importance · share of total effect range
● Human ◆ Synthetic
LLM results
3 configured models · latest per arm · settings and answer keys differ; not a controlled ranking.
Direct prompt
Simulated respondents
Exact metrics, all 3 saved runs and settings
| Configured model | Spearman ρ ↑ | Normalized RMSE ↓ | Scaled RMSE ↓ | Answer key / date | Arm / settings |
|---|---|---|---|---|---|
| claude-opus-5-5 ↗ | 0.993 | Not measured | 0.637 | hb-def-Gracia2014:v1 | Direct prompt
|
| jev-1.13.0 ↗ | 0.975 | Not measured | 0.799 | hb-def-Gracia2014:v1 | Direct prompt
|
| typesafe:jev-1.13.0 ↗ | 0.532 | Not measured | 1.109 | hb-def-Gracia2014:v1 | Simulated respondents
|
How to read these metrics
Spearman ρ measures agreement in level-effect ranking (−1 to 1; higher is better). Normalized RMSE divides unscaled RMSE by the standard deviation of human effects (unit-free; lower is better). Scaled RMSE fits a scale factor k and retains each run’s human-effect units; compare it only within the same reference and units. Missing values remain unmeasured.
Model names come from recorded configuration, unless a routing discrepancy is noted. A configured name alone does not verify the provider that answered.
When choices agree
Same winner by human preference margin · 39,345 pairs
Five equal-count groups, from closest choices to clearest preferences.
Effects on a common scale
● Human ◆ Calibrated synthetic · Human 95% intervals where saved
-3.2403.24
Method of Production: The method used for production is barn
Method of Production: The method used for production is free-range
Method of Production: The method used for production is organic
Origin of Production: The eggs were produced locally, in the province of Zaragoza
Origin of Production: The eggs were produced in the region of Aragon
Origin of Production: The eggs were produced in Spain
Price (Euros per half dozen): The price is €1.25 per half dozen
Price (Euros per half dozen): The price is €2.00 per half dozen
Price (Euros per half dozen): The price is €2.50 per half dozen
Exact effects and uncertainty (12 levels)
| Attribute / level | Human | Human 95% half-width | Calibrated synthetic |
|---|---|---|---|
| Method of Production: The method used for production is caged (reference) | 0.000 | Not measured | 0.000 |
| Method of Production: The method used for production is barn | 0.286 | 0.150 | 1.227 |
| Method of Production: The method used for production is free-range | 0.596 | Not measured | 0.836 |
| Method of Production: The method used for production is organic | 0.492 | Not measured | -0.275 |
| Origin of Production: The eggs were produced in Europe (reference) | 0.000 | Not measured | 0.000 |
| Origin of Production: The eggs were produced locally, in the province of Zaragoza | 1.109 | Not measured | 2.735 |
| Origin of Production: The eggs were produced in the region of Aragon | 0.762 | Not measured | 1.494 |
| Origin of Production: The eggs were produced in Spain | 0.641 | Not measured | 0.786 |
| Price (Euros per half dozen): The price is €0.75 per half dozen (reference) | 0.000 | Not measured | 0.000 |
| Price (Euros per half dozen): The price is €1.25 per half dozen | -0.799 | 0.041 | 1.368 |
| Price (Euros per half dozen): The price is €2.00 per half dozen | -1.999 | 0.103 | 0.714 |
| Price (Euros per half dozen): The price is €2.50 per half dozen | -2.798 | 0.144 | -0.047 |
Interpretation and limits
Wrong direction on 3 of 9 significant effects, weak ranking (ρ = 0.53); sizes not comparable (the paper reports logit coefficients).
Use it to generate hypotheses to test with people; sizes need the paper's respondent data, and confirm the 3 reversed effects with people.
- direction
- fails
- rank
- fails
- size
- not comparable
Spearman 95% interval: -0.099 to 0.858.
Supported uses
- Nothing yet without human confirmation.
Limits
- Ranking close levels against each other; the order diverges from humans.
- Decisions that hinge on the attributes whose direction flipped (listed below).
- Effect sizes in the paper's units; only direction, rank and shape are compared.
Calibration multiplies synthetic effects by k; agreement after scaling does not establish agreement in raw sizes. Missing metrics remain unmeasured.
About the study
01 / Human finding
What the study found
Spanish shoppers paid more for local eggs, and for free-range over caged.
Answer-key review: AI-reviewed
02 / Experiment design
Who chose, and what varied
- Country
- Spain
- Human respondents
- 803
- Attributes
- 3
- Task
- discrete choice
Registry and historical replication summaries
03 / Synthetic replication
Registry replication results
Paper-era reference
2023 CLM mean Spearman ρ
0.34Historical results use their original estimator and reference. They are not a matched comparison with the current run.
Sources and measurement provenance
Missing results mean unmeasured, not zero agreement.