Current replication
Spearman rank correlation
Not measuredNo current measurement is available in this registry.
Consumer Choice · 2011
Wezemael2013
European consumer preferences for beef with nutrition and health claims: A multi-country investigation using discrete choice experiments Paper ↗
- Spearman ρ ↑
- 0.566
- Normalized RMSE ↓
- Not measured
- Same direction
- 50%
- Same winner
- 67%
Saved run · 2026-11-04 · 6 paired effects · Inspect run ↗
Human vs synthetic effects
After scale calibration · logit coefficient
Dashed line: equal effects. × Opposite direction. Scale fitted with k = 5.0843; raw size agreement: not comparable.
What matters most
Attribute importance · share of total effect range
● Human ◆ Synthetic
LLM results
3 configured models · latest per arm · settings and answer keys differ; not a controlled ranking.
Direct prompt
Simulated respondents
Exact metrics, all 4 saved runs and settings
| Configured model | Spearman ρ ↑ | Normalized RMSE ↓ | Scaled RMSE ↓ | Answer key / date | Arm / settings |
|---|---|---|---|---|---|
| typesafe:jev-1.13.0 ↗ | 0.566 | Not measured | 1.322 | hb-def-Wezemael2013:v2 | Simulated respondents
|
| claude-opus-5-5 ↗ | 0.976 | Not measured | 0.624 | hb-def-Wezemael2013:v1 | Direct prompt
|
| jev-1.13.0 ↗ | 0.827 | Not measured | 1.123 | hb-def-Wezemael2013:v1 | Direct prompt
|
| typesafe:jev-1.13.0 ↗ | 0.566 | Not measured | 1.440 | hb-def-Wezemael2013:v1 | Simulated respondents
|
How to read these metrics
Spearman ρ measures agreement in level-effect ranking (−1 to 1; higher is better). Normalized RMSE divides unscaled RMSE by the standard deviation of human effects (unit-free; lower is better). Scaled RMSE fits a scale factor k and retains each run’s human-effect units; compare it only within the same reference and units. Missing values remain unmeasured.
Model names come from recorded configuration, unless a routing discrepancy is noted. A configured name alone does not verify the provider that answered.
When choices agree
Same winner by human preference margin · 37,440 pairs
Five equal-count groups, from closest choices to clearest preferences.
Effects on a common scale
● Human ◆ Calibrated synthetic · Human 95% intervals where saved
-3.9403.94
Nutrition and Health Claim: The steak's label says: 'Source of iron. Iron contributes to the normal cognitive function.'
Nutrition and Health Claim: The steak's label says: 'Rich in protein. Protein contributes to the growth or maintenance of muscle mass.'
Nutrition and Health Claim: The steak's label says: 'Poor in saturated fat. Consumption of saturated fat increases blood cholesterol concentration. Consumption of foods with reduced amounts of saturated fat may help to maintain normal blood cholesterol concentrations.'
Cost: The price is €15 per kg
Cost: The price is €18 per kg
Cost: The price is €21 per kg
Exact effects and uncertainty (8 levels)
| Attribute / level | Human | Human 95% half-width | Calibrated synthetic |
|---|---|---|---|
| Nutrition and Health Claim: The steak carries no nutrition or health claim (reference) | 0.000 | Not measured | 0.000 |
| Nutrition and Health Claim: The steak's label says: 'Source of iron. Iron contributes to the normal cognitive function.' | 1.510 | 0.295 | -0.854 |
| Nutrition and Health Claim: The steak's label says: 'Rich in protein. Protein contributes to the growth or maintenance of muscle mass.' | 1.120 | 0.291 | -0.318 |
| Nutrition and Health Claim: The steak's label says: 'Poor in saturated fat. Consumption of saturated fat increases blood cholesterol concentration. Consumption of foods with reduced amounts of saturated fat may help to maintain normal blood cholesterol concentrations.' | 2.580 | 0.424 | -0.566 |
| Cost: The price is €12 per kg (reference) | 0.000 | Not measured | 0.000 |
| Cost: The price is €15 per kg | -0.810 | 0.102 | -1.827 |
| Cost: The price is €18 per kg | -1.620 | 0.203 | -2.924 |
| Cost: The price is €21 per kg | -2.430 | 0.305 | -3.581 |
Interpretation and limits
Wrong direction on 3 of 6 significant effects, weak ranking (ρ = 0.57); sizes not comparable (the paper reports logit coefficients).
Use it to generate hypotheses to test with people; sizes need the paper's respondent data, and confirm the 3 reversed effects with people.
- direction
- fails
- rank
- fails
- size
- not comparable
Spearman 95% interval: -0.279 to 0.917.
Supported uses
- Nothing yet without human confirmation.
Limits
- Ranking close levels against each other; the order diverges from humans.
- Decisions that hinge on the attributes whose direction flipped (listed below).
- Effect sizes in the paper's units; only direction, rank and shape are compared.
Calibration multiplies synthetic effects by k; agreement after scaling does not establish agreement in raw sizes. Missing metrics remain unmeasured.
About the study
01 / Human finding
What the study found
Dutch beef buyers valued a saturated-fat health claim over iron or protein.
Answer-key review: AI-reviewed
02 / Experiment design
Who chose, and what varied
- Country
- Netherlands
- Human respondents
- 300
- Attributes
- 2
- Task
- discrete choice
Registry and historical replication summaries
03 / Synthetic replication
Registry replication results
Paper-era reference
2023 CLM mean Spearman ρ
0.53Historical results use their original estimator and reference. They are not a matched comparison with the current run.
Sources and measurement provenance
Missing results mean unmeasured, not zero agreement.