.pred_impaired .pred_control Class
1 0.40306887 0.5969311 control
3 0.66547506 0.3345249 control
5 0.71154002 0.2884600 control
7 0.29307658 0.7069234 control
9 0.01425278 0.9857472 control
11 0.56453589 0.4354641 impairedEvaluating a crystal ball - part 1
The future is now
The ABCD Framework
A useful way to remember prediction-model evaluation:
- A = Calibration-in-the-large
- B = Calibration slope
- C = C-statistic / Discrimination
- D = Decision curve / Clinical usefulness
Case Study: Cognitive Impairment Prediction
We evaluate a logistic regression model predicting cognitive impairment from:
- Age
- Sex (male)
- Tau protein
- Phosphorylated tau
- Amyloid beta 42
- APOE genotype
Outcome: Impaired vs Control (gold-standard neurophyschological test)
The Prediction Model
The model was trained on a separate training set:
\[ impairment_i \sim age + male + tau + p\_tau + Ab\_42 + genotype \]
where predictors are measured in cerebrospinal fluid via lumbar puncture.
Dataset Visualization
Tau vs p-tau by genotype and diagnosis
Biomarker Distributions
Distribution of biomarkers by diagnosis
Evaluating Prediction Models
What We Need
A prediction model can be evaluated from two things:
- What actually happened (observed outcome)
- What the model predicted (predicted probabilities)
Here we use predictions from a pre-trained logistic regression model.
The Validation Data
Class = observed outcome. .pred_impaired = predicted probability of impairment.
Confusion Matrix
How many were correctly classified?
Truth
Prediction impaired control
impaired 25 14
control 19 109Accuracy: How Often Correct?
How often does the model get the class right?
\[ Accuracy = \frac{TP + TN}{TP + TN + FP + FN} \]
# A tibble: 1 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 accuracy binary 0.802Accuracy = 0.802 (80.2% classified correctly)
The Problem with Accuracy
Accuracy is easy to understand but has an important limitation.
It does not tell us which class the model gets wrong.
It can be misleading when classes are unbalanced.
Example: If 95% are controls, a model that always predicts “control” has 95% accuracy, despite failing to identify any impaired participants.
Specificity: Avoiding False Positives
Among people who are not “impaired”, how many does the model correctly identify?
Specificity measures how well the model identifies the negative class (controls).
\[ Sensitivity = \frac{TP}{TP + FN} \]
# A tibble: 1 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 specificity binary 0.886Specificity = 0.886 (88.6% of controls correctly identified)
Sensitivity: Avoiding False Negatives
Among people who are “impaired”, how many does the model correctly identify?
Sensitivity measures how well the model identifies the positive class (impaired).
\[ Specificity = \frac{TN}{TN + FP} \]
# A tibble: 1 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 sensitivity binary 0.568Sensitivity = 0.568 (56.8% of impaired correctly identified)
Sensitivity vs Specificity
Specificity = 88.6%
- Good at identifying controls
- Few false positives
- Conservative about calling impairment
Sensitivity = 56.8%
- Misses many impaired cases
- More false negatives
- Not aggressive enough
Discussion Task
Discuss with your neighbor:
- Which diagnosis would you prefer a lower probability threshold? Why?
- Which diagnosis would you prefer a higher probability threshold? Why?
Consider consequences: - Cost of treatment - Side-effects - Effectiveness of treatment
Discrimination
ROC and AUC
Instead of choosing one threshold, we can look at model discrimination across all possible thresholds.
The ROC curve shows sensitivity against 1 − specificity.
ROC curve for cognitive impairment prediction
Area Under the Curve
# A tibble: 1 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 roc_auc binary 0.840AUC = area under the ROC curve
Calibration
Are the Probabilities Accurate?
Calibration asks a different question:
When the model says 70%, does the event happen about 70% of the time?
A simple calibration plot compares predicted probability with the observed proportion.
Calibration Plot
Calibration plot with breaks
A well-calibrated model follows the diagonal line.
Smooth Calibration Curve
Smooth calibration curve
ABCD Framework in Detail
A: Calibration-in-the-large
Are predictions systematically too high or too low?
B: Calibration Slope
Are predictions too extreme or not extreme enough?
C: C-statistic
How well does the model discriminate between outcomes? (AUC)
D: Decision Curve
Does using the model improve clinical decisions?
# A tibble: 2 × 5
term estimate std.error statistic p.value
<chr> <dbl> <dbl> <dbl> <dbl>
1 (Intercept) -0.402 0.214 -1.88 0.0603
2 logit_p 0.639 0.124 5.17 0.000000231Prediction vs Effect Estimation
Different Questions
Prediction
How accurately can we predict what will happen?
- Focus: accuracy of predictions
- Tools: accuracy, sensitivity, specificity, AUC, calibration
Effect Estimation
How does changing a variable relate to what happens?
- Focus: causal associations
- Tools: regression coefficients, odds ratios
Key Idea
No single metric evaluates a prediction model completely.
A useful model should: - Discriminate well - Produce sensible probabilities - Lead to better decisions
Summary: Evaluation Toolkit
| Question | Measure |
|---|---|
| Does the model classify correctly at this threshold? | Accuracy |
| Does it avoid false positives? | Specificity |
| Does it avoid false negatives? | Sensitivity |
| Can it distinguish the two outcomes? | ROC / AUC |
| Are the probabilities accurate? | Calibration |
| Does it improve clinical decisions? | Decision curve |
| What is the effect of a predictor? | Effect estimate |
The key idea: No single metric tells the whole story.
A useful model discriminates well, produces accurate probabilities, and improves decisions.
