Model | Discrimination: CV-AUC (± SD and 95% CI) | Calibration slope ± SD |
---|---|---|
Simulated true expected AUC | 0.73 | 1 |
Internal: | ||
1: CV-AUC | 0.71 ± 0.06 (0.59–0.81) | 0.93 ± 0.41 |
2: 20% holdout test | 0.70 ± 0.07 (0.57–0.82) | 0.89 ± 0.33 |
3: Bootstrap | 0.67 ± 0.02 (0.62–0.71) | 0.90 ± 0.37 |
External validation | ||
4: External test | ||
n = 100 | 0.69 ± 0.07 (0.56–0.83) | 1.05 ± 0.45 |
n = 200 | 0.70 ± 0.05 (0.61–0.80) | 1.01 ± 0.23 |
n = 500 | 0.70 ± 0.03 (0.64–0.74) | 1.04 ± 0.17 |
5: EARL 2 | 0.70 ± 0.07 (0.56–0.82) | 0.83 ± 0.33 |
6: Prevalence stage | ||
33–33–34 | 0.66 ± 0.04 (0.59–0.74) | 0.99 ± 0.21 |
100% 2 | 0.59 ± 0.04 (0.49–0.66) | 0.41 ± 0.14 |
100% 3 | 0.63 ± 0.04 (0.56–0.71) | 0.56 ± 0.12 |
100% 4 | 0.72 ± 0.03 (0.66–0.76) | 1.56 ± 0.28 |
7: Positivity rate | ||
0.10 | 0.73 ± 0.02 (0.69–0.78) | 1.07 ± 0.15 |
0.66 | 0.64 ± 0.04 (0.58–0.70) | 0.32 ± 0.12 |
0.90 | 0.61 ± 0.04 (0.53–0.66) | 0.25 ± 0.11 |
8: FPR/FNR | ||
0.12/0.12 | 0.78 ± 0.03 (0.73–0.83) | 1.78 ± 0.22 |
0.45/0.45 | 0.53 ± 0.02 (0.49–0.57) | 0.15 ± 0.10 |
0.25/0.25 | 0.66 ± 0.03 (0.61–0.71) | 0.86 ± 0.10 |