Skip to main content

Table 2 model performance expressed as CV-AUC and confidence interval and the standard deviation of models

From: External validation: a simulation study to compare cross-validation versus holdout or external testing to assess the performance of clinical prediction models using PET data from DLBCL patients

Model

Discrimination: CV-AUC (± SD and 95% CI)

Calibration slope ± SD

Simulated true expected AUC

0.73

1

Internal:

1: CV-AUC

0.71 ± 0.06 (0.59–0.81)

0.93 ± 0.41

2: 20% holdout test

0.70 ± 0.07 (0.57–0.82)

0.89 ± 0.33

3: Bootstrap

0.67 ± 0.02 (0.62–0.71)

0.90 ± 0.37

External validation

4: External test

n = 100

0.69 ± 0.07 (0.56–0.83)

1.05 ± 0.45

n = 200

0.70 ± 0.05 (0.61–0.80)

1.01 ± 0.23

n = 500

0.70 ± 0.03 (0.64–0.74)

1.04 ± 0.17

5: EARL 2

0.70 ± 0.07 (0.56–0.82)

0.83 ± 0.33

6: Prevalence stage

33–33–34

0.66 ± 0.04 (0.59–0.74)

0.99 ± 0.21

100% 2

0.59 ± 0.04 (0.49–0.66)

0.41 ± 0.14

100% 3

0.63 ± 0.04 (0.56–0.71)

0.56 ± 0.12

100% 4

0.72 ± 0.03 (0.66–0.76)

1.56 ± 0.28

7: Positivity rate

0.10

0.73 ± 0.02 (0.69–0.78)

1.07 ± 0.15

0.66

0.64 ± 0.04 (0.58–0.70)

0.32 ± 0.12

0.90

0.61 ± 0.04 (0.53–0.66)

0.25 ± 0.11

8: FPR/FNR

0.12/0.12

0.78 ± 0.03 (0.73–0.83)

1.78 ± 0.22

0.45/0.45

0.53 ± 0.02 (0.49–0.57)

0.15 ± 0.10

0.25/0.25

0.66 ± 0.03 (0.61–0.71)

0.86 ± 0.10

  1. CV-AUC cross-validated area under the curve, SD standard deviation, CI confidence interval, FPR false positive rate, FNR false negative rate