Table 3 Pairwise agreement within experts and between experts and RENEX

Log-linear model coefficients
Between experts RENEX and expert
E 1 E 2 E 1 E 3 E 2 E 3 R E 1 R E 2 R E 3
[ δ m1 , (SE)] [ δ m 2 , (SE)] [ δ m 3 , (SE)] [ θ m 1 , (SE)] [ θ m 2 , (SE)] [ θ m 3 , (SE)]
Non-obstructed 1.58* (0.42) 0.36 (0.40) 0.91 (0.46) 0.26 (0.40) 1.08* (0.41) 0.84* (0.32)
Equivocal -0.12 (0.40) 0.89 (0.37) -0.58 (0.44) -0.06 (0.37) -0.28 (0.39) 0.47 (0.33)
Obstructed 0.78 (0.46) -0.07 (0.43) 1.47* (0.45) 0.48 (0.43) 1.08 (0.44) 0.41 (0.38)
p values for tests of hypothesis
Among experts RENEX and expert Experts vs. RENEX
H 0: δ m 1= δ m 2= δ m 3 a H 0: θ m 1= θ m 2= θ m 3 b H 0: δ m 1+ δ m 2+ δ m 3= θ m 1+ θ m 2+ θ m 3 c
Non-obstructed 0.15 0.47 0.41
Equivocal 0.08 0.34 0.95
Obstructed 0.11 0.58 0.81
1. * Significant positive pairwise agreement at α = 0.01. aHypothesis that the overall pairwise agreement among experts is the same. The δ s reflect the strength of the beyond-chance agreement between two raters within each specific category. bHypothesis that the overall pairwise agreement between RENEX and each expert is the same. The θ s reflect the strength of the beyond-chance agreement between RENEX and an expert within each specific category. cHypothesis that the overall pairwise agreement among experts is the same as the overall pairwise agreement between each expert and RENEX.