1 Economic Review (Otaru University of Commerce), Vol.71, No.2 & 3, 1-16, December, 2020.
Supplement to the paper “A unified treatment of agreement coefficients and their asymptotic results: The formula of the weighted
mean of weighted ratios”
Haruhiko Ogasawara
This article supplements Ogasawara (2020), and gives (i) the partial derivatives of the sample chance-expected proportions with respect to the sample proportions of the associated multinomial distribution, evaluated at their population values in Section S1 and (ii) additional tables in Section S2.
Reference
Ogasawara, H. (2020). A unified treatment of agreement coefficients and their asymptotic results: The formula of the weighted mean of weighted ratios.
Journal of Classification (online published) https://doi.org/10.1007/s00357-020-09366-1.
S1. The partial derivatives and associated results
In Section S1, v i i
1
mis used rather than w i i
1
m( i j 1,..., ; k j 1,..., ) m . In some cases it is more natural to use w i i
1
mthan v i i
1
mas in K ˆ (G) and
ˆ (GO)
K (see Ogasawara, 2020, Appendix A2). In such cases,
1 m
1
1 m( 1,..., ; 1,..., )
i i i i j
v w i k j m can be used. That is, v i i
1
m’s in
the partial derivatives below can also be read as 1 w i i
1
m( i j 1,..., ; k j 1,..., ) m .
S1.1 The first partial derivatives and π ' e( ) ( ) v / π
(i) e(B) ( ) v e(B) ( ) v /
π 0 .
Supplement to the paper “A unified treatment of agreement coefficients and their asymptotic results: The formula of the weighted
mean of weighted ratios”
Haruhiko Ogasawara
Vo1. 71, No. 2 & 3, 35-50, December, 2020.
(ii) e(S) ( ) v
1 1
1 1
2 1 3
2 1 3
1 1
1 1
1
( ) , , 1 1
e(S)
1 1 1 1 1 1 1 1
1 2
1 1 1
( , )
(S) ( )
1
'
m j
m
m m
b m a b m a
m m
m b a
m
b m
m v k
l l l
l l j
i i i i
m m
m k k k k k
i l l l l i l l l
b l l a l l l a
a a
m
k k
i l i l
l l a
a m i j
k
v
v v
m
v
v 1
1 1 ( 1,..., ; 1,..., ),
m m
b j j
i k j m
m
where ( , ) i j (S)
bv is the k m 1 1 vector, whose elements are
1 1 1
1
( 1,..., ; 1,.., ; )
j b j m a
m
l l i l l l a
a a j
v
l k a m a j
.
2
1 2 1
1 3
1 3 2
1 1 1
1 1
2 2
e(S) ( )
,..., 1 1 1 1 1
1 1 1 1
1 1 1
,...
1 1
'
1
1
1
b m a
m m
b m a
m
m b a m
m m
b m
v k m k k m
i l l l
i i b l l a l
m
k k k
l i l l l
l l l a l
m
k k
l l i l i i
l l a l
i l l l
m v
v
v
m v
π π
1
1 3
1 3 2
1 1 , 1 1
, ,..., 1 1
1
1
a
b m
b m a
m
m k k m
b i l a l l
k m
l i l l l
l l l a l
v
1 1
1 1
( )
,,..., 1 1
1
m b a b
m m
k m
l l i l i b
l l a l
v
.
(iii) e(SO) ( ) v
1 1 1 1
1
1 1 1
2 1 3
2 1 3
1 1
( ) ( )
, , 1 1 1
, , 1 e(SO)
( ) ( )
1 1 1 1 1 1 1 1
1 2
1
/
1
m a
m m m
m
m m m
b m a b m a
m m
m
k m
k b
v l l l l l l l l l l b a l
i i i i i i
m m
m k k k m k
b b
i l l l l i l l l
b l l a l l l a
a a
k l l l
v m
v
v v
m
v
11
1
( )
1 1
( , )
(SO) ( )
1 1
' ( 1,..., ; 1,..., ),
m b a
m
b m
m
k b
i l
l a
a m m m i j
k
b j i j k j m
m
v 1
where ( , ) i j
b(SO)
v is the k m 1 1 vector, whose elements are
1 1 1
( ) 1
( 1,..., ; 1,..., ; )
j b j m a
m b
l l i l l l a
a a j
v
l k a m a j
.
Lemma S1. π ' e(SO) ( ) v / π m e(SO) ( ) v and π ' e(SO) ( ) w / π
e(SO) ( ) w
m
.
Proof.
2
1 2
1 3
1 3
e(SO) ( ) ( )
,..., 1 1 1 1 1
1
( )
1 1 1 1
2
' 1
b m am m
b m a
m
v k m k k m
i l l l b
i i b l l a
a m
k m k
l i l l l b
l l l a
a
m v
v
π π
1 1 1
1 1
( )
1
m1
m b1
a mm
k k
l l i l b i i
l l a
a m
v
2 1 3
2 1 3
1 2 3
1 1
( ) ( )
1 1 ,.., 1 1 , ,.., 1 1
1 2
( ) ( )
,.., 1 1
1
.
b m a b m a
b m m
m a b
m
m m
m k k k
b b
i l l l l i l l l
b i l l a l l l a
a a
m
k b b
l i l l l i
l l a
a m
v v
m
v
where the first term in braces on the right-hand side gives
2 1 2
2 1
1 2 1
( ) ( ) ( )
1 1 ,.., 1 1 1 ,.., 1 1
1
( ) ( )
e(SO)
,.., 1 1 1
1 1
/
b m b a m a
b m m
m a
m
m k k m m k m
b b b
i l l i l l l l l
b i l l a b l l a
a
k m m
b v
l l l l
l l b a
v v
m m
v m
and the remaining terms give the same e(SO) ( ) v ’s, yielding the required result.
The second result of Lemma S1 is similarly given by using e(SO) ( ) w and
1 m
( 1,..., ; 1,..., )
l l j
w i k j m . Q.E.D.
(iv) e(C) ( ) v
1 1
1 1
1 2 1 2 3
2 1 3
1 1
1 1
( ) ( )
1 1
e(C)
( ) ( )
1 1 1 1 1 1 1
1 2
( )
1 1 1
m a
m
m m
m a m a
m m
m m a
m
m
k a
v l l l l a l
i i i i
m m
k k k m k
a a
i l l l l i l l l
l l a l l l a
a a
m
k k
l l i l a
l l a
a m
v
v v
v
1
( )
(C) ( )
1
' ( 1,..., ; 1,..., )
j m
m i
k j j
i k j m
v 1 ,
where ( ) i (C)
jv is the k m 1 1 vector, whose elements are
1 1 1
( )
1 ( 1,..., ; 1,..., ; ).
j j j m a
m a
l l i l l l a
a a j
v
l k a m a j
Lemma S2. π ' e(C) ( ) v / π m e(C) ( ) v and π ' e(C) ( ) w / π m e(C) ( ) w .
Proof.
1 2 1
1 2 1
1 2 3 1
1 3 2
e(C) ( ) ( )
,.., 1 1 1 1 (1)
( )
1 1 1 1 (2)
' 1
1
m a m
m m
m a m
m
v k k k m
i l l l a i i
i i l l a l
m
k m k
l i l l l a i i
l l l a l
v
v
π π
1 1 1
1 1
( ) ( )
1 1 1
1 ,
m m a m
m m
m
k k
l l i l a m i i
l l a l
v
where the first term in braces on the right-hand side gives
6
1 2 1
1 2 2 2
1 2 1 1 2
1 2 1 2
( )
, ,.., 1 1 1 ,..., 1 2
(1) ( ) ( ) ( )
, ,.., 1 2 , ,.., 1 1 e(C)
,
m m a
m m m
m a m a
m m
k k k m m
a
i l l i i l
i l l l l i i a
k m k m
a a v
i l l i l l l l l
i l l a l l l a
v
v v
and the remaining terms give the same e(C) ( ) v ’s, yielding the required result.
The second result of Lemma S2 is similarly given by using e(C) ( ) w and
1 m
( 1,..., ; 1,..., )
l l j
w i k j m . Q.E.D.
(v) e(G) ( ) v
Note that e(G) ( ) w and e(G) ( ) v are defined only for unweighted pair-wise (dis)agreement ( m 2 or ' 2 m ; w ij ij , v ij 1 ij ( , i j 1,..., ) ) k . The following reduced expressions are used for e(G) ( ) w and e(G) ( ) v for simplicity
e(G) ( )
, 1 , 1 1
(1 ) (1 ) (1 )
1 1 1
k k k
w i i i i a a
ij ij
i j w i j a
k k k
and e(G) ( ) e(G) ( )
1
(1 )
1 1
1
v w k a a
a k
. Then, we have
e(G) ( ) 1
e(G) ( )
(1 ) / ( 1) 1 2 1 2 2(1 )
( 1) ( 1) ,
2(1 )
( , 1,.., ).
( 1)
w k
a a i j i j
a
ij ij
v i j
ij
k
m k m k
i j k
m k
Since k a 1 a 1 , the above result can be simplified as
e(G) ( ) 2( )
( 1)
w i j
ij m k
and
e(G) ( ) 2( )
( , 1,..., ) ( 1)
v i j
ij
i j k
m k
. However,
for the associated asymptotic variance, the earlier result can also be used since
the added terms will give no contribution to the variance due to the singular
property of cov(p). The equal results with different partial derivatives are
associated with the property that ij ( , i j 1,..., ) k are not independent
mathematical variables with their sum being fixed.
(vi) e(GO) ( ) v
As for e(G) ( ) w and e(G) ( ) v , e(GO) ( ) w and e(GO) ( ) v are defined only for unweighted pair-wise (dis)agreement ( m 2 or ' 2 m ; w
ij
ij, v
ij 1
ij;
, 1,..., )
i j k . Then, we have the following results.
( ) 2 e(GO)
1 1
( ) 2 e(GO)
1 1
1 1 ,
1 1 ,
1 1
k k
w a a a aa
a a
k k
v a a a aa
a a
k k
k k
( ) 2
e(GO) 1
(1) (2) (1) (2)
( ) / ( 1)
1 2 1 2 2(1 )
( 1) ( 1) ,
w k
a a
a
ij ij
i j i j
k
m k m k
( ) (1) (2)
e(GO) 2(1 )
( , 1,.., ).
( 1)
v i j
ij
i j k
m k
As before, the following simplified result can also be used
( ) (1) (2)
e(GO) 2( )
( 1)
w i j
ij m k
and
( ) (1) (2)
e(GO) 2( )
( , 1,..., ) ( 1)
v i j
ij
i j k
m k
.
S1.2 ( e( ) ( ) v / π ') diag( ) ( π e( ) ( ) v / ) π
(i) e(B) ( ) v
( ) ( )
e(B) diag( ) e(B) 0 '
v v
π
π π
(ii) e(S) ( ) v , e(SO) ( ) v , e(C) ( ) v , e(G) ( ) v and e(GO) ( ) v
8
1
1 1 1
1 1
1
( ) ( ) ( ) ( )
e( ) e( ) e( ) e( )
,..., 1
( ) 2
,..., 1 e( )
diag( ) '
( / ) .
m
m m m
m m
m
v v k v v
i i i i i i i i
k v
i i i i
i i
π π π
S1.3 The second partial derivatives (i) e(B) ( ) v
2 ( ) 2
e(B) v / ( )
π 0 .
(ii) e(S) ( ) v
(1) ( 2)
1 2
2 (1) (1) (2) ( 2)
1 1
2 ( )
e(S) ( , , 1, 2)
(S) ( )
1 1 2 1 1 1 2 1 1 2 (1) (2)
' ( , 1,..., ; 1,..., ),
b b
m
m m
v m m m m
i i j j
b b j j k
i i i i j j
j j
i i k j m
v 1
where ( i
b(1) ( 2)( )
1, i
b2, 1, 2) j j
S
v is the k m 2 1 vector, whose elements when j 1 j 2 are
(1) ( 2)
1 1 1 1 1 1 2 1 2 2 1
2 1
1, 2
1
a( 1,..., ; 1,.., ; 1, 2)
j b j j b j m
m
l a
l l i l l i l l
a a j j
v l k a m a j j
m
,
where 1
1, 2
a1
m a l a j j
when m = 2.
(iii) e(SO) ( ) v
(1) ( 2)
2
(1) (1) (2) (2)
1 1
2 ( )
e(SO) ( , , 1, 2)
(SO) ( )
1 1 1 2 1 1 2 (1) (2)
' ( , 1,..., ; 1,..., ),
b b
m
m m
v m m m
i i j j
b j j k
i i i i j j
j j
i i k j m
v 1
where ( i
b(1) ( 2)(SO) , i
b, 1, 2) j j
v is the k m 2 1 vector, whose elements when j 1 j 2
are
(1) ( 2)
1 1 1 1 1 2 1 2 1
( ) 1 1, 2
1
a( 1,..., ; 1,.., ; 1, 2)
j b j j b j m
m b
l a
l l i l l i l l
a a j j
v l k a m a j j
m
,
where 1 ( )
1, 2
a1
m b
a l a j j
when m = 2.
(iv) e(C) ( ) v
(1) ( 2)
1 2
2 (1) (1) (2) ( 2)
1 1
2 ( )
( , )
e(C)
(C) ( )
1 1 2 1 1 2 (1) (2)
' ( , 1,..., ; 1,..., ),
j j
m
m m
v m m
i i j j k
i i i i j j
j j
i i k j m
v 1
where (
(1) (2)(C)
j1,
j2) i i
v is the k m 2 1 vector, whose elements when j 1 j 2 are
(1) ( 2)
1 1 1 1 1 1 2 1 2 2 1
( ) 1 1, 2
1
a( 1,..., ; 1,.., ; 1, 2)
j j j j j j m
m a
l a
l l i l l i l l
a a j j
v l k a m a j j
m
,
where 1 ( )
1, 2
a1
m a
a l a j j
when m = 2.
(v) e(G) ( ) v , e(GO) ( ) v
1 1
1 1 2 2 2 2
1 2 1 2 1 2 1 2
1 2 1 2 1 2 1 2
1 1 2 2
2 ( ) e(G)
2 2 ( )
e(G)
2
2(1 ) /{ ( 1)}
2( )
( 1) ,
2( )
( 1) ,
w i j
i j i j i j
i i i j j i j j
v i i i j j i j j
i j i j
m k
m k m k
10
1 1 1 2 1 2
1 1 2 2 2 2
1 2 1 2
1 1 2 2
(1) (2)
2 ( ) e(GO)
2 ( ) e(GO)
1 2 1 2
2(1 ) / { ( 1)} 2( )
( 1) ,
2( )
( , , , 1,..., ).
( 1)
w i j i i j j
i j i j i j
v i i j j
i j i j
m k
m k i i j j k m k
S1.4 The third partial derivatives (i) e(B) ( ) v , e(G) ( ) v and e(GO) ( ) v
3 ( ) 3
e( ) v / ( )
π 0 .
(ii) e(S) ( ) v ( m 3)
(1) ( 2) (3)
1 2 3
3
(1) (1) ( 2) (2) (3) (3)
1 1 1
3 ( )
e(S) ( , , , 1, 2, 3)
(S) ( )
1 1 2 1 3 1 1, 2, 3 1 1 2 3 1 (1) (2) (3)
' ( , , 1,..., ; 1,..., ),
b b b
m
m m m
v m m m m
i i i j j j
b b b j j j k
i i i i i i j j j j
j j j
i i i k j m
v 1
where
(1) (2) (3)1 2 3
( , , , 1, 2, 3)
b(S)
b bi i i j j j
v is the k m 3 1 vector, whose elements when
1 2 3
j j j are
(1) (2) (3)
1 1 1 1 1 1 2 1 2 2 1 3 1 3 3 1
3 1
1, 2, 3
1
( 1,..., ; 1,.., ; 1, 2, 3),
j b j j b j j b j m a
m
l l i l l i l l i l l l
a a j j j
a
m v
l k a m a j j j
where 1
1, 2, 3
a1
m a l a j j j
when m = 3. When m = 2, 3 ( ) e(S) v / ( ) π 3 0 .
(iii) e(SO) ( ) v ( m 3)
(1) ( 2) (3)
3
(1) (1) ( 2) (2) (3) (3)
1 1 1
3 ( )
e(SO) ( , , , 1, 2, 3)
(SO) ( )
1 1, 2, 3 1 1 2 3 1 (1) (2) (3)
' ( , , 1,..., ; 1,..., ),
b b b
m
m m m
v m m
i i i j j j
b j j j k
i i i i i i j j j j
j j j
i i i k j m
v 1
where (
(1) (2) (3), , , 1, 2, 3)
b
(S)
b bi i i j j j
v is the k m 3 1 vector, whose elements when
1 2 3
j j j are
(1) ( 2) (3)
1 1 1 1 1 2 1 2 1 3 1 3 1
( ) 1 1, 2, 3
1
( 1,..., ; 1,.., ; 1, 2, 3),
j b j j b j j b j m a
m b
l l i l l i l l i l l l
a a j j j
a
m v
l k a m a j j j
where 1 ( )
1, 2, 3
a
1
m b
a l a j j j
when m = 3. When m = 2, 3 ( ) e(SO) v / ( ) π 3 0 .
(iv) e(C) ( ) v ( m 3)
(1) ( 2) (3)
1 2 3
3
(1) (1) (2) ( 2) (3) (3)
1 1 1
3 ( )
( , , )
e(C) (C) ( )
1, 2, 3 1 1 2 3 1 (1) (2) (3)
' ( , , 1,..., ; 1,..., ),
j j j
m
m m m
v m
i i i j j j k
i i i i i i j j j j
j j j
i i i k j m
v 1
where
(1) ( 2) (3)1 2 3
( , , )
(C)
j j ji i i
v is the k m 3 1 vector, whose elements when
1 2 3
j j j are
(1) ( 2) (3)
1 1 1 1 1 1 2 1 2 2 1 3 1 3 3 1
( ) 1 1, 2, 3
( 1,..., ; 1,.., ; 1, 2, 3).
j j j j j j j j j m a
m a
l l i l l i l l i l l l
a a j j j
a
v
l k a m a j j j
where 1 ( )
1, 2, 3
a
1
m a
a l a j j j
when m = 3. When m = 2, 3 ( ) e(C) v / ( ) π 3 0 .
S2. Additional tables
Table A1. Counts and degrees of seriousness of 4-wise disagreement for 2
4profiles using two rating categories given by dichotomization (different from that in Table 1; see below) for scores 1 to 5 evaluated by 4 raters in classification of the intensity of nuptial coloration for 29 fishes (reconstructed from Gwet (2014, Table A.4))
Profiles and their associated values Marginal proportions Profile Count v
4Av
4BRater Category 1 Category 2 1111 5 0 0.0
1112 1 1 0.5 1121 0 1 0.5 1122 2 1 1.0 1211 1 1 0.5 1212 0 1 1.0 1221 0 1 1.0 1222 1 1 0.5 2111 1 1 0.5 2112 0 1 1.0 2121 2 1 1.0 2122 1 1 0.5 2211 2 1 1.0 2212 1 1 0.5 2221 1 1 0.5 2222 11 0 0.0 Total 29
1 .345 .655 2 .414 .586 3 .379 .621 4 .414 .586 Average .388 .612 Dichotomization for scores:
Category 1 = scores 1 and 2 Category 2 = scores 3, 4 and 5
Note. v
4Aand v
4Bstand for the degrees of seriousness of 4-wise disagreement in each profile based on Method A with degrees 0 and 1, and Method B with degrees 0.0, 0.5 and 1.0, respectively. In the profiles 1 and 2 denote Categories 1 and 2, respectively, given by Raters 1, 2, 3 and 4.
13
Table A2. Asymptotic and simulated standard errors of sample coefficients of 4-wise agreement for 4 raters using two rating categories in Table A1 (the number of replications in simulations = 10,000)
4-wise agreement with k = 2 n = 29 n = 100 n = 200 K
( )
e( )( )vn ASE ASE SD/ASE SD
tSD/ASE SD
tSD/ASE SD
t( )ov
= .448 with v
4Afor 4-wise disagreement
K
(B).488 .875 .568 .106 .996 1.057 1.000 1.015 .996 1.003 K
(S).464 .837 .580 .108 1.008 1.077 1.009 1.024 1.003 1.011 K
(SO).463 .834 .586 .109 1.040 1.053 1.017 1.019 1.008 1.008 K
(C).465 .838 .578 .107 1.000 1.081 1.007 1.026 1.002 1.011
( ) ov
= .328 with v
4Bfor 4-wise disagreement
K
(B).476 .625 .640 .119 .996 1.077 .996 1.016 .987 .995
K
(S).443 .588 .670 .124 1.018 1.102 1.011 1.026 .999 1.004
K
(SO).440 .585 .681 .126 1.054 1.069 1.021 1.017 1.003 1.000
K
(C).443 .588 .667 .124 1.008 1.109 1.008 1.028 .997 1.005
Note. K
(B)= Bennett et al.-type K, K
(S)= Scott-type K, K
(SO)= modified Scott-type K, K
(C)=
Cohen-type K, n = sample size, ASE = asymptotic standard error, SD
t= the standard deviation of
studentized estimates of K
( )in a simulation, K
( ) 1
( )1 (
o( )v/
e( )( )v) .
14
Table A3. Asymptotic and simulated standard errors of sample coefficients of 3-wise agreement for 4 raters using two rating categories in Table A1 (the number of replications in simulations = 10,000)
3-wise agreement with k = 2 n = 29 n = 100 n = 200
( ) ov
= .388 K
( )
e( )( )vn ASE ASE SD/ASE SD
tSD/ASE SD
tSD/ASE SD
tThe formula of the ratio of means with v
3Afor 3-wise disagreement
K
(B).482759 .750 .58451 .109 .997 1.060 .997 1.013 .991 .998 K
(S).455191 .712 .60287 .112 1.015 1.083 1.010 1.024 1.001 1.007 K
(SO).453518 .710 .60994 .113 1.046 1.056 1.018 1.017 1.005 1.004 K
(C).456023 .713 .59963 .111 1.002 1.091 1.007 1.026 .999 1.008 The formula of the mean of ratios with v
3Afor 3-wise disagreement
K
(B).482759 * .58451 .109 .997 1.060 .997 1.013 .991 .998
K
(S).455186 * .60290 .112 1.016 1.084 1.010 1.024 1.001 1.007
K
(SO).453489 * .61008 .113 1.048 1.056 1.019 1.017 1.005 1.004
K
(C).456031 * .59962 .111 1.003 1.092 1.007 1.026 .999 1.008
Note. K
(B)= Bennett et al.-type K, K
(S)= Scott-type K, K
(SO)= modified Scott-type K, K
(C)=
Cohen-type K, n = sample size, ASE = asymptotic standard error, SD
t= the standard deviation of
studentized estimates of K
( )in a simulation, K
( ) 1
( )1 (
( )ov/
e( )( )v) . v
3A= 0 when a 3-wise
profile is 111 or 222 otherwise v
3A= 1 (compare v
3Ain Table A3 with v
4Ain Table A1). The
asterisks indicate that the corresponding common values of
e( )( )vare used.
15
Table A4. Asymptotic and simulated standard errors of sample coefficients of pair-wise agreement for 4 raters using two rating categories in Table A1 (the number of replications in simulations = 10,000)
Pair-wise agreement with k = 2 n = 29 n = 100 n = 200
( )ov
= .259 K
( )
e( )( )vn ASE ASE SD/ASE SD
tSD/ASE SD
tSD/ASE SD
tThe formula of the ratio of means with v
2Afor pair-wise disagreement
K
(B).48276 .500 .58451 .109 .997 1.060 .997 1.013 .991 .998 K
(S).45477 .474 .60457 .112 1.022 1.077 1.012 1.022 1.002 1.006 K
(SO).45352 .473 .60994 .113 1.046 1.056 1.018 1.017 1.005 1.004 K
(C).45602 .475 .59963 .111 1.002 1.091 1.007 1.026 .999 1.008 K
(G).50801 .526 .60954 .113 .976 1.081 .987 1.015 .985 .998 K
(GO).50903 .527 .60560 .112 .960 1.086 .982 1.017 .983 .999 The formula of the mean of ratios with v
2Afor pair-wise disagreement
K
(B).48276 * .58451 .109 .997 1.060 .997 1.013 .991 .998
K
(S).45474 * .60476 .112 1.022 1.081 1.012 1.023 1.002 1.006
K
(SO).45343 * .61040 .113 1.050 1.055 1.019 1.017 1.005 1.004
K
(C).45604 * .59966 .111 1.002 1.095 1.006 1.026 .999 1.009
K
(G).50783 * .60983 .113 .977 1.078 .987 1.014 .985 .998
K
(GO).50887 * .60579 .112 .960 1.084 .982 1.016 .983 .999
Note. K
(B)= Bennett et al.-type K, K
(S)= Scott-type K, K
(SO)= modified Scott-type K, K
(C)=
Cohen-type K, K
(G)= Gwet-type K, K
(GO)= modified Gwet-type K, n = sample size, ASE =
asymptotic standard error, SD
t= the standard deviation of studentized estimates of K
( )in a
simulation, K
( ) 1
( )1 (
( )ov/
e( )( )v) . v
2A= 0 when a pair-wise profile is 11 or 22 otherwise
v
2A= 1 (compare v
2Ain Table A4 with v
3Ain Table A3 and v
4Ain Table A1). The asterisks
indicate that the corresponding common values of
( )e( )vare used.
16
Table A5. Asymptotic and simulated correlations of p
( )ov, p
e( )( )vand K ˆ
( )’s for 3-wise agreement of 4 raters using the formula of the ratio of means and two rating categories in Table 1 (not Table A1; n
= 29 and the number of replications in simulations = 10,000) 3-wise agreement
Asymptotic correlations Simulated correlations
( ) ov
p 1 1
( )e(S)v
p .0894 1 .0947 1
( )e(SO)v
p .0136 .9957 1 .0079 .9943 1
( )e(C)v
p .1265 .9990 .9905 1 .1371 .9986 .9873 1 ˆ
(B)K 1 1
ˆ
(S)K .9072 1 .8442 1
ˆ
(SO)K .9104 .9997 1 .8468 .9987 1
ˆ
(C)K . 9054 .9999 .9994 1 .8421 .9997 .9972 1
Note. K ˆ
(B)= Bennett et al.-type ˆ K , K ˆ
(S)= Scott-type ˆ K , K ˆ
(SO)= modified Scott-type ˆ K , K ˆ
(C)= Cohen-type ˆ K , n = sample size. v
3A= 0 when a 3-wise profile is 111 or 222 otherwise v
3A= 1.
( ) ' 1
e(B)v