Assessing the reliability of ordered categorical scales using kappa-type statistics

Roberts, Chris; McNamee, Roseanne
October 2005
Statistical Methods in Medical Research;Oct2005, Vol. 14 Issue 5, p493
Academic Journal
journal article
Methods for the analysis of reliability of ordered categorical scales are discussed, focussing on the limitation of the single summary-weighted kappa coefficients. A symmetric matrix of kappa-type coefficients is suggested as an alternative. The method is proposed as being suitable for ordinal scale where there is no underlying continuum. Their application is illustrated using two data sets from reliability studies. If, instead, distances between categories can be specified, a weighted mean of the matrix terms can be used as a summary measure. This is equal to a weighted kappa coefficient with squared weights, provided distances between adjacent categories are equal. When a study design corresponds to a one-way random effects model, estimates of precision of kappa-type coefficient, including the coefficients described here, can be obtained using the delta-method, bootstrap resampling by subjects or jack-knifing by subjects. In the case of interobserver reliability studies, where there may be systematic differences between observers, the investigator may wish to generalise to a population of observers and subjects. In this case, jack-knifing by observer and subject is suggested. Empirical comparisons are made between standard error estimates based on the delta-method, on jack-knifing by subjects and a two-way jack-knife by subjects and observers. The results suggest that standard errors based on the delta-method or jack-knifing by subject alone may be overly precise.


Related Articles

  • Balancing Act. Entin, Carli; Williams, Virginia // Scholastic Math;02/12/2001, Vol. 21 Issue 9, p3 

    Presents a mathematics activity about a balanced scale.

  • ON BANDLIMITED SCALING FUNCTION*[sup 1]). Wei Chen; Qiao Yang; Wei-jun Jiang; Si-long Peng // Journal of Computational Mathematics;Jul2002, Vol. 20 Issue 4, p373 

    Discusses band-limited scaling functions. Interval band case and three interval bands case; Relationship of band-limited scaling functions to oversampling property; Explanation on translation invariance.

  • Counting errors in a real-time multichannel scaler Zielinski, Marek; Dygdala, Roman S.; Dygdafa, Roman S.; Plociennik, Przemyslaw; Pfociennik, Przemyslaw // Review of Scientific Instruments;Jan1998, Vol. 69 Issue 1, p299 

    Discusses the dead-time, double-pulse and gate width modulation errors in real-time multichannel scaler (RTMS). Sources of the errors; Influence of time gate width modulation on the floating allocation of RTMS; Reduction of double-pulse protection and dead-time using a two-phase gating system.

  • Spinoza's error: Memory for truth and falsity. Nadarevic, Lena; Erdfelder, Edgar // Memory & Cognition;Feb2013, Vol. 41 Issue 2, p176 

    Two theoretical frameworks have been proposed to account for the representation of truth and falsity in human memory: the Cartesian model and the Spinozan model. Both models presume that during information processing a mental representation of the information is stored along with a tag...

  • ASK PROFESSOR C. WORTHY. Rudow, Lenny // Boating;Dec2005, Vol. 78 Issue 12, p24 

    Presents the steps on how to get the approximate weight of a fish without using weighing scale. Formula; Presentation of example; Measurement of the fish.

  • Strong minimizers of the calculus of variations on time scales and the Weierstrass condition. Malinowska, Agnieszka B.; Torres, Delfim F. M. // Proceedings of the Estonian Academy of Sciences;2009, Vol. 58 Issue 4, p205 

    We introduce the notion of strong local minimizer for the problems of the calculus of variations on time scales. Simple examples show that on a time scale a weak minimum is not necessarily a strong minimum. A time scale form of the Weierstrass necessary optimality condition is proved, which...

  • A SENSE OF SCALE. NICHOLLS, JENNY // North & South;May2014, Issue 338, p89 

    The article presents the author's views on scale that can transform the understanding of worlds. He comments that fundamental concepts are as familiar to all as the relative size of the earth and the sun and could be taught at primary school. He opines on the results of an unscientific survey in...

  • A Comparative Study on Emotional Maturity among Adolescent Boys and Girls. Jose, Resmy; Sujatha, R. // International Journal of Nursing Education;Jul-Dec2012, Vol. 4 Issue 2, p73 

    In adolescents, emotional development refers to the attainment of emotional capabilities as they grow. In this study Emotional maturity was assessed and compared among adolescent boys and girls using Modified Emotional Maturity Scale. 300 adolescent boys and 300 adolescent girls were selected by...

  • Predictors of sources of self-confidence in collegiate athletes. Machida, Moe; Marie Ward, Rose; Vealey, Robin S. // International Journal of Sport & Exercise Psychology;Sep2012, Vol. 10 Issue 3, p172 

    The present study examined the antecedents of athletes' reliance on Vealey's nine sources of confidence. Achievement goal-orientation, perfectionism, coach-induced motivational climates were examined as predictors. Sources of confidence were categorized into controllable (e.g. mastery) and...


Read the Article


Sorry, but this item is not currently available from your library.

Try another library?
Sign out of this library

Other Topics