Evaluating Agreement: Conducting a Reliability Study

Karanicolas, Paul J.; Bhandari, Mohit; Kreder, Hans; Moroni, Antonio; Richardson, Martin; Walter, Stephen D.; Norman, Geoff R.; Guyatt, Gordon H.
May 2009
Journal of Bone & Joint Surgery, American Volume;May2009 Supplement 3, Vol. 91-A, p99
Academic Journal
Instruments that are useful in clinical or research practice will, when the object of measurement is stable, yield similar results when applied at different times, in different situations, or by different users. Studies that measure the relation of differences between patients or subjects and measurement error (reliability studies) are becoming increasingly common in the orthopaedic literature. In this paper, we identify common aspects of reliability studies and suggest features that improve the reader's confidence in the results. One concept serves as the foundation for all further consideration: in order for a reliability study to be relevant, the patients, raters, and test administration in the study must be similar to the clinical or research context in which the instrument will be used. We introduce the statistical measures that readers will most commonly encounter in reliability studies, and we suggest an approach to sample-size estimation. Readers interested in critically appraising reliability studies or in developing their own reliability studies may find this review helpful.


Related Articles

  • The use of Subject Matter Experts in Validating an Oral Health-Related Quality of Life measure in Korean. Jaesung Seo; MacEntee, Michael; Brondani, Mario // Health & Quality of Life Outcomes;9/4/2015, Vol. 13 Issue 1, p1 

    Objectives: This paper aimed to employ subject matter experts (SMEs) to assess the extent to which the Korean version of the short-form of the OHIP (OHIP-14 K) is culturally valid and equivalent in Korean. Methods: We approached 17 bilingual Korean SMEs from which 10 independently rated the...

  • Controller's Corner. Sadowski, Thomas J. // Journal of Government Financial Management;Fall2008, Vol. 57 Issue 3, p64 

    The article talks on the right reporting particularly in accounting practice. It suggests the importance of planning and reporting with the end in mind. It determines the connection between data and information as well as it defines words such as data, information, knowledge and wisdom in...

  • Methodological Issues in Questionnaire Design. Youngshin Song; Youn-Jung Son; Doonam Oh // Journal of Korean Academy of Nursing;2015, Vol. 45 Issue 3, p323 

    Purpose: The process of designing a questionnaire is complicated. Many questionnaires on nursing phenomena have been developed and used by nursing researchers. The purpose of this paper was to discuss questionnaire design and factors that should be considered when using existing scales. Methods:...

  • Kuvaileva kirjallisuuskatsaus: eteneminen tutkimuskysymyksestä jäsennettyyn tietoon. KANGASNIEMI, MARI; UTRIAINEN, KATI; AHONEN, SANNA-MARI; PIETILÄ, ANNA-MAIJA; JÄÄSKELÄINEN, PETRI; LIIKANEN, EEVA // Hoitotiede;2013, Vol. 25 Issue 4, p291 

    In recent decades, using literature reviews has become established and more versatile in research on nursing and health sciences. The spectrum of different literature reviews is extensive, and more attention has been paid to the methodical development of literature reviews. Descriptive...

  • Score Reliability: A Retrospective Look Back at 12 Years of Reliability Generalization Studies. Vacha-Haase, Tammi; Thompson, Bruce // Measurement & Evaluation in Counseling & Development;Jul2011, Vol. 44 Issue 3, p159 

    The present study was conducted to characterize (a) the features of the thousands of primary reports synthesized in 47 reliability generalization (RG) measurement meta-analysis studies and (b) typical methodological practice within the RG literature to date. With respect to the treatment of...

  • Measurements in Quantitative Research: How to Select and Report on Research Instruments. Hagan, Teresa L. // Oncology Nursing Forum;Jul2014, Vol. 41 Issue 4, p431 

    Measures exist to numerically represent degrees of attributes. Quantitative research is based on measurement and is conducted in a systematic, controlled manner. These measures enable researchers to perform statistical tests, analyze differences between groups, and determine the effectiveness of...

  • Test-Retest Correlations and the Reliability of Copy Testing. Silk, Alvin J. // Journal of Marketing Research (JMR);Nov77, Vol. 14 Issue 4, p476 

    Test-retest correlations often are reported as measures of the reliability of copy testing procedures with little or no attention to the conditions under which such an interpretation is meaningful. Some checks are suggested for assessing the consistency of test-retest data with certain...

  • Validation of the process criteria for assessment of a hospital nursing service. Bauer Feldman, Liliane; Kowal Olm Cunha, Isabel Cristina; D'lnnocenzo, Maria // Revista Latino-Americana de Enfermagem (RLAE);Jul/Aug2013, Vol. 21 Issue 4, p841 

    Objective: to validate an instrument containing process criteria for assessment of a hospital nursing service based on the National Accreditation Organization program. Method: a descriptive, quantitative methodological study performed in stages. An instrument constructed with 69 process criteria...

  • Reliability and validity of three questionnaires measuring context-specific sedentary behaviour and associated correlates in adolescents, adults and older adults. Busschaert, Cedric; De Bourdeaudhuij, Ilse; Van Holle, Veerle; Chastin, Sebastien F. M.; Cardon, Greet; De Cocker, Katrien // International Journal of Behavioral Nutrition & Physical Activit;9/18/2015, Vol. 12, p1 

    Background: Reliable and valid measures of total sedentary time, context-specific sedentary behaviour (SB) and its potential correlates are useful for the development of future interventions. The purpose was to examine test-retest reliability and criterion validity of three newly developed...


Read the Article


Sorry, but this item is not currently available from your library.

Try another library?
Sign out of this library

Other Topics