Representation of Ophthalmology Concepts by Electronic Systems: Intercoder Agreement among Physicians Using Controlled Terminologies
Received 17 August 2005; accepted 3 January 2006. published online 16 February 2006.
Objective
To assess intercoder agreement for ophthalmology concepts by 3 physician coders using 5 controlled terminologies (International Classification of Diseases 9, Clinical Modification [ICD9CM]; Current Procedural Terminology, fourth edition; Logical Observation Identifiers, Names, and Codes [LOINC]; Systematized Nomenclature of Medicine, Clinical Terms [SNOMED-CT]; and Medical Entities Dictionary).
Design
Noncomparative case series.
Participants
Five complete ophthalmology case presentations selected from a publicly available journal.
Methods
Each case was parsed into discrete concepts. Electronic or paper browsers were used independently by 3 physician coders to assign a code for every concept in each terminology. A match score representing adequacy of assignment for each concept was assigned on a 3-point scale (0, no match; 1, partial match; 2, complete match). For every concept, the level of intercoder agreement was determined by 2 methods: (1) based on exact code matching with assignment of complete agreement when all coders assigned the same code, partial agreement when 2 coders assigned the same code, and no agreement when all coders assigned different codes, and (2) based on manual review for semantic equivalence of all assigned codes by an independent ophthalmologist to classify intercoder agreement for each concept as complete agreement, partial agreement, or no agreement. Subsequently, intercoder agreement was calculated in the same manner for the subset of concepts judged to have adequate coverage by each terminology, based on receiving a match score of 2 by at least 2 of the 3 coders.
Main Outcome Measures
Intercoder agreement in each controlled terminology: complete, partial, or none.
Results
Cases were parsed into 242 unique concepts. When all concepts were analyzed by manual review, the proportion of complete intercoder agreement ranged from 12% (LOINC) to 44% (SNOMED-CT), and the difference in intercoder agreement between LOINC and all other terminologies was statistically significant (P<0.004). When only concepts with adequate terminology were analyzed by manual review, the proportion of complete intercoder agreement ranged from 33% (LOINC) to 64% (ICD9CM), and there were no statistically significant differences in intercoder agreement among any pairs of terminologies.
Conclusions
The level of intercoder agreement for ophthalmic concepts in existing controlled medical terminologies is imperfect. Intercoder reproducibility is essential for accurate and consistent electronic representation of medical data.
1Department of Ophthalmology, Columbia University College of Physicians and Surgeons, New York, New York.
2Department of Biomedical Informatics, Columbia University College of Physicians and Surgeons, New York, New York.
3Department of Radiology, Columbia University College of Physicians and Surgeons, New York, New York.
4Department of Medicine, Columbia University College of Physicians and Surgeons, New York, New York.
Correspondence and reprint requests to Michael F. Chiang, MD, Department of Ophthalmology, Columbia University College of Physicians and Surgeons, 635 West 165th Street, Box 92, New York, NY 10032.
Manuscript no. 2005-776
Supported by a Career Development Award from Research to Prevent Blindness, New York, New York (MFC); National Eye Institute, Bethesda, Maryland (grant no.: EY13972 [MFC]); and National Library of Medicine, Bethesda, Maryland (grant no.: LM07079 [ACY])
The authors have no commercial, proprietary, or financial interest in any of the products or companies described in the article
ABSTRACT