Agreement experiments: a method for quantitatively testing new medical image display approaches

Richard Eugene Johnston; Bonnie C. Yankaskas; John R. Perry; Stephen M. Pizer; David J. Delany; L. A. Parker

doi:10.1117/12.18983

1 August 1990 Agreement experiments: a method for quantitatively testing new medical image display approaches

Richard Eugene Johnston, Bonnie C. Yankaskas, John R. Perry, Stephen M. Pizer, David J. Delany, L. A. Parker

Proceedings Volume 1234, Medical Imaging IV: PACS Systems Design and Evaluation; (1990) https://doi.org/10.1117/12.18983
Event: Medical Imaging '90, 1990, Newport Beach, CA, United States

Abstract

New medical image display devices or processes are commonly evaluated by anecdotal reports or subjective evaluations which are informative and relatively easy to acquire but do not provide quantitative nieasures. On the other hand, experinients eniploying ROC analysis, yield quantitative measurements but are very laborious and demand pathological proof of outcome. We have designed and are employing a new approach, which we have termed "agreement experiments," to quantitatively test the equivalence of observer performance on two systems. This was specifically developed to test whether a radiologist using a new display technique, which has some clear advantages over the standard technique, will detect and interpret diagnostic signs as he would with the standard display technique. Agreement experiments use checklists and confidence ratings to measure how well two radiologists agree on the presence of diagnostic signs when both view images on the standard display. This yields a baseline measure of agreement. Agreement measurements are then obtained when the two radiologists view cases using the new display, or display method, compared to the standard technique. If the levels of agreement when one reads from the new and one reads from the standard display are not statistically different from the baseline measures of agreement, we conclude the two systems are equivalent in conveying diagnostic signs. We will report on an experiment using this test. The experiment compares the agreement of radiological findings for chest CT studies viewed on the conventional multiformat film/lightbox to agreement of radiological findings from chest CT images presented on a multiple screen video system. The study consists of 80 chest CT studies. The results were an 86% to 81% agreement between the two viewing modalities which fell within our criteria of showing agreement.

Citation Download Citation

Richard Eugene Johnston, Bonnie C. Yankaskas, John R. Perry, Stephen M. Pizer, David J. Delany, and L. A. Parker "Agreement experiments: a method for quantitatively testing new medical image display approaches", Proc. SPIE 1234, Medical Imaging IV: PACS Systems Design and Evaluation, (1 August 1990); https://doi.org/10.1117/12.18983

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available