Agreement between two versions of a CADx system: a simulation study

Berkman Sahiner; Nicholas Petrick; Sophie Paquerault; Weijie Chen; Tien Nguyen

doi:10.1117/12.877727

3 March 2011 Agreement between two versions of a CADx system: a simulation study

Berkman Sahiner, Nicholas Petrick, Sophie Paquerault, Weijie Chen, Tien Nguyen

Proceedings Volume 7966, Medical Imaging 2011: Image Perception, Observer Performance, and Technology Assessment; 79660J (2011) https://doi.org/10.1117/12.877727
Event: SPIE Medical Imaging, 2011, Lake Buena Vista (Orlando), Florida, United States

Abstract

A simulation study was conducted to investigate the agreement between original and updated versions of a computeraided diagnosis (CADx) system. Performances of two versions of a CADx system are traditionally compared using metrics derived from the receiver operating characteristic (ROC) curve. These aggregate standalone performance measures may reveal the overall improvement of the CADx system due to the update, but do not provide information about the specific change in CADx output for individual cases. To address this issue, we used the concordance measure, which compares the ordering of scores for pairs of cases between system versions (i.e., before and after the update of the system). In this preliminary study, the system update that we investigated was an enlargement of the training data set, which is often encountered in the development of a subsequent CADx system version for improving performance. We separately studied the effect of the size of the original training set, the number of features, and the distribution and separation of the two classes in the feature space on the concordance and AUC measures. When the effect of an update was compared among datasets with differences in intrinsic class separation, concordance was in general larger when the intrinsic class separation was larger. The amount of change in AUC between the original and updated CADx system did not always predict the degree of agreement between the two system versions. A large improvement in AUC could be accompanied with either a larger or smaller agreement between the original and updated systems. Quantification of the degree of agreement in standalone performance between different versions of a CADx system may serve to define a major algorithm update, and better depict the impact of that update.

Citation Download Citation

Berkman Sahiner, Nicholas Petrick, Sophie Paquerault, Weijie Chen, and Tien Nguyen "Agreement between two versions of a CADx system: a simulation study", Proc. SPIE 7966, Medical Imaging 2011: Image Perception, Observer Performance, and Technology Assessment, 79660J (3 March 2011); https://doi.org/10.1117/12.877727

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available