Predicting DNN classification performance on degraded imagery

Lotte Nijskens; Richard J. M. den Hollander; Jorge G. O. Melo; Frank P. A. Benders; Klamer Schutte

doi:10.1117/12.2682052

17 October 2023 Predicting DNN classification performance on degraded imagery

Lotte Nijskens, Richard J. M. den Hollander, Jorge G. O. Melo, Frank P. A. Benders, Klamer Schutte

Proceedings Volume 12742, Artificial Intelligence for Security and Defence Applications; 127420S (2023) https://doi.org/10.1117/12.2682052
Event: SPIE Security + Defence, 2023, Amsterdam, Netherlands

Abstract

The use of deep neural networks (DNNs) is the dominant approach for image classification, as it achieves state-of-the-art performance when sufficiently large training datasets are available. The best DNN performance is reached when test data conditions are similar to the training data conditions. However, if the test conditions differ, there will usually be a loss of classification performance, for instance when the test targets are more distant, blurry or occluded than those observed in the training data. It is desirable to have an estimate of the expected classification performance prior to using a DNN in practice. A low expected performance may deem the DNN unsuitable for the operational task at hand. While the effect on classification performance of a single changed test condition has been investigated before, this paper studies the combined effect of multiple changed test conditions. In particular, we will compare two prediction models for the estimation of the expected performance compared to the DNN performance on the development data. Our approach allows performance estimation in operation based on knowledge of the expected operational conditions, but without having access to operational data itself. We investigate the aforementioned steps for image classification on the MARVEL vessel dataset and the Stanford Cars dataset. The changing test conditions consist of several common image degradations that are imposed on the original images. We find that the prediction models produce acceptable results in case of small degradations, and when degradations show a constant accuracy falloff over their range.

Conference Presentation

(2023) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Lotte Nijskens, Richard J. M. den Hollander, Jorge G. O. Melo, Frank P. A. Benders, and Klamer Schutte "Predicting DNN classification performance on degraded imagery", Proc. SPIE 12742, Artificial Intelligence for Security and Defence Applications, 127420S (17 October 2023); https://doi.org/10.1117/12.2682052

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available