Simulated gold-standard for quantitative evaluation of monocular vision algorithms

Jack Akers; Andrew Buck; Raub Camaioni; Derek T. Anderson; Robert H. Luke III; James M. Keller; Matthew Deardorff; Brendan Alvey

doi:10.1117/12.2657567

15 June 2023 Simulated gold-standard for quantitative evaluation of monocular vision algorithms

Jack Akers, Andrew Buck, Raub Camaioni, Derek T. Anderson, Robert H. Luke III, James M. Keller, Matthew Deardorff, Brendan Alvey

Author Affiliations +

Proceedings Volume 12525, Geospatial Informatics XIII ; 125250A (2023) https://doi.org/10.1117/12.2657567
Event: SPIE Defense + Commercial Sensing, 2023, Orlando, Florida, United States

Abstract

In the physical universe, truth for computer vision (CV) is impractical if not impossible to obtain. As a result, the CV community has resorted to qualitative practices and sub-optimal quantitative measures. This is problematic because it limits our ability to train, evaluate, and ultimately understand algorithms such as single image depth estimation (SIDE) and structure from motion (SfM). How good are these algorithms, individually and relatively, and where do they break? Herein, we discuss that while truth evades both the real and simulated (SIM) universes, a SIM CV gold-standard can be achieved. We outline an extensible SIM framework and data collection workflow using unreal engine with the Robot Operating System (ROS) for three-dimensional mapping on low altitude aerial vehicles. Furthermore, voxel-based mapping measures from algorithm output to a SIM gold-standard are discussed. The proposed metrics are demonstrated by analyzing performance across changes in platform context. Ultimately, the current article is a step towards an improved process for comparing algorithms, evaluating their strengths and weaknesses, and automating algorithm design.

Conference Presentation

Citation Download Citation

Jack Akers, Andrew Buck, Raub Camaioni, Derek T. Anderson, Robert H. Luke III, James M. Keller, Matthew Deardorff, and Brendan Alvey "Simulated gold-standard for quantitative evaluation of monocular vision algorithms", Proc. SPIE 12525, Geospatial Informatics XIII , 125250A (15 June 2023); https://doi.org/10.1117/12.2657567

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available