KEYWORDS: Object detection, Ultrasonography, Education and training, Reliability, Video, Receivers, Surgery, Image information entropy, Deep learning, Data acquisition
Computed-based skill assessment relies on accurate metrics to provide comprehensive feedback to trainees. Improving the accuracy of video-based metrics computed using object detection is generally done by improving the performance of the object detection network, however increasing its performance requires resources that cannot always be obtained. This study aims to improve the accuracy of metrics in central venous catheterization without requiring a high performing object detection network by removing false positive predictions identified using uncertainty quantification. The uncertainty for each bounding box was calculated using an entropy equation. The uncertainties were then compared to an uncertainty threshold computed using the optimal point of a Receiver Operating Characteristic curve. Predictions were removed if the uncertainty fell below the predefined threshold. 50 videos were recorded and annotated with ground truth bounding boxes. These bounding boxes were used to train an object detection network, which was used to produce predictive bounding boxes for the test set. This method was evaluated by computing metrics for the predictive bounding boxes with and without having false positives removed and comparing them to ground truth labels using a Pearson Correlation. The Pearson Correlations for the baseline comparisons and the comparisons made using the results calculated using false positive removal were 0.922 and 0.816 for syringe path lengths, 0.753 and 0.510 for ultrasound path lengths, 0.831 and 0.489 for ultrasound usage times, and 0.857 and 0.805 for syringe usage times. This method consistently reduced inflated metrics, making it promising for improving metric accuracy.
Purpose: Computer-assisted skill assessment has traditionally been focused on general metrics related to tool motion and
usage time. While these metrics are important for an overall evaluation of skill, they do not address critical errors made
during the procedure. This study examines the effectiveness of utilizing object detection to quantify the critical error of
making multiple needle insertion attempts in central venous catheterization. Methods: 6860 images were annotated with
ground truth bounding boxes around the syringe attached to the needle. The images were registered using the location of
the phantom, and the bounding boxes from the training set were used to identify the regions where the needle was most
likely inserting the phantom. A Faster region-based convolutional neural network was trained to identify the syringe and
produce the bounding box location for images in the test set. A needle insertion attempt began when the location of the
predicted bounding box fell within the identified insertion region. To evaluate this method, we compared the computed
number of insertions to the number of insertions identified by human reviewers. Results: The object detection network
had an overall mean average precision (mAP) of 0.71. This tracking method computed an average of 4.40 insertion attempts
per recording compared to a reviewer count of 1.39 attempts per recording. Conclusions: The difference in the number of
insertion attempts identified by the computer and reviewers decreases with an increasing mAP, making this method suitable
for detecting multiple needle insertions using an object detection network with a high accuracy.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.