To create tumor “habitats” from the “signatures” discovered from multimodality metabolic and physiological images, we developed a framework of a processing pipeline. The processing pipeline consists of six major steps: (1) creating superpixels as a spatial unit in a tumor volume; (2) forming a data matrix {D} containing all multimodality image parameters at superpixels; (3) forming and clustering a covariance or correlation matrix {C} of the image parameters to discover major image “signatures;” (4) clustering the superpixels and organizing the parameter order of the {D} matrix according to the one found in step 3; (5) creating “habitats” in the image space from the superpixels associated with the “signatures;” and (6) pooling and clustering a matrix consisting of correlation coefficients of each pair of image parameters from all patients to discover subgroup patterns of the tumors. The pipeline was applied to a dataset of multimodality images in glioblastoma (GBM) first, which consisted of 10 image parameters. Three major image “signatures” were identified. The three major “habitats” plus their overlaps were created. To test generalizability of the processing pipeline, a second image dataset from GBM, acquired on the scanners different from the first one, was processed. Also, to demonstrate the clinical association of image-defined “signatures” and “habitats,” the patterns of recurrence of the patients were analyzed together with image parameters acquired prechemoradiation therapy. An association of the recurrence patterns with image-defined “signatures” and “habitats” was revealed. These image-defined “signatures” and “habitats” can be used to guide stereotactic tissue biopsy for genetic and mutation status analysis and to analyze for prediction of treatment outcomes, e.g., patterns of failure.
Modality filtering is an important feature in biomedical image searching systems and may significantly improve the retrieval performance of the system. This paper presents a new method for extracting endoscopic image figures from photograph images in biomedical literature, which are found to have highly diverse content and large variability in appearance. Our proposed method consists of three main stages: tissue image extraction, endoscopic image candidate extraction, and ophthalmic image filtering. For tissue image extraction we use image patch level clustering and MRF relabeling to detect images containing skin/tissue regions. Next, we find candidate endoscopic images by exploiting the round shape characteristics that commonly appear in these images. However, this step needs to compensate for images where endoscopic regions are not entirely round. In the third step we filter out the ophthalmic images which have shape characteristics very similar to the endoscopic images. We do this by using text information, specifically, anatomy terms, extracted from the figure caption. We tested and evaluated our method on a dataset of 115,370 photograph figures, and achieved promising precision and recall rates of 87% and 84%, respectively.
Contours, object blobs, and specific feature points are utilized to represent object shapes and extract shape descriptors that can then be used for object detection or image classification. In this research we develop a shape descriptor for biomedical image type (or, modality) classification. We adapt a feature extraction method used in optical character recognition (OCR) for character shape representation, and apply various image preprocessing methods to successfully adapt the method to our application. The proposed shape descriptor is applied to radiology images (e.g., MRI, CT, ultrasound, X-ray, etc.) to assess its usefulness for modality classification. In our experiment we compare our method with other visual descriptors such as CEDD, CLD, Tamura, and PHOG that extract color, texture, or shape information from images. The proposed method achieved the highest classification accuracy of 74.1% among all other individual descriptors in the test, and when combined with CSD (color structure descriptor) showed better performance (78.9%) than using the shape descriptor alone.
“Imaging signs” are a critical part of radiology’s language. They not only are important for conveying diagnosis, but may
also aid in indexing radiology literature and retrieving relevant cases and images. Here we report our work towards
representing and categorizing imaging signs of abdominal abnormalities in figures in the radiology literature. Given a
region-of-interest (ROI) from a figure, our goal was to assign a correct imaging sign label to that ROI from the following
seven: accordion, comb, ring, sandwich, small bowel feces, target, or whirl. As training and test data, we created our
own “gold standard” dataset of regions containing imaging signs. We computed 2997 feature attributes to represent
imaging sign characteristics for each ROI in training and test sets. Following feature selection they were reduced to 70
attributes and were input to a Support Vector Machine classifier. We applied image-enhancement methods to
compensate for variable quality of the images in radiology articles. In particular we developed a method for automatic
detection and removal of pointers/markers (arrows, arrowheads, and asterisk symbols) on the images. These
pointers/markers are valuable for approximately locating ROIs; however, they degrade the classification because they are
often (partially) included in the training ROIs. On a test set of 283 ROIs, our method achieved an overall accuracy of
70% in labeling the seven signs, which we believe is a promising result for using imaging signs to search/retrieve
radiology literature. This work is also potentially valuable for the creation of a visual ontology of biomedical imaging
entities.
Image modality classification is an important task toward achieving high performance in biomedical image and article retrieval. Imaging modality captures information about its appearance and use. Examples include X-ray, MRI, Histopathology, Ultrasound, etc. Modality classification reduces the search space in image retrieval. We have developed and evaluated several modality classification methods using visual and textual features extracted from images and text data, such as figure captions, article citations, and MeSH®. Our hierarchical classification method using multimodal (mixed textual and visual) and several class-specific features achieved the highest classification accuracy of 63.2%. The performance was among the best in ImageCLEF2012 evaluation.
Regions of interest (ROIs) that are pointed to by overlaid markers (arrows, asterisks, etc.) in biomedical images
are expected to contain more important and relevant information than other regions for biomedical article
indexing and retrieval. We have developed several algorithms that localize and extract the ROIs by recognizing
markers on images. Cropped ROIs then need to be annotated with contents describing them best. In most cases
accurate textual descriptions of the ROIs can be found from figure captions, and these need to be combined
with image ROIs for annotation. The annotated ROIs can then be used to, for example, train classifiers that
separate ROIs into known categories (medical concepts), or to build visual ontologies, for indexing and retrieval
of biomedical articles.
We propose an algorithm that pairs visual and textual ROIs that are extracted from images and figure
captions, respectively. This algorithm based on dynamic time warping (DTW) clusters recognized pointers into
groups, each of which contains pointers with identical visual properties (shape, size, color, etc.). Then a rule-based
matching algorithm finds the best matching group for each textual ROI mention. Our method yields a
precision and recall of 96% and 79%, respectively, when ground truth textual ROI data is used.
Pointers (arrows and symbols) are frequently used in biomedical images to highlight specific image regions of
interest (ROIs) that are mentioned in figure captions and/or text discussion. Detection of pointers is the first
step toward extracting relevant visual features from ROIs and combining them with textual descriptions for a
multimodal (text and image) biomedical article retrieval system.
Recently we developed a pointer recognition algorithm based on an edge-based pointer segmentation method,
and subsequently reported improvements made on our initial approach involving the use of Active Shape Models
(ASM) for pointer recognition and region growing-based method for pointer segmentation. These methods
contributed to improving the recall of pointer recognition but not much to the precision. The method discussed
in this article is our recent effort to improve the precision rate. Evaluation performed on two datasets and
compared with other pointer segmentation methods show significantly improved precision and the highest F1
score.
Biomedical images are often referenced for clinical decision support (CDS), educational purposes, and research. They
appear in specialized databases or in biomedical publications and are not meaningfully retrievable using primarily textbased
retrieval systems. The task of automatically finding the images in an article that are most useful for the purpose of
determining relevance to a clinical situation is quite challenging. An approach is to automatically annotate images
extracted from scientific publications with respect to their usefulness for CDS. As an important step toward achieving
the goal, we proposed figure image analysis for localizing pointers (arrows, symbols) to extract regions of interest (ROI)
that can then be used to obtain meaningful local image content. Content-based image retrieval (CBIR) techniques can
then associate local image ROIs with identified biomedical concepts in figure captions for improved hybrid (text and
image) retrieval of biomedical articles.
In this work we present methods that make robust our previous Markov random field (MRF)-based approach for pointer
recognition and ROI extraction. These include use of Active Shape Models (ASM) to overcome problems in recognizing
distorted pointer shapes and a region segmentation method for ROI extraction.
We measure the performance of our methods on two criteria: (i) effectiveness in recognizing pointers in images, and (ii)
improved document retrieval through use of extracted ROIs. Evaluation on three test sets shows 87% accuracy in the
first criterion. Further, the quality of document retrieval using local visual features and text is shown to be better than
using visual features alone.
Biomedical images are invaluable in establishing diagnosis, acquiring technical skills, and implementing best practices in
many areas of medicine. At present, images needed for instructional purposes or in support of clinical decisions appear in
specialized databases and in biomedical articles, and are often not easily accessible to retrieval tools. Our goal is to
automatically annotate images extracted from scientific publications with respect to their usefulness for clinical decision
support and instructional purposes, and project the annotations onto images stored in databases by linking images
through content-based image similarity.
Authors often use text labels and pointers overlaid on figures and illustrations in the articles to highlight regions of
interest (ROI). These annotations are then referenced in the caption text or figure citations in the article text. In previous
research we have developed two methods (a heuristic and dynamic time warping-based methods) for localizing and
recognizing such pointers on biomedical images. In this work, we add robustness to our previous efforts by using a
machine learning based approach to localizing and recognizing the pointers. Identifying these can assist in extracting
relevant image content at regions within the image that are likely to be highly relevant to the discussion in the article
text. Image regions can then be annotated using biomedical concepts from extracted snippets of text pertaining to images
in scientific biomedical articles that are identified using National Library of Medicine's Unified Medical Language
System® (UMLS) Metathesaurus. The resulting regional annotation and extracted image content are then used as indices
for biomedical article retrieval using the multimodal features and region-based content-based image retrieval (CBIR)
techniques. The hypothesis that such an approach would improve biomedical document retrieval is validated through
experiments on an expert-marked biomedical article dataset.
Biomedical images are invaluable in medical education and establishing clinical diagnosis. Clinical decision support
(CDS) can be improved by combining biomedical text with automatically annotated images extracted from relevant
biomedical publications. In a previous study we reported 76.6% accuracy using supervised machine learning on the
feasibility of automatically classifying images by combining figure captions and image content for usefulness in finding
clinical evidence. Image content extraction is traditionally applied on entire images or on pre-determined image regions.
Figure images articles vary greatly limiting benefit of whole image extraction beyond gross categorization for CDS due
to the large variety. However, text annotations and pointers on them indicate regions of interest (ROI) that are then
referenced in the caption or discussion in the article text. We have previously reported 72.02% accuracy in text and
symbols localization but we failed to take advantage of the referenced image locality.
In this work we combine article text analysis and figure image analysis for localizing pointer (arrows, symbols) to extract
ROI pointed that can then be used to measure meaningful image content and associate it with the identified biomedical
concepts for improved (text and image) content-based retrieval of biomedical articles. Biomedical concepts are identified
using National Library of Medicine's Unified Medical Language System (UMLS) Metathesaurus. Our methods report
an average precision and recall of 92.3% and 75.3%, respectively on identifying pointing symbols in images from a
randomly selected image subset made available through the ImageCLEF 2008 campaign.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.