1.

Introduction

Colorectal cancer is the third-leading cause of cancer-related death worldwide.¹ The early diagnosis of colorectal cancers and adenomatous polyps facilitates the application of early-stage therapies, and significantly increases the potential for survival.^2,3 To detect and monitor the progression of various pathologies at an early stage, spectral measurement and analysis, which provide accurate quantifications of microvascular and morphological properties of the mucosa, are useful. However, point spectroscopy does not account for tissue spatial heterogeneity as it detects light from a single point, and hence, is ineffective for mapping lesion area. However, hyperspectral imaging (HSI) accounts for tissue spatial heterogeneity and can be effectively used for spatial mapping of tissue morphology and physiology.^4–10 This technique involves the acquisition and analysis of a series of reflected two-dimensional images sampled at different wavelengths.^11–16

In the gastrointestinal field, most HSI studies have involved ex vivo biopsies, resected tumor tissues, or surficial organs such as the skin, tongue, or larynx. Clancy et al.¹⁶ developed a laparoscope HSI system based on liquid-crystal tunable filter, Martin et al.^14,15 developed an HSI system with fluorescence for imaging of the larynx, and Leitner et al.¹⁷ developed an HSI laryngoscope system based on acoustic-optical tunable filter, which can collect 51 images with a bandwidth of 5 nm in 1.25 s. Shimoyama et al.¹⁸ developed a micro Fabry–Perot interference filter placed at the tip of a flexible endoscope to create a wavelength-adjustable spectral endoscope, although this has not been used in the clinic thus far. Kiyotoki et al.¹⁹ proposed the use of a diagnostic support system for resected gastric cancer tissues with an HSI camera, and indicated that HSI could be used to measure the spectral reflectance (SR) in gastric tumors and could differentiate between tumorous and normal mucosa. However, the color of the resected tissues differed from that observed in vivo, which suggests that the spectral properties of tissue may change after they are resected. In the present study, we aimed to assess the difference in the in vivo SR of malignant colorectal tumors and normal mucosa, and sought to clinically apply the HSI technology in vivo to detect colorectal tumors or adenomatous polyps with high accuracy.

2.

Methods

2.1.

Equipment Setup

Figure 1 shows a diagram of the flexible hyperspectral (HS) colonoscopy system, which can be used to obtain a series of reflected HS images in a noncontact manner in the wavelength range of 405 to 665 nm. This system was created by modifying the commercialized endoscope AQ-100, which was donated by Shanghai Aohua Photoelectric Endoscope Co. Ltd. The system comprises four units^20–24: an HS light source, endoscope imaging unit, image processing unit, and monitor unit (LMD-2451MD, Sony). The HS light source unit contains a xenon arc lamp (PE300AF, PerkinElmer Optoelectronics), which has an integrated parabolic reflector that collects light from the light bulb and produces a collimated output beam. Twenty-seven sequential narrowband interference filters (Shenyang HB Optical Technology Co., Ltd., China; HB405T10K is the model number of one of the filters) that are centered from 405 to 665 nm at 10-nm intervals—with a full-width at half maximum (FWHM) of 10 nm, peak transmission of $\sim 60\%$, and two all-pass holes—are mounted on two motorized filter wheels positioned in the path of the collimated beam. Figure 2 shows the wheels, which is the major difference from the parts of the conventional flexible endoscopy system. In the conventional white light illumination mode, RGB color filters are used and the all-pass holes in the HS wheels are positioned in the illumination beam path. When the suspected disease area is targeted and the HSI mode is enabled, 27 monochromatic spectral-channel light beams are produced sequentially in 4.2 s as the filter wheel rotates. An optical fiber bundle directs the beams to the colorectal mucosa after the optical taper focuses the light onto the bundle. A combination of lenses is attached to the distal end of the bundle. This combination results in a 5.0-cm-diameter illumination spot at a working distance of 2.0 cm. A $582\times 752\text{-}\text{pixel}$ monochromatic charge-coupled device (CCD; ICX279AL, Sony) positioned at the distal end of the optical fiber bundle is used to spectrally resolve the reflected images from the mucosa. Analog signals read from the CCD are sequentially transformed into digital signals in the image-processing unit. A cable is used to connect the image-processing unit to the medical monitor, which is used to display the reflected images of the colorectal mucosa.

Fig. 1

Diagram of flexible HSI endoscope system.

Fig. 2

The optical filter wheel used in the HSI system. The use of this wheel is the major difference from the conventional flexible endoscopy system.

2.4.

Band Selection

A band selection algorithm based on recursive divergence (RD)³¹ was employed to evaluate the HS images, and to determine the wavelength subset that contained the most useful information. With regard to the training samples of the obtained HS images, we manually categorized the samples into two classes: disease pixels $w_i$ and normal tissue pixels $w_j$. Each pixel is a vector along the spectral axis. The RD is defined³¹ as the total average information $J_{ij}(x)$ for discriminating class $w_i$ from $w_j$, and is estimated by

Eq. (1)

$$J_{ij}(x)=\int [p_i(x)-p_j(x)]\ln \frac{p_i(x)}{p_j(x)}\mathrm{d}x,$$

where $p_i(x)$ is the probability density function of $x$ in class $w_i$. The divergence is the symmetric version of the Kullback–Leibler distance, and it is non-negative, monotonic, and additive for independent variables.

$J_{ij}(x_p^{*})$ represents the divergence with $p$ selected bands, and $J_{ij}(x_p^{*},x_{p+1}^{*})$ represents the divergence with the additional band $x_{p+1}^{*}$. If the signal classes are characterized by $p$-dimensional multivariate normal distributions $\mathbf{N}({\mathbf{\theta }}_{\mathbf{i}},{\mathbf{\Sigma }}_{\mathbf{i}})$, where ${\mathbf{\theta }}_{\mathbf{i}}$ and ${\mathbf{\Sigma }}_{\mathbf{i}}$ are the mean vector and covariance matrix of class $w_i$, respectively. $tr$ would represent the trace of a matrix. Moreover,

$$z_{ij}=\{\begin{array}{cc}1,& \text{if}{x}_j\in w_i\\ 0,& \text{otherwise}\end{array},$$

$n_{\mathrm{i}}=\sum _{j=1}^n{z}_{ij}$, where $n$ is total number of samples. If the additional band $x_{p+1}^{*}$ has a mean value of ${\theta }_k^{*}$, variance of ${\sigma }_k^2$, covariance vector of $z_k$ between $x_{p+1}^{*}$ and the elements of $x_p$, for class $k$ ($k=i$ or $j$), then the divergence with an additional band $x_{p+1}^{*}$ can be recursively calculated in an efficient manner, as follows:

Eq. (2)

$$J_{ij}(x_p^{*},x_{p+1}^{*})=J_{ij}(x_p^{*})+{\mathrm{\Delta }}_{ij}(x_{p+1}^{*}),$$

where ${\mathrm{\Delta }}_{ij}(x_{p+1}^{*})$ is the incremental divergence due to the addition of $x_{p+1}^{*}$ and can be calculated by the following equation:

Eq. (3)

$${\mathrm{\Delta }}_{ij}(x_{p+1}^{*})=\frac{1}{2{\delta }_i}({\theta }_i^{*}-{\theta }_j^{*})[({\theta }_i^{*}-{\theta }_j^{*})-{({\mathbf{\theta }}_{\mathbf{i},\mathbf{p}}^{*}-{\mathbf{\theta }}_{\mathbf{j},\mathbf{p}}^{*})}^t{\gamma }_i]+\frac{1}{2{\delta }_j}({\theta }_i^{*}-{\theta }_j^{*})[({\theta }_i^{*}-{\theta }_j^{*})-{({\mathbf{\theta }}_{\mathbf{i},\mathbf{p}}^{*}-{\mathbf{\theta }}_{\mathbf{j},\mathbf{p}}^{*})}^t{\gamma }_j]+\frac{1}{2}(z_i^t-z_j^t)({\delta }_i^{-1}{\gamma }_i-{\delta }_j^{-1}{\gamma }_j)+\frac{1}{2}({\sigma }_i^2-{\sigma }_j^2)({\delta }_j^{-1}-{\delta }_i^{-1})+\frac{1}{2}tr\{{\mathbf{\Sigma }}_{\mathbf{i},\mathbf{p}}({\gamma }_j-{\gamma }_i){\delta }_j^{-1}{\gamma }_j^t-{\mathbf{\Sigma }}_{\mathbf{j},\mathbf{p}}({\gamma }_j-{\gamma }_i){\delta }_i^{-1}{\gamma }_i^t+[({\gamma }_i{\delta }_i^{-1}{\gamma }_i^t+{\gamma }_j{\delta }_j^{-1}{\gamma }_j^t)({\mathbf{\theta }}_{\mathbf{i},\mathbf{p}}^{*}-{\mathbf{\theta }}_{\mathbf{j},\mathbf{p}}^{*})]-({\gamma }_i{\delta }_i^{-1}+{\gamma }_j{\delta }_j^{-1})({\theta }_i^{*}-{\theta }_j^{*})]{({\mathbf{\theta }}_{\mathbf{i},\mathbf{p}}^{*}-{\mathbf{\theta }}_{\mathbf{j},\mathbf{p}}^{*})}^t\},$$

where ${\gamma }_k={\mathrm{\Sigma }}_{k,p}^{-1}{z}_k$ and ${\delta }_k={\sigma }_k^2-z_k^t{\mathrm{\Sigma }}_{k,p}^{-1}{z}_k$.

The detailed derivation of the incremental divergence has been described in Ref. 31. Equation (3) provides an efficient manner to calculate the divergence with the additional band. The procedure for efficient band selection based on the recursive equation of divergence can be described as follows (the corresponding block diagram is shown in Fig. 3):

• Step 1: Set $D$ to the initial band set and $S$ to the empty set. Select a starting band (${\lambda }_i$) by exhaustively determining all bands and identifying one with the maximum divergence.

• Step 2: Calculate ${\mathrm{\Delta }}_{ij}(x_{p+1}^{*})$ according to Eq. (6) for all the remaining bands.

• Step 3: Select the band with the largest incremental effectiveness (${\lambda }_k$), and add it to the selected band set. The algorithm will stop when a certain number of bands are selected. Otherwise, proceed to Step 2.

Fig. 3

Block diagram for band selection based on RD.

3.

Results

We obtained 21 HS image cubes of malignant colorectal tumors or adenomatous polyps from 12 patients in the clinical trials. Figure 4 shows a color image of a malignant colorectal tumor and its surrounding normal tissue, along with a graph of the SR of the tumor and normal mucosa in the wavelength range of 405 to 655 nm. Both SRs are normalized using preprocessing methods and are different from the raw reflectance data. Figure 5 shows some images from an HS image cube of a colorectal tumor, which are centered at 405, 455, 545, 585, 595, and 655 nm, respectively. The spectral characteristics in the tumor and normal tissue regions were found to be significantly different according to the wavelengths. Except for the image centered at 405 nm, which represents low light intensity, the microvascular patterns were clearer on the images centered at 455, 545, and 585 nm than those centered at 595 and 655 nm.

Fig. 4

Color image of a malignant colorectal tumor and a graph showing the SR of the tumor and normal mucosa in the wavelength range of 405 to 655 nm.

Fig. 5

Images from an HS image cube of malignant colorectal tumors. Images are centered at 405, 455, 545, 585, 595, and 655 nm, respectively. The spectral characteristics in the tumor and normal tissue regions are significantly different according to the wavelengths.

Figure 6 shows the SR for all pixels in the tumor region and normal mucosa. The SRs in the tumor regions tend to be high in wavelength bands $>585\mathrm{nm}$ and low in the region of 475 to 585 nm. One reason for this finding may be the presence of significant angiogenesis in the mucosa in the tumor region. Hemoglobin absorbs more light in the wavelength region $<585\mathrm{nm}$ and less light in the region $>585\mathrm{nm}$. In particular, there were seven abnormal curves that had reflectance peaks among the normal tissue signatures. This finding may be associated with the occasional interferences when the system obtains the HS data, or with noise amplification during the preprocessing. As there were $>\mathrm{20,000}\text{pixels}$ in the normal tissue region, we removed these seven curves from our study set and ignored their influences in the subsequent analyses.

Fig.6

SR for pixels in the tumor region and in the normal mucosa. The SRs in the tumor regions tend to be high in wavelength bands $>585\mathrm{nm}$ and low in the region of 475 to 585 nm.

We processed the 21 HS image cubes of colorectal tumors and adenomatous polyps using the RD band selection method. Table 1 shows the band selection results of 10 image cubes. We observed that 625, 525 to 545, 415 to 425, 465, and 495 to 505 nm were among the most frequently selected wavelengths in the studied wavelength region.

Table 1

Band selection results using the RD method.

Figure 7(a) shows the manually marked tumor region, and Fig. 7(b) shows the region which was classified based on the five selected bands. This image was obtained from one of the 21 HS cubes from the study cases. As expected, the classification with a greater number of bands had a higher accuracy and specificity (Table 2). In fact, the classification based on all HS images had the highest accuracy and specificity. Moreover, the classification with all the HS bands showed high sensitivity, whereas no marked difference in sensitivity was observed in classification with 1 band to 5 bands. Although the results exhibit a large variance among the 21 HS cubes (Table 2), which may be due to the small number of studied cases, they indicate that spectral imaging could help tumor region detection and classification, as spectral imaging can extract accurate spectral information, which is usually diluted in conventional color imaging.

Fig. 7

(a) The manually marked tumor region and (b) the tumor region classified based on the five selected bands. In (b), some single points classified by SVM are removed and the outline is smoothed, as we are only concerned about region detection.

Table 2

Accuracy, sensitivity, and specificity of the classification results based on different bands (SVM), compared to those based on the method of manual classification.

We merged the adjacently selected bands (such as 525 to 545 nm) into a single band with a broader FWHM (30 nm), and used these five selected bands for spectral imaging of colorectal tumor and adenomatous polyps in clinical trials. Pseudocolors were assigned to pixels corresponding to the five spectral components. Figure 8 shows the contrast of colorectal adenomatous polyp images under white illumination and spectral imaging. We observed that spectral imaging using the selected wavelengths is useful for outlining the disease region. As indicated by the arrow, potential angiogenesis occurs in an extended region surrounding the polyp region. Under white light illumination, the margin of the extended region cannot be clearly observed; however, with spectral imaging, the region can be easily outlined.

Fig. 8

(a) A colorectal adenomatous polyp under white light illumination; (b) the color-enhanced adenomatous polyps under spectral illumination of the five selected bands (625, 525 to 545, 415 to 425, 465, and 495 to 505 nm). As indicated by the arrow, the margin of the polyp region is outlined clearly. The boxes show that the microvascular pattern of the polyp mucosa is enhanced under spectral illumination with the five selected bands.

Moreover, spectral imaging is useful for enhancing the microvascular pattern of the polyp mucosa in vivo. As indicated by the boxes in Fig. 8, the pit pattern can be observed more clearly on the spectral image than on the white light image. This could help doctors determine the type of polyps or stage of the tumor.

4.

Discussion

For an endoscopy system with CCDs operating at low speeds, the use of five bands may be excessive for meeting the real-time requirements. Hence, four, three, and even two bands can be employed to aid diagnosis, as all the bands are selected according to the HS images of the colorectal tumor and adenomatous polyps. Narrowband imaging (NBI) technology, which is widely used in the clinical setting, uses two wavelengths: 415 and 540 nm.³⁴ In the early stage, some NBI systems also apply a third wavelength—600 nm.³⁵ The first three wavelengths we selected for spectral imaging are similar to those used in NBI. However, the wavelengths selected in our technique are more accurate and specific for colorectal tumors and adenomatous polyps. Moreover, in the future, we can employ different wavelength subsets based on different types of tumors or environments, in comparison to the two defined wavelengths used in NBI.

Spectral imaging can easily be adopted in endoscopy systems. For endoscopy systems with color CCDs, only a filter wheel needs to be added. However, for systems with monochromatic CCDs, a fast wavelength switching device, such as an acoustic-optical tunable filter, is also required.

Despite the favorable results observed in the present study, we need to further verify the selected bands in clinical trials in the future. Moreover, as the reflected spectral images may differ based on various factors, such as the disease stage, and individual and environmental changes, we do not expect that these band selection results would apply to all individuals. In the present study, we have simply proved that this spectral imaging technique involving band selection from the HSI of colorectal tumors can be useful when employed in a clinic in vivo, and can aid in the outlining of a disease region and enhancing of the pit pattern of the mucosa.

In the future, after summarizing a larger number of cases, we can determine an optimal spectral band subset that can be effective for most patients with colorectal tumor or adenomatous polyps. Furthermore, optimal spectral band subsets can be selected that are individual-specific and tumor stage-specific.

5.

Conclusion

This study demonstrated that HSI can be used in vivo to measure the SR in malignant colorectal tumors and adenomatous polyps, and to differentiate between tumors and normal mucosa. A flexible endoscope system was modified with HS technology and was used to obtain in vivo HS images. A band selection algorithm, based on the RD method, was used to analyze HS images of residual colorectal regions, and five bands (625, 525 to 545, 415 to 425, 465, and 495 to 505 nm) were identified that corresponded to colorectal tumors. We then applied these bands to spectral imaging in clinical trials for colorectal tumors or adenomatous polyps. We observed that the application of these five bands to spectral imaging is useful for outlining the disease region and enhancing the microvascular network on the mucosa surface. Although further study is required, this diagnostic support system has great potential for further application in the clinical setting.