1.

Introduction

Colorectal cancer (CRC) is the third most prevalent cancer in the United States¹ and worldwide,² resulting in an estimated number of 608,000 deaths annually.³ Treatment of CRC is frequently performed using endoscopic approaches, in particular laparoscopic intervention, which is a preferred method over open surgery.^4,5 Cancer identification during surgical treatment relies on the physician’s visual ability to discriminate healthy from malignant tissue under white light illumination. However, the low intrinsic contrast of carcinomas, especially with respect to the propagating tumor margin or smaller foci of disease, makes it difficult to precisely demarcate and completely excise during surgery. In addition, color endoscopy cannot visualize under the surface, making it virtually impossible to discriminate deep-seated cancer or accurately evaluate the penetration extend of carcinomas. As a result, color endoscopy is known to miss tumors in up to 24% of the cases.⁶

Optical contrast enhancement using narrow band imaging (NBI), chromoendoscopy and autofluorescence imaging (AFI) has been considered in imaging CRC during colonoscopy. NBI uses narrow bandpass filters to enhance the hemoglobin contrast and thus visualize the tumor vascularization.⁷ A randomized study published by Adler et al.⁸ concluded that adenomata are detected more frequently using NBI compared to color endoscopy, however the detection difference observed between the two methods was not statistically significant. In chromoendoscopy nonspecific dyes are applied to visualize micro-anatomical changes in the mucosa, but the technique is mostly used for biopsy guidance. AFI is based on the emission of intravital fluorescent molecules and has been used for the detection of lesions⁹ and polyps¹⁰ in colonoscopy as well as in oral oncology,¹¹ among others. While AFI can help to identify molecular differences between tissues, the method lacks sufficient sensitivity and specificity for improving clinical outcome. These techniques have been mostly applied for screening and diagnostic purposes in colonoscopy and their use in surgical laparoscopy of CRC is not established.

The administration of fluorescent agents has been also considered to improve endoscopic detection of cancer. Indocyanine green (ICG) is an FDA approved fluorescent dye that has been used in medical procedures for over 50 years. ICG is a cyanine dye used as a blood-pool agent that can be administered intravenously (i.v.) to visualize blood vessels, for example in eye angiography.¹² ICG may demarcate vascular tumors and extravagate in cases of elevated vascular permeability, however it has not shown highly specific cancer identification. Similar cancer detection performance has been observed with other nonspecific dyes, such as fluorescein. Alternatively, fluorescent molecules with targeting capacity to cancer moieties have been instead considered in recent years to increase the sensitivity and specificity of cancer detection.^13,14 Their ability to specifically target molecular signatures of cancer promises a great increase of efficiency and recognition rates also in colonoscopic screening.¹⁵ Importantly, the first clinical translation of a targeted agent was recently achieved for surgical guidance in open ovarian cancer surgery,¹⁶ possibly heralding the use of targeted fluorescent agents in endoscopic procedures as well. The study showed that targeted fluorochromes resulted in significantly higher detection of cancer over visual inspection, offering a promising outlook for surgical molecular guidance.

While the first fluorescence molecular imaging for surgical guidance was performed with a lens-based system developed for optical surgery applications, surgical guidance in CRC surgery would require the detection of color and fluorescence images and video during endoscopy. Acquisition of color reflectance and fluorescence information through a single imaging channel can be facilitated by using a filter wheel as demonstrated in Ref. 17, however, such a setup measures the two channels sequentially, thus limiting the frame-rate and reducing the clinical utility of the approach. Instead, methods that can detect simultaneously color and fluorescent signals offer improved utility; for example, endoscopes utilizing dichroic beam-splitters to separate near-infrared (NIR) and color images.¹⁸ However, the systems developed so far for endoscopic fluorescence imaging are either not adapted to clinical devices or cannot provide simultaneous color and high-sensitivity fluorescence measurements at video frame-rates. Clinical propagation of fluorescence molecular imaging will require the development of video-rate acquisition of color and fluorescence images, their processing and real-time display capability, which is essential for fast and easy surveillance during endoscopy. Video-rate color and fluorescence imaging can give the physician important feedback during biopsies and surgical intervention. However, a current limitation is that while most clinical endoscopes have been developed for color video in the visible, it is unclear if they are optimally suited for operation with fluorescence molecular probes developed for operation in the NIR. The use of the NIR for fluorescence molecular imaging provides important advantages as it allows the seamless separation of conventional color measurements and fluorescence measurements. Furthermore it offers operation under reduced tissue auto-fluorescence compared to the visible and allows for the detection of deep-seated signals, typically at depths of 5 to 10 mm.

In this work, we characterized and present herein the attenuation of conventional clinical endoscopes for the NIR and tabulate the differences observed. Based on this analysis we selected a clinical scope and developed a system for CRC surgery video-rate color/NIR fluorescence imaging capacity. To achieve high integration and seamless video acquisition from the visible and NIR we employed dedicated graphics processing unit (GPU) processing, which allows for alignment and image processing operations which improve the visualization accuracy. Furthermore, we characterized the sensitivity and signal-to-noise ratio (SNR) achieved as a function of frame rate. Finally we demonstrate the ability to detect cancer, based on targeted fluorescence signals in vivo, in animal models at video-rate performance. The characterized system has been found appropriate for clinical use as a next step.

2.

Methods

2.2.

Real-Time Intraoperative Imaging System

2.2.1.

System design

The imaging setup was designed to offer video-rate simultaneous color and NIR fluorescence endoscopy. The system employed the 10-mm-diameter laparoscope (26901-00, Dufner Instruments, Tuttlingen, Germany) with an aperture angle of 45 deg and 310 mm length. White light illumination for color imaging was provided by a 250 W halogen lamp (KL-2500 LCD, Schott AG, Mainz, Germany) while the fluorescence excitation was provided by a 750 nm CW laser diode (BWF2-750-0, B&W Tek, Newark, Delaware) with a maximum power of 300 mW. Both light sources were coupled into a multimode bifurcated fiber bundle (Leoni FiberOptics, Neuhaus-Schierschnitz, Germany) connected to the light guide of the laparoscope. The bundle was manufactured from 80 μm fibers to connect one white light source and up to three laser outputs to the scope. The optical signal was separated by a beam splitter (700DCXXR, AHF analysentechnik AG, Tübingen, Germany) into a visible and a NIR component and relayed to two CCD cameras by relay lenses. The color image was recorded by a 12-bit color camera (PCO AG, Kelheim, Germany) while the fluorescence image was relayed to an iXon 3 CCD (DU-888, Andor Technology, Belfast, Northern Ireland) that offers a quantum efficiency of over 75% in the NIR region. Shortpass filters (E700SP-2P, Chroma, Rockingham, Vermont) were placed in front of the white light source to minimize cross-talk of white light into the NIR channel. The fluorescence channel contained a bandpass filter (ET810/90, Chroma) for collecting fluorescence emitted in the 765 to 855 nm spectral band.

To implement real-time operation, both cameras were connected to a PCI acquisition card that transferred the data via direct memory access (DMA) into the computer’s RAM. The raw data are saved on a solid-state-drive (256 GB, SATA-III 6 Gb/s, Liteon, Taiwan) that can handle the high throughput of video-rate images with up to 1.3 MPixel and up to 14-bits. The images are then transferred to the GPU memory for postprocessing. Significant speedups have been reported for architectures using the GPU as a coprocessor ¹⁹ and, since many image processing operations can easily be parallelized, we opted for a GPU-based processing architecture, even though we think an implementation on the CPU would have been viable, too. The implemented pipeline for image transfer and processing is shown in Fig. 1(b). The fluorescence images are resized to the color channel resolution and registered by applying a projective transformation that is calculated from user-selected control points in an initial calibration step. The custom designed system software displays the coregistered color and fluorescence channel images as well as the composite overlay. Furthermore the interface offers easy control of all relevant system parameters and the possibility to capture both individual images and H.264/MPEG-4 recordings of the color and fluorescence channels and their composite overlay. Figure 1(c) shows the transmission over wavelength of the filters and dichroic employed in the setup. The fluorescence channel collects light mainly from the NIR passband, while the visible range is guided to the color channel.

Fig. 1

(a) Schematic of the intraoperative imaging system developed for laparoscopy. White light and laser are connected via a multimodal fiber to the laparoscope and used to illuminate the field of view. A dichroic beamsplitter separates the reflectance image into a color and near-infrared (NIR) fluorescence component, which are relayed to two CCD sensors. (b) Schematic of software pipeline. Camera data are acquired via direct memory access (DMA) and transferred to the graphics processing unit (GPU), where postprocessing is performed and the overlay is generated. Finally, the images are displayed in 8-bit resolution on the monitor. (c) Transmission spectra of the utilized emission bandpass, dichroic mirror, and shortpass filters.

2.2.2.

Optical characterization

The system’s spatial resolution was determined by a standard three-bar resolving test target (USAF 1951, NT53-714, Edmund Optics, Barrington, New Jersey). It consists of groups of three vertical and horizontal bars that make it possible to determine the contrast transfer function (CTF) from the contrast equation

Eq. (1)

$$\mathrm{CTF}(f)=\frac{I_{\text{max}}(f)-I_{\text{min}}(f)}{I_{\text{max}}(f)+I_{\text{min}}(f)}.$$

Thereby, $I_{\text{max}}$ and $I_{\text{min}}$ are the maximum and minimum intensity values of neighboring bar groups and $f$ denotes the spatial frequency of line pairs (lines/mm) for a group element. An element’s spatial frequency is standardized to be $2^{-1/6}$ times the frequency of the adjacent higher element. Color images of the target were taken at the laparoscopic working distance of 2 cm.

2.2.3.

Sensitivity measurements

The minimal detectable fluorochrome concentration was determined by measuring a dilution series of Alexa 750 (Invitrogen, Eugene, Oregon) in phosphate buffered saline (PBS). Starting at 1.56 μM, a total of 17 dilutions were generated. The concentration was halved at each step, thus the final phantom contained 0.02 nM of Alexa 750. A tube with 500 μL of the respective concentration was placed at a distance of 2 cm from the laparoscope tip so that the entire tube is in the field of view. White light and laser illumination were delivered solely through the laparoscope in order to include the attenuation of the laparoscope optics and the multimodal fiber in the measurement. The exposure time was chosen individually for each image in such a way, that at least 60% of the camera’s dynamic range is used. All other acquisition parameters remained constant. After each measurement a control tube containing PBS was measured with the same parameters and in the same position. Image statistics were calculated on a region of interest (ROI) that was selected for each measurement to represent the inside of the tube. The selected ROI corresponded to an active volume of about 40 μL, which contained the amount of 62.5 pmol of fluorochrome for the first phantom and 0.95 fmol for the final one. Dark images were acquired for the respective exposure times and subtracted from the fluorescence and control images. Therein the image average ranged from 390.9 to 392.0 and the standard deviation from 2.25 to 2.35 pixel counts. The SNR in dB was calculated as

Eq. (2)

$${\mathrm{SNR}}_{\mathrm{dB}}=20\log \left(\frac{\mathrm{S}}{\mathrm{RMSN}}\right).$$

Thereby, $S$ is the mean signal intensity of the raw fluorescence image within the ROI, while the root-mean-square noise (RMSN) is calculated as the RMS of the control image’s ROI. In this way the SNR is normalized for signal variations that are introduced by the PBS, reflections from the tube or inhomogeneous illumination. For a normal signal distribution a confidence level of 95% is achieved if the signal is two times higher than the noise and a 99.7% confidence for a signal that is three times higher. On the dB scale these values correspond to 6 and 9.5 dB, respectively.

A second experiment was conducted to characterize the imaging performance at video-rate conditions. Thereby, three representative concentrations of 500, 50, and 5 nM (20, 2, and 0.2 pmol, respectively in 40 μL) were imaged at fluorescence exposure times varying from 60 to 200 ms. The resulting image SNR was calculated as described above.

3.

Results

3.2.

Real-Time Intraoperative Imaging System

3.2.1.

Optical characterization

The spatial resolution for the laparoscopic imaging system was determined from the USAF 1951 test target image displayed in Fig. 2. By measuring the maximum and minimum pixel values within a bar group the CTF for the groups corresponding spatial frequency $f$ was calculated. This yields the transverse and lateral CTF for the color channel. The focusing of point objects through a circular aperture creates a diffraction pattern called the airy disk. The Rayleigh criterion defines resolution as the distance between two objects, where one objects diffraction pattern minimum overlaps with the others’ maximum. This corresponds to a contrast of 26.4% between the maximum and minimum intensity values. The resolution was determined by approximating the CTF with a linear regression line and applying the Rayleigh criterion. The transverse resolution was determined to be 49 μm while the lateral resolution was calculated as 54 μm.

Fig. 2

Color channel image of USAF 1951 standard resolution target taken with the laparoscopic imaging system under white light illumination.

3.2.2.

Sensitivity measurements

Seventeen images of the different fluorochrome concentrations as well as the corresponding control images were taken with exposure times ranging from 60 ms to 9.5 s. The calculated SNR for the entire series is plotted in Fig. 3(a). As expected, the SNR decreases with the fluorochrome concentration, even though the exposure time for each image was set to a value that used at least 60% of the dynamic range in order to avoid underexposure. The measurements were approximated by a linear regression line from which the minimum concentrations that can be detected with 95% and 99.7% confidence were determined. For the 9.5 dB criterion, the sensitivity limit was calculated as 0.8 nM (32 fmol in the 40 μL volume imaged), while an SNR of 6 dB is achieved for concentrations higher than 0.5 nM (20 fmol in 40 μL). These results correspond well to the visual evaluation of the fluorescence images.

Fig. 3

(a) Graph showing the signal-to-noise ratio (SNR) in dB over AF750 concentration (blue) and calculated regression line (red). (b) SNR over fluorescence camera exposure time for concentrations of 500 nM (blue), 50 nM (green), and 5 nM (red).

Figure 3(b) shows the SNR over fluorescence exposure time for concentrations of 500, 50, and 5 nM. This is an important measurement to understand the frame rate that can be achieved with the laparoscope, assuming different required sensitivity levels. Over the measured integration times the three concentrations exhibit a similar behavior, with the image quality decreasing more rapidly at low acquisition times. For the two higher concentrations a satisfactory SNR of over 20 dB can be achieved even for exposure times below 100 ms (blue and green scale). The lowest concentration of 5 nM yields an SNR of 7.4 dB at 100 ms exposure (red scale). Hence, increased detection accuracy, e.g., the 9.5 dB criterion used above, can only be achieved when the video frame rate is decreased below 10 fps.

3.3.

Tissue Imaging

The feasibility of video-rate color and NIR fluorescence endoscopy was showcased on induced colon tumors in mice. The epi-illumination images obtained from the colon of the mouse are shown in Fig. 4. This model showcases the system’s ability to image fluorescent markers with high resolution and high sensitivity in tissue.

Fig. 4

Laparascopic imaging of the mouse colon performed ex vivo in the abdomen (a)–(c) and on the excised colon (e)–(g). (a) and (e) Color channel images. (b) and (f) Fluorescence images of the IntegriSense 750 distribution within the colon. (c) and (g) Composite overlay image, where the fluorescence is thresholded and superimposed via alpha blending on the color image for better visualization. (d) Hematoxylin and eosin (H&E) stained “Swiss roll” of the colon. (h) Detail ($20\times$) of H&E staining.

The color images displayed in Fig. 4(a) and 4(e) directly relate to the surgeon’s vision, however, the lack of contrast between healthy and malignant tissue is apparent. The fluorescence images in Fig. 4(b) and 4(f) show a distinct accumulation of the NIR contrast agent in the colon. This fluorescence signal is thresholded and coregistered with the color channel in real-time and visualized as a transparent green overlay, as shown in Fig. 4(c) and 4(g). Histopathological examination using H&E staining was performed on the excised colon to confirm the malignant nature of the tissue. In addition to image snapshots, video recordings were acquired during the experiments, with a frame rate of 10 fps or more, to showcase the system’s capability for real-time imaging. The video-rate display of the molecular contrast provided by IntegriSense not only helps in localizing the tumor but can also give instant feedback during surgical resection.

4.

Discussion

Targeted fluorescent probes that operate in the NIR spectral range provide molecular contrast that can improve the endoscopic detection of tumor. Skeptics of the approach note the barriers of clinical translation of novel fluorescent agents with molecular specificity. Conversely, such translation was recently accomplished¹⁶ and more agents are expected now to undergo clinical translation in the near future. In addition, strategies that minimize the clinical translation risk have been also recently described.²⁸

Correspondingly, we presented in this work the development of a system that facilitates video-rate color and NIR fluorescence-based molecular imaging through a human laparoscope. A dual-optical-path approach was combined with highly sensitive fluorescence detection and GPU-based processing to facilitate video-rate sensing and rendering of low fluorescence signals and computationally intensive image processing tasks. Due to the computational ability of the GPU, images acquired from different cameras and CCD chips could be synchronized, aligned, scaled, and superimposed in real-time. The fluorescence distribution was then visualized in pseudo-color on a composite image. However, different visualization schemes may be implemented, including synchronized visualization of raw color and fluorescence video. Another reason we opted for the GPU as a coprocessor was the possibility to implement sophisticated fluorescence correction schemes in the future that reduce the influence of tissue optical properties on the signal.

Alternative methods for combining and optimizing white-light illumination and fluorescence excitation into the illumination port of the laparoscope could be considered. These could be coupled into a single guide light using conventional optics, specifically aspheric lenses for collimation followed by posterior refocusing. Alternatively, and in order to maintain the robustness of the bifurcated fiber approach considered here, fiber-optic or light cones or tapers could be incorporated to maximize the efficiency in the light coupling between guides of different cross-sections.

The development of this system included the characterization of clinical scopes as to their transmission characteristics in the NIR. Typically, most clinical scopes are optimized for operation in the visible and there is a characteristic lack of technical specifications for NIR operation. We found attenuation differences between scopes, reaching up to an order of magnitude for one-way transmission. This indicates up to two-orders of magnitude signal loss for collecting NIR fluorescence images and video, a suboptimal performance. Instead, color and NIR-dedicated scopes should be developed and employed in the design of clinical systems.

Of importance in this study was to examine the sensitivity achieved as a function of frame-rate and fluorochrome concentration under standard operation parameters. Despite the scope attenuation, concentrations in the subnanomolar range could be detected, owing to the use of cooled EMCCD camera detection with high NIR quantum efficiency to facilitate highly sensitive fluorescence measurements. Concentrations in the subnanomolar range, however, require slowing down the video frame rate at rates below 10 fps for high SNR detection. In an ideal case, increasing signal collection times would increase the SNR by the square root of the time allowed, however, the results indicate that the improvement gradually decreases for higher exposure times, showing that other factors such as readout electronics, gain, and quantization noise also play a role. System characterization further yielded 50 μm lateral and transverse resolution. Obviously, the resolution strongly depends on the imaging distance and the laparoscope’s optical path, thus different values would result for different endoscopic devices.

Applied to tissue measurements, the system was shown capable to detect physiologically relevant fluorescence signals from tumors when using targeted fluorochromes at frame-rates of 10 fps. The average SNR in the tissue measurements was approximately 21 dB, which was well above the detection limit of the camera. Clinical application of the imaging system will make high demands on sensitivity. However, by adapting the illumination intensity within ANSI limits and by possibly slowing down the video rate, even more sensitive detection could be achieved for areas of controversial signal appearance. Using the GPU processing power, such adaptive processes could be applied dynamically during the measurement.

The ability to simultaneously visualize color and fluorescence images in video mode can be beneficial during a surgical intervention in comparison with systems that switch between color and fluorescence measurements in a time-share mode. In addition, the use of a low-end camera may not be an optimal solution for high-performance fluorescence imaging. Therefore, a dual channel, dual CCD-chip system would be necessary for endoscopic fluorescence molecular imaging, for example in laparoscopic colorectal surgery.

Overall, NIR fluorescence endoscopy could significantly improve clinical practice in endoscopic procedures. Besides surgery, colonoscopy also suffers by high miss rates. Colonoscopy is recommended as a CRC screening method every 10 years for patients older than 50 years.^29,30 Screening strategies based on sigmoidoscopy and colonoscopy are cost-effective³¹ and commonly used³² procedures that are expected to reduce the incidence of CRC by about 80%.^33,34 However, they currently miss up to 22% of polyps or flat adenomata.³⁵ In contrast to surgical procedures which are invasive and high-risk, the administration of targeted agents in screening studies complicates the clinical burden and cost. Conversely, it was shown by van Dam et al.¹⁶ that a targeted fluorescent agent helped to identify almost five times more ovarian tumor deposits than visual observation under white light. If such performance was to be reproduced in CRC screening, it could possibly justify the use of fluorescence molecular imaging in screening protocols as well. This is particularly relevant in colonoscopy due to the long time interval allowed between screening sessions.

Fluorescent molecular agents against cancer and other diseases are a subject of strong ongoing research. The high fluorescence detection sensitivity that can be achieved, as showcased herein, enables imaging at subtherapeutic doses, which minimizes the risk to the patient.²⁸ The clinical use of targeted fluorochromes could shift the paradigm of endoscopic imaging from visual inspection to tumor biomarker-based detection. This approach can lead to a form of “real-time pathology” which, while not expected to substitute the postoperative pathological analysis, is foreseen as a powerful approach to enhance clinical decision making during surgery by molecular contrast.