1.

Introduction

Near-infrared (NIR) fluorescence imaging is becoming a prevalent tool for image guidance during surgery.^1–3 In these applications, fluorescent agents are introduced during surgery to enhance contrast of the operating field and to improve delineation of anatomical features. Indocyanine green (ICG) is an FDA-approved contrast agent commonly used in such applications. Approximate wavelengths of ICG absorption and emission are 780 and 820 nm respectively, a desirable spectral range minimizing interactions with biological tissue and maximizing imaging depths.⁴ In neurovascular surgery, ICG is often intravenously administered to evaluate blood flow and the vascular environment.⁵ ICG has also been used to better delineate cancerous from normal tissue due to extravasation of dye into tumor tissue.⁶

Certain surgical specialties, most notably neurosurgery, extensively use surgical microscopes—highly specialized stereomicroscopes installed on articulated mounts and providing a long working distance, multiuser access, and certain functional enhancements.^7–9 Implementation of NIR fluorescence imaging in these applications creates the added challenge of merging various imaging modalities within one format that is both familiar and beneficial to the neurosurgeons.

To facilitate some of the intraoperative NIR imaging techniques, fluorescence videoangiography can be integrated into surgical microscopes (e.g., Zeiss OPMI Pentero Infrared 800, Leica FL800) routinely used during complex vascular surgeries. These commercial microscopes feature a switchable design where the surgeon is required to switch from the fully optical bright-field view to the fully electronic projection of NIR fluorescence within the ocular or onto a liquid-crystal display monitor. The NIR image alone lacks the spatial cues which would normally help the surgeon identify anatomical points of reference and, therefore, the surgeon mentally coregisters fluorescence in the NIR image to the respective structure(s) in the bright-field view. Several research-stage devices aid in intraoperative visualization of NIR fluorescence.^10–13 However, none of these are capable of real-time display of the merged NIR and bright-field stereoscopic images.^14–16 The augmented microscope we recently introduced uses an optical technology to produce a simultaneous view of the surgical field (real object) and computer-processed NIR fluorescence (synthetic object) superimposed in real time.^17,18 Using a newly completed prototype of this device, we demonstrated fluorescence angiography with augmented microscopy enhancement (FAAME) in animal models.¹⁹ In cerebrovascular surgeries performed to resolve aneurysm and arteriovenous malformations, intraoperative fluorescence angiography is applied to assess blood flow and vessel patency. The augmented microscope was capable of producing a composite image in which fluorescence of ICG presented in false color helped localize vessels, branching points, and direction of flow under full white light illumination of the surgical field.¹⁹ With cerebrovascular surgeries routinely performed under specialized stereomicroscopes, this demonstration provides a relevant example of how the augmented microscope improves intraoperative diagnostic assessment within the standard configuration of the surgical stereomicroscope.

Here, we describe the detailed design of the augmented microscope, provide a set of performance tests using a tissue phantom and solutions of ICG, and provide examples of applications. The prototype includes augmentation in one of the two optical paths of the stereomicroscope. We used polyacrylamide gels with embedded polystyrene beads to represent optical properties of the white matter. We used capillary tubes to test various concentrations of ICG. Working under bright-field illumination representing a typical luminance range at the surgical field, we determined the augmented microscope’s ability to detect and display NIR fluorescence of ICG overlaid with bright-field images within the ocular of the microscope. Finally, we demonstrated applications of augmented microscopy for visualization of ICG in vivo and visualization of an NIR laser beam.

2.

Material and Methods

2.1.

Augmented Microscope Prototype Design

For demonstration of augmented microscopy, we modified a stereomicroscope of the common main objective, or the Galilean, type. This design, common in surgical microscopes, produces two parallel beams of collimated light so various optical modules, such as filters or beamsplitters, can be inserted into the optical pathway. Without loss of generality, we provided augmentation to one of the two optical paths of stereovision. The prototype utilizes an Olympus modular stereomicroscope (SZX7, Olympus, Center Valley, Pennsylvania). We designed a custom augmentation module to integrate the bright-field image with a processed image of NIR fluorescence within the optical path of the stereomicroscope (Fig. 1). The visible (bright-field) and NIR fluorescence images are acquired through the same optical system and overlaid in real time, ensuring correct spatial and temporal coregistration of the composite image presented to the user through the ocular of the microscope.

Fig. 1

Schematic of optical pathway in augmented microscope. Current prototype provides augmentation in the right ocular.

As shown in Fig. 1, NIR excitation is provided by a 160 mW 780 nm light-emitting diode (LED; M780L2, Thorlabs, Newton, New Jersey) mounted on the left optical path of the microscope. The collimated NIR light passes through a short-pass filter (793AESP, Omega, Brattleboro, Vermont), removing longer wavelengths produced by the LED, then reflected down the left optical path toward the operating field via a dichroic mirror (730DCSP, Chroma, Bellows Falls, Vermont). The microscope uses a $1\times$ NA 0.10 objective lens (DFPLAPO $1\times$, Olympus). NIR fluorescence signal is collected through the right optical path, where it passes through two $50/50$ beamsplitters, passes through an emission band-pass filter (840HBP50 EM, Omega), and, finally, is detected using a scientific complementary metal oxide semiconductor (CMOS) camera (ORCA-Flash4.0, Hamamatsu, Hamamatsu City). The CMOS camera is attached to the microscope via an adapter (U-TV0.5 XC-3, Olympus). The NIR fluorescence image is processed via computer software (HCImage, Hamamatsu) and projected back into the right ocular path using an $800\times 600\text{pixel}$ monochromatic green organic light emitting diode (OLED) (Green OLED-XL, eMagin, Bellevue, Washington) controlled by monocular interface and design kit (IDK) EMA-200002 (eMagin). The IDK board utilizes a microchip PIC16F876, and a reprogrammable CMOS flash microcontroller. The board acts as the interface between analog VGA ($640\times 480\mathrm{pixels}$) or SVGA ($800\times 600\mathrm{pixels}$) video signals (with RS 232 and composite ports also available) and the eMagin SVGA OLED microdisplay. The board automatically determines the frame rate, and provides user-controlled adjustments of image orientation and luminance in 255 steps. The OLED is coupled to the system using SM1 lens tube components (Thorlabs). A custom holder for the OLED was printed using a three-dimensional (3-D) printer (Objet350 Connex, Stratasys, Eden Prairie, Minnesota). This holder, placed at the far end of the lens tube, allows for $\pm 1.5\mathrm{mm}$ lateral adjustment to ensure image coregistration. Moving toward the microscope body, a plano-convex lens, diameter 25 mm, focal length 100 mm, is placed in the lens tube at approximately the focal length from the OLED array. The adjustable lens tube length enables formation of the OLED image at the intermediate image plane within the optical path of the microscope. Finally, a custom-made adaptor is used to mechanically couple the lens tube to the accessory port of the beamsplitter module. Image processing creates the false color image and corrects its spatial placement for coregistration. This monochromatic synthetic image representing NIR fluorescence is combined with the real (or bright field) image of the surgical field of view using the $50/50$ beamsplitter, yielding the composite image presented in the right ocular. The monochromatic OLED provides for high image intensity and desirable overlay color. Bright-field illumination of the surgical field is provided by a metal halide-based ring illuminator (Fiber Lite MH-100, Dolan-Jenner Industries, Boxborough, Massachusetts) with an NIR blocking short-pass filter (SP OD4 675 nm, Edmund Optics, Barrington, New Jersey), and a set of LED lamps with 800 lm output, 86 CRI, 4000K (16P30S/840FL32, Toshiba, Irvine, California), representative of lighting conditions within the operating room. Each ocular is fitted with short-pass filters (E700SP, Chroma) to protect the user from transmitted NIR light. The microscope has been fitted onto a Zeiss OPMI surgical microscope stand (Fig. 2). Visible, NIR, and augmented images are also displayed on the two high-definition monitors. The total weight of the microscope head (without the stand) before modifications required for augmentation of images is 2.6 kg. With all the accessories added, the weight of the microscope head is 4.8 kg. For comparison, Zeiss Pentero allows for the total weight of accessories up to 6 kg.

Fig. 2

Augmented microscope system design: (a) on a stand with extendable arm and motorized telescoping base; (b) close-up view of augmentation module.

2.5.

Fluorescence Imaging Sensitivity

The detection limit of ICG was evaluated using two methods: (1) directly, that is, by imaging ICG serial dilutions in capillary tubes placed in the direct view of the imaging system; and (2) through the gel, that is, by imaging ICG in capillary tubes placed under tissue phantoms. Using ImageJ, regions of interests were manually drawn around the capillary tube and compared to a control background signal captured at the beginning of experimentation. Signal-to-noise ratio (SNR) was calculated as $\mathrm{SNR}=({\mu }_{\text{signal}}-{\mu }_{\text{background}}/{\sigma }_{\text{background}})$, where ${\mu }_{\text{signal}}$ and ${\mu }_{\text{background}}$ is the mean of the signal and background, respectively, and ${\sigma }_{\text{background}}$ is the standard deviation of the background. Images were subjectively evaluated by the user and objectively with MATLAB using a standard edge-detection algorithm utilizing the Canny filter for finding the edge of the capillary tube.

2.5.1.

Indocyanine green dilutions

Twenty-five serial dilutions of ICG in two-fold increments were prepared in capillary tubes, ranging in concentration from 6.19 mM to 369 pM. Each tube was individually imaged to determine the concentration that produced the maximal fluorescence signal intensity. This concentration was used for the following tissue phantom experiments.

2.5.2.

Indocyanine green imaging under tissue phantoms

A 1-mm diameter capillary tube filled with ICG was embedded between layers of polyacrylamide gels. A 5-mm gel was placed beneath the tube and various layers of gels were stacked on top to make up the appropriate thickness $\mathrm{\Delta }Y$ (Fig. 3). ICG was imaged through the tissue phantoms at a constant concentration of $96.8\mu \mathrm{M}$ and at varying gel thicknesses ($\mathrm{\Delta }Y=0–10\mathrm{mm}$).

Fig. 3

Polyacrylamide gels were layered to resemble various tissue depths. $G0$ indicates the 5-mm base gel below the 1-mm diameter capillary tube (T). $G1$ through Gn indicates the various layers of gels that were stacked to increase imaging depth from 0 to 10 mm ($\mathrm{\Delta }Y$).

3.

Results

3.1.

Augmented Microscope Performance

3.1.1.

Microscope resolution

Using the air force target, the augmented microscope optical system was capable of resolving group 7, element 6 with full magnification at $56\times$, equivalent to 228 line pairs per millimeter; and group 5, element 4 at $8\times$ magnification, equivalent to 45 line pairs per millimeter.

3.1.2.

Serial dilutions in capillary tubes

Maximal fluorescence intensity from the capillary tube appeared at $96.8\mu \mathrm{M}$ ICG concentration. At higher concentrations, the fluorescence intensity decreased, which we attributed to self-quenching. We observed a linear increase in fluorescence emission versus excitation power when imaging $96.8\mu \mathrm{M}$ solution with no evidence of photobleaching. We were able to demonstrate a linear relationship with fluorescence intensity and concentration between 189 nM and $6.05\mu \mathrm{M}$ [$R^2=0.996$; Fig. 4(a)]. The minimum detectable concentration of ICG imaged directly (method 1, Sec. 2.5) was 189 nM under direct observation. Fluorescence from the capillary tube was not visually detectable below an SNR of 2 [Fig. 4(b)]. The MATLAB algorithm was not able to find any capillary tube edge below an SNR of 2.

Fig. 4

Serial dilutions in capillary tubes: (a) indocyanine green (ICG; $11.8\mathrm{nM}–6.05\mu \mathrm{M}$) in capillary tubes on a log–log scale. Each capillary tube was independently imaged. The graph shows the linear range of ICG fluorescence emission versus concentration. Dotted line represents signal-to-noise ratio (SNR) of 2. Inset shows all 25 dilutions on a log scale. (b) Image collage of ICG filled capillary tubes which were imaged directly ($\mathrm{\Delta }Y=0\mathrm{mm}$). Images shown: visible (top), near-infrared (NIR; middle), and augmented (bottom).

3.1.3.

Detection of indocyanine green through tissue phantom

When imaging through the gel (method 2, Sec. 2.5), the augmented microscope was able to detect fluorescence emission of the ICG filled capillary tube ($96.8\mu \mathrm{M}$) through 10 mm of tissue phantom [Fig. 5(a)]. However, with the increasing thickness of the tissue phantom, the edges of the capillary tube are increasingly diffuse and the tube becomes indiscernible at 3 mm and greater thicknesses of tissue phantom [Fig. 5(b)], both subjectively and using the MATLAB algorithm.

Fig. 5

Detection of ICG through tissue phantom: (a) constant ICG concentration ($96.8\mu \mathrm{M}$) imaged at increasing depths within tissue phantoms (0 mm–10 mm). Dotted line represents detection limit threshold at $\mathrm{SNR}=2$. (b) Image collage of ICG filled capillary tubes which were imaged through the gel ($\mathrm{\Delta }Y=1–4\mathrm{mm}$). Images shown: NIR (top) and augmented (bottom).

3.2.

Demonstration of Augmented Imaging

3.2.1.

Color balance with augmented imaging

A simple illustration of augmented microscopy is provided in Fig. 6. Images shown here represent the eyepiece view of the augmented microscope operating in (a) bright field, (b) NIR fluorescence, and (c) augmented configurations, producing real, synthetic, and composite images, respectively. In the augmented configuration, NIR fluorescence, not visible to the operator, is converted into green signal and overlaid with the bright-field image. White rings of light seen in some of the wells in Figs. 6(a) and 6(c) are reflections of the ring illuminator from the surface of the aqueous solution of the dye.

Fig. 6

Custom colored dye solutions with randomly plated ICG solution: (a) visible view through microscope, (b) NIR view seen on computer monitor, and (c) augmented view seen in real time through the ocular of the augmented microscope.

3.2.2.

Near infrared laser beam

An 820-nm NIR laser beam was introduced into a handheld laser ablation wand and imaged with the microscope (Fig. 7). It can be seen that without NIR augmentation of the images, one cannot visualize the laser beam or the visible environment in the visible (a) and NIR (b) images, respectively. The augmented image (c) allows for complete visualization of treatment beam and visible environment.

Fig. 7

NIR laser light delivered through handheld laser wand for demonstration of NIR laser beam guidance by augmented microscope: (a) visible image conceals location of NIR laser beam, (b) NIR image lacks any spatial information, and (c) augmented image visualizes both NIR light and visible spatial ques.

3.2.3.

In vivo imaging

We demonstrated image-guided surgery in anesthetized rats using the augmented microscope. First, the surgeon exposed the left carotid artery under the microscope operating in the bright-field mode. The working distance of 81 mm with $8\times$ magnification provided a field of view of 25 mm across, which is appropriate for this procedure. The synthetic channel was activated to generate superposition of bright field and NIR fluorescence. The ICG was injected and a bright green signal appeared in areas filled with ICG solution (Fig. 8 and Video 1). Examples of images generated in real time are shown in Fig. 8. The surgeon has the ability to control the brightness of the synthetic image (NIR fluorescence color coded in green) added on top of the real image (bright field). The image in Fig. 8(c) exemplifies a balance of these two. The composite image illustrated there is generated at 20 fps, enabling a video rate monitoring of ICG distribution. We estimate that with an NIR sensor format of $512\times 512\text{pixel}$ and 25 mm field of view corresponding to $8\times$ magnification, blood vessels as small as $50\mu \mathrm{m}$ in diameter can be visualized, and $7\mu \mathrm{m}$ at $56\times$ magnification (i.e., human arterials are on the order of tens of micrometers). This spatial resolution is approaching the limit of optical resolution of the system operating at 820 nm with the NA 0.1 objective lens.

Fig. 8

Left rat carotid artery: (a) visible image, (b) NIR image, and (c) augmented image (also shown in Video 1). Ruler increments are 1 mm. Spatial differences between simultaneously acquired NIR and augmented images are a result of capturing these images at different microscope ports. ICG injection into rat model and visualization of carotid artery (Video 1 MOV, 12 MB) [URL: http://dx.doi.org/10.1117/1.JBO.20.10.106002.1].

4.

Discussion

We described the practical realization of augmented microscopy, where bright-field images are superimposed with electronically processed images of NIR fluorescence of ICG within the optical path of the stereomicroscope. This augmented microscope is designed to produce such composite images in real time, under illumination conditions typically encountered in the operating room. Operating under luminance of 100,000 lx provided by a metal halide lamp and auxiliary LED lamps, and capturing NIR fluorescence at 20 Hz frame rate, the microscope can detect ICG concentrations down to 189 nM. Typical dose given to a patient is at least $3\mu \mathrm{M}$. We observed a linear relationship between the fluorescence intensity and ICG concentration within the range of $189\mathrm{nM}–6.05\mu \mathrm{M}$ ($R^2=0.996$), with self-quenching becoming evident at higher concentrations. The concentration that produced maximal fluorescence emission ($96.8\mu \mathrm{M}$) is different than that reported in the literature.²² This may be attributed to differences in experimental conditions as well as differences in specific serum albumin contributions to the fluorescence intensities observed by various authors. Even at higher concentrations of ICG, $194\mu \mathrm{M}$ and $96.8\mu \mathrm{M}$, we observed a linear increase in fluorescence emission versus excitation power with no evidence of photobleaching.

The SNR appeared to be lower for 0 mm gel depth compared to 1 mm gel depth. At 0 mm, the capillary tube is partially exposed at the air/gel interface, and this interface may in fact enhance forward and lateral scatter from the tube and gel with concomitant reduction in SNR, as observed in Fig. 5.

Fluorescence images observed using the augmented microscope appear sharp and well defined (Figs. 4 and 8). This ensures accurate visualization of superficial anatomical structures. The ability to detect signal at greater depths within tissues may be beneficial for observing hidden structures of importance. At the same time, however, fluorescence from within a greater depth may contribute to diffuse images, displaying superimposed layers of fluorescence originating from various depths, similar to those shown in Fig. 5. Extracting useful information from such images may require further development of the instrument, including structured illumination or image processing. In its current realization, the augmented microscope was able to display augmented fluorescence images down to approximately 3 mm. Beyond this depth, structures are no longer discernable and fluorescence signal is diffused producing no useful anatomical information.

The novelty of the presented augmented microscope is threefold: (1) NIR augmentation of bright-field images is provided within the optical path of a conventional stereomicroscope via an add-on module; (2) by utilizing the optical path of the stereomicroscope, we maintain full 3-D stereoscopic vision, lost in fully digital display systems; and (3) we retain the imaging environment familiar to surgeons, including integral features of surgical microscopes such as real-time magnification and focus adjustments, camera mounting, and multiuser access.

The optical stereomicroscope remains a standard tool in neurosurgery and other surgical procedures. It retains all cues critical for 3-D perception: binocular disparity, motion parallax, convergence, and accommodative distance. A microscope-based intraoperative imaging system provides control of the magnification and the field of view while maintaining uninterrupted spatial coregistration and multiviewer access as required in many surgical applications of microscopes.

The concept of the augmented stereomicroscope described here should be contrasted with stereoscopic rendition systems based on two cameras, often found in computer vision systems, including the da Vinci robotic system (Intuitive Surgical, Sunnyvale, California). Such electronic systems simplify image processing and overlays. In the implementation presented here, the microscope retains the full stereoscopic capability, with the bright-field (real) image being presented in both oculars. The false-colored NIR fluorescence (synthetic) image appears in only one ocular where it is superimposed with the matching channel of the bright-field image. Therefore, the primary cues for 3-D vision, convergence, and accommodation depth are completely defined by the bright-field image. We believe that this solution reduces strain and fatigue associated with fully electronic stereoscopic displays, such as that used in the da Vinci Surgical Systems, where the accommodative distance and the convergence angle are decoupled and are pointing to different spatial locations.²³ Such distorted, even conflicting, spatial relations often contribute to fatigue and strain as well as subpar outcomes in certain performance tests.^24,25 Also, our system will likely reduce the strain and fatigue in comparison to systems where the left and right oculars separately present the real and synthetic images, with no clear cues for spatial coregistration of these two types of images, such as the dual imaging color module (Leica Microsystems, AG). We envision that a fully developed augmented microscope will generate a stereoscopic composite image in both oculars, using separate NIR fluorescence images captured from the left and right optical paths of the microscope, providing complete and correct spatial cues for 3-D vision, therefore significantly reducing strain and fatigue.

Developments of intraoperative NIR fluorescence imaging techniques may enable surgical manipulation while simultaneously monitoring distribution of NIR contrast agents. The augmented microscope described here uniquely addresses this need in many situations routinely requiring the use of specialized surgical microscopes, such as neurological, cerebrovascular, otolaryngological, and ophthalmic procedures. As such, it complements the ever-increasing range of superb commercial and research devices relying on the synthetic (predominantly false-colored NIR fluorescence) images projected on auxiliary monitors,^26–28 wearable see-through display,^29,30 or dynamic optical projection onto the surgical field³¹ introduced recently and excelling in their own applicable medical specialties.

Procedures that traditionally are performed using surgical microscopes will likely benefit from image augmentation as we described in this report. In one example, we demonstrated FAAME, a technique in which the augmented microscope is employed to improve the performance of ICG videoangiography, to directly visualize the relationship between angiographic data and the tissue.¹⁹ ICG videoangiography has already become a standard element of vascular neurosurgeries.³² Therefore, the surgeon reviews the black and white angiography video loops displayed on a monitor to assess aneurysm and vessel patency, vascular architecture within the surgical field, obliteration status of arteriovenous pathology, direction of blood flow, and assessment of bypass grafts and collateral blood flow.³³ FAAME merges existing imaging technologies to simultaneously present information available from an intraoperative angiogram within the ocular of the microscope, while maintaining full functionality of the surgical microscope.¹⁹

Another area of potential application of the augmented microscope is in surgical treatment of brain tumors. Current treatment modalities include surgical resection of tumors and their adjacent tissue, which often contains infiltrative neoplastic cells.^34,35 Aggressive resection is associated with the risk of removing normal brain tissue and impairing functions of the patient. On the other hand, incomplete resection of tumor results in its immediate relapse in 90% of patients.³³ Intraoperative NIR imaging may aid in resection of these challenging tumors.^19,26–39 The long-term interest in the intraoperative NIR fluorescence visualization is driven by the development of new agents capable of improved delineation of brain tumor margins. Examples of dyes include various derivatives of NIR fluorescent dye currently undergoing preclinical or clinical evaluations, such as BLZ-100, a chlorotoxin-ICG conjugate;^36,37 CLR1502, an alkylphopholine derivative of a cyanine dye;^40,41 anti-EGFR antibody fragments conjugated to IRDye 800 CW;^42,43 and others.⁴⁴ We believe that the ability to simultaneously visualize the surgical field and contrast agent within the augmented microscope will enable broad implementation of image guidance in surgeries.

The presented imaging system development is concerned with processing and display of NIR images, such as those obtained from fluorescence emission. Because scattering and absorptive properties of the tissue obscure fluorescence signal, various forms of data corrections may be required to yield quantitative diagnostic data related to the true distribution of NIR contrast agents in the tissue. Examples of such processing methods are discussed in the literature.^45–47 These are generally based on some form of processing fluorescence and reflectance data, and may require acquisition of spectrally resolved images. Because the visible and NIR channels are separated in the presented design, processing images transmitted in the visible channel, 350 to 750 nm, would not affect functionality of augmentation; however, implementation of such processing may require further testing in the augmented microscope. The image augmentation we describe allows for seamless integration of such quantitative fluorescence images with the optical system of the microscope. The system has the ability to create image-processing scripts, such as arbitrary thresholding of live data before projecting the synthetic images through the augmentation module. With the possible implementation of color OLED, the thresholded data can also be color-coded in a user defined scheme. We envision that display of such quantitative fluorescence images could benefit from the color OLED display (in place of green monochrome OLED of the current microscope) to help highlight diagnostically relevant information.

Future development of the augmented microscope will focus on introducing stereoscopic augmentation, whereby both channels of the stereoscopic microscope will produce the composite of real and synthetic images, with proper coregistration of left and right ocular images. In addition, we will explore applications of the augmented microscope in guiding laser surgery using an NIR laser beam, such as Ti:Sapph and Nd:YAG,^48,49 as well as for superimposing other types of information, such as prerecorded magnetic resonance imaging images or single-point data readouts.