2.1.

Neurosurgery

Neurosurgery is the branch of medicine that treats disorders of the central nervous system (CNS) or peripheral nervous system (PNS) by physical manipulation, modification, or modulation of anatomical (e.g., the subthalamic nucleus for deep brain stimulation) and pathological (e.g., aneurysm clipping and resection of brain tumors) structures.^33–35 In terms of research and clinical techniques, neurosurgery is among the most rapidly developing subspecialties of medicine,³⁶ propelled by the interdisciplinary integration of tools from imaging, molecular biology, cancer neuroscience, electrophysiology, brain mapping, neuroengineering, computational biology, bioinformatics, and robotics. Clinically, neurosurgery is composed of the following subspecialties:

“Neurosurgical oncology” is the surgical branch of neuro-oncology focused on the diagnosis, treatment, and long-term management of tumors of the CNS and PNS. Surgical resection is the primary course of treatment for a large set of tumors. The success of tumor resection is one of the most important initial predictors of overall survival and quality of life.^37,38 Therefore, the goal of tumor surgery is to maximize the extent of tumor resection (EOR) while preserving the functional brain to ensure high post-operative functional outcomes (i.e., achieving an oncofunctional balance^39–44). However, rates of EOR can be as low as 30% as reported by post-operative,⁴⁵ standard-of-care magnetic resonance imaging (MRI) using conventional surgical techniques.

Conventional resections are performed under white light illumination with or without magnification (e.g., using microscopes or surgical loupes). In these procedures, the surgeon uses the cues from visual white light and tactile feedback to determine which tissue to resect and which to preserve.⁴¹ However, because brain tumors often appear visually similar to normal brain tissue, residual tumors often remain unresected, leading to low rates of maximal EOR. This is especially problematic in infiltrative areas of the most aggressive malignant tumors, such as glioblastomas (GBMs).⁴¹ Surgical adjuncts such as intraoperative MRI (iMRI), intraoperative ultrasound (US), and neuronavigation can improve visualization and intraoperative surgical decision-making. Despite their benefits, these tools have limitations including disruption of the surgical workflow, inaccurate spatial information due to brain shift, low contrast (normal tissue versus pathology), and high costs.⁴⁶ Therefore, there is an acute need for real-time, high-resolution technologies that accurately delineate tumors from normal brain tissue in neurosurgical oncology.^47–52

“Vascular neurosurgery” is the branch of neurosurgery focused on the diagnosis and surgical treatment of blood vessel pathologies of the nervous system.³³ This encompasses a variety of conditions including aneurysms, arteriovenous malformations (AVMs), stroke, and hemorrhage. The primary aims of surgical treatments include restoring normal blood flow to the brain, preventing blood clot formation and stroke, repairing vascular pathologies (e.g., aneurysms and fistulas), and resecting vascular lesions (e.g., AVMs and cavernomas). Given that the spatial scale of vascular structures in the nervous system is of the order of millimeters, submillimeter precision and real-time intraoperative feedback are critical to safely treat pathologies while preserving normal vasculature. Although intraoperative three-dimensional (3D) digital subtraction angiography provides visualization of the neurovasculature in 3D as well as differentiates its venous and arterial components,⁵³ it does not provide direct intraoperative visualization of vasculature and pathology at the tissue level. Intraoperative Doppler US can detect blood flow,⁵⁴ but it is constrained in resolution (i.e., millimeters) and field of view (i.e., single point detection) and is sensitive to patient motion. Intraoperative indocyanine green (ICG) fluorescence angiography provides real-time intraoperative feedback with surface visualization of vasculature using ICG fluorescence, which accumulates in the blood vessels.⁵⁵ However, visualization of vasculature and pathologies is transient (i.e., ICG signal washes out shortly after administration), is useful only for surface imaging, and is not specific to pathologies as it accumulates in all normal and abnormal vasculature.⁵⁶ Therefore, there is an acute need for real-time, non-transient, and highly specific intraoperative imaging technologies that can distinguish between normal and pathological neurovasculature for visual feedback in vascular neurosurgery.

“Functional neurosurgery” is the surgical branch of neurosurgery that treats various chronic neurologic disorders of the brain through functional modification. These disorders include epilepsy, movement disorders, pain, spasticity, and psychiatric illnesses.³³ One example of functional neurosurgery is the treatment of intractable epilepsy via surgically resecting the epileptogenic area, which is the area of the brain where seizures are believed to originate. The goal of this surgery is to eliminate or decrease the frequency and severity of seizures.⁵⁷ In epilepsy surgery it is important to map out the affected area of the brain, typically with intraoperative electrocorticography (ECoG).⁵⁸ During this procedure, a grid of electrodes is placed on the cortex to measure electrical activity and identify regions with abnormal signals that might indicate seizure origin. However, intraoperative ECoG interrupts the surgical workflow by requiring electrode placement, signal measurement, signal interpretation, electrode removal, and co-registration of electrode locations with signal origins on the brain. In addition, recordings can take a few minutes to complete and interpret. The resolution of ECoG is dependent on the intrinsic spacing within the electrode array, with spatial resolutions of up to a centimeter using conventional grids. There is also a risk of infection associated with the use of such an electrode array with long-term monitoring. As such, imaging techniques that provide visualization of the epileptogenic regions would enable real-time feedback and ideally more accurate identification of the seizure-causing regions. Overall, there is a need for imaging technologies that provide functional neurosurgeons with real-time and highly specific identification of normal and abnormal functions in the nervous system.

“Spine surgery” is the surgical branch of neurosurgery that treats disorders affecting the spinal cord.³³ Spine surgery can address issues such as spinal deformity, nerve compression, pain, and neurological deficits due to disorders of the spinal cord and nerves. Surgical navigation has become critical in spine surgery to perform accurate manipulation of bony structures while preventing damage to the spinal cord and its surrounding neural elements. Such navigation is typically done with fiducial markers placed on the skin and spine, but these can get obscured, deformed, or displaced during surgery,⁵⁹ compromising accurate real-time guidance. It is therefore clear that to enhance the accuracy and safety of spine surgery, there is a pressing need for non-invasive real-time tracking systems and algorithms. These advanced technologies will provide better guidance during surgical procedures, ensuring more effective treatment of spinal disorders and improved patient outcomes.

“Other subspecialties” of neurosurgery include trauma and peripheral nerve surgery. However, there has been no clinical work with HSI in these subspecialties, so we will not discuss them here.

2.2.

Hyperspectral Imaging

HSI is the acquisition of high-resolution spectra over a wide field of view. HSI allows for capturing a 3D hyperspectral cube of size $H\times W\times N$, where $H$ and $W$ are the height and width of images in the cube, respectively, and $N$ is the number of wavelength channels [Fig. 1(a)]. The value of $N$ roughly distinguishes it from multispectral imaging, a spectrally resolved imaging paradigm that uses fewer, broader spectral bins. Here, we define a multispectral system to have less than 10 wavelength channels ($N<10$) and a hyperspectral system to have more than 10 ($N>10$). Each $H\times W$ channel in the cube is equivalent to a two-dimensional (2D) image that would be captured by placing an appropriate bandpass spectral filter in front of the camera. Capturing spectral data in addition to spatial information can be used to determine the composition of the contents of the imaged scene.^31,32,60 An in-depth review of the construction and properties of such systems can be found in the literature,^31,32 and we discuss only the essentials here. HSI technologies relevant to neurosurgery and their general specifications are illustrated in Fig. 1. Acquisition of a 3D hyperspectral image cube with a 2D camera sensor, however, is not straightforward. Thus, several techniques for the capture of hyperspectral image cubes have been developed, each with its own unique advantages and pitfalls.^61,62

Fig. 1

Hyperspectral imaging technologies used in neurosurgery. (a) Hyperspectral image cube is an array of size $W\times H\times N$, where $W$ and $H$ are the width and height, respectively, of images in the cube along the $x$ and $y$ spatial dimensions, and $N$ is the number of wavelength channels along the $\lambda$ dimension. Each $W\times H$ channel in the cube is equivalent to an image that would be captured by placing an appropriate bandpass spectral filter in front of the camera. (b) Point scanning methods acquire a complete spectrum at a single $(x,y)$ pixel coordinate (i.e., “point”), scanning along the $x$ and $y$ spatial dimensions to reconstruct the full 3D hyperspectral cube. (c) Line scanning methods acquire 2D data of size $W\times N$ along one $x$ spatial dimension, scanning along the $y$ spatial dimension (i.e., “line”) to reconstruct the full 3D hyperspectral cube. (d) Spectral scanning methods acquire 2D data images of size $W\times H$ at one $\lambda$ wavelength channel, scanning along the $\lambda$ wavelength dimension (i.e., “spectral”) to reconstruct the full 3D hyperspectral cube. (e) Snapshot methods acquire the full 3D hyperspectral image cube of size $W\times H\times N$ with each single acquisition (i.e., “snapshot”).

“Point scanning methods” (also referred to as whiskbroom scanners) operate using a single detector or a small array of detectors to sequentially scan the scene, capturing spectral data pixel by pixel. Although this method provides high spectral resolution, the point-scanning approach needs $M=HW$ acquisitions, which for megapixel-sized images is time-consuming and limits their use to imaging static scenes and/or small fields of view [Fig. 1(b)].

“Line scanning methods” (also referred to as pushbroom scanners) encode spectral data in one spatial dimension, allowing parallel measurement of the other spatial dimension. Typically, these methods use a linear array of detectors aligned perpendicular to the scanning direction (say, along the $H$ dimension), capturing spectral data row by row. This approach reduces the number of acquisitions to $M=W$, which significantly reduces acquisition time compared with point scanners. However, the acquisition of thousands of line scans still comes at a high time cost. These are the most widely available systems^63–66 used abundantly in HSI applications [Fig. 1(c)].

“Spectral scanning methods” image one spectral channel (i.e., one waveband) in the hyperspectral cube at a time and employ a tunable bandpass spectral filter to capture sequentially 2D images at each spectral channel. Spectral scanners offer the flexibility to acquire cubes over a programmable set of wavelengths with selectable spectral resolution. High-spectral-resolution cubes come at a high time cost, especially when considering their use in the dynamic, fast-paced surgical setting. Typical tunable filters used are liquid crystal tunable filters (LCTFs)⁶⁷ and acousto-optic tunable filters⁶⁸ [Fig. 1(d)].

“Snapshot methods”^69–71 capture a hyperspectral cube with complete spatial and spectral information in a single exposure. Snapshot acquisition is achieved by space division multiplexing of the sensor over the spatial and spectral dimensions, similar to a plenoptic camera.⁷² In this approach, the sensor area is distributed over a number of parts equal to the number of spectral channels. Each of these parts images a wide-field image corresponding to one spectral channel, and these parts are stacked together to form the hyperspectral cube. This technology is facilitated by new optical designs incorporating lenslet arrays^70,71,73,74 and varying filtering and dispersion strategies. This rapid acquisition enables the use of snapshot systems in applications requiring real-time hyperspectral feedback, such as in intraoperative image guidance, where long scan times or bulky scanning hardware can interfere with the surgical workflow. However, space division multiplexing requires a trade-off between spatial and spectral resolutions for equivalent acquisition times—as we increase the number of parts, the sensor is segmented into fewer pixels available for each part [Fig. 1(e)].

“Snapscan systems” combine the benefits of snapshot and line scanning hyperspectral systems. Such systems are built with mosaic filter arrays as in snapshot systems but employ internal scanning of the mosaic and computational reconstructions to yield fast, high-resolution hyperspectral cubes.⁷⁵

“Compressed sensing methods” exploit the regularity in natural signals to obtain an approximation to the hyperspectral cube.⁷⁶ An example of such regularity is the sparsity of individual spectral channels in the spatial frequency domain, which is the subject of a classic signal processing technique called compressed sensing. Such systems have the capability to provide video-rate hyperspectral acquisition with high spatial resolution for scenes that follow its assumptions.⁷⁷ In addition, such methods can also implement programmable spectral filters⁷⁸ in addition to bandpass filters, which allow for matched filtering of spectral signals for classification and segmentation applications.

2.3.

Benefits of HSI in Neurosurgery

As mentioned before, the spectrum in one pixel of the hyperspectral cube contains the optical signature or “optical fingerprint” of the imaged scene point at that spatial coordinate (Fig. 2). This fingerprint can include fluorophores [e.g., protoporphyrin IX (PpIX)] and/or chromophores (e.g., oxy- and deoxyhemoglobin) that differentially accumulate in tissues. This fingerprint is representative of the tissue composition of the imaged scene point—typically, bulk brain tissue, arterial blood vessels, venous blood vessels, various types of tumors, and background. HSI is particularly useful when classifying these kinds of tissue because reflectance and fluorescence spectra obtained with the hyperspectral cubes have high discriminative power that has been widely characterized.^79–83

Fig. 2

Spectra of fluorophores, chromophores, and reflectance in the visible to near-infrared (NIR) used in HSI for neurosurgery. HSI in neurosurgery has used both exogenous agents, such as 5-aminolevulinic acid that leads to the production of protoporphyrin IX (PpIX), and ICG as key fluorescence biomarkers in fluorescence guided surgery (FGS), with their fluorescence spectra shown in black. Other endogenous fluorophores (e.g., FAD, NADH) are shown in blue, and PpIX photoproducts as well as tissue reflectance and chromophores (e.g., oxy- and deoxyhemoglobin) are shown in red. The $y$-axis shows the intensity of fluorescence emission, reflectance, or absorption in arbitrary units, and the $x$-axis shows the wavelength $\lambda$, in nanometers.

As an example of this high discriminative power in the context of vascular neurosurgery, consider a pixel consisting of a blood vessel. The main chromophores involved in the reflectance spectrum of this pixel are oxyhemoglobin and deoxyhemoglobin. The reflectance spectra of deoxyhemoglobin and oxyhemoglobin, which are equal at 545 nm, change rapidly in opposite directions between 545 and 560 nm. Therefore, spectrally resolved imaging in the visible range of the spectrum allows for highly accurate estimates of the relative concentrations of deoxyhemoglobin and oxyhemoglobin, allowing optical measurements of oxygen saturation.

In addition to pixel-wise classification of tissue constituents, hyperspectral data enable other kinds of optical characterization across the surgical field of view. The rich data encoded in each hyperspectral cube offer the potential to extract optical features that would otherwise be impossible to detect visually with the naked eye or with a conventional color image.^67,84 For example, spectrally resolved wide-field data have been shown to correct for the distorting effects of tissue optical properties on emitted fluorescence signals,⁸⁵ which opens the possibility for using HSI to evaluate the surgical field of view and provide quantitative, objective measures of fluorescence and therefore absolute fluorophore molar concentrations.⁶⁷

Putting all these capabilities together with modern acquisition techniques from optics and computational imaging, advances in computational methods and hardware, and segmentation and classification with artificial intelligence,⁸⁶ HSI has the potential to be a powerful tool for real-time intraoperative guidance.

2.4.

Challenges in Current Neurosurgical HSI Approaches

Translating an optical system for clinical use into the neurosurgical operating room presents unique challenges not encountered in traditional benchtop imaging settings for pre-clinical studies⁸⁷ (Fig. 3). The fundamental principle for translation of a novel HSI system into the operating room is that any system and imaging process must not significantly interfere with or interrupt the neurosurgical workflow; it should enable ease of integration, safety, and efficiency for dynamic intraoperative use. A major practical consideration is the size of the imaging system. The spatial footprint of the optical setup must be as small as possible to seamlessly integrate and “fit” into the already instrument-dense neurosurgical operating room (consisting of, for example, the surgical microscope, US imager, ultrasonic aspirator, neuronavigation, drill, and suction control).

Fig. 3

HSI systems in neurosurgery. (a) HELICoiD system uses an exoscope with two line-scan hyperspectral cameras mounted in a confocal configuration. The HELICoiD system fits within a $60\times 60\times 90\mathrm{cm}$ bounding and requires removing the surgical microscope for acquisition, thus interrupting the surgical workflow. (b) Small footprint handheld HSI snapshot system does not require removing the surgical microscope but does not provide the same field of view as seen from the surgeon’s oculars. (c) and (d) HSI systems [spectral scanning in panel (c) and snapscan in panel (d)] mounted on one of the side ports of the surgical microscope enable the acquisition of 3D hyperspectral image cubes co-registered with the surgeon’s field of view with a small physical footprint to seamlessly integrate into the already space-constrained neurosurgical operating room. (a) Adapted from Leon et al.,⁸⁸ under CC-BY 4.0. (b) Adapted from MacCormac et al.,⁸⁹ under CC-BY 4.0. (c) Reproduced from Valdés et al.,⁶⁷ under CC-NC-SA 3.0. (d) Adapted from Kifle et al.,⁹⁰ under CC-BY 4.0.

Next, the hyperspectral image captured by the system should be as high-quality as possible, while being as close to real-time as possible ($\sim 10\mathrm{Hz}$), consistent with other intraoperative imaging modalities such as US imaging, neuronavigation feedback, microscope visualization, and 3D exoscope imaging. For the hyperspectral data to be useful for surgical guidance, it must fulfill certain basic constraints in addition to real-time acquisition. First, structures in the brain visualized intraoperatively are of the order of millimeters. Therefore, submillimeter resolution over a surgical field of view of the order of centimeters is critical. Second, the spectral bandwidth of the fluorescence peaks of commonly used fluorophores may be as narrow as nanometers, requiring spectral resolutions of a few nanometers. Lastly, as light is split into spectral channels in the already light-starved conditions of fluorescence imaging, the hyperspectral system sensor should have high quantum efficiency, high bit depth, and low dark noise to enable short exposure times.

The speed of hyperspectral acquisition is constrained by the space–spectrum–sensitivity trade-off. Therefore, these conditions are all difficult to satisfy together. The most common, line-scan hyperspectral imagers provide high spectral and spatial resolutions in one spatial dimension [Fig. 3(a)]. However, providing equivalent resolution in the second spatial dimension for surgically relevant scales is time-consuming (typically tens to hundreds of seconds). To be more sensitive to low-intensity fluorescence signals, existing spectral scanning methods [Fig. 3(c)] typically increases exposure times, decreasing hyperspectral cube acquisition rates. Snapshot and snapscan HSI systems^70,71,75,91 [Figs. 3(b) and 3(d)] can potentially provide fast frame rates for hyperspectral acquisition.^{67,87,92–94} However, they sacrifice spatial resolution to do so, also increasing exposure if increased sensitivity is needed. Managing this balance among the imaging parameters to construct clinically practical and effective systems is one of the most important open problems in neurosurgical HSI.

3.1.

Imaging Hardware and Software

3.1.1.

Neurosurgical oncology

HSI for use in neurosurgical oncology was introduced by Gebhart et al.¹⁰⁷ in 2007 with the use of a Varispec VIS-20 LCTF from Cambridge Research Instruments, Inc.¹⁰⁸ coupled with a $512\times 512$ PhotonMax electron multiplying charge-coupled device (EMCCD) camera¹⁰⁹ mounted on a surgical microscope to measure intraoperative autofluorescence and diffuse reflectance spectra with acquisition times of 5 min. Here, the authors did not solely use reflectance but rather both reflectance and autofluorescence measurements to determine a reflectance/autofluorescence ratio for optimal identification of tumor tissue. Similar to the previous approach, Valdés et al.⁶⁷ used a Varispec LCTF coupled with a pco.pixelfly charge-coupled device (CCD) camera¹¹⁰ on a surgical microscope (Zeiss OPMI Pentero) [Fig. 3(c)] to measure the reflectance and fluorescence spectra in a fluorescence correction algorithm to enable more accurate measurement of tissue fluorophores during brain tumor resection. Here, both approaches did not solely use reflectance measurements for tissue identification. Instead, they coupled their reflectance measurements with fluorescence to enable tumor tissue identification, which will be discussed in more detail later (see Sec. 4). It was not until 2016 with the kickoff of the European Hyperspectral Imaging Cancer Detection (HELICoiD) project¹¹¹ and the development of the HELICoiD demonstrator by Salvador et al.¹¹² and Fabelo et al.,¹¹³ where HSI of reflectance was used solely for tumor tissue identification.

The HELICoiD demonstrator consists of a pair of line sensor hyperspectral cameras mounted on a custom optical breadboard in the operating room [Fig. 3(a)]. These cameras, bought off-the-shelf from Headwall Photonics,⁶⁴ are the CCD-based Hyperspec^® VNIR A-series operating in the VIS-NIR wavelength range (400 to 1000 nm, 826 spectral bands, 2- to 3-nm resolution, $90\text{frames}/\mathrm{s}$) and the InGaS-based Hyperspec^® NIR 100/U operating in the NIR short-wave infrared (SWIR) wavelength range (900 to 1700 nm, 172 spectral bands, 5-nm resolution, $100\text{frames}/\mathrm{s}$). The cameras are set up in a confocal stereo configuration with matched fields of view, at an imaging distance of 40 cm and surgical field clearance of 29 cm. The entire imaging assembly is mounted on a translation stage to implement pushbroom scanning [Fig. 1(c)]. The demonstrator system uses a 150-W quartz–tungsten–halogen (QTH) bulb with a spectral range of 400 to 2200 nm, passed through an optical fiber to a cold light emitter. This ensures that the heat from the QTH bulb is not transmitted to the tissue to avoid tissue damage. Follow-up work in the HELICoiD project used other hyperspectral line cameras, such as the Specim ImSpector^® VNIR V10-E spectrograph⁶⁶ (400 to 1000 nm, 2.8-nm resolution) by Madroñal et al.⁹⁷ and the Headwall Hyperspec^® NIR X-Series⁶³ (900 to 1700 nm, 166 spectral bands, $100\text{frames}/\mathrm{s}$) by Ravi et al.¹¹⁴ in linear scanning configurations to capture hyperspectral datasets.

In the initial HELICoiD pilot study, several pixel-wise classification algorithms were used on the data collected with the HELICoiD demonstrator to test the potential of reflectance spectra in tumor resection. These include support vector machines (SVMs), multilayer perceptrons (MLPs), and random forests (RFs) implemented on parallel processing platforms such as the Headwall Hyperspec^® Data Processing Unit^112,113 (31 images from 22 procedures on primary glioblastomas and 135k labeled spectra from the HELICoiD demonstrator) and the Kalray MPPA-256-N HPC device⁹⁶ (13 images from 13 procedures on glioblastomas and metastases and 25k labeled spectra from the HELICoiD demonstrator). The training data consisted of mixed-patient pixel-wise spectra from intraoperative hyperspectral cubes with pathologist-labeled ground truth classification labels. These were tested on data from both HELICoiD cameras separately, and the VIS-NIR data were shown to be most effective with the RF classifier, providing cross-validated accuracy, sensitivity, and specificity greater than 99% for mixed-patient pixel-wise three-class classification.^96,113 Subsequently, this classification scheme with a larger dataset (36 cubes from 22 patients, $>375\mathrm{k}$ labeled spectra from the HELICoiD demonstrator) has been integrated into a mixed supervised–unsupervised framework to provide fast intraoperative visualization¹¹⁵ with a total per-frame acquisition and processing time of 1 min at an overall accuracy greater than 98% for five-class classification (including blood vessels). Further work has extended and improved these results with techniques such as blind linear unmixing^116,117 and empirical mode decomposition,¹¹⁸ shown SVMs effective for identifying malignant tumor phenotypes,¹¹⁹ and demonstrated estimation of the molecular composition of brain tissues in real time.¹²⁰

Further, to ease the time and computational complexity of working with high-dimensional hyperspectral data (hundreds of wavelength channels across millions of pixels) and improve the semantic consistency of segmentation, dimensionality reduction with manifold embedding has been employed.¹¹⁴ This method uses a deep learning–based modified version of the $T$-distributed stochastic neighbor (t-SNE) manifold embedding algorithm,¹²¹ called fixed-reference t-SNE (FR-t-SNE). This non-linear embedding method attempts to preserve local spatial regularity (nearby pixels represent the same class with high probability) while still capturing high-level global features (pixel classes). The possibility for generalization of this method was evaluated by testing the model on patient data from a different set of individuals, with around 72% overall accuracy and 53% tumor sensitivity for four-class classification (33 images from 18 patients, captured with the HELICoiD demonstrator). A combination of the above pixel-wise and dimensionality-reduced classifiers to create a joint spatio-spectral classifier has been shown by Fabelo et al.¹²² to have an overall accuracy greater than 99%, with a speed-up of $>4.5$ to 8.5× achieved with hardware acceleration (five cubes from five patients and 45k labeled spectra from HELICoiD demonstrator).

Various hardware acceleration platforms have been explored to speed up the classification computation by individually optimizing the components of these classifiers. The linear kernel SVM¹¹³ has been sped up 3 to 5× on massively parallel processor arrays⁹⁷ and system-on-chip architectures^123,124 and 90× on graphics processing unit (GPUs);¹²⁵ dimensionality reduction with principal component analysis (PCA) for data preprocessing¹¹⁵ has been sped up 36× using multiple central processing unit (CPU) compute cores;¹²⁶ $k$-nearest neighbor classification^115,122,127 has been sped up 30 to 66× on GPUs; and k-means clustering^115,122 has been sped up 150× on GPUs.¹²⁸ Jointly implementing the entire pipeline with PCA on a multi-GPU¹²⁹ platform has resulted in a total speed-up of 180× over the serial platforms, resulting in processing times being reduced from several hundreds of seconds to tens of seconds.¹²⁹ The effect of optimizing the data-type representation of the hyperspectral images and their storage in memory has been explored for lower-throughput processing.¹³⁰

Recently, deep learning has been applied to tumor identification in both deep fully connected per-pixel and convolutional spatio-spectral configurations.^131,132 This generalizes the hyperspectral data embedding and classification features for the embedded data while allowing for fast computation on the GPU. In combination with unsupervised clustering techniques and minimal user guidance, these accuracies rise to 77% to 78% for one-dimensional (1D) spectral deep neural networks (DNNs),^131,132 72% to 77% for 2D convolutional neural networks (CNNs),^131,132 80% for a combination of 1D DNN and 2D CNN,¹³¹ and 80% for 3D spatio-spectral CNNs¹³² (with datasets consisting of eight cubes from six patients and 82k labeled spectra;¹³¹ 12 cubes from 12 patients and 116k spectra,¹³² both from the HELICoiD). Other deep learning architectures^133–143 have also produced comparable results with the potential for fast hyperspectral brain structure classification. Figure 5 shows examples of the HELICoiD demonstrator during brain tumor surgery for tissue classification using unmixing methods and deep neural networks.

Fig. 5

Classifying brain tissue types based on reflectance spectra. Left to right: intraoperative hyperspectral reflectance imaging on four patients with glioma grades 2 and 4 using the HELICoiD system (patient 1 in row 1, patient 2 in rows 2 and 4, patient 3 in row 3, and patient 4 in row 5), white-light synthetic RGB image reconstructed from the hyperspectral cube with tumor regions marked in yellow and biopsy sites with black circles, ground truth–labeled pixels and pixel classifications using linear unmixing methods [extended blind end-member and abundance extraction (EBEAE)],^117,144 and a two-layer pixel-wise DNN.¹³¹ The four classes are NT, TT, BV, and BG. EBEAE yields around 60% overall accuracy, 30% tumor sensitivity, and 85% tumor specificity, whereas the DNN yields 85% overall accuracy, 65% tumor sensitivity, and 95% tumor specificity with fivefold cross-validation on mixed-patient pixel-wise data. GBM, glioblastoma; OD, oligodedrogioma; A, astrocytoma. Adapted from Leon et al.,⁸⁸ under CC-BY 4.0.

Manual initial feature engineering has also been attempted to provide better pre-processed data as input for classification algorithms. For example, by selecting the most relevant spectral bands using iterative combinatorial optimization algorithms,⁹⁹ correlation-based ranking,¹⁴⁵ and deep learning.¹⁴¹ In addition, registered pairs of VIS-NIR and NIR images from the HELICoiD demonstrator have been analyzed for spectral similarities between classes to ignore non-distinctive samples.¹⁴⁶

The two data streams from the visible and near-infrared (VNIR) and NIR cameras in the original HELICoiD setup^112,113 need to be fused to create a single hyperspectral cube¹⁴⁶ to add more useful data to the computational methods described above. Therefore, a new version of the demonstrator has been proposed by Leon et al.,¹⁴⁷ where the confocal stereo configuration is changed to make the camera axes parallel. This changes the transformation between the two camera viewpoints from a projection to a translation, allowing for less spatial and radiometric distortion of the captured spectra. Combined with spatial and spectral upsampling, hyperspectral cubes are generated at the original spatial resolution and two wavelength ranges (641 spectral bands between 435 and 901 nm and 144 spectral bands between 956 and 1638 nm), resulting in a 21% accuracy increase as compared with using just the VNIR camera on a synthetic material database.

Because the HELICoiD system is mounted on a platform separate from the surgical microscope, it interrupts the surgical workflow due to the need for physical translation of the HELICoiD system prior to data acquisition [Fig. 3(a)]. To prevent such movement, Mühle et al.⁸⁷ designed a workflow with a TIVITA^® VIS-NIR tissue hyperspectral camera (500 to 1000 nm, 100 spectral bands, 5-nm spectral resolution, $640\times 480$ output pixels, $\sim 100\text{frames}/\mathrm{s}$, $\sim 6\mathrm{s}/\text{cube}$)⁶⁵ mounted onto surgical microscope oculars. However, as the cameras used in the above projects can capture only one-dimensional spatial slices, physical scanning of the cameras in one dimension across the surgical field of view is required to capture the entire hyperspectral cube. Thus, this system can capture nanometer-resolution megapixel intraoperative surgical datasets (comparable with previous systems^{88,122,147,148}) at the cost of $\sim 5\mathrm{s}$ per capture. Data captured from this system yields 99% accuracy and greater than 98% sensitivity for tumor detection (one patient, 29k labeled spectra). However, given the time requirement for data acquisition of a single hyperspectral cube, it has had limited utility for routine clinical use as it significantly interrupts the surgical workflow, which precludes performing the resection under continuous feedback from the HSI system.

Therefore, snapshot HSI systems such as the Ximea Corporation MQ022HG-IM-SM5X5-NIR (665 to 975 nm, 25 spectral bands, $409\times 217\text{pixels}$, $170\text{frames}/\mathrm{s}$)¹⁴⁹ based on the IMEC SM5x5 NIR sensor, the Cubert Ultris X50 (350 to 1000 nm, 155 spectral bands, $570\times 570\text{pixels}$, $1.5\text{frames}/\mathrm{s}$),⁹¹ the Senop HSC-2 (freely selectable bandwidths and resolutions)⁷³ and the BaySpec OCI-2000 Series snapshot hyperspectral imagers (475 to 875 nm, 35 to 40 spectral bands, $50\text{frames}/\mathrm{s}$)⁷⁴ have been explored as potential alternatives^{89,90,98,150–154} [Fig. 1(d)]. These can be mounted either by themselves^{89,98,150–152} or coupled to a surgical microscope^{90,93,153,154} to minimize disturbance to the surgical workflow [Fig. 3(d)]. In addition, systems that fuse the advantages of snapshot and line scanning hyperspectral acquisition, called snapscan systems (such as the IMEC Snapscan VNIR,^75,93 470 to 900 nm, 150 spectral bands, $3600\times 2048\text{pixels}$, 2- to 20-s acquisition), coupled with surgical microscopes have been used for intraoperative imaging.¹⁴¹ These systems have been used to develop machine learning-based classification (e.g., SVM, decision tree, and RF classifiers^90,93,98,151) and convolutional neural networks,¹⁵³ with similar results—for instance, a system with the Senop HSC-2 camera reported accuracies around 98%.¹⁵³

3.1.3.

Functional neurosurgery

Epilepsy surgery requires the mapping of metabolically active brain regions, including epileptogenic regions, that demand more oxygen and blood. This link between neuronal activity and changes in blood flow and oxygenation is commonly referred to as neurovascular coupling.¹⁵⁹ As seizures result from intense, uncontrolled neuronal activity, regions of the brain exhibiting seizure activity are highly metabolically active and as such display differences in their neurovascular coupling compared with regions not exhibiting seizure activity.

The first use of HSI for evaluating neurovascular coupling dynamics in epilepsy intraoperatively was in 2013 by Noordmans et al.,¹⁰⁴ where one patient with intractable sensorimotor seizures of the left hand was imaged using an LCTF-based system (Varispec VIS¹⁰⁸ filter with a pco.pixelfly camera,¹¹⁰ $1392\times 1024\text{pixels}$) coupled to a Zeiss Pentero surgical microscope (Fig. 6). In this work, the entire cerebral cortex was imaged over the span of 7 min, and the area of increased oxyhemoglobin at the start of seizure activity matched the epileptogenic zone. Subsequently, Laurence et al.¹⁰⁵ further validated this finding in 12 epilepsy patients, which included non-lesional, focal cortical dysplasia type and heterotopia. The authors found that regions of seizure activity were isolated with an intraoperative HSI system.

Fig. 6

HSI to map seizures intraoperatively. (a) Local increase in oxygenation during seizure: oxygenation changes estimated from oxyhemoglobin concentration during a seizure. (b) Area matched to a photo of the cortex: overlay of oxygenation changes on an RGB image of the brain cortex, which correlates with electrical recordings of seizure activity measured via electrocorticography. Position 20 corresponds to the sensory cortex of the hand where positive seizure activity was recorded and HSI measured higher oxygenation. Reproduced from Noordmans et al.,¹⁰⁴ with permission from John Wiley & Sons, Inc. (c) Relative concentration as a function of time.

Further, a snapshot hyperspectral system from IMEC with filters mosaiced on a CCD sensor (480 to 630 nm, 16 spectral bands, $256\times 512\text{pixels}$, 10 to $20\text{frames}/\mathrm{s}$) coupled with a Zeiss Pentero microscope was used for intraoperative hemodynamic imaging on one patient undergoing epilepsy surgery resection by Pichette et al.⁹² at video rates. Laurence at al.¹⁶⁰ tested this system to measure the interictal discharges in eight patients with non-lesional or subcortical heterotopias undergoing epilepsy surgery, where unsupervised clustering of oxygenation correlated well with direct electrical measurements of the imaged cortex.

Lastly, HSI has been used for intraoperative optical functional brain mapping with a three-chromophore [oxyhemoglobin, deoxyhemoglobin, and oxygenated cytochrome-c-oxidase (oxCCO)^161,162] system by Caredda et al.¹⁶³ Incorporating oxCCO into the model introduces a direct measure of cellular metabolism. This work used a Ximea Corporation MQ022HG-IM-SM5X5-NIR hyperspectral camera (665 to 960 nm, 25 spectral bands, $409\times 217\text{pixels}$, $14\text{frames}/\mathrm{s}$)¹⁴⁹ to measure the tissue reflectance spectra while the patient was repetitively clenching his fist. These reflectance spectra were fit to the model, and the resulting concentration maps were thresholded to identify areas of high oxygenation and metabolism, which were found to strongly correlate with those identified with gold standard direct electric brain stimulation. In addition to incorporating oxCCO, Caredda et al.¹⁶⁴ have demonstrated blind unmixing using non-negative matrix factorization to account for two metabolic biomarkers strongly correlating with direct electrical brain stimulation on 12 patients undergoing resection for a brain tumor near the motor cortex.

HSI techniques in vascular and functional neurosurgery have both used oxygen saturation and hemodynamics. Therefore, optimal schemes for measuring the two simultaneously have been studied in Caredda et al.¹⁶⁵ with Monte Carlo simulations of hemodynamic signals following neuronal firings. These schemes select specific combinations of NIR spectral bands from the hyperspectral image to ensure minimal errors in estimating the proportions of oxyhemoglobin, deoxyhemoglobin, and oxCCO, therefore seeking to achieve accurate metabolic and hemodynamic inferences. Simulations for the specific system designed and implemented in previous work¹⁶⁶ augmented with a Ximea MQ022HG-IM-SM5X5-NIR hyperspectral camera¹⁴⁹ were performed considering the effect of realistic factors such as spectral cross-talk and Gaussian noise on the estimation error. This study found that 21 to 22 spectral bands were enough to compute tissue chromophore proportions accurately (0.5% error for oxyhemoglobin, 4.4% error for deoxyhemoglobin, and 15% error for oxCCO), whereas 10 to 12 spectral bands provided a similar performance. The general approach implemented with this Monte Carlo simulation can potentially be used outside hemodynamic imaging in neurosurgical oncology and spine surgery to determine the optimal spectral signatures for tissue identification tasks using HSI.

3.2.

Datasets

The HSI systems described in Sec. 3.1 have produced rich datasets of intraoperative hyperspectral data. Some of this data are available in the public domain for use by researchers who do not have access to or do not have the resources for constructing and deploying their own HSI systems. We describe publicly available datasets, including those captured for individual projects.

3.3.

Visualization Techniques

3.4.

Clinical Results

Clinical studies using the optical systems and computational methods described above have shown the potential for surgical utility of HSI in reflectance mode for neurosurgery. Here, we review the results from clinical studies performed and present a summary of their statistics and findings in Table 3.

Table 3

Clinical validation of hyperspectral imaging systems for neurosurgical applications.

4.1.

Imaging Hardware and Software

The first demonstration of multispectral fluorescence imaging in neurosurgical oncology was in 2003 using a wide-field five-band (bandpass spectral filters from Omega Optical²⁰⁷ at 495-, 543-, 600-, 640-, and 720-nm center wavelengths; 20-nm bandwidth, $755\times 484$ DVC CCD detector²⁰⁸) multispectral system. Here, the authors imaged a fluorescent tumor after exogenous administration of the fluorescent agent, Photofrin,²⁰⁹ with a total acquisition time of 15 s. This study concluded that multispectral imaging has the capability to separate Photofrin fluorescence from a background with a 10:1 signal-to-background ratio. Further, it hypothesized that multispectral data could estimate Photofrin concentrations, with a detection limit of 50 to $100\mathrm{ng}/\mathrm{ml}$ at 0.5-mm depth inside tissue-mimicking phantoms. However, this work assumed that tissue is homogeneous, causing these estimates to be accurate only when tissue optical properties matched the validation phantoms.

As noted before, the first hyperspectral fluorescence imaging was in 2007, where Gebhart et al.¹⁰⁷ developed an HSI system that consisted of a Varispec VIS-20 LCTF from Cambridge Research Instruments, Inc.¹⁰⁸ coupled with a $512\times 512$ PhotonMax EMCCD camera¹⁰⁹ mounted on a surgical microscope to measure intraoperative autofluorescence and diffuse reflectance spectra in one patient. The authors found that a value less than 1.25 for the ratio of autofluorescence at 460 nm to diffuse reflectance at 700 nm was highly diagnostic for tumor tissue.

Valdés et al.⁶⁷ developed a similar hyperspectral system and implemented the first intraoperative approach to correct fluorescence signals for the distorting and attenuating effects of tissue optical properties in 12 patients with brain tumors [Fig. 3(c)]. They imaged the diffuse reflectance at excitation and emission wavelengths and fluorescence, followed by implementation of a correction algorithm^67,210,211 (i.e., a spectrally constrained dual-band normalization algorithm) for use in 5-ALA-PpIX FGS. Similar to the work by Gebhart et al.,¹⁰⁷ this approach used a Varispec LCTF coupled to a pco.pixelfly camera and custom optical adapter¹¹⁰ unto a surgical microscope modified for fluorescence imaging (Zeiss OPMI Pentero). The surgical field was imaged under white light and 405-nm illumination respectively^67,84,211 to measure fluorescence spectra and reflectance with a total maximum acquisition time of $<16\mathrm{s}$. The measured fluorescence spectrum $F_{\text{raw}}(\lambda )$ was corrected by an empirical factor inversely proportional to the excitation reflectance $R_{\mathrm{exc}}$ and power law proportional to the emission reflectance $R_{\mathrm{em}}$.

The corrected fluorescence spectrum was fit to a weighted sum of basis spectra for fluorophores of interest (e.g., PpIX, fluorescein sodium, and tissue autofluorescence) to isolate only PpIX or FS fluorescence. Thus, the estimated corrected PpIX values were found to be directly proportional to absolute PpIX concentrations. This correction allowed the detection of PpIX concentrations as low as $20\mathrm{ng}/\mathrm{ml}$, which was significantly lower than the lowest concentrations of 600 to $1000\mathrm{ng}/\mathrm{ml}$ found in visually fluorescent (i.e., red-pink visual fluorescence through surgical oculars) high-grade glioma tissues. Further, these results were encouraging as they indicate the ability to detect low yet diagnostically significant PpIX concentrations to identify low-grade glioma and infiltrative margins that are usually “invisible” with conventional techniques^{49,51,67,189,212–214} (Fig. 7). This work concluded that a threshold of $100\mathrm{ng}/\mathrm{ml}$ had a positive predictive power of $>90\%$ for tumor tissues. The HSI approach by Valdés et al.⁶⁷ was further validated in additional studies demonstrating improved detection capabilities in clinical ALA-PpIX FGS.⁸⁴ In further work by Valdés et al.,²¹¹ a more sensitive pco.edge camera²¹⁵ allowed lower acquisition times of 1 to 2 s with the same detection limit. An even more sensitive EMCCD camera²¹⁶ from Nüvü cooled to $-85°\mathrm{C}$ further decreased the limit of detection to $1\mathrm{ng}/\mathrm{ml}$, comparable to point spectroscopy methods²¹⁷ at a maximum total acquisition time of 5 s. This correction method was further applied to pediatric brain tumors, where the limit of visual detection was determined to be $200\mathrm{ng}/\mathrm{ml}$,²¹⁸ and the lower limit of detection for PpIX was $20\mathrm{ng}/\mathrm{ml}$. These were all validated with tissue-mimicking phantoms consisting of a solution of PpIX mixed with an absorber (e.g., hemoglobin and yellow food dye) and a scatterer (e.g., intralipid emulsion).⁶⁷ Known fluorophore concentrations in these phantoms can be used to map the corrected fluorescence to absolute PpIX concentrations and evaluated for accuracy metrics such as linearity (i.e., $R^2$ value and mean percentage errors).

Spectrally constrained dual-band normalization has been systematically evaluated for its accuracy in correcting the raw fluorescence signal for tissue optical properties, its highly sensitive estimates of fluorophore concentrations (i.e., PpIX),^{67,210,211,219,220} its reproducibility by different clinical and research teams and HSI systems,^94,221–223 and its diagnostic utility with greater sensitivity, negative predictive values, and overall accuracy for tumor detection compared with visual expert evaluation. Specifically, Lehtonen et al.²⁰⁶ found that visual assessments yielded 63% accuracy, 48% sensitivity, 92% specificity, and $340\mathrm{ng}/\mathrm{ml}$ minimum limit of detection for PpIX. Meanwhile, an HSI system based on a standalone Senop HSC-2 camera (500 to 900 nm, up to 1000 spectral bands, $1024\times 1024\text{pixels}$, $150\text{frames}/\mathrm{s}$)²²⁴ yielded 96% accuracy, 100% sensitivity, and 86% specificity, and $16\mathrm{ng}/\mathrm{ml}$ minimum limit of detection (16 samples with PpIX and eight control samples; number of patients not specified).

Bravo et al.²¹⁹ have shown in three patients that corrected concentration estimates (with spectral fitting to isolate PpIX) correlate strongly with point spectroscopy estimates²²⁰ (linear fit $r=0.98$) when compared with uncorrected estimates (linear fit $r=0.91$ accounting for other fluorophores, linear fit $r=0.82$ not accounting for other fluorophores) (Fig. 8).

Fig. 8

Comparison of HSI to point spectroscopy. Point spectroscopy provides gold standard spectrally resolved measurements and PpIX concentration estimates that can be used to validate the estimates from hyperspectral processing. HSI extends the applicability of fluorescence guidance to WHO grade III anaplastic oligoastrocytomas (AOA) (a)–(e) and meningiomas (MEN) (f)–(j), where the PpIX concentration is significantly less than the limit for visual fluorescence. (k) Fluorescence spectra fit and estimated PpIX concentrations from HSI (top) and point spectroscopy measurements (bottom). MR texture map = matching MRI 2D image; Zeiss—white = white-light image from a conventional Zeiss microscope; Zeiss—blue = fluorescence image from a conventional Zeiss microscope; integrated fluorescence = map of fluorescence calculated from the area under the fluorescence spectrum from 620 to 650 nm; quantitative PpIX = map of PpIX concentration estimates. Reproduced from Bravo et al.,²¹⁹ under CC-BY 4.0.

Xie et al.²²¹ developed a Bayesian reconstruction method based on spatial regularization and tested it on one tissue specimen from a glioblastoma patient. This approach defines reconstruction in terms of a total variation-regularized minimization problem

The first term here, based on previous point spectroscopy analysis,⁸⁵ attempts to make the reconstruction of $C(x,y)$ faithful to the measurement of $F_{\text{raw}}(\lambda )$. Here, $\mathrm{\Omega }$ is a factor that maps corrected fluorescence intensity to concentration, and $\mathrm{\Gamma }$ is a regularization factor that decides the smoothness of the reconstruction. This reconstruction lowers the detection limit to $10\mathrm{ng}/\mathrm{ml}$ using an uncooled ORCA-Flash4.0 EMCCD sensor from Hamamatsu Photonics with 26 s of total acquisition and processing time. Such low detection levels would be particularly useful for detecting low, but diagnostically significant PpIX levels in low-grade gliomas.²²⁰ Further computational work used an unspecified sCMOS camera²²⁵ with the Sony IMX252 sensor by Black et al.¹⁹⁹ and a pco.edge camera ($14\mathrm{ng}/\mathrm{ml}$ minimum detection limit).^94,219,226

Finally, Black et al.²²² used machine learning–based approaches on the unmixed fluorophore contributions to predict the following tumor properties in 891 hyperspectral measurements from 184 patients with multiple brain tumor histology types: tumor type (12 categories, test accuracy 85%), tumor margin location (tumor bulk, infiltrative margin, and healthy tissue altered due to tumor, test accuracy 96%), isocitrate dehydrogenase enzyme (IDH) gene mutation type (mutated and normal, test accuracy 86%), and tumor grade (II–IV, test accuracy 93%). In addition, PCA variation analysis revealed that the five fluorophores mentioned above were the most likely components explaining the dataset spectra under the assumption of Gaussian noise.²²² Incorporating the more physically accurate Poisson unmixing model, with a dataset containing 555,666 spectra, allowed Black et al.²²³ to unmix fluorophores previously impossible due to their small proportion and thus building up a “spectral library” containing ${\mathrm{PpIX}}_{620}$ (see next paragraph), ${\mathrm{PpIX}}_{634}$, reduced nicotinamide-adenine dinucleotide (NADH), flavin adenine dinucleotide (FAD), flavins, lipofuscin, melanin, elastin, and collagen as its members. Finally, deep learning–based architectures have incorporated the non-linear wavelength-dependent effects not taken into account by the previous algorithms to correct and unmix fluorescence spectra with a semi-supervised architecture.²²⁶ This approach yielded more realistic and smooth estimates of PpIX concentration maps as well as removing imaging artifacts such as specularities.

As mentioned above, correction methods, such as spectrally constrained dual-band normalization, commonly undergo validation using fluorescent tissue-mimicking liquid phantoms. However, a recent study by Suero Molina et al.²²⁷ has proposed a photostate of PpIX that contributes a fluorescence spectrum shifted to a peak at 620 nm that likely occurs naturally in tissue, but not in such phantoms. The presence of this photostate (called ${\mathrm{PpIX}}_{620}$ as opposed to the usual ${\mathrm{PpIX}}_{634}$) impacts the accuracy of conventional linear fitting models which use the basis spectra of ${\mathrm{PpIX}}_{634}$, PpIX photoproducts and autofluorescence from NADH, lipofuscin, and flavins. Therefore, incorporation of the ${\mathrm{PpIX}}_{620}$ spectrum into linear fitting models has been proposed to improve the accuracy of the spectral fit in dimly fluorescent areas (e.g., low-grade gliomas and infiltrative regions of high-grade gliomas). This also lowers false positives by removing the spurious contribution of ${\mathrm{PpIX}}_{620}$, yielding the true ${\mathrm{PpIX}}_{634}$ spectrum and therefore accurate, lower ${\mathrm{PpIX}}_{634}$ estimates.¹⁹⁹ Further, additional studies have noted the proportion of the two photostates (i.e., the overall blue shift of the PpIX spectrum) correlates with tumor grades of tissues.²¹⁴

This LCTF design provided a small footprint to enable HSI with high spatial resolution at user-defined spectral resolutions and acquisition times in the order of seconds. Although this HSI design and subsequent implementations have been translated into the operating room given their integration with commercial surgical microscopes, they suffer from one major limitation for widespread surgical use: image acquisition from these systems requires spectral scanning (i.e., an image for every wavelength of interest with a finite amount of camera exposure for each wavelength to reconstruct a full hyperspectral cube). As such, these HSI systems have limited intraoperative utility for widespread use because they do not provide real-time surgical guidance. To address this limitation, a recent snapshot HSI system that used a series of birefringent crystals was developed by Marois et al.²²⁸ to capture 64 spectral channels at a time. This system achieved a frame rate of 4 to $6\text{frames}/\mathrm{s}$ over a broad wavelength range (425 to 825 nm, 64 spectral bands, $600\times 400\text{pixels}$) and subsequently implemented a spectrally constrained dual-band normalization technique as well.

4.2.

Clinical Results

Clinical studies using HSI in FGS have focused mostly on tissue classification for improving tumor detection (Table 3). The first study sought to detect residual tumors with a limited number of (multispectral) images²⁰⁹ coupled to visual inspection of these channels. The first quantitative clinical studies, carried out by Valdés et al.,⁶⁷ performed unmixing of fluorescent components of tissue via fluorescence spectrum fitting and correction of PpIX fluorescence intensity for attenuation due to light-tissue interaction to estimate absolute pixel-wise tissue concentrations^67,84 on 12 patients undergoing brain tumor resection (Fig. 7). Subsequent work from this group showed improvements in accuracy and sensitivity for PpIX detection.²¹⁹ These corrections were further incorporated into a spatially regularized optimization for smooth and accurate estimates of PpIX concentration maps.^221,226 Further, the autofluorescence properties of tissue were characterized in two studies to incorporate them into the unmixing algorithms, using an increasing number of components and known compounds (e.g., PpIX photoproducts and differing PpIX states)—one analyzing 2692 in vivo spectra from 128 patients¹⁹⁹ and one building a spectral endmember library from 555,666 fluorescence spectra measured from 891 ex vivo sample measurements.²²³ The coefficients of the resulting fluorescence spectrum fit were shown to be useful for predicting properties of tumors such as type, margin, grade, and IDH mutation status²²² in 891 spectra from 184 patients. Further, to optimize the dose and administration time of 5-ALA, hyperspectral studies were performed to estimate the pharmaco-kinetics of PpIX inside tissue—one with 81 spectra from 25 patients for low-grade gliomas²⁰¹ and one with 201 spectra from 68 patients for malignant gliomas.²⁰² These studies showed an optimal post-dose time of 7 to 8 h at which PpIX tumor fluorescence signal peaks.

The results of these studies point toward the potential for HSI to enhance fluorescence feedback to serve as an improved surgical adjunct. One of these HSI studies has made its dataset available upon request,²¹⁹ whereas another offers the spectral library constructed during its analysis²²³ to facilitate further research. Exact PpIX concentrations, which are determined by correcting its fluorescence spectrum from the distorting effects of tissue optical properties and unmixed from autofluorescent and other fluorescent components in tissue, can increase the accuracy of predicting tumor presence, whereas the unmixed autofluorescent parts predict tumor properties with machine learning. This, combined with the optical functional and vasculature mapping from the previous section, will allow for all-optical joint visualization of anatomy and tumor for safe and accurate tumor resection.