1.

Introduction

Fluorescence molecular imaging (FMI) has made great advances in clinical translation over the last few years.¹ Driven by these advances, technologies at the forefront of the field are evolving rapidly, particularly in the areas of device design, fluorescent agents, image processing algorithms, and performance assessment metrics.² Consequently, the number of imaging devices and their applications is increasing.

Moreover, following the first-in-human application of FMI in 2011 by van Dam et al.,³ numerous clinical studies have been completed or are currently ongoing. A major outcome of all this activity is the recent approvals by the US Food and Drug Administration (FDA) of $\sim 20$ fluorescence-guided clinical imaging systems⁴ as well as 3 tracers for surgical guidance: (1) 5-aminolevulinic acid (5-ALA/Gleolan®; Photonamic GmbH and Co., KG, Pinneberg, Germany) for use as an intra-operative optical imaging agent in patients with suspected high-grade gliomas,⁵ (2) hexaminolevulinate (HAL, available as Hexvix, Photocure ASA, Oslo, Norway, and Cysview Photocure Inc., Princeton, New Jersey, United States) for use in non-muscle-invasive bladder cancer,⁶ and (3) pafolacianine (Cytalux, On Target Laboratories LLC, West Lafayette, Indiana, United States) for intraoperative imaging of folate receptor-positive ovarian and lung cancers.^7,8 All this activity has highlighted the need for better and user-independent standardization procedures that would allow for system characterization, performance monitoring, data referencing, and comparison, even among markedly different systems. This is, also, very relevant to the fluorescent image-guided surgery (FIGS), given the FDA clearance of multiple FIGS devices for imaging with indocyanine green (ICG) and other contrast agents.⁹ Addressing this need is essential for ensuring optimal impact and wider clinical acceptance of FMI and FIGS. ¹⁰

Over the past few years, numerous studies on phantom development and standardization procedures, as well as attempts to achieve consensus in the community, have been reported.^{1,2,9,11–19} Thus far, methods and reference targets for system evaluation and comparison have been developed on an individual basis, but a universal cross-platform metric for image fidelity evaluation has yet to be developed.¹⁶

Currently, the sensitivity of FMI systems is assessed mostly using the signal-to-noise ratio (SNR) and/or contrast metrics.^2,12,17,20 It has been shown, however, that the definition of the background can play a significant role in the interpretation of the acquired images, especially during tissue imaging.^21–24 For example, Chen et al.²² and Hoogstins et al.²¹ reported that background estimation significantly affected quantification results for bulk-stained tissue fluorescence imaging and intraoperative/ex vivo fluorescence imaging, respectively, using metrics including SNR, signal-to-background ratio (SBR), and contrast-to-noise ratio (CNR). Widen et al.²³ demonstrated the impact of region of interest (ROI) sizes on overall signal and mean fluorescence intensity by analyzing fluorescent probes in animal experiments. Dijkhuis et al.²⁴ also demonstrated the effect of manually selected ROI in fluorescent data analysis and proposed semi-automatic methods for objective assessment of fluorescent signals in resected tissue. In view of the theoretical effect described above, Azargoshasb et al.²⁵ quantified how fluorescent SBR influences the robotic surgical performance of participants ($n=16$) during an exercise with a custom grid phantom. On the other hand, Palma-Chavez et al.²⁶ reported 15 different SNRs and five contrast formulas that are currently used in the field of optoacoustics, indicating that the lack of consensus is not only limited to FMI applications. The plethora of background definitions, as well as the different quantification formulas used across multiple studies, emphasize the importance of reaching a wide consensus for performance assessment and quality control of FMI systems.

Indeed, despite the fact that SNR and contrast are the most commonly used metrics for the sensitivity assessment of various systems,^1,26 there are only a few studies comparing different FIGS systems, most of which are optimized for ICG imaging.^9,19,27 In addition, the formulas used to calculate SNR and contrast and methods for evaluating background ROIs vary across different studies. In a recent study, LaRochelle et al.²⁸ demonstrated the influence of background definition in SBR, SNR, CNR, and contrast-to-variability ratio through measurements on anthropomorphic three-dimensional (3D)-printed phantoms. However, to the best of our knowledge, there is no study quantifying the effect of the combined variation (ROIs and metrics formulas) on performance assessment. An in-depth testing and evaluation of current strategies are crucial to raise community awareness of existing limitations, to spur effective development of the technology, and to set the performance limits that are required for regulatory approvals.

Building on the assumption that the SNR and contrast metrics depend on the selection of background ROIs and quantification formulas, herein, for the first time, we systematically investigate and showcase this dependence with regard to the sensitivity assessment of markedly different FMI systems.

In specifics, using six near-infrared FMI systems, we captured fluorescence images of a composite rigid phantom previously developed by our group.^11,18,19 We then assessed the sensitivity¹⁹ of those systems using six previously published formulas for SNR and contrast^17,29–33 and two background locations. Moreover, based on these metrics, we quantified the corresponding benchmarking (BM) scores,¹⁹ and the systems were ranked based on these scores.

Recently, we called attention to the need for a commonly accepted phantom to promote good imaging practices during the development of FMI systems or their use in clinics.¹ We now pinpoint additional needs to consistently define ROIs and use common quantification formulas for SNR and contrast. Answering these needs will enable consistency, allow data comparison and referencing, and advance the quality and performance of FMI systems. These improvements will promote wide acceptance and usage of FMI as a tool for interventional and endoscopic procedures.

2.

Materials and Methods

2.1.

FMISystems

For this study, we used six fluorescence imaging systems distributed in different labs in the United States and Europe. The main specifications of each system, as well as the adopted phantom imaging protocols, are summarized in Table 1, while the corresponding system schematics are presented in Fig. 1. All measurements were conducted in darkness to eliminate the influence of ambient light on the results.

Table 1

FMI systems used in this study and the corresponding imaging protocols.

Name	Sensor	Bit depth (bit)	Resolution (pixels)	Wavelengths (nm, excitation/emission)	Exposure time (sec)	Working distance (mm)	Fluence ratea (mW/cm2)	Reference
Mob	CMOS camera	8	$3264\times 2448$	785/825	0.1	150	12.5	34
NIRF I	Apogee Camera-Alta U2020ML	16	$1600\times 1200$	785/825	1	750	0.74	35
NIRF II	Kodak KAI-2020M	16	$2758\times 2208$	785/825	1	750	12.5	Not available
Solaris	Fluorescence sCMOS camera	16	$2200\times 2500$	730/800	dependent on the video rate	700	10	PerkinElmer®Solaris
RawFl	sCMOS camera	16	$1024\times 1024$	760/800	0.2	450	1.5	36
Hybrid	EMCCD camera	16	$512\times 512$	750/810-90	0.1	150	15.5	37

^aAt the phantom surface.

Fig. 1

Schematic diagrams of the fluorescence imaging systems used in the study. (a) Mob—adapted with permission from Ghassemi et al.³⁴ (b) NIRF I/NIRF II—adapted from Kanniyappan et al.³⁵ (c) Solaris—adapted and modified from Behrooz et al.³⁸ (d) RawFl—adapted from Ségaud et al.³⁶ (e) Hybrid—adapted with permission from Glatz et al.³⁷ CCD, charge-coupled device; F, filter; LED, light-emitting diode; LD, laser diode; L, lens; EMCCD, electron-multiplying charge-coupled device; RL, relay lens; DM, dichroic mirror; CW, continuous wave; sCMOS, scientific complementary metal oxide semiconductor; LCTF, liquid crystal tunable filter; D, diffuser; DMD, digital micro-mirror device; NIR2, near-infrared camera.

Mob is a mobile phone-based near-infrared fluorescence (NIRF) imaging system previously,³⁴ where its spectral sensitivity was documented. It involves a 1W 785-nm laser diode, an 800-nm short-pass excitation filter (84-729, Edmund Optics, Barrington, New Jersey, United States), and a long-pass emission filter with a cutoff wavelength at 825 nm (86-078, Edmund Optics) for the detection. The phone camera is based on an 8-bit complementary metal oxide semiconductor (CMOS) sensor with an f/2.4 aperture lens (Eigen Imaging, Inc., San Diego, California, United States) and a near-infrared blocking filter, which was removed during this study.

NIRF I is a custom benchtop NIRF imaging system³⁵ with a light-emitting diode (M780L3, Thorlabs, Inc., Newton, New Jersey, United States) centered at 780 nm and power of 200 mW. The same optical filters used with the Mob system were also used in the NIRF I imaging system. A 16-bit charge-coupled device camera (Alta U2000, Apogee Imaging Systems, Roseville, California, United States) coupled with a zoom lens (7-mm focal length, f/3.9, Tamron, Commack, New York, United States) was used for the detection of the emitted fluorescence.

NIRF II is an updated version of the NIRF I imaging system. Its main improvement is the replacement of the imaging sensor with the more sensitive Kodak KAI-2020M (Image Sensor Solutions Eastman Kodak Company, Rochester, New York, United States), while fluorescence was induced by a laser diode at 785 nm and 1W power, instead of the light-emitting diode present in NIRF I system.

Solaris is an open-air commercially available fluorescence imaging system by PerkinElmer (Waltham, Massachusetts, United States). The Solaris system is designed for research applications, including preclinical studies for advanced molecular-guided surgery, and drug efficacy and safety measurements.

RawFl is a custom-built setup³⁶ with a filtered 760-nm laser diode (LDX Optronics, Maryville, Tennessee, United States) light source, a 16-bit scientific complementary metal oxide semiconductor (sCMOS) camera (pco.edge 5.5, PCO AG, Kelheim, Germany) as a detector and polarizers (PPL05C; Moxtek, Orem, Utah, United States) for minimizing the contribution from specular reflections at the surface of the sample.

Hybrid is a custom-built system combining fluorescence and color imaging and has been described previously.³⁷ Fluorescence excitation is achieved using a laser diode (FLX-750-1500 M-100-9 MM Frankfurt Laser Company, Friedrichsdorf, Germany) and detection with an electron-multiplying charge-coupled device (EMCCD, DV897DCS-BV, Andor Technology, Belfast, United Kingdom).

2.2.

Standardization Phantom

The composite phantom shown in Fig. 2(a)¹⁹ was used to quantify the SNR and contrast from images acquired by the six systems. The application of the phantom as a fluorescence standard for performance assessment, quality control, and comparison of markedly different systems through a single image has been described in detail in previous studies.^11,18,19 In the current study, however, the SNR and contrast were evaluated only on the “sensitivity versus depth” region of the phantom [see Fig. 2(a)]. This region includes (1) the transparent polyurethane (WC-783 A/B, BJB Enterprises, Tustin, California, United States) matrix base, with 0.00875 mg/g alcohol-soluble nigrosin (Sigma Aldrich, St. Louis, Missouri, United States) and 1.5 mg/g ${\mathrm{TiO}}_2$ nanoparticles (titanium IV oxide; Sigma Aldrich) for mimicking absorption and scattering, and (2) nine equally sized circular wells, made of the same polyurethane base with $20\mu \mathrm{g}/\mathrm{g}$ bovine hemin ($\ge 90\%$ pure; Sigma Aldrich) and 0.66 mg/g ${\mathrm{TiO}}_2$ for absorption and scattering and 10-nM organic quantum dots (Qdot 800 ITK, Thermofisher Scientific, Waltham, Massachusetts, United States) for fluorescence. As shown in Fig. 2(a), the nine wells were embedded into the phantom matrix at distances of 0.2, 0.4, 0.6, 0.8, 1.0, 1.33, 1.66, 2.0, and 3.0 mm, respectively, from the phantom’s top surface.

Fig. 2

Sensitivity versus depth phantom region. (a) An illustration of the composite phantom used in this study, with the sensitivity versus depth wells highlighted and enlarged. Arrows denote two areas (b1 and b2) used as background regions. The depth of the phantom wells (bottom left, $Dx$ where $x=a,b,c\dots$) indicates the distance from the top surface of the phantom to each fluorescent inclusion. The concentrations of different constituents are the same for all inclusions. Qdots, quantum dots for fluorescence; Hemin, bovine hemin; and ${\mathrm{TiO}}_2$, nanoparticles (see Sec. 2.1). (b) Fluorescence images normalized to their corresponding maxima as acquired by the six systems employed in the study (see Table 1 for the description of each system).

3.

Results

Employing the six FMI systems described in Table 1, we imaged the composite phantom of Fig. 2(a) and isolated the sensitivity versus depth region from the acquired images, as shown in Fig. 2(b). As expected, the markedly different systems yield subjectively different images from the same area of the same phantom, which highlights the importance and need for FMI standardization to ensure a consistently high degree of performance and facilitate clinical translation.

The ROIs used for the quantification of SNR and contrast are shown in the top-right inset of Fig. 2(a). The arrows point out the two areas (b1 and b2) used for background calculation. These locations were chosen based on different studies assessing the performance of FMI systems^13,28 and according to phantom constituents and geometry.

Figures 3 and 4 illustrate the calculated SNR and contrast metrics, respectively, as functions of depth. The results obtained using ${\mathrm{SNR}}_1$ and ${\mathrm{SNR}}_2$ show the same trend for all systems, but not equivalent values [Fig. 3(b)]. Moreover, the results obtained using ${\mathrm{SNR}}_3$ and ${\mathrm{SNR}}_4$ not only differ from ${\mathrm{SNR}}_1$ and ${\mathrm{SNR}}_2$ but are also influenced by the chosen background area. For instance, when comparing Mob and NIRF I for ${\mathrm{SNR}}_2^{b1}$, it is evident from Fig. 3(b) that the NIRF I system has a greater SNR than the Mob system. However, a comparison of the NIRF I system using ${\mathrm{SNR}}_3^{b1}$ with the Mob system using ${\mathrm{SNR}}_2^{b1}$ yields the opposite conclusion.

Fig. 3

Dependence of SNR on the two background locations shown in Fig. 2(a) and/or the quantification formulas of Table 2 for different FMI systems. (a) SNR values for all systems at each depth. ${\mathrm{SNR}}_1$ shows the same behavior for each system as a function of depth. ${\mathrm{SNR}}_2$ shows a similar trend to ${\mathrm{SNR}}_1$ for all systems, regardless of the background employed. ${\mathrm{SNR}}_3$ and ${\mathrm{SNR}}_4$ show different trends compared with ${\mathrm{SNR}}_1$ and ${\mathrm{SNR}}_2$, depending on the background. (b) SNR values of the phantom well with depth = 1 mm for all systems. The values correspond to the dashed area highlighted in panel (a).

Fig. 4

Dependence of contrast on the two background locations shown in Fig. 2(a) and/or the quantification formulas of Table 2 for different FMI systems. (a) The contrast metric results for all systems at each depth. $C_M$ and $C_W$ show similar trends when either b1 or b2 is employed for both calculations. (b) Contrast results for the phantom well with depth = 1 mm for all systems. The values correspond to the dashed area highlighted in panel (a).

Figure 4 demonstrates the results of the contrast metrics with respect to the applied formula (i.e., $C_M$—Michelson contrast and $C_W$—Weber contrast) and considered background ROI. The trends for both $C_M$ and $C_W$ metrics are similar for each system when the same background values are considered (i.e., b1 or b2 for both formulas). Conversely, when comparing the trends observed in $C_M$ under the two background values, the background influence on the quantification of the contrast metrics becomes evident [Fig. 4(b)]. For example, the Mob system has a higher contrast than the RawFl system when the Michelson contrast is applied under the b1 background for both systems. This is not true, however, when the Michelson contrast is used under b1 for the Mob system and b2 for the RawFl system. In that case, the RawFl system has a higher contrast than the Mob one [see Fig. 4(b)].

The influence of the applied formula and background ROI shown in Fig. 3 for SNR and Fig. 4 for contrast becomes even stronger when both metrics are combined to assess the performance of different FMI systems. Figure 5 depicts the ranking of the six systems used in this study based on the corresponding BM scores, which were calculated from the SNR and contrast metrics. Figure 5(a) illustrates the effect of combining the different formulas and background locations on the quantification of the BM scores per system. Moreover, Figs. 5(b)–5(e) demonstrate exemplary BM scores for each system as selected from the four marked squares in Fig. 5(a). The four squares were selected after a visual inspection of the map in Fig. 5(a) to showcase the variability in the quantified BM scores. As can be seen, the BM scores not only have different values, but also their trend is different per combination of formulas and background ROIs. This trend becomes clear in Fig. 5(f), where the systems’ ranking (i.e., 1—worst through 6—best) is shown for the various BM scores of Figs. 5(b)–5(e). For example, the hybrid system’s rank is superior to Solaris’ rank if their BM scores result from the combination of ${\mathrm{SNR}}_2^{\mathrm{b}1}$ is used for the BM score calculation [Figs. 5(a), 5(c), and 5(f)].

Fig. 5

BM scores calculated according to Gorpas et al.¹⁹ for each system. (a) Map of the BM scores quantified using different SNRs and contrast (C) formulas (see Table 2) and two different backgrounds [see Fig. 2(a)]. The squares marked with numbers 1, 2, 3, and 4 correspond to the representative graphs of BM scores in panel (b) for square 1, ${\mathrm{SNR}}_2^{\mathbf{b}1}$ and ${\mathrm{C}}_{\mathit{M}}^{\mathit{b}1}$; (c) for square 2, ${\mathrm{SNR}}_1$; (d) for square 3, ${\mathrm{SNR}}_3^{\mathbf{b}1}$; and (e) for square 4, ${\mathrm{SNR}}_4^{\mathbf{b}2}$. (f) The rank of each system as a result of the BM scores for all squares of panel (a).

4.

Discussion and Conclusion

In the current work, through the comparison of six near-infrared FMI systems, we showed that the assessment of system performance and standardization via SNR and contrast is highly dependent on the definition of background ROI and the formulas used. This proves the need for careful attention to test a method’s clinical relevance, as well as consistency in defining metrics for objective, quantitative assessment of FMI system performance.

We used fluorescence data from the sensitivity versus depth areas of a multiparametric phantom to quantify SNR and contrast by means of different formulas obtained from the literature (Table 2). It was demonstrated that resultant SNR values can be affected by both the selected background location and the formulas applied (Fig. 3). In the case of contrast values, resultant trends appear similar for both Michelson and Weber formulas ($C_M$ and $C_W$ in Table 2), but the employed background ROI is still observed to impact the trends. Indeed, as we show in Fig. 4, the contrast ($C_W$) for the hybrid system changes by a factor of 8.3 depending on the background, while for the Mob system by 2.9. The dependence of the applied formula and/or background becomes more evident for SNR, where the Mob system shows a variation by a factor of 19.6 in the SNR estimation. This indicates a pressing need for common quantification formulas for SNR and contrast and consistent ROI definition for both signal and background. All measurements in this study were conducted in darkness to minimize ambient illumination that would further complicate the quantification of SNR and contrast. However, illumination is another critical factor that must be accounted for when darkness is not possible. One way to address this challenge is by acquiring a “dark” image with the excitation sources turned off and subsequently subtracting that image from the fluorescence image. This step should be performed before quantifying and reporting any performance assessment and quality control metrics. Meeting these requirements is crucial to achieving reliable results and standardization guidelines for FMI.^2,14 Having this internal consistency during the development of FMI systems will lead to the establishment of international consensus across the field and will contribute to the widespread acceptance and use of FMI.

Our goal, however, was not only to assess the performance of each system in different SNRs and contrast definitions but also to show how these definitions affect the comparison of markedly different systems. The results of our contrast and SNR calculations were translated into BM scores and then to rank values. This analysis revealed the dependence of the ranking on the definition of background ROIs or the adopted formulas (Fig. 5). For example, the rank value for the Solaris system was lower than the corresponding values for the NIRF II and the hybrid systems if the performance assessment was based on the Michelson contrast and SNR₂ formula with background defined as b1. However, the Solaris system ranks higher than the NIRF II and the hybrid systems when SNR is evaluated as SNR₁ and contrast through the Weber formula with the b2. This inconsistency in the determinants of the metrics for system evaluation can affect the development and comparison of systems and ultimately the design and efficacy of clinical or pre-clinical studies. In a recent report, the American Association of Physicists in Medicine (AAPM) proposed SNR₄ and $C_M$, as metrics for the performance assessment of fluorescence imaging systems.¹⁷ Moreover, the suggested background region proposed in these guidelines for the estimation of SNR corresponds to a region with the same optical properties as the interrogated wells, but without fluorescent dye. This corresponds to the ROI b₂ in our study since the wells are gradually covered with the phantom matrix material which has no fluorescent dye. On the other hand, in the AAPM study, the contrast is associated with the resolution of a system and not the signal contrast as employed herein. Thus, although there is agreement in the SNR definition (${\mathrm{SNR}}_4^{\mathrm{b}1}$) between the AAPM and our study, we additionally employed the contrast as a means of sensitivity assessment. Nevertheless, these recommendations represent a promising initial step toward establishing a widely accepted protocol for standardizing FMI systems, thereby addressing the inconsistencies demonstrated herein.

Similar limitations for the quantification of SNR and contrast have also been reported during the use of FMI systems in pre-clinical and clinical applications. For example, LaRochelle et al.²⁸ discussed the variability of the methods used for reporting the quantitative sensitivity metrics using 3D anthropomorphic phantoms with incorporated NIR fluorescent tumor parts. On the other hand, Hoogstins et al.²¹ used data from both animal and human studies with multiple fluorescence tracers to show that background noise and background selection have a significant influence on the quantification of SBR and contrast-to-background ratio. Similarly, Azargoshasb et al.²⁵ showed that SBR quantification can impact the surgical discrimination of fluorescence signals, highlighting the importance of the applied quantification approach in intraoperative decision-making. Herein, we present, to the best of our knowledge, the first study that showcases not only how the adopted formulas and the used background affect the performance assessment of an FMI system but also how the lack of consensus on quantification methods of SNR and contrast can result to misleading interpretation of system comparison measurements.

Moreover, for the quantification of the BM scores, we assumed normal signal distributions, according to which a measurement represents 95% confidence when its value is twice the magnitude of the noise level.¹⁹ Thus, the reference threshold values applied here are user-independent in comparison to another value commonly used in fluorescence imaging, the Rose criterion.¹⁷ The Rose criterion method also sets a limit of detection for fluorescence imaging for which the CNR values must be greater than 3 to 5.⁴⁰ However, the range of a particular threshold value varies from study to study^35,41,42 and depends on several parameters such as object shape, edge sharpness, viewing distance, and observer experience. Besides the parameters affecting the threshold value, Rose’s studies were intended for electronic imaging systems (i.e., photography, television, and optical and visual systems)⁴³ and were focused on human perception of signal detectability.⁴⁴ However, threshold values that are constrained by aspects of the human visual system might no longer be relevant with the advent of artificial intelligence (AI) imaging and signal processing. AI algorithms will allow for lower thresholds according to definitions that are not subject to intra- and inter-human observer variability.^45–47 The criterion adopted herein follows a more simplistic statistical approach that evaluates system performance without depending on human perception and thus is more relevant for assessing the detection limits of FMI systems.

The findings of this study are also relevant to existing ICG-based FIGS systems. Similar to FMI, most FIGS system sensitivity assessment and quality control approaches are still based on the quantification of SNR and contrast metrics. However, the quantification methods for these metrics still represent a major limitation factor for cross-platform system comparisons and affect the design and/or repeatability of preclinical or clinical trials. Moreover, consistency in quantification and reporting of the various performance assessment metrics is especially important for FIGS systems, as no established quality control protocols currently exist despite the wide clinical use of such systems. The quantitative assessment of the system performance presented herein advances the current standardization strategies, which is critical for the further development of this technology and for establishing the performance limits that are a prerequisite for regulatory approvals.

Finally, similar challenges in the quantification of SNR and contrast are present in other optical technologies that are currently under investigation. For example, Palma-Chavez et al.²⁶ showcased variability in SNR and contrast quantification methods within the field of optoacoustics. Fluorescence lifetime imaging is another emerging and very promising technology that also lacks consensus in the quantification of SNR, despite its frequent use in assessing the reliability of lifetime measurements. Under appropriate modifications, our study can also be adapted for such technologies, thereby contributing to the development of performance assessment and quality control protocols for imaging methods beyond FMI and FIGS.

Disclaimer

The mention of commercial products, their sources, or their use in connection with material reported herein is not to be construed as either an actual or implied endorsement of such products by the U.S. Department of Health and Human Services. This paper reflects the views of the authors and should not be construed to represent the U.S. FDA’s views or policies.