1.

Introduction

Over the past few decades, minimally invasive surgery (MIS) techniques have become prevalent in various fields of surgical diagnostics and therapy,¹ offering many benefits over traditional surgery. These benefits include less tissue damage and, therefore, less pain for the patients, shorter recovery time as well as increased safety and precision.^2–4 Moreover, the use of MIS procedures instead of traditional open surgery, especially for routine operations, can lead to significantly reduced health care costs.^5,6 Researchers at Johns Hopkins University School of Medicine recently found that by increasing the use of laparoscopic techniques by 50%, US-American hospitals collectively could prevent thousands of postsurgical complications and save up to $280 million per year.⁶

Currently, the majority of modern MIS systems are video based, giving surgeons an extended, in some cases, even three-dimensional view inside the patient’s body revealing its anatomical topography and, therefore, allowing for better visual support during surgery. Making use of digital endoscopes, such systems enable surgeons to operate through a much smaller incision than would otherwise be required for traditional open surgery. At the same time, the surgeon’s exact view of the surgical field can be provided to nurses or assisting surgeons with the same image quality by projecting the digital image of the MIS system on an additional monitor. This allows for better communication of surgical workflows and strategies and, therefore, more efficient collaboration in the operating room.

Despite the advantages of video-based MIS technologies and their increasing dissemination, there is still a lack of international standards and norms regarding the color reproduction properties of such fully digital visual systems. Modern endoscopic devices usually consist of a digital RGB camera system attached to a thin rigid or flexible tube, which can be inserted through small incisions or natural orifices (e.g., mouth or nostrils) into the patient’s body. The endoscopic tube is designed in such a way that tiny surgical instruments can be inserted and used by the surgeon in order to reach out for and operate on the internal organs. It further contains a carefully optimized optical fiber and lens system. In combination with an external light source and the attached camera system, this ensures both the proper illumination of the organs under inspection and the correct transmission of the image from the objective lens to the capturing device, i.e., the digital camera.

However, image quality and color reproduction still remain a big issue for such medical applications, which—according to the authors’ best knowledge—is addressed by only a few manufacturers because of missing obligatory standards in this field. Regarding other fields of application of digital camera systems, it is a known consensus that the issue of color reproduction has to be considered thoroughly to obtain qualitatively good and pleasant results. In this context, an interview-based investigation of the color image preference of surgeons revealed that a high acceptance of fully digital MIS devices could only be achieved if the experienced visual impressions resemble those usually perceived when using purely optical systems, i.e., systems they got used to over the years, which basically provide genuine colorimetric information.⁷ Hence, even though a good surgeon under the assumption of sufficient image contrast might still efficiently perform a surgery, in case various organic tissues were represented in colors deviating from their actual ground truth, there is strong evidence that a proper color reproduction is desirable to achieve high user acceptance. Furthermore, it is the authors’ belief that a colorimetrically genuine representation of the captured image data also provides an indispensable part for appropriate digital archiving and teaching purposes.

Assuming raw image capturing, the color reproduction of digital endoscopes and MIS devices is mainly influenced by the camera color filters, the optical transmission properties of the endoscopic tube, and the spectrum of the illuminating light source, which in modern solutions is formed by a highly efficient, multispectral LED light engine usually tuned to the emission of cool white light. In this context, perfect color reproduction means that the reproduced colors of, for example, inner organs would exactly match their direct perception under the same illumination. With this in mind, one can easily imagine that a more or less perfect color reproduction and, therefore, negligibly small colorimetric errors are crucial for the success of a surgery, where different kinds of tissues must be distinguished by the surgeon not only through their texture but also through their color. However, perfect color reproduction is usually not the case because, in general, the RGB-sensor of a digital camera system does not fulfill the required Luther–Ives condition,^8–10 i.e., the spectral response curves of the RGB-sensor cannot be described as a linear combination of the eye cone response functions. As a consequence, more or less severe colorimetric errors occur when comparing the direct perception of the surgical field to its reproduction.

To correct for these systematic errors of digital cameras, various linear and nonlinear characterization approaches and methods have been proposed in the literature.^11–20 Despite their varying degree of complexity, they all have in common that, for a given illuminant, they try to minimize the perceived color differences between the reproduction of some test colors and the direct perception of the same test colors in a device-independant uniform color space (UCS). In practice, most manufacturers apply a simple linear least-mean-squares optimization based on some standardized color charts, as described by Hubel et al.²⁰ However, as one might expect, the reflectance properties of such printed color charts generally have nothing to do with those of real human tissue a surgeon is confronted with in the operating room and, therefore, such an optimization can only be considered as a rough estimate rather than a colorimetrically proper characterization. A significant improvement could only be achieved if the in-situ reflectance properties of the human tissue under inspection were known, which is usually not the case. The aim of this paper, therefore, is to provide an easy-to-implement reconstruction approach based on the Wiener filter²¹ that estimates the spectral reflectances of organic tissue from a series of images captured under varying LED illumination. It should be further shown that once the in-situ spectral reflectances are known, significantly improved color correction is feasible.

The paper is organized as follows. In Sec. 2, the applied Wiener-filter approach for estimating the spectral reflectances is introduced. Section 3 gives an overview of the experimental setup, whereas in Sec. 4, some results of the spectral reflectance estimation are presented. In Sec. 5, a method based on in-situ measured reflectance spectra is discussed, which attempts to enhance the color correction properties of video-assisted MIS devices. Finally, the paper closes in Sec. 6 with some concluding remarks and an outlook on future research intentions.

2.

Wiener-Filter Estimation Approach

Estimating in-situ spectral reflectances from digital camera responses of a modern endoscope generally is an ill-posed problem^22,23 since the dimensionality of the camera signals is much smaller than the number of spectral components that should be reconstructed. In the literature, several strategies have been proposed to tackle this problem making use of a priori knowledge of the underlying acquisition process.^12,22–36 In this context, one of the most important and widely used reconstruction methods is the so-called Wiener estimation, which is also the method of choice in the present work. Despite its simplicity, the Wiener estimation approach has been proven to provide accurate results regarding the spectral reconstruction of nonflourescent natural reflectances.^22–26

Assuming a normal distribution and statistic independence of reflectances and system noise, the reflectance-estimating Wiener Filter, which is applied to the camera responses, minimizes the mean-squared error (MSE) between the estimated and the actual reflectance of a surface color. Basically, the responses ${\mathbf{c}}^{(i,j)}={[c_R^{(i,j)},c_G^{(i,j)},c_B^{(i,j)}]}^T$ of a linear trichromatic RGB camera system at a pixel location $(i,j)$ on the camera sensor are given by

For the sake of simplicity, the pixel position indices $(i,j)$ are omitted in the following and Eq. (1) can be rewritten as follows:

According to Refs. 2324.–25 and 37, the above reflectance-estimating Wiener filter matrix is defined in such a way that it minimizes the MSE between the estimated and the actual reflectance and, therefore, can be derived as follows:

3.

Experimental Setup

Lacking a real MIS device equipped with a multispectral LED light source, we decided to set up an experimental environment, as shown in Figs. 1(a) and 1(b), which simulates the surgical workflow using a commercially available single lens reflex (SLR) camera and a customized six-channel LED light engine. Here, the abdomen or thorax of a patient is abstracted by a 40 cm × 40 cm × 60 cm wooden box with diffusely reflecting white painted walls on the inside in order to guarantee homogeneous illumination of the organic tissue under inspection. Please keep in mind that this setup should only provide a first proof-of-concept rather than mirroring reality.

Fig. 1

Illustrations of the artificial abdomen/thorax used in the experiments. The surgical workflow of a video-assisted MIS device should be simulated using a commercially available SLR camera and a customized six-channel PWM-controlled LED light engine. The artificial abdomen/thorax is formed by a $40\mathrm{cm}\times 40\mathrm{cm}\times 60\mathrm{cm}$ wooden box with diffusely reflecting white painted walls on the inside in order to guarantee homogeneous illumination. (a) Schematic representation of the experimental setup. The organic tissue sample is placed inside the box right beneath the measurement device, which is kept in position by a specially designed fixture. The illumination is realized using a multicolor edge-LED arrangement covered by diffusing panels and installed on all four sides of the lid. (b) Image showing the experimental setup as it can be found in our laboratory. Here, the radiance measuring head of the employed array spectrometer is attached to the fixture facing down toward the homogeneously illuminated test sample inside the box. Measurements were always performed in this $0\hspace{0.17em}\mathrm{deg}/0\hspace{0.17em}\mathrm{deg}$ geometry.

As can be seen from Fig. 1(a), the illumination is realized from above by a six-channel (red, deep red, green, blue, warm-white, and cool-white) edge-LED arrangement consisting of multicolor LED stripes being installed on all four sides of the box’s lid. Several diffusing panels covering the LEDs and forming the bottom of the lid guarantee diffuse and homogeneous light emission into the artificial abdomen/thorax. An aperture ring with a diameter of 10 cm embedded in the center of the lid together with a specially designed fixture allows for an easy insertion and alignment of different measurement devices. Furthermore, each LED channel can be controlled separately via pulse-width modulated (PWM) dimming and the corresponding normalized SPDs are illustrated in Fig. 2. They were measured in $0\mathrm{deg}/0\mathrm{deg}$ geometry using a white reflectance standard placed centrally inside the experimental box in combination with an Instrument Systems CAS 140CT array spectrometer connected to a TOP 200 radiance measuring head, as shown in Fig. 1(b).

Fig. 2

Normalized SPDs of the six different PWM-controlled LED channels red, deep red, green, blue, warm-white (WW), and cool-white (CW). The SPDs were measured in $0\mathrm{deg}/0\mathrm{deg}$ geometry using a white reflectance standard and an array spectrometer.

During the actual experiment, i.e., the image capturing and subsequent spectral reconstruction of organic tissue reflectances, the spectrometer head is removed from the box and a Canon 750D SLR digital camera is inserted at exactly the same position with its objective facing perpendicularly down toward the test object being placed right beneath the camera in the center of the bottom of the experimental box. Instead of human organs, freshly butchered pig and cattle organs were used as test objects to investigate the performance of the reconstruction algorithm. Due to biological similarity, it is supposed that results can easily be transferred to MIS applications involving human tissue.

As mentioned in Sec. 2, an easy-to-implement Wiener filter approach is used to estimate the spectral reflectances of the different test organs from the digital $RGB$ camera responses of the Canon 750D. One problem that arises with this estimation method is that more than three camera responses are usually needed to obtain high accuracy.³⁸ This is because of the extremely ill-posed nature of the calculation when mapping only three values $R$, $G$, and $B$ to reflectance intensities at a few hundred different wavelengths (e.g., 401 different wavelengths from 380 to 780 nm if 1 nm sampling is applied). In order to overcome this problem, we combine multiple captures of the test organs illuminated by different LED SPDs [six in our case, see Fig. 2]. As stated in Ref. 39, this allows us to significantly increase the effective dimension and, as a result, make a more accurate estimate of the spectral reflectances without the need of additional constraining assumptions on their properties (positivity, boundedness, and smoothness must be fulfilled for naturally occurring reflectance spectra).

Hence, the reflectance-estimating Wiener filter of Eq. (4) needs to be adjusted accordingly. Considering the six different illumination conditions ${\mathbf{l}}_1,\dots ,{\mathbf{l}}_6$ of our experimental box, the combined lighting-sensitivity matrix $\mathbf{\Lambda }$ must be rewritten as follows:

Last but not least, the noise autocorrelation matrix ${\mathbf{K}}_{\mathbf{nn}}$ still needs to be determined. Here, we follow the recommendation given in Ref. 40, where the noise of camera channel $\mu =R,G,B$ under the $v$’th illuminant was estimated through the variance ${\sigma }_{\mu \nu }^2$ in the signal value of some central region $A$ surrounding the evaluation point $p(i,j)$ of which the spectral reflectance should be reconstructed. Assuming the noise of different camera channels to be independent, as stated in Sec. 2, its autocorrelation matrix can finally be expressed as follows:

It should be noted that in the present study, the digital camera is operated in raw image capturing mode. Instead of using a debayering algorithm, the camera response vector ${\mathbf{c}}^{(i,j)}$ belonging to $p(i,j)$ is therefore obtained by averaging the corresponding signal values within the surrounding pixel region $A$. Furthermore, the spectral responsivities ${\mathbf{s}}_R$, ${\mathbf{s}}_G$, and ${\mathbf{s}}_B$ of the Canon 750D, which are necessary to construct the Wiener Filter, were measured using a monochromator setup, as described in Ref. 41. Results are illustrated in Fig. 3. As can be seen, the Canon 750D used in our experiments is unresponsive to incident light below 400 nm and above 700 nm due to its built-in ultraviolet and infrared band-elimination filters. For this reason, we limit the range of all subsequent measurements and calculations to a spectral range between these 400 and 700 nm with a 1 nm sampling, i.e., $n=301$.

Fig. 3

Relative spectral responsivity of the Canon 750D color channels $(RGB)$ measured by using a monochromator setup. Note that the spectral responsivity of a camera system results from the combination of the spectral transmittance of the used objective, the spectral transmittance of the Bayer pattern, and the spectral sensitivity of the monochrome camera sensor underneath.

4.

Results of the Spectral Reflectance Estimation

In order to evaluate the accuracy of the Wiener estimation approach described in this paper, the results of both measured and estimated reflectances should be compared for a certain variety of organic tissue. Here, the term “measured” refers to the results obtained from direct radiometric measurements provided by a spectrometer, while the term “estimated” denotes the spectral reflectances that were reconstructed from $RGB$ camera responses by applying the Wiener filter approach discussed above. The following tissue samples have been chosen: a pig’s heart, tongue, stomach, lung, liver, spleen, and kidney as well as a freshly butchered beef steak representing muscle tissue.

For the measurements of the spectral reflectances, the same measurement equipment and geometry, as shown in Fig. 1, was used with the radiance measuring head of the array spectrometer being placed on the top of the experimental box and connected to the fixture holding it in position. The measuring aperture was adjusted to 0.3 deg. The PWM values of the six LED channels were optimized to simulate CIE D65 illuminant and we measured the resulting SPD ${\mathbf{l}}_{\mathrm{w}}$ reflected by a calibrated white standard with known spectral reflectance ${\mathbf{r}}_{\mathrm{w}}$, which had been positioned right beneath the spectrometer head, see Fig. 1(a). Afterward, the white standard was removed from the box and a tissue sample was placed at exactly the same position. We measured the light ${\mathbf{l}}_{\text{sample}}$ being reflected from its surface. The spectral reflectance of the tissue sample can eventually be calculated by

Next, the measuring head was removed and replaced by the Canon 750D without touching or moving the organic tissue sample under inspection. Note that due to the specially designed fixture, the optical axis of the camera and of the spectrometer head could be adjusted to virtually lie exactly on top of each other so that, in both cases, the same evaluation point $p_0$ given by the intersection of the optical axis with the tissue surface could be chosen for direct measurement and reconstruction.

During image capturing, the six different LED channels were separately switch on and off one after the other and, for each of these illumination conditions, an image of the organic tissue sample was taken resulting in a total number of six $RGB$ raw images per sample. The images were stored to disk and subsequently processed by applying the Wiener filter matrix $\mathbf{W}$ to the extracted camera response vector $\mathbf{c}$ at location $p_0$, as described in Secs. 2 and 3, to obtain an estimate $\widehat{\mathbf{r}}$ for the actual spectral reflectance ${\mathbf{r}}_{\text{sample}}$. To guarantee that the same spectral information reflected from the organic tissue surface was used for both the direct measurement and the estimation, the pixel area $A$ surrounding $p_0$, which is needed to calculate the variances of the noise matrix and the camera response vector (see Sec. 3), was limited to the size of the measuring spot of the spectrometer projected onto the surface of the tissue sample. This additional constraint allows for a better and more meaningful comparison between the direct measurement and the performance of the Wiener filter algorithm, since it eliminates in sufficiently good approximation, the uncertainties attributed to the different measuring geometries that otherwise would have had a large impact and, therefore, enables the proper determination of the scaling factor $\delta$ in the current experimental setup. Here, this factor was optimized in such a way that the spectral reflectance of a known color sample measured at the same position and under the same geometry as later the organic tissue samples was reconstructed as good as possible. With the same calibrated white standard as mentioned previously being chosen for this purpose, negligibly small deviations of less than 0.5% at each wavelength were observed between the known reflectance spectrum of the white standard and its Wiener filter estimate obtained for the optimal scaling factor ${\delta }_{\mathrm{opt.}}$, which was subsequently used as a fixed value in the definition of the lighting-sensitivity matrix $\mathbf{\Lambda }$ given by Eq. (5) and, therefore, in the final Wiener filter matrix $\mathbf{W}$ to be applied to the camera response vector $\mathbf{c}$. Please note that with the scaling factor $\delta$ being dependent on the measuring geometry, especially on the distance between tissue sample and illuminating light source, a thorough precalibration similar to the one described here must be provided by the manufacturers of such future MIS devices in order to allow for a proper in-situ reflectance estimation of organic tissue.

Finally, regarding the performance validation of the Wiener filter ansatz for such a purpose, the measurement protocol and experimental setup described in this and the preceding section, respectively, were used to reconstruct the spectral reflectances of several different organic tissue samples from digital camera responses. Figure 4 summarizes the obtained reconstruction results and compares them with the corresponding measured reflectances derived from Eq. (9). With the exception of the noisy part of the measured spectra below 430 nm, pretty good agreement can be observed for all samples. In Table 1, the degree of deviation between the estimated and measured reflectances is expressed in terms of the MSE metric and the CIEDE2000 color differences $\mathrm{\Delta }{E}_{00}$.⁴² Although the former is pretty common in statistics to be used for comparative purposes with a value of zero indicating perfect agreement, the latter being the current CIE recommendation for computing small color differences allows for drawing conclusions about the similarity of the visual appearance of the estimation and the measurement when hypothetically viewed under identical illumination conditions, which in the present case are assumed to be defined by reference illuminant D65. For completeness, the chromatic-only differences $\mathrm{\Delta }{E}_{00}^{\mathrm{chrom.}}$ are also tabulated.

Fig. 4

Comparison of the measured (dashed blue line) and estimated (solid red line) reflectance spectra of eight different organic tissue samples. The relatively large noise in the measured spectra below 430 nm is due to the very low light emission of the LED light source at small wavelengths (see Fig. 2) leading to a bad signal-to-noise ratio in the measurement device. When calculating the quotient given by Eq. (9), this eventually results in large noise values. (a) Steak, muscle, (b) heart, fat, (c) liver, (d) kidney, (e) lung, (f) stomach, (g) tongue, and (h) spleen.

Table 1

Overview of the observed degree of deviation between the estimated and measured reflectance spectra of various organic tissue samples in terms of MSE, ΔE00, and ΔE00chrom. values. Due to the large measurement noise below 430 nm, only signals at larger wavelengths were considered for the calculations. In addition, reference illuminant D65 was assumed for computing the color differences.

As can be seen, relatively small MSE values ranging between $7.28\times {10}^{-4}$ and $2.61\times {10}^{-5}$ with an average of $2.15\times {10}^{-4}$ are obtained for various tissue samples. Compared to the MSE performance of the Wiener filter approach reported by other researchers,^22,38,43,44 where typical values were found to be of the order of $\sim 7\times {10}^{-4}$, this basically indicates good to excellent absolute agreement between the estimated and measured reflectances. Furthermore, regarding the corresponding CIEDE2000 measure, most of the tissue samples, with the exception of lung and spleen, exhibit barely noticeable color differences ranging between 0.74 and $2.10\mathrm{\Delta }{E}_{00}$ with an average of $1.39\mathrm{\Delta }{E}_{00}$ close to the empirically determined just noticable limit defined by a $\mathrm{\Delta }{E}_{00}$ of unity.⁴⁵ Again, these results indicate excellent performance of the Wiener filter estimation and conform with previous findings.⁴⁴ Even the larger color differences obtained for the tissue samples of lung and spleen, which might be noticed also by unexperienced observers⁴⁵ (but only if they had the chance of comparing the reconstructed and the original tissue samples side by side), are not critical and still in an acceptable range because for the intended application of the spectral reconstruction algorithm in the present work, it is more important to reproduce the general trend of the in-situ reflectance spectra, as shown in Fig. 4, than to exactly match their theoretical visual appearance. In other words, even though the estimated tissue reflectances, which are used for optimizing the color correction matrix as described in the next section, show some kind of offset in comparison to their ground truth, the resulting linear color correction matrix—as we will see—still features excellent performance with respect to reproducing the color appearance of the original tissue samples.

5.

Enhanced Color Correction for Video-Assisted Surgery

5.2.

Results of the Optimized Color Correction and Further Implementation Notes

Assuming D65 reference illuminant, the resulting CIEDE2000 color differences $\mathrm{\Delta }{E}_{00}$ between the color corrected and the actual tristimulus values of the eight original tissue samples selected for performance validation can be calculated using either of the two color correction matrices. The corresponding distributions of the obtained color differences are eventually compared and visualized in Fig. 5.

Fig. 5

Illustration of the CIEDE2000 color differences $\mathrm{\Delta }{E}_{00}$ between the color corrected and the actual tristimulus values of the original tissue samples shown in Fig. 4. The results of the standard color correction matrix ${\mathbf{M}}_{\mathrm{SMI}}$ are compared to those obtained for the specifically optimized matrix ${\mathbf{M}}_{\mathrm{opt.}}$. As further emphasized by the corresponding box plots shown in the inset, the latter gives much smaller color differences for all tissue samples indicating superior color reproduction properties. With the statistical comparison of both methods showing a $p$ value of 0.0016 and an effect size $d$ of 2.21, a significant enhancement of the color correction for video-assisted surgery based on in-situ reflectance estimation could clearly be confirmed.

As can be seen, the application of the specifically optimized matrix ${\mathbf{M}}_{\mathrm{opt.}}$ for performing the color correction leads to heavily reduced color differences $\mathrm{\Delta }{E}_{00}$ for all eight tissue samples when compared to the application of the standard matrix ${\mathbf{M}}_{\mathrm{SMI}}$. While the former shows an average color difference of only $0.42\mathrm{\Delta }{E}_{00}$, the latter performs much worse giving an average color difference of $1.22\mathrm{\Delta }{E}_{00}$, which is approximately three times larger. By making use of a two-sample $t$-test for unequal variances,^50,51 whose application requirements have been tested using the Shapiro–Wilk test^52–54 (checking for normality) in combination with the Brown–Forsythe extension of Levene’s test⁵⁵ (checking for equal variances), the significance of the observed improvement for the optimized color correction matrix ${\mathbf{M}}_{\mathrm{opt.}}$ could clearly be confirmed ($T=4.42$, $p=0.0016$, $\alpha =0.05$) showing a large effect size $d$⁵⁶ of 2.21. Hence, it can be concluded that especially in cases where a proper distinction of different kinds of tissue is necessary, significant enhancement in color correction by a factor of three might be crucial for the success of a surgery. Furthermore, with a demand for improved color reproduction properties of digital systems being reported for such kind of applications (see Sec. 1), these results clearly emphasize the importance of our in-situ approach for modern MIS surgery. In the attempt of providing a better visualization of the benefit of our proposed method, Fig. 6 exemplarily compares for the organic samples of lung and muscle tissue the respective color image representations of the debayered raw image data as being extracted from the Canon 750D imaging device before color correction and the color corrected image data obtained by applying ${\mathbf{M}}_{\mathrm{opt.}}$ to the raw camera responses at each pixel location. As can be seen, the image quality on the right-hand side of Fig. 6 associated with the optimized color correction seems to be much improved compared to what raw image processing would deliver: it is not just that the colors and tissue structures appear to be much more realistic, the color corrected images also provide more perceived depth and visual information than their uncorrected counterparts. Please note that in all cases, the same white balance was applied.

Fig. 6

Comparison between the exemplaric color image representations of the debayered raw image data (a and c) and the color corrected image data using ${\mathbf{M}}_{\mathrm{opt.}}$ (b and d) for the organic tissue samples of lung (upper row) and muscle (lower row). The blue square-shaped targets define the measurement points at various locations on the surface of the respective tissue sample. Please note that in all cases, the same white balance was applied.

In practice, the whole process of recovering the true color information from captured $RGB$ images can be implemented to work reliably and without any noticeable disruption of the surgical workflow simply by adopting a short flashing sequence of the different LED channels combined with a fast image capturing to create the necessary input data for running the Wiener filter estimation (see Sec. 3). Including the subsequent reconstruction of the targeted tissue reflectance spectra from the $RGB$ camera responses, this can principally be realized to be accomplished within a few hundred milliseconds. With the application of the Wiener filter approach being basically given by a simple and fast-performing matrix multiplication [see Eq. (3)], the limiting factor in this context is the integration time of the actual imaging system. Assuming, for example, an MIS device which offers 60 fps, it would only take 100 ms for the flashing sequence and image capturing to be finished in case that the device light source consists of six different individual LED channels similar to the ones used in the present study. Hence, it can be concluded that the higher the frame rate the faster the flashing sequence and, therefore, the spectral reconstruction can be performed.

Regarding the issue of explicitly selecting the tissue samples, which are intended for being spectrally reconstructed, two complementary modes of operation are favored during a surgery.⁷ First, the surgeon should be given the option to run an auto-tuned optimization of the color correction applied to the digital reproduction of the perceived surgical field. For this purpose, a certain number of arbitrary yet suitable pixel positions has to be selected automatically by the MIS device (the corresponding algorithm still needs to be developed) in order to perform the reflectance estimation of meaningful in-situ tissue samples that can eventually be used to enhance the color reproduction of the digital system following the methodology described above. Second, they should have the option to define the pixel positions on their own by using an appropriate, device-dependent input method. Besides allowing for an improved device color correction, which explicitly considers the spectral characteristics of (critical) tissue samples judiciously selected by the surgeon, it also enables them to add further information and tissue metadata for archiving and teaching purposes. As a long-term goal, this kind of data in combination with the corresponding reconstructed reflectance spectra and, for example, an intelligent machine-learning algorithm could additionally be used (i) for automated tissue recognition as a support of the surgical workflow and (ii) to continuously extend and customize the database applied to train the Wiener filter leading to an improved performance of the MIS device for specific applications.

In any case, after having defined a certain number of device pixel positions (either automatically by the system or manually by the surgeon), the flashing sequence for enabling the reflectance estimation of the corresponding tissue samples must only be triggered once, i.e., with the exception of this quite short period of time, where the individual LED channels are switched on and off one after the other, the real-time image capturing is not affected by the applied Wiener filter estimation. The subsequent optimization of the color correction matrix can efficiently be performed on the microcontroller or CPU of the MIS device, which takes another few milliseconds and can easily be decoupled from the continuous image capturing so that like in the case of the Wiener filter estimation, this parallelly performed computation has no negative effect on the image quality, the contrast, or the actual imaging time of the MIS device. On the contrary, once the optimized color correction matrix has been calculated, it can be used to simply replace its standard counterpart in the imaging processing pipeline eventually leading to superior image quality and color reproduction properties as it has been shown previously in this work. However, it should again be noted here that the final implementation of the proposed method into a real video-assisted MIS device is still pending and, therefore, might entail further problems that one is not aware of yet.

6.

Conclusion and Outlook

In this paper, an approach of optimizing the color reproduction properties of modern MIS devices based on the estimation of in-situ spectral reflectances applying the Wiener filter approach was proposed. Even though the Wiener filtering is widely used in image processing as a common way of reducing noise and image blurring^21,57–59 as well as in color science as a method for estimating camera sensitivities³⁹ and object reflectances from camera response data,^{22–26,37,38,40} its usage described in the current paper is the first time that the results of such an estimation were adopted for the in-situ enhancement of the color reproduction properties of an imaging system based on the context-specific optimization of the corresponding color correction matrix for surgical applications. From this perspective, the current work should be considered as a first proof-of-concept, which, nonetheless, already clearly emphasizes the potential benefits resulting from the newly proposed method when being implemented into modern video-assisted MIS devices. In addition, it gives a detailed overview on how such an implementation can efficiently be performed and, at the same time, highlights further possible advantages such a system would offer like, for example, automated tissue recognition and true color archiving. Hence, it can be concluded that the present paper provides a significant contribution for future developments on the topic of video-assisted MIS surgery giving guidance for researchers and manufacturers working in this field on how to achieve superior image quality and additional system benefits.

Lacking an appropriate MIS device and the opportunity to test our approach under realistic conditions, an artificial abdomen/thorax was constructed to simulate the surgical workflow in the laboratory. While an array spectrometer connected to a radiance measuring head was used for the direct measurements of the spectral reflectances of test samples placed inside the thorax, the necessary input data for the Wiener filter to perform an accurate reflectance estimation were provided by a combination of a commercially available SLR digital camera and a customized six-channel LED light source. With this experimental setup, good to excellent agreement between the direct measurements and the reflectance estimation could be observed for a variety of different organic tissue samples.

Once the in-situ reflectance spectra were known from spectral reconstruction—at least to a sufficiently good approximation—it could be shown that an optimized color correction was feasible. Based on the results of the spectral reflectance estimation, a variation of the ISO standard method originally designated for the color characterization of digital cameras was applied here to derive a more sophisticated color correction matrix specifically optimized for accurately reproducing the color properties of the organic tissue samples under inspection. Statistical analysis eventually confirmed a significant enhancement in color reproduction when the proposed method is applied compared to what could be achieved by the standard procedure of colorimetric characterization. On average, three times smaller color differences could be observed, which, in a surgical application, may lead to less tissue damage and an increase in safety for the patient, especially in cases where a proper distinction of different kinds of tissue might be crucial for the success of a surgery.

Future research intentions mainly aim for porting, testing, and optimizing our approach on a real MIS device. In this context, it should be analyzed which LED channels are indispensable and which can be considered to be redundant in order to provide sufficiently good reconstruction results for achieving significantly enhanced color reproduction properties. So far, the LED selection in the current work was more or less arbitrary with the sole objective of being able to imitate reference illuminant D65 as good as possible. Even though verification is still pending, the blue LED channel in particular is supposed to be superfluous for the intended application of reconstructing the spectral reflectance properties of human tissue and could most likely be omitted in a prospective implementation.

Since the reported results so far were only obtained for ex vivo animal tissue samples, a further research question that still remained unanswered but should be addressed in the near future is whether the Wiener filter-based approach implemented for the current work is also suitable to be used for applications involving in vivo real human tissue. Due to biological similarity, it is indeed assumed that comparably excellent results could also be obtained in these cases—nevertheless, proof is still necessary and can only be provided after the proposed method has been successfully ported to a feature-complete MIS test device, which, as stated at the beginning of the last paragraph, should therefore be the main focus of further considerations.