2.1.

Multiview Imaging

The multiview imaging setup consists of a Pentax Kx digital single-lens reflex (DSLR) camera with a resolution of $4288\times 2848\text{pixels}$ (Ricoh Imaging Company Ltd., Tokyo, Japan) and a square mirror set. The square mirror set was composed of four plane mirrors that were cohered together with a square $70\times 70\mathrm{mm}$ cross section. This configuration allowed us to obtain five different views of the target within a single image at exactly the same time. With a 3-D reconstruction algorithm, the 3-D superficial characteristics of the tissue could be obtained.

The basic principle underlying the multiviewpoint imaging system is shown in Fig. 1. The multiview camera is placed on one end of the square mirror, and its optical axis is coincident with the axis of the square mirror set. Each of the four mirrors produces a virtual subcamera on the other side. Thus the virtual subcameras are symmetric with respect to the optical axis, and the distance between a real subcamera and a virtual subcamera is equal to that between the opposite mirrors. The five “cameras” will take pictures of point $P$ from different viewpoints. As a result, 3-D topography of the scene can be achieved by correlating the matching pixels between the objective image and the reflective images.

Fig. 1

Principle of the multiview imaging system.

2.2.

Reconstruction of Structural Characteristics

2.2.2.

Image undistortion

The camera in the multiview imaging was a Pentax Kx camera with a Pentax DAL18-55 mm lens. Since a wide-angle view including the objective image and all the reflection images was needed, the focal length was set to 18 mm. However, wide-angle lenses brings distortions to the captured images, and those distortions can be corrected by the camera calibration. As shown in Fig. 2(a), 15 images of a flat checker board placed at different positions and orientations were captured. Knowing that the size of each square on the checker board is $15\times 15\mathrm{mm}$, the focal length, principal points, skew coefficient, distortion coefficient, and its extrinsic parameters were obtained using the camera calibration toolbox.⁷ Then any captured images with distortion can be undistorted. Figure 2(b) shows the reconstructed checker board position with respect to the camera in each captured image.

Fig. 2

Calibration of the multiview camera: (a) 15 images of a flat checker board placed at different positions and orientations; and (b) the position of the reconstructed checker board with respect to the camera in each captured image.

2.2.3.

Stereo calibration of the real and the virtual subcameras

The 3-D coordinate of a space point can be calculated by any two of the real and virtual subcameras shown in Fig. 1. In our system, the real subcamera was stereo-calibrated with each virtual subcamera, respectively, so we were able to use four stereo image pairs to reconstruct the same point. Viewing from these five subcameras at the same time can significantly reduce the occlusion caused by the fluctuant surface of the object. For each real-virtual subcamera pair, we calibrated it as a stereo system intrinsically and extrinsically using the method provided in the toolbox. These parameters were used to rectify the image pairs of the sample and convert the 2-D pixel coordinate pair into 3-D world point coordinate in the future. Similarly, 15 images of a small flat checker board with the size of $5\times 5{\mathrm{mm}}^2$ squares were taken. Each of the images was cut into five small images corresponding to the real and the virtual subcameras. Those virtual subcamera images were flipped as they were reflected by mirrors. Thus we got four groups of small checker board calibration images: middle-left, middle-right, middle-top, and middle-bottom. Figure 3(a) shows one of the small checker board calibration images, and Fig. 3(b) shows the cut and flipped images. Figure 3(c) displays one pair of cameras and the reconstructed position of the calibration planes after the stereo calibration.

Fig. 3

Stereo calibration of real-virtual subcamera pairs: (a) the image of the checker board; (b) the five cut and flipped views of the checker board; and (c) one pair of cameras and the calibration planes after stereo calibration.

2.3.

Multiview Hyperspectral Imaging

The procedure of multiview hyperspectral imaging consists of the following three consecutive steps: (1) multiview images were acquired using the reflective mirror set and the high-resolution camera in order to reconstruct 3-D topography of the objective tissue; (2) hyperspectral data cubes were acquired for calculation of a tissue oxygenation map; and (3) 3-D tissue topography and the oxygenation map were fused for 3-D mapping of tissue functional parameters.

Two methods were used to obtain the hyperspectral images of the objective tissue. The first one is to utilize OL 490 Agile Light Source (Gooch & Housego, Orlando, Florida) as the tunable light source that provided a wide wavelength range (380 to 780) nm at a spectral resolution of 2 nm. A fiber optic ring light guide (Edmund Optics, Barrington, New Jersey) was connected to the tunable light source to provide uniform illumination. The second method is to mount a liquid crystal tunable filter (Cambridge Research Inc., Cambridge, Massachusetts) with the wavelength range of 400 to 720 nm and a bandwidth of 7 nm in front of the camera lens to enable imaging at designated wavelengths. A Hamamatsu C4742-80-12AG air-cooled digital CCD camera (Hamamatsu, Bridgewater, New Jersey) with $1344\times 1024\text{pixels}$ resolution was mounted at the same position of the DSLR camera to capture hyperspectral images.

The experimental system as shown in Fig. 4 was carefully aligned for the coincident optical axes of two cameras. Two cameras were alternately used to acquire multiview and hyperspectral images to guarantee sufficient resolution for the multiview imaging and the high sensitivity for hyperspectral imaging simultaneously. The camera for multiview imaging has a resolution of $4288\times 2848\text{pixels}$, which is similar to the resolution of the hyperspectral camera after being equally divided into five separate views. The hyperspectral camera has a high sensitivity (quantum efficiency $>50\%$) at wavelengths from 400 to 700 nm. Switching between the two cameras and coregistering their corresponding images would be the best way to obtain both high-quality structural and functional information of the objective tissue. We took out the middle view from the multiview camera and registered it with the hyperspectral image. The scale-invariant feature transform method was used to find at least three sets of feature points on the two images, and the transform matrix was calculated.¹¹ Since the 3-D model was reconstructed based on the middle view, with the transform matrix we were able to map any functional information obtained from hyperspectral imaging onto the 3-D results. After obtaining the hyperspectral images of the objective tissue, a wide gap second-derivative spectroscopic technique was used to reconstruct tissue oxygenation maps based on the hyperspectral images.¹²

Fig. 4

Multiview hyperspectral imaging system. The 3-D structural characteristics of the object obtained by the multiview camera, and the functional characteristics of the object were obtained through oxygenation algorithm on the hyperspectral images captured by the hyperspectral camera.

2.4.

Validation Experiment

2.4.1.

Three-dimensional reconstruction accuracy and axial resolution measurement

To determine the reconstruction accuracy of the multiview system, we used two known geometry objects as the imaging targets, a flat checker board and a cylindrical checker board with a diameter of 57.70 mm, as shown in Fig. 5(a). The captured images are shown in Figs. 5(b) and 5(c). Sixty-four corner points of the black and white squares were extracted, and their spatial coordinates were reconstructed using the calibrated parameter obtained in the previous work. Then a plane and a cylindrical surface were fitted using these set of points as the reconstructed surfaces. The root mean square error between the reconstructed points and the best-fit surface can indicate the reconstruction accuracy of the multiview system.

Fig. 5

Checker board validation for multiview imaging: (a) a flat checker board and a cylindrical checker board; (b) multiview image of the flat checker board; and (c) multiview image of the cylindrical checker board.

To measure the axial reconstruction resolution, we mounted the flat checker board onto a linear motorized translation stage PT1-Z8 (Thorlabs Inc., Newton, New Jersey) that has a minimal achievable incremental movement of $0.05\mu \mathrm{m}$ and a bidirectional repeatability of $<1.5\mu \mathrm{m}$. The translation stage moved 7 mm in total along the axis of the multiview system with a step size of 0.5 mm. At each step, we captured one image and reconstructed the plane at that position. The minimal axial distance of the system can be resolved by comparing the distance between each reconstructed plane. Figure 6(a) shows the configuration of the accuracy test experiment. Figure 6(b) shows the images we captured at different axial positions.

Fig. 6

Axial accuracy measurement of the multiview system: (a) experimental set-up of the accuracy measurement and (b) an illustration of the images acquired at different height with a step size of 0.5 mm.

2.4.2.

Three-dimensional topography of a pressure ulcer foot model

A pressure ulcer foot model (3BScientific, Tucker, Georgia) with four chronic wounds representing different phases of the pressure-induced chronic wound was used in experiment to validate the multiview system, as shown in Figs. 7(b) and 7(c). One of the wounds was measured by the multiview imaging system, as shown in Fig. 7(a). The wound model was mounted on a gypsum base manufactured especially for it. A flat board with a 1-cm grid was also mounted onto the base to illustrate the size of the model. With the algorithm described in Sec. 2.2, we could reconstruct the 3-D morphology of the wound.

Fig. 7

Multiview imaging for 3-D reconstruction of the chronic wound model: (a) multiview imaging system for the wound model; (b) the wound model; (c) four representative wounds on the wound model; and (d) a photograph of the wound taken by the multiview imaging technique.

2.4.3.

Three-dimensional oxygenation mapping of a tissue-simulating blood phantom

We also combined multiview imaging with hyperspectral imaging for 3-D mapping of oxygenation. The experimental setup consists of the high resolution DSLR camera, the hyperspectral camera, the multiview reflective mirror set, the tunable light source, a light ring, and a tube simulated blood phantom, as displayed in Fig. 8(a). The tunable light source was used to illuminate the test objects at the designated wavelengths. To verify the performance of the multiview hyperspectral system, a tube-shaped blood phantom was placed at a tilted angle within the field of view. The topography of the phantom was reconstructed based on the multiview image, while the oxygenation map was reconstructed from the hyperspectral images using a wide gap second derivative spectroscopic technique. The tube-shaped blood phantom was prepared in advance by the following protocol: (1) add 20-g milk powder into deionized water to make 200-ml milk suspension; (2) mix 200 ml of milk, 70 ml of deionized water, and 12 ml of donated blood; and (3) put the mixture in a polycarbonate centrifuge tube of 60 ml. The oxygenation level of the phantom was adjusted by the drop-wise addition of sodium hydrosulfite solution into the phantom mixture by a syringe. The sodium hydrosulfite solution was prepared in advance by adding 1 g of hydrosulfite to 40 ml of deionized water. A series of blood phantoms with consequentially decreased oxygenation levels were produced by adding the following amount of sodium hydrosulfite solution into the centrifuge tube: 0, 0.1, 0.15, 0.15, 0.20, 0.25, and 0.30 ml, respectively. After the sodium hydrosulfite solution was added, the centrifuge tube was sealed and stirred for uniform distribution of oxygenation. A fiber optical sensor of an OOISensors system (Ocean Optics Inc., Dunedin, Florida) was inserted in the phantom to monitor the partial pressure of dissolved oxygen and compare with the oxygenation map acquired by hyperspectral imaging. The measurement of oxygen partial pressure was used to validate the multispectral oximetry technique because of an approximately linear correlation between oxygen saturation and oxygen tension in an oxygen saturation range from 20% to 80%.¹³

Fig. 8

Multiview hyperspectral imaging experiments: (a) tissue-simulating blood phantom and (b) in vivo tissue model.

3.1.

Accuracy and Axial Resolution for Three-Dimensional Topographic Reconstruction

Figure 9(a) shows the reconstructed points of the flat checker board imaged in Fig. 5(b) and their best-fit plane. The RMS error obtained by fitting the reconstructed points with a plane is only 0.0120 mm. Figure 9(b) shows a slice of the best-fit plane and the reconstructed points on it, indicating its high reconstruction accuracy. Figures 9(c) and 9(d) show the reconstructed points of the cylindrical checker board imaged in Fig. 5(c) and a slice across the best-fit cylinder, respectively. We used a cylindrical surface to fit these points. The RMS error between the reconstructed points and the best cylinder is 0.1491 mm. The diameter of the best-fit cylinder surface is 57.348 mm, which shows a very good reconstruction accuracy compared to the real value of 57.70 mm. The different colors in Figs. 9(a) and 9(c) represent different height values of the reconstructed images, with the scales shown as the $z$-axis.

Fig. 9

3-D reconstruction accuracy measurement results: (a) reconstructed points of the flat checker board; (b) a slice of the best-fit plane with the reconstructed points; (c) reconstructed points of the cylindrical checker board; and (d) a slice across the best-fit cylinder with the reconstructed points.

Figure 10 shows the results of the axial resolution measurement. Fifteen planes were reconstructed corresponding to the 15 axial positions of the flat checker board, with a total axial range of 7 mm, as shown in Fig. 10(a). Figure 10(b) shows a slice across the reconstructed points and the best-fit planes that are indicated in Fig. 10(a), where each layer of the best-fit planes can be easily distinguished. Thus, the axial resolution of 0.5 mm (translation stage movement step size) can be achieved by the multiview imaging technique.

Fig. 10

Axial resolution measurement results: (a) reconstructed result of 15 axial positions of the flat checker board; and (b) a slice across the reconstructed points and the best-fit planes.

3.2.

Reconstruction of Wound Model

Figure 11 shows the reconstructed topography of the wound model using the multiview imaging system. Figure 11(a) illustrates the 3-D reconstructed surface, where the different colors represent different height values of the reconstructed surface. Then we project the color textured image captured by the middle “real subcamera” onto the 3-D surface, as shown in Fig. 11(b).

Fig. 11

Topography of the wound model obtained by the multiview technique. (a) 3-D surface topology reconstructed from multiview images; and (b) 3-D image of the wound model after pasting the color textured image on 3-D surface topology.

3.3.

Multiview Hyperspectral Reconstruction of the Simulated Blood Phantom

Figure 12(a) shows the reconstructed topography of the tube, where the different colors represent different height values of the reconstructed surface. Figures 12(b)–12(h) display oxygenation maps of the tube-shaped blood phantom with the topography characteristics. From the figure, it can be easily observed that the oxygenation saturation of the blood phantom decreased along with the increased volume of the sodium hydrosulfite solution. This result indicated that the structural and functional characteristics of the simulated blood phantom could be easily achieved with the simple multiview hyperspectral method. The mean ${\mathrm{StO}}_2$ of the region of interest on the blood phantom obtained from the hyperspectral imaging and the partial pressure (${\mathrm{PO}}_2$) of dissolved oxygen in the blood phantom measured by the fiber optics probe connected to an OOISensors system are shown as a green curve and a blue curve, respectively, in Figs. 12(i) and 12(j). The consistency of the change of ${\mathrm{StO}}_2$ and ${\mathrm{PO}}_2$ from the recipe 1 to 7 also validates the multiview hyperspectral method.

Fig. 12

Multiview hyperspectral imaging of a tube simulating blood phantom: (a) topography of the tube; (b)–(h) topographic ${\mathrm{StO}}_2$ map of the tube simulated blood phantom from recipe 1 to 7, red color denotes a higher oxygen saturation, while blue color denotes a lower oxygen saturation; and (i) and (j) ${\mathrm{StO}}_2$ of the region of interest obtained from the hyperspectral imaging and the partial pressure of dissolved oxygen in the blood phantom measured by the fiber optics probe that connected to an OOISensors system.

3.4.

Multiview Hyperspectral Imaging of an In Vivo Tissue Model

Figure 13 displays the result of the occlusion experiment on the healthy subject. Figure 13(a) is the dynamics of the blood perfusion of the fingers during the occlusion experiment, while Fig. 13(b) is the dynamics of ${\mathrm{tcpO}}_2$ of the fingers during the occlusion experiment. Figures 13(c)–13(e) are the topography of the fingers with oxygenation characteristics at one moment during the baseline, occlusion, and reperfusion stage, respectively. The result indicated the occlusion effect and hyperemia effect clearly, which also implies that the system simultaneously can obtain topographic and oxygenation characteristics in live tissue.

Fig. 13

Multiview hyperspectral imaging of a healthy subject’s fingers in an occlusion experiment: (a) dynamics of blood perfusion of fingers during occlusion experiment; (b) dynamics of ${\mathrm{tcpO}}_2$ of fingers during occlusion experiment; (c) 3-D oxygenation distribution of the fingers at one moment during the baseline stage; (d) 3-D oxygenation distribution of the fingers at one moment during the occlusion stage; (e) 3-D oxygenation distribution of the fingers at one moment during the reperfusion stage; and (f) 3-D topography of the fingers.

4.1.

Camera Selection

In our system, a DSLR camera is used to acquire a multiview image while a hyperspectral camera is used to acquire multispectral images. Ideally speaking, it is better to use a single camera to acquire all the images. However, a single camera system cannot be implemented because a hyperspectral camera has a much lower total pixel amount ($1344\times 1024\text{pixels}$) than that of the DSLR camera. If we use a hyperspectral camera to capture all the images, these pixels have to be reassigned to nine views (including five views used for image processing), with only $448\times 448\text{pixels}$ for each view. In contrast, using a relatively low-cost Pentax DSLR camera can readily increase the pixel per view level to $1429\times 1429\text{pixels}$ so that 3-D reconstruction will yield a higher spatial resolution. A hyperspectral camera will acquire the center view of the test subject without sacrificing its achievable resolution. Our next step is to integrate multiview and hyperspectral imaging procedures using a single high-resolution hyperspectral camera.

4.2.

Image Time and System Alignment

Since we use two cameras to obtain structural and functional information of the subject, the cameras should be easily switched in order to shorten the operating time. The total imaging time includes multiview imaging time, hyperspectral imaging time, and the time to switch between two cameras. Multiview imaging only takes one shot on the target object. The exposure time ranges from 0.05 to 0.5 s, depending on the illumination and the reflectance conditions. Hyperspectral imaging takes five images of the object at five wavelengths for the blood oxygen measurement. The exposure time ranges from 0.05 to 1 s, depending on the illumination wavelengths. The time of switching cameras is less than 10 s. Therefore, the total time for acquiring a complete set of images is estimated less than 20 s.

To minimize the positional error induced by switching two cameras, we used a coaxial alignment for the optical axes of two cameras and coregistered the hyperspectral images with the center view image. First, both cameras were mounted onto clamping slides (Thorlabs Inc., Newton, New Jersey), slid along on a horizontal 66 mm optical construction rail (Thorlabs Inc., Newton, New Jersey), and fixed at specific positions. Second, a dovetail optical rail (Thorlabs Inc., Newton, New Jersey) was vertically aligned with a rail carrier slide (Thorlabs Inc., Newton, New Jersey) on the operation table below the camera. A round alignment disk target (Thorlabs Inc., Newton, New Jersey) was then mounted onto the slide with the horizontal position of the rail fixed so that the target can only move along the rail vertically. Third, the DSLR camera was aligned so that the target remained at the center of the acquired images as it was placed at different heights from 0 to 15 cm. The calibration procedure ensured that the camera was coaxial with the path of the moving target. The same calibration procedure was repeated for the hyperspectral camera so that the optical axes of both cameras were precisely aligned and readily switchable in a short time with accuracy. After each experiment, the acquired hyperspectral images and the center view image were coregistered in order to obtain the 3-D structural and functional information.

4.3.

Imaging Distortion Induced by Target–Camera Distance and Target Curvature

Target–camera distance and target curvature contribute to the distortion of the acquired images by the multiview system. To minimize the influence of target–camera distance on imaging distortion, the target is placed at a designated target–camera distance (around 10 mm to the end of the square mirror set) and the corresponding area in the five views is maximized. With the designated target–camera distance in the experiment, its influence on the imaging distortion can be effectively compensated. To minimize the influence of target curvature on imaging distortion, we calibrate the multiview system in advance by placing a checker board at the designated target–camera distance. The checkerboard is orientated in different directions corresponding to different shapes and curvatures for a target object in 3-D space. Based on the results of the calibration test, the radial and the tangential factors of the distortion can be calculated and corrected. In a $40\times 40\times 10{\mathrm{mm}}^3$ space volume, reconstruction of a flat board after the above calibration procedures is able to reach an RMS error of $<0.0120\mathrm{mm}$ in comparison with the best-fit plane. Similarly, reconstruction of a cylindrical board after the above calibration procedures is able to reach an RMS error of less than 0.1491 mm in comparison with the best-fit cylinder. These results imply that the proposed calibration procedures can effectively reduce the imaging distortion induced by target–camera distance and target curvature.

4.4.

Scattering of Biological Tissue

For multiview imaging of biological tissue, the scattering of light will also affect the imaging reconstruction accuracy. On one hand, light scattering in deep tissue may blur the surface texture and induce difficulty for stereo matching. On the other hand, depending on different view angles, light scattering may cause different levels of dislocation for deep layers of tissue. This effect exists in not only multiview imaging, but also other 3-D surface reconstruction methods such as surface scan and structured light illumination. Since it is difficult to eliminate imaging distortion due to light scattering, the alternative effect is to reduce the light scattering from deep tissue. Jacques et al. used polarized light to differentiate between superficial and deep tissue.^14,15 O’Toole et al. used structured-light-transport imaging to geometrically get rid of the strong subsurface scattering.¹⁶ Despite these effects, 3-D imaging distortion induced by light scattering has not been fully studied or compensated. Many researchers simply neglected the scattering effect as they estimated wound tissue topography using various 3-D imaging techniques such as multiview imaging, laser scanning, and structured light illumination.^17–20 By using polarized light and combining multiple imaging modalities, it is possible to acquire only the surface reflectance of biological tissue for reducing imaging distortion in 3-D reconstruction and for obtaining other pathophysiologic information of biological tissue.¹⁴