2.1.

Optomechanical Design

In the following, the development of an illumination and imaging device that can be used for eye tracking and pupil monitoring will be discussed. The digital displaying stereo microscope used is a prototype that has been developed by Carl Zeiss Meditec. The system to develop should easily be attached to the existing digital displaying stereomicroscope. At first, the digital displaying stereo microscope was characterized to determine important optical parameters. The parameters are as follows: magnification of 0.97, field of view of $4\times 4{\mathrm{mm}}^2$ and numerical aperture of the eyepiece of $\mathrm{NA}=0.16$. With these parameters at hand, the illumination and imaging system have been optomechanically designed, as shown in Fig. 2 for the mechanical design and Fig. 3 for the optical design.

Fig. 2

Schematic setup of the illumination and imaging system for pupil monitoring.

Fig. 3

Imaging design for the pupil monitoring system with a smaller and displaced aperture stop.

The practically implemented optomechanical setup can be seen in Fig. 4 and the optical parameters of the lenses employed are listed in Table 1. The limited field of view accessible by the digital stereomicroscope of $4\times 4{\mathrm{mm}}^2$ imposes some restrictions on the maximum amount of pupil movement. In fact, in addition to the lateral pupil movement, the surgeon has to turn his head to access the outmost positions of the display. The initial illumination design was based on infrared light-emitting diode illumination, emitting at a wavelength at which the visual perception is not disturbed (830 nm). However, internal reflection effects of the digital displaying stereo microscope, which is optimized for visible light applications, resulted in strongly disturbed images (bright background due to reflections), which made the application of near-infrared illumination and imaging impossible. Therefore, the entire design of the illumination system had to be changed. A good solution was found by simultaneously using the stereo displays as a light source. In that manner, the issue with disturbing reflections effects could be minimized without any disturbing artifacts arising for the visual perception of the surgeon. A beam splitter plate was implemented in the digital displaying stereo microscope at a position where the light beams are collimated. This location was originally occupied by a reflection prism, which had been removed and replaced. In that manner, it was possible to direct the reflected light from the eye to the camera. Furthermore, during the design, it was taken care that the high NA provided by the eyepiece of the digital stereomicroscope could be adjusted via an iris diaphragm to monitor the pupil without being too restrictive on the positioning stability of the surgeon’s eye. It was later realized that due to the reduced irradiance caused by a smaller diameter of the iris diaphragm, the integration time had to be increased, resulting in motion blur due to the eye’s movement within the integration time. Therefore, in combination with a chin-forehead rest, to ease stable positioning of the surgeon, the diameter of the diaphragm was set to a value of $\sim 1.7\mathrm{mm}$ resulting in an object sided NA of 0.033, which still enables a sufficiently large irradiance on the sensor so that motion blur does not yet occur. Moreover, $2\times 2$ binning was applied to the CCD sensor (Vistek eco415MVGE, SNR 50 dB) to increase the integration area³⁴ and hence keep the integration time within the required temporal constrain of 50 ms (temporal frequency should be faster than the latency of the human eye, which is 20Hz). The images obtained from the eye through the digital displaying stereomicroscope with changing pupil position are shown in Fig. 5. In Ref. 35, the image postprocessing routines applied to obtain the pupil positions for these images are described.

Fig. 4

Practically implemented setup of the illumination and imaging system for pupil monitoring: (a) scaffolding of construction and (b) system attached to the existing digital displaying Zeiss stereomicroscope.

Fig. 5

Recorded images from a human eye pupil with different pupil positons.

Motivated by the tight constrains, temporal and optical, of the current pupil imaging system, an additional solution has been developed, which is based on endoscopic monitoring in the eyepiece. In that manner, the limitations imposed on recoverable field of view, the irradiance, and the positioning stability could all be relaxed. Another possibility to track the pupil in surgical microscopy has been discussed in Ref. 36. It is based on an optical system that needs to be attached to the eyepiece and therefore relies on long eyepiece to eye distance to accommodate such an optical pupil tracking device, which has the drawback that the field of view that can be accessed by the surgeon’s eye is smaller than in the conventional stereomicroscopy. Hence, in addition to the current state of the art, two systems have been developed for imaging the pupil. The first is based on an additional monitoring system, as described above, which involves imaging through the existing digital displaying stereomicroscope, and a second system that is based on endoscopic imaging close to the surgeon’s eye without any interaction with the digital stereomicroscope’s optical elements (see Fig. 6).

Fig. 6

(a) Endoscopic setup for pupil monitoring, (b) holes in the eyepiece for the endoscope fiber, and (c) with smartphone and endoscope recorded image of a human eye.

The latter is more favorable since it is not constrained by the optical system of the stereomicroscope, which has been optimized to image the picture provided by the digital displays with sufficiently high resolution on the surgeon’s retina. Once the pupil has been detected and its position has been determined, the necessary information has been obtained to calculate the viewing direction, which can be used as input parameters for the digital recording stereomicroscope as to change the pupil position or to adjust the focus. The focus setting can be obtained from the point at which the viewing directions of both eyes intersect and from a priori knowledge about the surface topography of the investigated region.

3.1.

Optomechanical Design

In this section, a description of the digital recording stereo microscope is presented, which allows for accommodation and pupil shift. The temporal requirement on the system was to deliver a temporal resolution better than 50 ms (20 Hz) in accordance with the human eye’s latency.³⁷ Furthermore, the optical requirements on the system were a spatial resolution of $\sim 50\mu \mathrm{m}$ and a field of view of $30\times 30{\mathrm{mm}}^2$, which are common specifications among surgical microscopes. The optical design is shown in Fig. 7, and the mechanical design in Fig. 8(a). The practically implemented imaging system is shown in Fig. 8(b) Important optical parameters are listed in Table 2. Special attention was given to the optical design of the recording system to match the imaging parameters and quality of a conventional stereo microscope (Zeiss OPMI Pico).

Fig. 7

Optical design of the adaptive recording system.

Fig. 8

Recording system: (a) mechanical design, (b) experimentally implemented prototype system, and (c) simulation result for input object of 27-mm diameter in ZEMAX.

The imaging performance of the recording system is shown in Fig. 8(c), which illustrates the image quality obtained from an example input picture of 27-mm diameter. In Fig. 9(b), the modulation transfer function (MTF) and Seidel aberration coefficients are displayed, which demonstrate a good imaging performance of the adaptive digital stereo microscope near the optical axis. Toward the edge, a reduction of the performance of the optical system is apparent. This is due to an increase of aberrations with increasing distance to the optical axis, as shown in Fig. 9(b), and vignetting, which causes a loss of brightness toward the edge of the simulated image.

Fig. 9

Imaging performance of recording system displayed in Fig. 7: (a) MTF curve with $x$-axis representing the spatial frequency in lp/mm and $y$-axis representing the contrast and (b) field-dependent Seidel coefficients.

The optical performance parameters obtained for the digital adaptive system in comparison to a conventional (Zeiss OPMI pico) stereo microscope are shown in Table 3. The refocusing range of the conventional stereo microscope (see Table 3) is based on the effect of the average human eye accommodation range from infinity (relaxed eye) to 250 mm (accommodated eye).

Table 1

Lens data of lenses shown in Fig. 3.

Table 2

Lens data of achromatic lenses shown in Fig. 7.

Table 3

Comparison of optical parameters conventional and digital stereo microscope.

In the digital recording stereo microscope (see Figs. 7 and 8), accommodation is implemented via the addition of two tunable lenses (EL-10-30-C-VIS-LD) from the company Optotune AG,¹³ one for each arm of the stereoscopic system. The Optotune lens enables the application of a variable focal length ranging from 80 to 200 mm. The root-mean-square wavefront error at 525 nm and 0 mA is smaller than 0.15 $\lambda (78.75\mathrm{nm})$ in horizontal orientation and 0.25 $\lambda$ in vertical orientation (121.25 nm). The difference between the two orientations is mainly caused by gravity-induced coma,³⁸ as demonstrated in Fig. 10, which represents the wavefront error with respect to the orientation and focus setting applied for 525 nm. The Optotune lens can be set to a new position within 15 ms, which is within the temporal requirements imposed on our system (faster than 20 Hz).

Fig. 10

Root-mean-square wave aberration error with respect to the focal length applied to the Optotune lens EL-10-30-VIS-LD, which have kindly been provided by Optotune Switzerland AG.

With this Optotune lens in place, the refocusing range of our system becomes 55 mm.

To implement active displacement of the pupil position, two different possibilities have been considered: At first, a two axis tilt mirror positioned in the intermediate image plane has been taken into account. The second possibility is based on positioning a diaphragm in an intermediate pupil plane, serving as an aperture stop for the entire system, and in that manner modeling the function of the eye’s pupil. This diaphragm will be translated via a motorized $x-y$ stage in order to simulate the change of the eye’s pupil position.

The latter of the two has been implemented due to the following reasons:

Based on the latency of the human eyes with regards to changing pupil position, the temporal requirement imposed is 50 ms (20 Hz).³⁷ Therefore, a conventional manual, $x-y$ stage, was modified by the attachment of two linear actuators [see Fig. 11(a)] from the company Nanotec Electronic GmbH & Co. KG (L2818L0604-T5X5), which enable a maximum speed of $140\mathrm{mm}/\mathrm{s}$.³⁹ To cover the same amount of shift at the entrance pupil in our digital recording stereo microscope in comparison to the conventional system, the diaphragm has to be displaced by $\pm 0.6\mathrm{mm}$. However, the total amount of shift that can be applied without significant loss caused by vignetting is $\pm 7\mathrm{mm}$.

Fig. 11

(a) Motorized $x-y$ stage for incorporating the translation of the monitored human eye’s position and (b) two prisms have a bigger free diameter $(D1>D2)$ than two lenses.

Despite the requirement to match the optical parameters as previously discussed, special attention was also given to the design of the mechanical devices, in particular to host the motorized stage, which covers 53-mm axial dimension in which no optical components can be mounted. Furthermore, care was taken during the optomechanical design to:

3.2.

Optical Resolution

The camera employed (XIMEA MQ013MG-E2) has a pixel pitch $\mathrm{\Delta }x$ of $5.3\mu \mathrm{m}$. To ensure sufficient sampling of the high-resolution elements, the pixel pitch was multiplied by a factor of 2.5 so that a period [black and transparent strip of United States Air Force (USAF) test target] can unambiguously be displayed. For instance, a factor of 2, as suggested by Nyquist criteria, could result in a homogenous gray level over the entire test-target element of interest, in case the image of the black strip falls between two pixels. Consequently, it would be impossible to image such a test-target group.

The magnification of the system shown in Fig. 7 is ${\beta }^{\prime }=0.25$. Hence, our system can deliver a resolution of

To confirm this theoretically calculated resolution, a USAF test target has been employed. Rather than taking into account only a single cross section for a considered test-target element, the optical resolution was determined by averaging all cross sections across the test-target element under investigation. This approach enables a more accurate determination of the optical resolution. The averaged cross sections are shown in Fig. 12. Care was taken so that the three local minima (black strips) are visible and the ratio of the largest local minima (green dot Fig. 12) to the smallest local maxima (red dot in Fig. 12) is less than 0.73 according to Rayleigh’s resolution criterion for a circular aperture.⁴² A maximum spatial frequency of 17.96 line pairs/mm (group 4, element 2) could be achieved (see Fig. 12), which corresponds to a lateral resolution of $55.68\mu \mathrm{m}$. This matches well the predicted value calculated in Eq. (1).

Fig. 12

Recorded image of USAF test target with normalized cross-section plots of corresponding test target elements that could still be resolved.

3.3.

Depth from Focus

Studies on the magnification ratio $\beta$ and the focal length of the Optotune lens $f$ both with respect to the distance $z$ between the object and the main objective of the optical system are displayed in Figs. 13(a) and 13(b).

Fig. 13

Performance curves: (a) linear magnification by changing the working distance and (b) nonlinear Optotune’s focal length by changing the distance between the object and the main objective.

It can be seen that the magnification follows a linear trend. The average magnification is 0.252, and the max/min are $0.244/0.262$. The focal length of the optune lens on the other side can best be described by an exponential curve. The information provided by the latter of the two curves has been used to construct a cuboid with the dimensions $27\mathrm{mm}\times 27\mathrm{mm}\times 55\mathrm{mm}$ ($\text{width}\times \text{depth}\times \text{height}$).

This cuboid was used to demonstrate the recovery of depth information as acquired by the eye in the process of accommodation (Fig. 14). The cuboid has been printed in house via the application of a 3-D printer from the company Formlabs, see Fig. 14(a). Twelve different steps have been printed, with 5-mm height difference between each step. The printing accuracy is determined by the selected layer thickness, which in our case is 0.05 mm for our Formlabs $1+$ machine.

Fig. 14

Cuboid, (a) SolidWorks design with different line pairs/mm (min 0.5 and max $5\mathrm{LP}/\mathrm{mm}$) in $z$-direction, (b) obtained images from different planes of cuboid when applying different foci to the Optotune lens (distance between object and main objective is displayed on top of each image, whereas the corresponding focal length is shown in brackets); the refocusing capability can be appreciated watching the video (Video 1, MOV, 657 kB [URL: http://dx.doi.org/10.1117/1.JBO.22.5.056007.1]), where all still images shown in (b) are present, (c) topography map obtained using sliding window approach with white scale bar representing 4.5 mm.

Applying different voltage settings to the Optotune lens enables the recovery of different axial planes for the digital recording stereo microscope. The results obtained are shown in Fig. 14(b).

We start changing the distance by $z_{\min }=167\mathrm{mm}$ and end at $z_{\max }=222\mathrm{mm}$. Images at different refocusing distances are shown in Fig. 14(b). From these images, it is possible to obtain a 3-D image while the magnification also has to be taken into account. The generation of the topograpy map shown in Fig. 14(c) is based on the calculation of the variance within a sliding window that is moved across each of the images, as discussed in Ref. 43. A structured object, which is in focus, results in a high variance value. Comparing the different variance values in a pixelwise manner for all the processed images and picking out the largest enables the recovery of the in-focus plane for the pixel under consideration. In that manner, all the pixels are processed. We have been using a window size of $10\times 10\text{pixels}$ to obtain the topography map. The topography map obtained can be used to actively steer the optotune lens in combination with pupil tracking, so that the object region the surgeons looks at automatically becomes refocused.

3.4.

Depth Information by Stereoscopic Vision

The field of view at which stereoscopic vision is enabled depends on the working distance applied and can slightly be changed when shifting the pupil. This is in analogy to the human eye, where the pupil is shifted toward the nose for near-sided objects and in central position for far sided objects. From Fig. 15, it can be seen that the largest stereoscopic field of view can be recovered at a working distance of 170 mm when the pupil is as at central and inside shifted position and amounts to $24\mathrm{mm}\times 27.5\mathrm{mm}$. The latter of the two dimensions remains the same for all pupil positions, since the stereoscopic vision is predominately determined by the horizontal pupil position (first dimension). At 200-mm working distance and central pupil position, the overall largest stereoscopic field of view can be obtained covering $27.5\mathrm{mm}\times 27.5\mathrm{mm}$. At 220-mm working distance, the largest field of view ($26\mathrm{mm}\times 27.5\mathrm{mm}$) can be recovered when the pupil is displaced by 3-mm away from the optical axis of the main objective. Hence, changing the focal length applied to the Optotune lens ought to be accompanied with a change of the pupil position in order to enable recovering a large stereoscopic field of view, see Fig. 15(a). Another important parameter for stereoscopic vision is the stereo angle, which determines the stereoscopic depth discrimination. According to Ref. 41, the smallest stereo angle $\mathrm{\Delta }\vartheta$ that can still be resolved by the human eye is 10′′ under appropriate lighting conditions (at least $0.1\mathrm{cd}/{\mathrm{m}}^2$). The corresponding depth discrimination $\mathrm{\Delta }s$ is a function of the stereobasis distance $d_{\mathrm{s}}$ and the in-focus working distance $d_{\mathrm{w}}$

Fig. 15

(a) Relationship between in-focus working distance and recoverable stereoscopic field of view, (b) stereo angle ($\vartheta$ displayed by green line) with respect to the working distance and (c) corresponding stereoscopic depth discrimination ($\mathrm{\Delta }s$ displayed by blue line).

Fig. 16

Schematic sketch of the stereoscopic angles $\vartheta$ at different pupil positions, indicated by different colors and corresponding depth $d_{\mathrm{s}}$.

Therefore, a pupil shift in the direction of the main objective’s optical axis results in an entrance pupil shift in the opposite direction, away from the optical axis, resulting in an increased stereobasis and consequently in an improved depth discrimination, as shown in Fig. 15(c) and Eq. (2). The recorded images for left and right eyes and the corresponding color-encoded stereoscopic image (left eye red colored image and right eye cyan colored image) are shown in Fig. 17. The anaglyph image shown in Fig. 17(c) can be viewed using 3-D red-cyan glasses.

Fig. 17

Images recorded with the digital recording stereo microscope for (a) left arm, (b) right arm of the system, and (c) color-encoded stereoscopic image

3.5.

Perspective Imaging

The perspective, an additional depth clue of the eye, which is realized by different viewing directions, as when laterally moving the eye and/or the head, is investigated in the following. The digital recording microscope enables perspective imaging by means of moving the diaphragm in the intermediate main objective’s pupil plane by the motorized $x-y$ stage. As a proof of principle, a 3-D device with two bars occluding the field of view has been printed in house, as shown in Fig. 18(a). The two bars have a distance of 4.1 mm and are located at a distance of 70 mm to the in-focus object plane, which project a shadow on the in-focus object plane. Changing the location of the aperture in the intermediate pupil plane enables the recovery of object details, which have previously been hidden by the shadow. This is in analogy to the perspective imaging experience when moving the eye’s pupil often accompanied with lateral head movement.

Fig. 18

Perspective view: (a) device to demonstrate the possibility to apply different perspective views, (b) results obtained from different viewing direction when moving the aperture in $x$-direction; to appreciate the perspective impression, a movie of the still images represented and further recorded images while shifting the pupil are shown in the video (Video 2, MOV, 739 kB [URL: http://dx.doi.org/10.1117/1.JBO.22.5.056007.2]).

The results obtained for different aperture positions with an aperture shift of 2 mm between consecutive recordings are displayed in Fig. 18(b). The aperture shift was accompanied by refocusing procedure using the tunable lens. It could be demonstrated that different parts of the zebra are obscured, so that it is in principle possible to recover an image of the zebra, which is free of any obscuration.

To test this, which is based on an algorithm similar to the one discussed in Sec. 3.3 that enables the recovery of a topography map, the sliding window, within which the variance is calculated, has a size $30\times 30\text{pixels}$ and is moved across each of the six images. With the sliding window variance approach, it is possible to separate the shadow pattern (low to no contrast) from the object details (high contrast). In that manner, an image is generated that is predominately composed of the object details only, see Fig. 19.

Fig. 19

Combined image, which takes into account only high spatial frequency information.

More sophisticated routines, such as histogram adjustment, can help to improve the final combined image.