1.

Introduction

The human eye’s separation of the field of view into two regions—one of high and one of low resolution, the fovea centralis and periphery—has inspired a number of attempts at doing the same thing with digital imaging systems.^1–6 The advantage is the ability to see over a wide field of view (FOV) while maintaining high resolution capability in the central region of view. A foveated system thus possesses the potential to solve the main difficulty that has prevented the translation of endomicroscopy to direct clinical use: current endomicroscopes lack the field of view needed to give clinicians the proper context for the high-resolution measurements needed to detect lesions. Pathologists, for example, routinely scan entire tissue sections with low resolution, and then focus down on areas of interest with high magnification and resolution. The high resolution images allow one to determine the health and morphology of individual cells, but this information is only useful when one knows the cells’ location within the overall tissue.

The first demonstrated foveated imaging system of which we are aware is that of Spiegel et al.,⁷ in which the authors describe the design of an “artificial retina”—a custom-designed sensor array with spatially variant response. Following on this work, a number of other research teams attempted alternative approaches for customized on-chip foveated sensor arrays.^8–10 McCarley et al.⁴ take a different direction and develop an on-chip binning method that allows one to transmit regions of the image at the full sampling rate of the detector array while binning pixels outside these regions into larger “super-pixels” for lower resolution.

The first attempt to perform foveation with optical components appears to be Suematsu et al.,¹¹ where the authors use a door peephole lens together with distortion-correction software to obtain foveated images. The authors later designed a higher performance lens to achieve similar distortion characteristics over a larger image.¹² This elegant approach to foveation was taken further by Wakamiya et al.,¹³ in which the authors design a wide-angle camera lens adapted to a standard CCD array format.

An alternative approach to foveation is to use a spatially tunable phase element to correct lens aberrations in a localized region of the field of view. Martinez et al.,¹ for example, use a tunable transmissive liquid crystal spatial light modulator (SLM) to correct for aberrations across a $\pm 45\mathrm{deg}$ FOV. The same basic method is also possible using a reflective SLM.¹⁴ A pragmatic approach is taken by Ude et al.,¹⁵ in which the authors use a pair of cameras viewing the same scene—one high-resolution narrow-FOV camera and one low-resolution wide-FOV camera—to obtain foveation. Since this can produce parallax problems when viewing nearby objects, Hua and Liu⁵ adapt the method to use a beamsplitter to separate the objective lens optical path into two paths of different magnification, each of which is sampled by its own detector array.

For endoscopic imaging, many of these prior implementations are problematic. The SLM approach requires that the modulator be an integral part of the objective lens, so that an SLM’s large size becomes an obstacle. The on-chip method of McCarley et al. provides no benefits to endoscopic imaging, where bandwidth constraints are generally minimal due to the low pixel counts typically transmitted by imaging fiber bundles. The dual detector method of Hua and Liu, while compatible with endoscopic imaging if implemented with a pair of imaging fiber bundles, does not substantially ease the optical design problem of achieving simultaneously a high-resolution central field and a wide total field of view. Our own approach follows that of Suematsu¹² and Wakamiya¹³ by using optical distortion within a customized objective lens, but designed for endoscopy rather than camera use. The resulting foveated endoscopic objective lens attempts to capture all of the benefits of foveation (high-resolution central field, wide total field, and easier optical design) at the cost of requiring highly aspheric optical elements.

Foveation also has the potential of getting around the sampling limitation available with current imaging fiber bundles. If high resolution is achieved everywhere within the field of view, the limited number of fibers (bundles with up to 100,000 fibers are available) means that the bundle rather than the lens restricts the overall FOV. With foveated imaging we can dedicate a fraction of fibers to the fovea and use the remaining fibers for wide field imaging.

2.

Lens Design

The goal of the foveated endoscopic lens design is to achieve $\sim 5\mathrm{\mu m}$ resolution in the central field of view ($<0.5\mathrm{mm}$), smoothly transitioning to 50 μm resolution at the edge of the field. In order to allow the lens to access remote areas of the body, the design attempts to keep the optical elements under $\sim 3\mathrm{mm}$ diameter. The design starts with a three-element system in which the front and rear elements are diamond-turned plastic aspheric surfaces while the middle element is a commercial off-the-shelf spherical glass lens, following loosely along a previous design.¹⁶ The layout of this objective design and its optical performance is shown in Fig. 1, while the optical prescription is given in Table 1. In the final design, the lens maps a $\pm 2.7\mathrm{mm}$ field of view onto a $\pm 0.7\mathrm{mm}$ image. The central field has a magnification $M$ of 0.5, falling to $M=0.04$ at the edges of the field, and is diffraction limited throughout the entire FOV. The design is telecentric in image space in order to provide high coupling efficiency into an imaging fiber bundle.

Fig. 1

(a) The optical layout of the foveated lens, (b) spotsize diagrams, and (c) MTF curves. The spotsize diagrams include a circle showing the 2.6 μm radius Airy disk for the on-axis NA. The MTF plot uses a maximum spatial frequency corresponding to the optical cut-off frequency of the on-axis NA.

Table 1

The optical prescription for the prototype foveated lens, designed for a wavelength of 633 nm. Surfaces 3 to 4 comprise a commercial glass lens (Qioptiq 311330000). The surface sag z is represented by the equation z=(1/R)r2+α4r4+α6r6, where r is the transverse coordinate, R the radius of curvature (the second column below), and α4 and α6 are aspheric coefficients. All dimensioned units are in millimeters.

The numerical aperture of the lens varies with field angle as indicated in Table 2, determined by numerical ray tracing at each field position (interested readers can consult details in the Zemax macro NA_calcs.zpl given at the authors’ website.¹⁷) Because the lens operates with large field angles (up to $\pm 41\mathrm{deg}$ at the edges of the field), the standard paraxial formula for object-space NA is split in two in order to factor in the oblique angle: for a system with low distortion, the tangential and sagittal NA values at a given field point are related to one another by approximately a factor of ${\cos }^2(\theta )$, where $\theta$ is the chief ray angle in object space.^18,19 For the semimajor $r_{\tan }$ and semiminor $r_{\mathrm{sag}}$ axes of the Airy ellipse, one still uses the familar equation $r=0.61\lambda /\mathrm{NA}$ by substituting ${\mathrm{NA}}_{\tan }$ or ${\mathrm{NA}}_{\mathrm{sag}}$ for the numerical aperture value. The depth of field (DOF) is calculated from the ray tracing model and is defined as distance between the positive and negative longitudinal offsets at which the defocus aberration rms spotsize exceeds the diffraction-limited spotsize. The outer fields, due to their small NA, have an extremely long depth of focus. Due to the telecentric geometry, on the other hand, the image-side NA is approximately the same everywhere across the field, at $\mathrm{NA}=0.145$. Note that the reduction in NA with field position does not result in a large falloff of light level at the edges of the raw (distorted) image because of the corresponding decrease in magnification there.

Table 2

The design parameters for the prototype foveated lens. In a given column below, a pair of numbers separated by a slash indicates the value for tangential/sagittal fields. For the on-axis fields, these two numbers are identical. DOF is the depth of field and M the magnification.

The first and last surfaces in the system both create large negative distortion. Since the front element contains negative power and the rear elements are both positive in power, the design is roughly anti-symmetric about the stop, so that odd aberrations (i.e. distortion and coma) sum together. The first surface is concave toward the object—a common configuration in microscope objectives that allows for the reduction of spherical aberration and field curvature.^20–22 The lens was designed for monochromatic 633 nm light but remains diffraction-limited to about a bandwidth of $\pm 50\mathrm{nm}$ from the design wavelength. That is, at $\pm 50\mathrm{nm}$, the RMS polychromatic geometric spotsize is approximately the same as the Airy disk radius.

While we have indicated that the lens design is diffraction-limited across the entire FOV [as indicated by Fig. 1(b)], the model MTF curves [Fig. 1(c)] show substantial departure from the behavior of a conventional diffraction-limited lens. The off-axis curves at low and medium spatial frequencies fall well below the MTF curves for the on-axis field, but this is due almost entirely to the reduction in the NA with increasing field position rather than to lens aberrations. In addition, the large off-axis angles in image space cause the pupil to become elliptical, so that the tangential and meridional MTF curves show different behavior.

Figures 2(a) and 2(b) show the field curvature and distortion of the design, indicating that the distortion reaches a value of 50% at the maximum field point. Figures 2(c) and 2(d) shows the effect of the distortion on a simulated object. The resulting image will be sampled well in the central region while the edge regions will be sampled sparsely.

Fig. 2

(a) Field curvature and (b) distortion in the design. The two diagrams in the bottom row show a ray tracing simulation of an image acquired with the foveated objective: (c) the object in view, (d) its image. (Note that the image has been inverted for easy comparison with the object.)

3.

Lens Tolerancing, Manufacture, and Testing

Tolerance analysis on the design requires a small modification to standard procedures because rays near the edge of the field can experience extreme deviations from their designed location. These extreme rays completely dominate the resulting merit function, despite the poor system resolution there and so need to be treated separately from rays corresponding to the other field points. Thus, system tolerancing was performed using the inner set of fields alone (i.e. field heights of 0, 1, and 2 mm). From the analysis, one finds that lens decenter is the dominating source of system error for the manufactured system, requiring that element and surface decenters be held to less than 40 μm.

The lens design consists of two plastic elements and one commercial glass lens (Qioptiq 311330000). The two plastic lenses [the first and third elements shown in Fig. 1(a)] are cut from 38 mm diameter disks of PMMA (acrylic) on a 4-axis Nanotech 250UPL precision lathe using single point diamond turning. Only the central 3 mm of the plastic disks contain the optical surfaces; the remainder of the disks were painted blue in order to prevent stray light.

In order to mount and align the assembled system, we designed three mounts and an aperture stop using SolidWorks CAD software and manufactured these with a ProJet SD3000 3D printer. The resulting system is shown in Fig. 3.

Fig. 3

The experimental setup for testing the prototype foveated lens. Shown at left an Olympus UMPlanFl $5\times$ microscope objective lens used for relaying and magnifying the foveated lens’ image onto a detector array. At right are the (white) lens disc mounts, with the optical surfaces themselves comprising only the central 3 mm of each disc, which have been painted blue to prevent stray light.

Because the back working distance of the foveated lens is only 3.6 mm, one cannot place a commonly available detector array directly at the image plane in order to detect the signal. Instead, for testing the system, we constructed an optical relay using an Olympus UMPlanFl $5\times$ microscope objective lens together with a 100 mm tube lens (Thorlabs LA1509-A) to re-image the signal onto a detector array (QImaging Retiga 2000R). The input NA and field of view of the optical relay (0.15 and $3\times 4\mathrm{mm}$ respectively) have been closely matched to the image size and image-space NA of the foveated lens. All images were taken using a 660 nm LED light source (Thorlabs M660L2) for illumination.

4.

Calibration and Experimental Results

For practical use, the displayed images from the system must be unwarped in order to remove the distortion. As a first step toward correcting the distortion present in the images, we created a set of calibration targets comprising a field of dots laid out in a Cartesian grid pattern, printed onto transparency film using a standard desktop laser printer.* Figure 4 shows an ideal target, the corresponding image through the prototype lens, and the distortion-corrected image after unwarping. Note that the square-shaped appearance of the circular dots is a result of resolution degradation by the printer.

Fig. 4

(a) The ideal calibration target, (b) its image through the prototype lens, and (c) the distortion-corrected image. The target consists of a Cartesian grid of dots spaced 0.628 mm apart and designed to overfill the field of view. Note that the bottom image contains about 2.4 times the number of pixels ($1181\times 1166$) as the measured image ($752\times 752$).

Using the measured images, we can locate the dot coordinates on the detector array and compare to the known dot coordinates at the object plane. One can then use least squares analysis to estimate the mapping parameters for performing image correction.^23–28 Since the mapping of object to image is in principle a purely radial function, the mapping parameters are the image coordinates of the optical axis $(x_c^{\prime },y_c^{\prime })$ together with the $a_i$ coefficients used to express the radial form of the mapping from image plane $(x^{\prime },y^{\prime })$ to object plane $(x,y)$:²⁵

Using the prototype lens and distortion correction algorithm, we measured a 1951 USAF resolution target in the lens’ central field of view (see Fig. 5). (With the exception of Fig. 8, all images were taken with the object placed at the 1.9 mm working distance.) Looking at the closeup of the highest-resolution groups, we can see the bars are resolved down to elements 1 or 2 in group 7, corresponding to 128 or 144 line pairs per mm—a resolution of $7\sim 8\mathrm{\mu m}$. This number is worse than the initial design of $\sim 5\mathrm{\mu m}$ and appears to be a result of damage to one of our lenses (surface 6) during system assembly and alignment as well as to difficulty in reaching the required mechanical tolerances using the SD3000 3D printer for manufacturing our mounts.

Fig. 5

A closeup of a 1951 USAF resolution target imaged through the prototype lens, showing a closeup of (a) groups 4 to 7, and (b) groups 6 and 7 on the target (shown by the red box outline in the full image), indicating that either element 1 or 2 of group 7 is just resolved (i.e. a resolution of $7\sim 8\mathrm{\mu m}$). The fields of view of the two images are 1.19 mm and 238 μm respectively.

While Fig. 5 shows the resolution in the fovea, Video 1 (Fig. 6) shows the wide field of view achieved by the system. In this example we translate the resolution target across the field of view and show simultaneously the raw measured image and the distortion-corrected image as the object moves across. While the distortion correction algorithm has difficulty in achieving full correction at the edges of the FOV, one can see that the corrected images show little of the hemispherical warping present in the raw measurements.

Fig. 6

An image sequence showing the resolution target as it translates across the field of view, in raw data (left) and after distortion correction (right). (Video 1, 5.4MB MPG) (URL: http://dx.doi.org/10.1117/1.JBO.17.2.021104.1)

Figure 7 shows a hematoxylin and eosin (H&E) stained slice of mouse esophagus tissue imaged through the prototype lens after the distortion correction algorithm has been applied. The fovea allows one to see individual nuclei, while the entire system allows mapping of a wide field of view. The overall field of view shown in the images (Figs. 6 and 7) is 4.6 mm in diameter. This is smaller than the designed 5.4 mm due to the presence of reflections at the edges of the field. (These are especially apparent in the raw data of Fig. 6.) When cutting small lenses into larger disk substrates, the edges of the lens have a fillet due to the finite radius of curvature of the cutting tool. The resulting fillet has a curvature different from the lens itself, the result of which produces reflections at the edge of the field. In order to remove these reflections, we have truncated the field to a 4.6 mm diameter.

Fig. 7

(Top) An image of a hematoxylin and eosin (H&E) stained slice of mouse esophagus, with diameter at the object of 4.6 mm. In the foveal image (bottom, 1.1 mm diameter at the object) one can see individual nuclei (dark spots). The annotated tissue regions are: (1) epithelium, (2) muscle/cartilage, (3) thyroid, and (4) salivary gland. Note that the 660 nm illumination results in poor contrast for the tissue’s eosinophilic structures.

In order to illustrate the lens’ long depth of focus, Fig. 8 shows a video of an experiment in which a user waves a hand 350 mm in front of the lens’ nominal object plane. Both the user’s hand and the raw image from the lens on the computer screen are simultaneously in view, and show that, with the exception of the fovea, the shape of the hand is visible even at such a long distance.

Fig. 8

An illustration of the long depth of field of the lens. While a hand is waved about 350 mm in front of the foveated lens’ nominal object plane, the raw image data (see the computer screen shown in the image) shows the shape of the hand is resolved except in the high-resolution central field. (Video 2, 1.1 MB MPG) (URL: http://dx.doi.org/10.1117/1.JBO.17.2.021104.2)

5.

Conclusion

Endoscopes with wide fields of view are commonly used in clinical settings to view organs inside the body. On the other hand, endomicroscopes, despite their proven ability to diagnose tissue diseases, have seen little direct application in the clinic due to practical difficulties with their use: although they enable viewing of cellular morphology, they cannot provide a view of the large-scale tissue structure that is needed to determine context for morphology-based diagnosis. Bringing together these two techniques in a single instrument has proven difficult, but we have been able to show that distortion-induced foveation should allow one to achieve a high-resolution fovea together with wide field of view in order to marry the strengths of endoscopy with endomicroscopy. Our prototype foveated lens shows that this can also be done in the small system diameters needed for agile endoscope probing within the body.

For ease of use, another advantage shown by our prototype lens is a long depth of field for the periphery. Not only does this make the probe much more practical for viewing the contorted surface geometry of organs inside the body, it also allows users to scan whole organs from a distance, and then to move in closer to regions of interest for obtaining morphological measurements.

The current proof-of-concept lens design has a low object space NA on axis for performing endomicroscopy. In future work, we plan to modify the current design, increasing the number of lens elements to achieve $\mathrm{NA}=0.25$ on axis. We are also developing new manufacturing methods for these miniature lenses, allowing for the tighter tolerances required by high NA optics.