1.

Introduction

Flow and image cytometers are instruments that measure cells and are in wide use in clinical and research settings. Flow cytometers, which began in the 1950s with the Coulter counter and attained fluorimetry capabilities in the 1960s,¹ are much more prevalent because of advantages in speed and automation, despite the superior information content of 2-D images relative to 1-D flow data. Commercial flow cytometers can reach $\mathrm{70,000}\mathrm{cells}∕\mathrm{s}$ ,² while image cytometers typically process cells at under $1000\mathrm{cells}∕\mathrm{s}$ at moderate magnification/resolution.³ Speeds of more than $\mathrm{100,000}\mathrm{cells}∕\mathrm{s}$ have been reported in imaging⁴ at low magnification where depths of field are large and autofocus performance is not critical. Although image cytometers have recently achieved the same level of walk-away automation for large cell populations³ [see also high content screening instruments from Beckman Coulter (Fullerton, California), Cellomics (Thermo Fisher Scientific, Pittsburgh, Pennsylvania), GE-Amersham (Buckinghamshire, United Kingdom), Evotec Technologies (Hamburg, Germany), BD Biosciences (Boston, Massachusetts), Molecular Devices Corporation (Sunnyvale, California), and the Compucyte Corporation (Cambridge, Massachusetts)], experience with flow cytometry and the demands of applications in fields like clinical diagnostics and drug discovery suggest that image cytometry use may expand dramatically with similarly high throughput. Throughputs an order of magnitude greater can be achieved by moving the sample in a continuous motion.^{5, 6} But robust autofocus performance in high-resolution optical microscopy, where depths of field are on the order of $1\mathrm{\mu }\mathrm{m}$ , has been challenging^{7, 8} and has not previously achieved the performance required for widespread use of continuous-scanning image cytometry.

The two primary approaches to autofocus are: 1. position sensing, usually based on light reflection off of the specimen surface substrates,^{9, 10, 11} and 2. image content sharpness measurements, wherein the image quality is maximized directly to achieve best focus. ^{7, 8, 12, 13, 14} Surface sensing is best for total internal reflect fluorescence (TIRF) microscopy, because the position of the focal plane directly adjacent to the surface is fundamental to the physics of fluorescence generation (e.g., the Nikon Perfect Focus System). It is also widely used for lower resolution, large depth of field $(\mathrm{NA}<0.5)$ image cytometry. Image-content-based autofocus works best for medium- and high-resolution automated microscopy $(\mathrm{NA}\ge 0.5)$ , where the depths of field approach or become even smaller than the thickness of the tissues and where the specimens vary in height and thickness. The drive to use higher NA objectives arises from the need to improve resolution, which is a function of NA, and increase fluorescence light gathering power, which is a function of ${\mathrm{NA}}^4$ .¹⁵ The ${\mathrm{NA}}^4$ dependence of fluorescence light gathering is particularly dramatic and in many cases may explain, even more than the increase in resolution, why tiny details become so much more visible in fluorescence as NA is increased. Since depth of field is a function of ${\mathrm{NA}}^2$ ,¹⁵ autofocus challenges increase substantially with NA. These challenges and tradeoffs have made autofocus an active microscopy research topic for many years. Here we report adapting our image-content-based autofocus methods to create dynamic image-content-based autofocus for medium- and high-resolution continuous scanning. The system parallelizes our previous autofocus methods to image and measure the sharpness of multiple focal planes in parallel for on-the-fly autofocus.

For use in a wide range of applications, a high-performance image cytometer should acquire and process large quantities of data at submicron resolution, e.g., about 20,000 1-Mpixel images for an area of one $25\times 75\text{-}{\mathrm{mm}}^2$ microscope slide at $0.3\times 0.3\text{-}\mathrm{\mu }{\mathrm{m}}^2$ /pixel sampling [with a $0.6\text{-}\mathrm{\mu }\mathrm{m}$ resolution objective (Rayleigh, $\mathrm{NA}=0.5$ and $\lambda =500\mathrm{nm}$ ) and a $1\mathrm{K}\times 1\mathrm{K}$ CCD camera Nyquist sampling $\left(0.3\mathrm{\mu }\mathrm{m}\right)$ , the field of view is $0.307\times 0.307{\mathrm{mm}}^2$ and 19,868 images are needed to cover the area of a $75\times 25\text{-}{\mathrm{mm}}^2$ microscope slide]. Conventionally, the cells are scanned by moving a stage between each field of view (FOV) and stopping to autofocus and capture an image.³ Good autofocus is fundamental to the quality of a microscope image in an automated instrument; the quality of the measurements and observations drawn from the image are dependent on sharp focus. Autofocus is perhaps even more important in automated image cytometry, because image segmentation (as well as other image processing and analysis operations) and the resulting measurements are sensitive to focus. Autofocus is critical for maintaining consistent image sharpness because slides, coverslips, and plates are far from flat (relative to the depth of field), and mechanical instabilities that include thermal expansion, gear backlash, and settling are best countered by corrective feedback. In lower resolution systems, autofocus is sometimes performed on only every five or ten fields to increase speed. For NAs of 0.5 or more where depths of field approach $1.0\mathrm{\mu }\mathrm{m}$ or less, however, measurement precision can be compromised by infrequent autofocus.

Here, we report our exploration of parallel multiplanar image sharpness measurements for continuous autofocus to overcome the speed limitations of our current systems, in which the repeated acceleration and deceleration of a relatively massive stage to sequentially image many adjacent fields of view fundamentally limits speed. We found that acceleration-induced vibration that degrades imaging and autofocus performance can be easily achieved with the commercial motorized stages routinely used in microscopy. Sequential stage motion, autofocus, and image capture are also inefficient simply because the camera is not acquiring an image for cytometry during autofocus and stage movement. Previous designs for moving the stage continuously to overcome these problems and increase speed are summarized in Table 1 . The system by Netten ¹⁶ followed a prerecorded focus path during continuous scanning, as does the Aperio Technologies (Vista, California) brightfield system for scanning tissue sections. The system by Castleman¹⁷ apparently paused to perform static autofocus at regular intervals, and those by Shippey ¹⁸ and Tucker ^{19, 20} autofocused dynamically during scanning. Both Shippey ¹⁸ and Netten ¹⁶ reported that the focus error was greater than the depth of field, and Tucker ¹⁹ reported that a $1\text{-}\mathrm{\mu }\mathrm{m}$ focus error produced a 12% error in the integrated optical density of the cell nucleus. Autofocus accuracy was reported to be very dependent on the density of cells by Castleman,¹⁷ and the need for many cells in each field of view was common to all of the systems that included autofocus. None of these designs have been widely adopted. The techniques and problems reported with them in these studies point to limitations in automatic operation, large focus errors that would limit use in high-resolution microscopy, relatively few focus updates per image field, and too much dependence on cell density for reliable focus tracking, perhaps due to limited sharpness measurement sensitivity and dynamic range.

Table 1

Previously reported continuous-scanning instruments. AF autofocus, NR denotes not reported in listed reference, IOD is integrated optical density, and CV is the coefficient of variance.

More recently, the GE-Amersham (Buckinghamshire, United Kingdom) In Cell Analyzer 3000, which uses reflection positioning off of the surface of the substrate for focusing a laser slit-illuminated partial confocal light beam that is scanned continuously, was introduced for high content screening. The partial confocal imaging corrects for medium resolution $(\mathrm{NA}\sim 0.6)$ focus errors by removing some out-of-focus image information to perform optical sectioning, which makes it less critical to find the average best focus across the field. Similarly, the Evotec Opera (Woburn, Massachusetts) and the Atto Pathway HT^TM system use spinning disk confocal image acquisition to remove out-of-focus light. These confocal systems tend to be 2- to 3-fold more expensive than wide field fluorescence instruments. These and most other automated microscope manufacturers have not published autofocus performance.

To maximize light collection efficiency and begin with the simplest optics, we chose to base the system reported here on wide-field fluorescence imaging. Because thermal and mechanical instability can cause focus to change between scans of the same area, and prescan focus profiling can be time consuming for high-resolution images, we also assumed that dynamic autofocus during scanning is a requirement for high fidelity and ease of use. With high-resolution scanning $(\mathrm{NA}\ge 0.5)$ , measurement accuracy degrades if autofocus is not performed on each field of view. The designs by Shippey ¹⁸ and Tucker ^{19, 20} that included dynamic autofocusing sampled only two focus planes and utilized integrated optical density (IOD) for measuring focus. Static (fixed lateral position) image-content-based autofocus experiments have shown that 7 to 11 planes are usually required for robust autofocus, that intensity statistics (e.g., mean, or IOD, and SD) are poor measures of focus ^{7, 8, 12, 13} and that the power of the upper-half spatial frequency image spectrum as a measure of sharpness is the best measure of focus.¹⁴ We therefore combined this sharpness measurement with multiplanar image acquisition to build and test on-the-fly continuous-scanning closed-loop autofocus.

Concurrent with the task of high-resolution autofocus, the use of continuous-motion time-delay-and-integrate (TDI) scanning has the potential to dramatically speed image acquisition while retaining high light sensitivity. TDI scanning is a special CCD readout mode that increases the integration period by allowing multiexposure of a moving object when its motion is synchronized to the CCD line rate, producing a 2-D image. A TDI CCD has multiple adjacent linear stages (e.g., 96), that transfer and integrate the photo-generated charge from line to line synchronously with the motion of the imaged object, increasing sensitivity (e.g., 80 times for 96 stages) over a single line array. This method has been adopted in our image cytometer to increase sensitivity in fast image acquisition, and we have developed a dynamic autofocus system that maintains spatial high resolution by tracking focus during the microscope stage motion.

A continuously moving object can be imaged by strobed lighting or by synchronizing a linear CCD with the motion. Because fluorescence saturation fundamentally limits the amount of light per unit time that can be obtained from the specimen, we incorporated TDI cameras, which increase photosensitivity by increasing the integration time proportional to the number of CCD lines. TDI sensors are often used in machine vision applications to detect and identify defects on continuously moving objects (e.g., manufactured parts on an assembly line). For web inspection, the object in the field of view of the sensor can be assumed to be flat because the depths of field of macroscopic optics are usually large, enabling focus to be adjusted to include the extent of the object.²¹ In these applications, resolution (which drives higher NAs/lower F-stops that reduce depths of field) is not usually a limiting factor. Where depths of field are limited, the autofocus technique described here may also be useful for web inspection. For any TDI application, proper synchronization of image acquisition with motion is required to avoid blurring and maintain proper horizontal to vertical aspect ratios in the acquired image (i.e., square pixels).

TDI technology has also been applied to create an imaging flow cytometer with high resolution and fluorescence sensitivity at speeds of up to $300\mathrm{cells}∕\mathrm{s}$ .^{22, 23, 24} The advantage in flow cytometry is that hydrodynamic focusing can be used to keep the cells in the focal plane. The image acquisition is synchronized to the fluid flow for TDI imaging.²⁵ With cells flowing in a single file, the additional lateral area on the TDI CCD is utilized for additional analyses, including spectral imaging and 3-D structure.^{24, 26, 27} Autofocus is performed by moving the objective in response to a difference signal generated by two detectors sensing light modulated by two Ronchi rulings deployed behind a 50/50 beamsplitter.²⁸ It is not known if this autofocus method would work for image cytometry, but other intensity-based autofocus methods have not performed well for microscopy of specimens on a substrate.¹²

Based on our experience with static autofocus ^{7, 8, 14, 29} in automated microscopy of specimens on substrates, we developed and tested dynamic autofocus for a continuous-scanning system. Our static image-content-based techniques were converted to dynamic autofocus by designing an electro-optical system for acquiring multiple image planes at predetermined focal depths in parallel, which we named the “volume camera.” The volume camera also uses TDI CCD sensors for high sensitivity during continuous, constant-velocity motion of the specimen, and images nine focal planes in parallel.^{5, 30} Although more complex than the methods summarized in Table 1, this continuous-scanning design incorporates the features we believed necessary for achieving robust dynamic autofocus. In this first report of our design, we describe the software control system that enables all of the hardware components to work together to achieve dynamic autofocus in continuous-scanning image acquisition.

2.

Materials and Methods

A cartoon of the continuous image scanning system with dynamic autofocus is shown in Fig. 1 . The volume camera acquires images via light that is transmitted through coherent fiber optic bundles from the image plane of the microscope to the array of CCD cameras. Each fiber bundle images a different focal plane, as shown in Fig. 1. An array of nine focus measurement circuits^{7, 31} measures image sharpness directly from the nine analog videos of the volume camera. And a computer processes the focus data and corrects focus by moving the objective lens. Unlike static autofocus, where several sequential focus measurements are used to calculate best focus once for each field of view, dynamic autofocus utilizes a closed-loop feedback control system. This control system must manage stage motion, buffer the continuous stream of incoming focus measurement data, capture fluorescence image frames, calculate the best foci, and perform corrections while focus is constantly changing as a function of specimen and environmental variations. Figure 2 is a block diagram of the continuous scanning system showing the microscope, the autofocus loop, the fluorescence image capture, and the synchronization functions. A specimen of cells on a slide is the input to the system, while the in-focus fluorescence image is the output. The primary components and methods important in the performance of the cytometer are summarized as follows.

Fig. 1

A cartoon of the fiber optic continuous-scanning autofocus is shown. The host computer receives focus measurements from an array of autofocus circuits (AF) via the volume camera—an array of CCDs bonded to optical fiber bundles—and updates focus as the stage moves. The axial displacements (staircase) of the optical fibers control the focal plane viewed by each CCD. The control system buffers and realigns the data to correct for the image delays of adjacent fibers. Another CCD camera (not shown) simultaneously collects the in-focus fluorescent image.

Fig. 2

A block diagram of the continuous-scanning system is shown. A timing sequencer controls the CCD sensors in the volume camera and the components in the focus measurement circuit array. The fiber optically coupled array of CCDs and focus circuits generate the multiplanar focus measurements and feed them to the computer for calculation of best focus. The bold arrows represent “volume data” that consist of multiple images and focus data streams that are processed in parallel. The computer synchronizes stage motion with the cooled TDI CCD camera to acquire the in-focus fluorescence image. Fvol is the line rate of volume camera, Ftrail is the line rate of trailing camera, Vs is the stage velocity, and EOF is the end of frame.

2.2.

Microscope

The continuous-scanning autofocus instrumentation was added to a modified Nikon Diaphot-300 inverted microscope with Plan $10\times 0.30\text{-}\mathrm{NA}$ Ph1 DL, Fluor $20\times 0.75\text{-}\mathrm{NA}$ Ph3 DL, and $40\times 0.75\text{-}\mathrm{NA}$ Plan Fluor Ph3 DLL objectives, with the 0.52-NA condenser. The objectives were designed for both phase contrast and fluorescence illumination modes, the Fluor objectives have fluorite optical elements for high UV transmission, and the Plan objectives are flat field corrected. Depth of field and NA are interrelated and are application dependent (specimen thickness). The depth of field is proportional to the wavelength, and is inversely proportional to the square of the NA.¹⁵ It is not uncommon to observe specimens thicker than the depth of field of objectives with $\mathrm{NAs}>0.5$ . So submicron errors in focus can cause substantial image degradation.⁵ For the $20\times 0.75\text{-}\mathrm{NA}$ objective, the depth of field was $0.8\mathrm{\mu }\mathrm{m}$ , using the Rayleigh criterion in fluorescence with $\lambda =450\mathrm{nm}$ .¹⁵ We tested autofocus in brightfield, darkfield (not reported), and phase contrast. For the experiments reported here, autofocus was carried out using phase contrast illumination while simultaneously imaging in fluorescence by splitting the light to a different path, as shown in the light path diagram in Fig. 3 . Autofocus can be performed directly with fluorescence images, but performance is better in phase, and because of possible photobleaching and toxic fluorescence by-products, it is best to minimize fluorescence exposure, especially for live cells.⁸ Bright field illumination is also often brighter than fluorescence, which simplifies providing adequate signal to the nine sensors in the volume camera. For fluorescence, we used a DAPI epifluorescent filter cube (Chroma Technology, Brattleboro, Vermont) that included a $365\text{-}\mathrm{nm}+∕-10\text{-}\mathrm{nm}$ excitation filter, a 400-nm dichroic mirror, and no emission filter. It was followed by a 615-nm longpass dichroic mirror (Chroma Technology, Brattleboro, Vermont) that we used to replace the 20/80 eyepiece/sideport beamsplitter prism inside the base of the microscope. This second dichroic mirror separated the phase contrast illumination from the fluorescent light; the fluorescent light was imaged at the sideport and phase contrast on the trinocular head camera port above the eyepieces. Two autofocus light sources were used: a high-luminance Hewlett Packard LED (HPWT-MH00, 626-nm wavelength) and the standard Nikon 50-W halogen bulb in conjunction with a red 665-nm cutoff longpass glass filter (Melles Griot RG665). The halogen bulb was brighter, but the LED can be strobed, and much brighter LEDs are now available. A Nikon 0.9 to $2.25\times$ C-mount zoom lens was used to couple the volume camera to the microscope’s trinocular port.

Fig. 3

A diagram of the light paths for simultaneous phase contrast (to the volume camera) and fluorescence (to the trailing cooled TDI CCD camera) is shown. The bottom “dichroic mirror” assumes a second mirror (not shown) that reflects transmitted fluorescence light to the left.

2.5.

Virtual Frame and Hardware Synchronization

To synchronize the TDI CCD sensor boards in the volume camera array, an external pulse (start of line) was sent to all cameras by a timing sequencer board, which establishes the beginning of a horizontal scan line and set the volume camera line rate ( $F_{\mathrm{vol}}$ in Fig. 2). This pulse was also transmitted to the focus measurement array, as shown in Fig. 2. Since the line scan CCDs of the volume camera produced a video signal with only horizontal blanks between video lines, the concept of end-of-frame (EOF) was absent (no vertical blank sync). Therefore, to define a focus measurement interval or virtual frame that consisted of a predefined number of lines, the timing sequencer board created and transmited an end-of-frame signal to the array of focus measuring circuits (EOF in Fig. 2). The timing sequencer worked as the central controller of the parallel focus circuits and the array of TDI cameras, synchronizing the focus measurement and acquisition of the volume camera image planes.

The virtual frame shown in Fig. 4 is a subfield, or a fraction of the field of view of the specimen that moves across the CCD sensors as the stage moves. Each virtual frame first crosses each of the volume camera CCD sensors in succession and finally reaches the trailing camera. Therefore, there is a spatial offset in the focus measure of each imaging channel, which was taken into account in the control software that buffered the data and images and performed the best focus calculations. The focus measurement electronics design limited the virtual frame size to a minimum of two 1024-pixel lines, and a size of 64 lines was found to be the best tradeoff between more rapidly updating focus (less lines of integration) versus increasing SNR (more lines).

Fig. 4

The virtual frame corresponds to a number of 1024 pixel-wide video lines that produce a single focus measurement, and its length can be smaller or larger than the 96 lines of the TDI CCD. Image data moves across the active area of individual volume camera fibers and the trailing camera simultaneously. Focus measurements are acquired asynchronously and registered spatially for best focus comparison. The nine fibers and the trailing camera active areas are physically laid out within the full microscope field of view.

2.7.

Simultaneous Imaging Paths and Focus Tracking Experiments

With the simultaneous use of two independent imaging paths that had different electro-optical components, one for autofocus and the other for the in-focus image capture, focus and image capture calibration/synchronization was required. Even though the two microscopy modes (phase and fluorescence) share the optical path from the specimen, dividing them into two sets of sensors resulted in the need to calibrate for parfocality and centration, and synchronize acquisition in a manner that compensated for differences in sensitivity and magnification. A prescan calibration was carried out for each set of experimental conditions to obtain adequate signal levels for both output paths and compensate for positional errors caused by optical aberration and mechanical focus differences between the two light paths.

Calibration was first carried out with a structured pattern as the input object (e.g., a Ronchi ruling) with its respective image observed in the output ports. When the line features were positioned parallel to the direction of stage travel, no stage motion was necessary to create a 2-D TDI image. Images were also obtained from other specimens by strobing with the LED while in TDI mode. The Hammamatsu Orca 100 CCD trailing camera was operated in full frame mode (as a static imager) for some of the alignment procedures; incremental autofocus was carried out using conventional image cytometry methods such as comparison with static autofocusing and for manually leveling the slide to within the $100\text{-}\mathrm{\mu }\mathrm{m}$ range of the PIFOC. The detailed calibration procedures were as follows.

• 1. Magnification settings. With a selected objective, a test target with known spacings (Ronchi ruling or micrometer) was imaged, and pixel sampling density and magnification were calculated. The zoom lens in the trinocular port (autofocus light path) was adjusted to obtain the appropriate magnification and line rate [Eq. 1] to match the fluorescence image capture sampling density.

• 2. Parfocality, centration, and orientation. The cross sectional active area of the volume camera was set to the middle of the microscope visual field using adjustment screws. The position of the trailing camera was set using a custom-made $xyz$ microposition stage integrated into its mount, enabling lateral and parfocal alignments. Ronchi-rulings test bars were set parallel to stage motion and the custom-made rotational mounts of both sets of cameras were used to align the Ronchi rulings perpendicular to the CCD lines; adjustment was completed when the lines in the output images appeared sharpest. The trailing camera $z$ -micropositioner was adjusted for parfocality with the center fiber optic CCD channel of the volume camera. Parfocality was refined and confirmed by repeatedly plotting the focus function curves of these two cameras. The trailing camera was centered laterally $\left(xy\right)$ with the volume camera along the direction of image motion, and the end of its active area was adjusted backward in the field of view to coincide with the position of the last sensor in the volume camera.

• 3. Light intensity adjustment. Even with a bright light source, phase contrast required additional input gain at the focus measurement circuits, which were also adjusted for best image contrast in part by adjusting the level (or offset) to decrease background. These adjustments kept the signal below the circuit saturation points and optimized the focus indices per virtual frame time. For the fluorescence imaging path, the pixel intensity level was adjusted by the camera digital gain. After an integration period was set in relationship to the stage velocity and TDI dwell time, the corresponding light sources and gains were adjusted to avoid image saturation on any of the cameras with a test scan on a region of the sample with a high density of cells. The trailing camera exhibited high sensitivity at the scanning speeds tested here, and a neutral density filter and a bandpass filter at the side port were used to further attenuate fluorescence light intensity as needed.

• 4. Parallel focus measurement calibration. The multiple detectors in the volume camera are placed in different $z$ positions, which introduced optical path length differences that led to the need for axial positioning via focus function calibration, as shown in Fig. 5 . It was straightforward to utilize the focus function curves from the Ronchi rulings to locate and adjust the best focus for each camera. The focus function curves were sharp and monotonic, giving a peak that indicated the focal point of each CCD sensor relative to the object plane. The volume camera family of focus function curves also best established the position of the object and was used to verify that trailing camera focus had been set to the focus of the center fiber optic sensor in step 2. See Fig. 5 and the following section for the detailed description of volume camera focus calibration.

Fig. 5

A set of calibration plots consisting of the focus measurement responses of each of the channels is shown. The normalized focus function responses of the fiber optically coupled imaging channels were obtained using a $10\text{-}\mathrm{\mu }\mathrm{m}$ period micrometer under brightfield microscopy and a $z$ axis through focus excursion of $20\mathrm{\mu }\mathrm{m}$ . Channels 3 and 7 are not shown. The trailing camera focus response is also shown, and during calibration its peak is adjusted to the middle of the focus search range.

After this calibration of the imaging paths, closed-loop autofocus performance was evaluated in static- and moving-stage experiments to determine the stability of: static focus with no stage motion, dynamic focus with no stage motion with perturbations provided by manual defocus (data not shown), and tracking tilted artificial (Ronchi rulings) and cell monolayer specimens during lateral stage motion. The fluorescence image was stored for later analysis, and the volume camera signal was used only for focus tracking (in phase contrast). During scanning, a log of the objective lens positions was recorded along with the time of each recording. The time recordings enabled the percentage of focus updates missed (likely due to operating system uncertainties) to be calculated. A missing objective lens position meant a failed focus update at the unrecorded time point. The throughput of the system in cells/s depended on the cell density, the speed of the stage, and sampling density. The trailing camera recorded the continuous fluorescence image in a sequence of image frames of $1024\times 1024$ pixels, and this frame size and the stage speed was defined as the imaging field rate (and corresponds to a conventional 2-D or incremental imaging rate). The measurements of cell features were performed field by field on these $1024\times 1024$ images by a software module that can work either off-line or asynchronously with image acquisition via an on-disk image storage buffer.

3.

Control System Design for Fiber Optic Continuous Autofocus

The control system software communicated with the hardware elements to close the autofocus feedback loop and acquire the fluorescence image. The control system was organized into five software modules that are designated M1 to M5 in Fig. 6 . These five modules managed the complexity of several components running in parallel, organized multiple data streams, and minimized and managed delays in the feedback loop. The nine continuous focus data streams from the volume camera were stored dynamically in multiple raw data acquisition buffers. Data were sorted via a virtual moving window (defined in software). The data was field of view (FOV) registered and placed in focus index and $XYZ$ position buffers for best focus calculation. Software modules M1 to M5 and the data buffering relied on $\mathrm{C}++$ classes and the main National Instruments interface functions, and are described in further detail as follows.

Fig. 6

This data flow diagram for the real-time asynchronous software modules shows the processing of a specimen to extract a data store of images and cell biology features. The focusing feedback loop and trailing camera TDI acquisition occur simultaneously and are linked through timing signals based on stage movement velocity. Labels M1 to M5 designate the corresponding software modules.

4.

Results

First, an analysis of the software and hardware latencies was carried out, since these are fundamental to the feedback characteristics of the system. These latencies include the time required for software sorting and processing, the PIFOC response, and the time delays caused by the sequentially lateral displacement of the fiber optic imaging faces. The complexity of the parallel hardware system latencies was managed by synchronization in the software modules to enable closed-loop autofocus. Table 2 lists the software modules along with their indexed processes, execution times, order of execution dependency, and state interdependencies. The timing-critical software functions were: sorting the data, calculating best focus, and checking progress of the moving focus-image acquisition window. The sorting function proved to be the slowest of the processes, but was still 2 orders of magnitude faster than closed loop electromechanical objective repositioning (PIFOC response of $10\mathrm{ms}$ ). Calculating the best focus and checking the scan progress, which were performed for each feedback adjustment of the objective focus position, required 10% the overhead of the sorting function. The system design thus successfully limited and managed these latencies, and understanding them helped in setting the feedback gain.

Static- and moving-stage experiments were carried out to better understand the effects of the latencies, determine the best gain settings, and characterize the closed-loop autofocus performance. In the static-stage experiment, the microscope stage was immobile, but environmental conditions including mechanical vibrations and thermal focus drift were present. Closed-loop autofocus was performed at a focus update rate of $27\mathrm{Hz}$ , and the static-stage focus position was maintained with a SD of $14\mathrm{nm}$ over a 70-s period. In the moving-stage autofocus experiment, a comparison of tracking on a moving, tilted slide with etched Ronchi rulings (of $2000\mathrm{lines}∕\mathrm{in.}$ ) in continuous and incremental autofocus is shown in Fig. 7 . Focus was updated up to every $41\mathrm{\mu }\mathrm{m}$ along the direction of stage travel with the $20\times$ objective. Increasing the focus update gain $K$ in Eq. 5 improved tracking later in the scan, but higher gains demonstrated more oscillatory behavior at the beginning. There can be transient out-of-focus instability or drift at the beginning of the scan because the focus measurements are in error until the stage reaches a constant velocity and traverses the distance between the outer fibers. This was corrected by a prestart period of acceleration where autofocus is not performed.

Fig. 7

Continuous focus tracking over $20\mathrm{mm}$ of a tilted specimen at different gains is compared to incremental focus (at four positions). Ronchi rulings of $2000\mathrm{lines}∕\mathrm{in.}$ were imaged with a Fluor $20\times 0.75\text{-}\mathrm{NA}$ 160-mm tube length objective at a stage speed of $1.13\mathrm{mm}∕\mathrm{s}$ . Focus was updated at $27.13\mathrm{Hz}$ with a focus update distance of $41\mathrm{\mu }\mathrm{m}$ .

The ultimate goal was to acquire in-focus fluorescent images of cells during continuous stage motion using on-the-fly focus tracking. Continuous fluorescence imaging was broken into image frames of $1024\times 1024$ pixels $\left(1\mathrm{MB}\right)$ , and an image strip was composed by the concatenation of these frames. This is demonstrated with the fluorescence image shown in Fig. 8a , which is a typical sharp image strip acquired during continuous scanning of DAPI-stained cells cultured on a coverslip. Focus tracking of cells on four 30-mm-long strips included more than 5700 focus updates, of which only 1.74% were missed for an average of one miss every 57 fluorescence images. The scanned area comprised more than 700 1-MB fluorescence images. Focus was measured and updated just over 8 times per one fluorescence image field captured with the trailing camera, or 8-fold more often than with our incremental scanning system. For this reason, the occasional failure to update focus was not thought to compromise autofocus performance. As can be seen in Fig. 8b, focus tracking was well behaved and study of the data revealed no unexpected behaviors associated with missed update misses. The speed of the stage in these experiments was $0.572\mathrm{mm}∕\mathrm{s}$ or $\sim 0.1{\mathrm{mm}}^2∕\mathrm{s}$ with fluorescence image sampling of $167\times 167\text{-}\mathrm{nm}∕\mathrm{pixel}$ , which resulted in a field rate of $3.3\mathrm{frames}∕\mathrm{s}$ . In other similar cell scanning experiments, the stage speed was set to $2.34\mathrm{mm}∕\mathrm{s}$ and a fluorescence image sampling of $335\mathrm{nm}∕\mathrm{pixel}$ , providing an imaging rate of $6.8\mathrm{fields}∕\mathrm{s}$ . We achieved a maximum focus update rate of $56\mathrm{Hz}$ reliably; failure to update focus started to increase beyond this value. The stage speed was limited to $2.54\mathrm{mm}∕\mathrm{s}$ by the prototype volume camera line rate, which corresponded to a fluorescence field rate of $7.4\mathrm{fields}∕\mathrm{s}$ with the $20\times$ objective. All of the captured images were sharp, indicating that dynamic autofocus performed well.

Fig. 8

(a) A fluorescence image strip acquired with a resolution of $167\times 167\text{-}{\mathrm{nm}}^2∕\mathrm{pixel}$ is shown. The left strip represents a 4-mm length of the specimen. The right image is a $0.5\times 0.171\text{-}{\mathrm{mm}}^2$ area and 24 focus updates occurred on it as shown by the marks along the right side of the image. These images were extracted from a 30-mm-long continuous scan of NIH 3T3 cells cultured on a coverslip and stained with DAPI. Stage speed was $0.572\mathrm{mm}∕\mathrm{s}$ (for $\sim 0.1{\mathrm{mm}}^2∕\mathrm{s}$ ). With an ideal, densely populated specimen $(\mathrm{10,000}\mathrm{cells}∕{\mathrm{mm}}^2)$ , these conditions would scan up to $978\mathrm{cells}∕\mathrm{s}$ . (b) A plot of the focus positions over the full 30-mm scan is shown. The focus update rate was $27\mathrm{Hz}$ . The images were acquired with a $40\times 0.75\mathrm{NA}$ Plan Fluor Ph3 objective. The stage speed was $0.572\mathrm{mm}∕\mathrm{s}$ and it took $\sim 52\mathrm{s}$ to scan the $30\mathrm{mm}$ .

Table 3

Summary of performance of the new continuous scan and autofocus system. “a” Test conditions: closed-loop autofocus with no stage motion. Specimen: Ronchi rulings (2000lines∕in.) , focus update rate= 27Hz . Microscope in brightfield mode, tests were carried out with 20× and 40× optics, and focus feedback gain of 0.2 and 0.4. “b” Specimen: NIH 3T3 cells with DAPI stain. There were 769 cells detected, mostly in the G0 phase.

Table 3 summarizes the experimental conditions and performance results for continuous scanning and autofocus. The autofocus error was obtained from several 70-s-long closed-loop autofocus trials, which were carried out at an update rate of $27\mathrm{Hz}$ with the stage not moving. For all of the trials, the error was the standard deviation of the focus position, and the “best” experiment was the one with the lowest standard deviation. For the G0/G1 CV reported in Table 3, commercial DNA analysis software (Phoenix Flow System, San Diego, California) was used to fit the DNA content histogram measurements that were performed on a $17.1\text{-}\mathrm{mm}\times 171\text{-}\mathrm{\mu }\mathrm{m}$ section of a slide on which NIH 3T3 cells were cultured and stained with DAPI. 769 cells were detected and focus tracking performed well (with a stage speed of $0.572\mathrm{mm}∕\mathrm{s}$ ), even with the relatively low cell density.

5.

Discussion and Conclusions

This combination of optics, electronics, and software control functions successfully achieved closed-loop continuous-image-content-based autofocus on a microscope for the first time. Careful design, testing, and calibration were critical for determining the delays and divergence of focus responses in each channel and compensating in software for these differences. The software control system provided functions for calibration, buffering, and processing the raw data, measuring and maintaining closed-loop feedback autofocus, and acquiring the fluorescence image. Sharp focus of cell monolayers on microscope slides demonstrated that parallel multiplanar focus measurements are a powerful basis for autofocus in TDI image acquisition for microscopy.

The high static focus precision of $14\mathrm{nm}$ resulted from simultaneous focus sampling of multiple focal planes, enabling a much higher effective focus measurement rate. This precision was consistently better than the best SD of $23\mathrm{nm}$ and the average of $56\mathrm{nm}$ that we previously achieved in sequential autofocus.⁷ The absolute focus positions of dynamic autofocus matched the incremental autofocus carried out on the same field of view well. In comparing dynamic autofocus positions to those obtained during incremental scanning, repositioning errors due to system thermal and mechanical instabilities are likely on the order of $1.0\mathrm{\mu }\mathrm{m}$ , so caution must be used in overinterpreting the results in Fig. 7. Given the repositioning errors, the different gains tracked the specimen well and it can be concluded that decreasing the gain enough to eliminate oscillation resulted in good tracking for specimen gradients within $\pm 0.37\mathrm{\mu }\mathrm{m}∕\mathrm{mm}$ (for the experiments of Fig. 7). Furthermore, for a typical focus search range (or vertical sampling of the fiber optic staircase in specimen space) of $2.7\mathrm{\mu }\mathrm{m}$ , continuously tracking a specimen slope of $8\mathrm{deg}$ worked for all of the slides we prepared. The system was tested with $10\times 0.30\mathrm{NA}$ , $20\times 0.75\mathrm{NA}$ , and $40\times 0.75\mathrm{NA}$ objectives. Autofocus is simpler with lower NA objectives where depth of field is larger. Performance was also excellent on the higher 0.75-NA objectives, confirming that this image-content-based autofocus system is also capable of performing well with higher resolution objectives. Autofocus performance better than the depth of field for a range of objectives will enable a broad range of applications.

The seven software modules operated concurrently under Windows NT. The raw tracking data were collected asynchronously in the background using the National Instruments boards and DMA transfers, which do not require CPU overhead. The tracking data acquisition memory requirements were manageable, and the computer was fast enough to carry out the software processes without requiring explicit real-time operating system functions. Even though acquiring the tracking data asynchronously from the focus measurement circuit data-valid signal is inefficient because both valid and invalid autofocus data are acquired and a search is required to locate the valid data, this extra overhead was not limiting. This practical buffering and sorting solution prevented the additional headaches of software design for specialized real-time operating systems and synchronous hardware interfaces. The penalty was an average of one missed focus update for every 57 conventional 2-D fields of view, which did not appear to adversely affect autofocus performance. Another possible reason for these occasional missed focus updates was the moving window jitter, which was previously noted to omit the reading of buffered samples during the valid focus trigger. However, short term measurements indicated that only one of the three valid buffered samples was missed due to window jitter, which would not have prevented the system from updating a new best focus. But it is possible that all three samples were missed very rarely. Whatever the source of missed focus updates, focus was updated frequently enough (8 times/conventional 2-D field) to keep the high-resolution fluorescence image sharply focused during continuous stage motion.

The lateral offsets between fields of view (or virtual frames) of the fiber optic imaging faces in the volume camera required buffering and registration for correct comparison of focus measurements and continuous focus tracking operation. These offsets introduced a delay that slowed tracking response and might adversely affect performance under conditions not yet demonstrated. The delay may reduce tracking response to sharp transient gradients in the specimen surface. Direct use of the measurements without registration increased the response to sharp gradients, but also increased uncertainty around best focus position (data not shown). It is possible that a feedback model that explicitly accounts for these different delays might perform better than the proportional feedback model implemented here. A related problem is that a linear staircase pattern for the axial positions of the fiber faces might coincidently be exactly matched by a long linear specimen gradient. At the appropriate velocity, this condition would yield no focus measurement differences at best focus on which to base feedback control, which might in turn lead to oscillatory behavior not yet encountered. A staggered pattern of the fiber faces would reduce the probability of encountering a specimen-fiber optic gradient match. Other possible disadvantages of the imaging fiber bundles were: their fragility (especially the small-area glued CCD-bundle interface), errors in magnification due to variation in the packing density, degradation of the imaging MTF, irregular CCD-to-fiber bonding distance, end-to-end cross sectional area mismatches, $\sim 50\%$ loss in image transmission, and occasional defects that introduced fixed distortion patterns. Overall, however, the fiber imaging quality was high and these defects, which probably largely produced variability in the noise floor between imaging channels, did not appear to greatly degrade performance.

It is instructive to estimate possible cell analysis rates using the principle of continuous TDI scanning. Cell density varies and increased cell density can add complexity to image segmentation,³⁹ but with a densely populated specimen of, e.g., white cells smeared on a slide at $\mathrm{10,000}\mathrm{cells}∕{\mathrm{mm}}^2$ (allowing $10\times 10\mathrm{\mu }{\mathrm{m}}^2∕\mathrm{cell}$ ), the present experimental conditions would result in a rate of about $1000\mathrm{cells}∕\mathrm{s}$ . And at the maximum line rate of $4\mathrm{KHz}$ permitted by the volume camera prototype (or $7.76\mathrm{KHz}$ for the trailing camera) and a $20\times$ objective (versus the $40\times$ objective in these experiments), the rate would be $8000\mathrm{cells}∕\mathrm{s}$ . Further increases in line rate or CCD line length would proportionally increase analysis speeds. The 4-kHz maximum line rate of the volume camera should result in a maximum focus position update rate of $(4000\mathrm{lines}∕\mathrm{s})∕(64$ lines/focus $\mathrm{update})=62.5\mathrm{Hz}$ , which is still below the characteristic PIFOC controller system frequency of about $100\mathrm{Hz}$ (or a 10-ms response period). In our experiments, we achieved a maximum update rate of $56\mathrm{Hz}$ reliably.

Successful closed-loop feedback autofocus in continuous-motion scanning enabled high-resolution $(\mathrm{NA}\ge 0.5)$ image cytometry operation at speeds more comparable to flow cytometry. Sequential start-and-stop stage motion and focus measurement in conventional automatic image cytometry previously placed substantial limitations on both speed and imaging efficiency. Imaging multiple focal planes in parallel and acquiring the fluorescence image at the same time enabled continuous scanning that overcame the image cytometry limitations (microscope stage acceleration and sequential imaging of the focal planes for autofocus) of incremental scanning. With continuous scanning, image acquisition speed is limited only by the combinations of camera sensitivity, fluorescence intensity, and CCD line width. For bright fluorescence, scanning speed will continue to increase with faster CCD clocking. And for any given fluorescence brightness and optical setup, increasing the CCD line width will result in a directly proportional increase in speed through adding imaging parallelism, limited only by the objective field of view.

Using the most recent Nikon infinity-corrected objectives, Nyquist sampling with 1,024-pixel video lines acquires an image comprising about 1/3 of the field of view. Longer CCD lines will proportionally increase scanning speed at the same brightness. Thus, increasing parallelism is a powerful tool that can continue to be exploited for further increases in scanning speed. Many cells can be imaged and analyzed in parallel with continuous-motion image cytometry, whereas flow cytometry has been limited to sequential operation or a single file stream of cells thus far. If this fundamental difference persists, image cytometry speed can continue to increase as technologies improve without sacrificing sensitivity, while flow cytometry speed increases will be fundamentally limited by single-file sequential cell measurement operation. With both the speed and information advantage, we predict that image cytometry instrumentation will eventually supplant flow cytometry, especially for applications where the cells are natively anchorage-dependent.