2.1.

Mosaicing Method

Our previous mosaicing method required stitching square-shaped images, with $\sim 1:1$ aspect ratio, in two dimensions, as shown in Fig. 1(a). The newly developed method acquires rectangular-shaped long strips of images,²⁰ with $1:25$ aspect ratio for a 10-mm long strip, and stitches along the length of the strip as shown in Fig. 1(b). The elimination of a stitching dimension reduces the processing time and the artifacts due to illumination fall-off by half.

Fig. 1

Comparison of mosaicing concepts: (a) mosaicing of square-shaped images in two dimensions; (b) mosaicing of rectangular-shaped long strips in one dimension (this figure is reproduced from Ref. 20).

The strip mosaicing scheme, with combined optical and mechanical scanning, is shown in Fig. 2. The fast optical scanner produces a scan-line (in the $x$ direction) that defines the width of a strip. Scanning the length of the strip (in the $y$ direction) is performed by translating the stage in the direction orthogonal to the line. When the acquisition of a strip image is completed (“forward scan” in $y$ direction), the stage moves in the $x$ direction a distance equal to or less than the width of the strip. Then the stage moves along the strip length but in the opposite direction (“return scan”), and another strip image is acquired. This process of acquiring strip images, sequentially in opposite $y$ directions (forward scan followed by return scan) is repeated until the entire tissue is imaged. The aspect ratio of a strip is restricted only by the mechanical limits of the scanning stage.

Fig. 2

Schematic of the strip-scanning mechanism. The fast optical scan ($x$ axis scan) defines the width of a strip. The slow mechanical scan along the length of the strip ($y$ axis scan) is performed by translating the stage in the direction orthogonal to the $x$ axis. When the acquisition of a strip image is completed (“forward scan”), the stage moves in the $x$ direction a distance equal to or less than the width of the strip. Then the stage moves along the strip length, but in the opposite direction (“return scan”), and another strip image is acquired. This process of acquiring strip images is repeated until the entire tissue is imaged.

2.2.

Instrumentation

2.2.3.

Synchronization of strips in the y direction

In our earlier configuration,²⁰ the translation stage mechanical scan (slow scan in $y$ direction) was driven by a stepper-motor without a position encoder. We accomplished the synchronization of the polygon driven optical scan (fast scan in $x$ direction) and the translation stage mechanical scan (slow scan in $y$ direction) by counting the steps from the stepper-motor. This method turned out to be unreliable. We observed mismatch of up to 75 lines, between any two strips, due to missing steps in the motors and the resulting lack of position accuracy. This mismatch was corrected with software during stitching of the strips but at the expense of more computer processing and increased time for creating mosaics. However, the current position synchronization in the $y$ direction reduces the work of the stitching algorithm, thus reducing the time required to complete the mosaic. The synchronization mechanism that we report in this paper ensures that the theoretical maximum mismatch between any two strips is one line in the $y$ direction. Figure 3 shows the synchronizing scheme for the strips.

Fig. 3

The synchronizing scheme for the strips. A $y$ translation stage cycle consists of two mechanical movements that produce two image strips: the “forward scan” $P_0$ to $P_3$, and the “return scan” $P_3$ to $P_0$. The velocity profiles are depicted in red for the forward movement, and blue for the return. The image is acquired within the constant velocity portion (d), between $P_1$ and $P_2$, of the scans. This avoids distortion of the mosaic due to compression or elongation of pixels. Therefore, we choose a scan distance (D) for the $y$ translation stage such that the region of constant velocity (d) is larger than the size of the tissue sample. After the forward scan is initiated and the stage reaches $P_1$, the “position trigger” signal is asserted to arm the counter that monitors horizontal synchronization pulses (HSYNC) from the asynchronous optical scanner. When the HSYNC counter receives the next HSYNC pulse, the data acquisition begins. When the stage reaches $P_2$, the HSYNC counter resets and the acquisition continues until the last line is complete in the strip image. Then the stage decelerates and stops at $P_3$. Once the $x$ translation stage moves in the $x$ direction to a predetermined position that sets the width of the strip and the overlap between adjacent strips, the return scan is initiated from $P_3$ to $P_0$. It follows a similar mechanism to the forward scan. This cyclical process, forward scan and return scan, is repeated until the entire tissue is imaged.

2.2.4.

Tissue flattening and stage leveling

In order to image a large area, the surface of the tissue is scanned in a two-dimensional (2-D) “image plane,” which is perpendicular to the optical axis of the objective lens in the microscope (Fig. 4). However, surgically excised tissue has a three-dimensional (3-D) topography with varying shape and size. Furthermore, the tissue has variable mechanical compliance; for example, skin tissue is relatively stiff whereas breast tissue is soft and gel-like. This requires that the tissue surface be flattened onto a 2-D plane that conforms to the image plane of the microscope [Fig. 4(c)]. For this purpose we designed a tissue flattening and mounting device, as shown in Fig. 4(a).

Fig. 4

Schematic representation of the tissue-flattening device: (a) 2-D view of the tissue (in red) inside the cassette in its natural shape, as often seen in Mohs surgical excisions, with the edges curved up from the image plane and the lower (to be imaged) surface not flattened. The cassette is shown to be open with no force is applied to the water-filled bladder; (b) view of the closed tissue-cassette before mounting on the tip-tilt plate; the thumbscrews A, B, and C can be used to adjust the tip and tilt of the image plane relative to the optical axis, so as to orient the two at exactly 90 deg to each other; (c) schematic representation shows how the tissue is flattened, relative to the image plane and optical axis, by the water filled bladder when force is applied to it.

We use a rubber bladder filled with water to flatten the tissue onto a 1-mm-thick BK7 window inside a cassette Fig. 4(a). To flatten the desired (lower) surface, the user places the tissue at the center of the cassette, the water-bladder on top and closes the cassette cover. Then the tissue-cassette is placed on the tip-tilt plate, secured with two spring-loaded clips and placed on the microscope.

Acquisition of a large number of images over a large area at constant depth requires the imaging and scanning planes to be coincident. This is obtained by adjusting the tip-tilt plate with spring-loaded thumbscrews [Fig. 4(b)]. The three thumbscrews are placed at right angles to each other, and tip/tilt can be adjusted by initially turning either thumbscrew A or C [Fig. 4(a)], while observing images of the glass window at four corners. The image of the window may be either a circular spot when focused on the surface or a ring due to field curvature when focused past the glass. This adjustment is performed in an iterative manner until the images at the four corners appear identical to each other. This device and the alignment method allow large tissues to be matched to the image plane of the microscope, and enables mosaicing over large areas. The mounting of the tissue cassette and the adjustment of the tip-tilt are robust enough to allow repeatable operation during extended periods without the need to realign.

2.3.

Acquisition of Mosaics

For acquisition of mosaics, the resolution is set in our acquisition system to the level of nuclear and other morphologic detail needed to match standard histology. While relatively lower resolution may be adequate for imaging skin tissue, higher resolution is needed to analyze breast tissue.

The images are acquired in LabView and displayed in 8-bits. The 8-bit pixel depth was chosen because of our previous experience and for practical reasons. Clinical results from our previous studies^18,19 have validated that 8-bits are sufficient for this particular application. Our Mohs surgeons found the quality of the resulting images to be clinically acceptable and were able to evaluate mosaics, in a manner similar to their reading of histopathology, with sensitivity of 96.6% and specificity of 89.2%. Another consideration is that 8-bit mosaics of large areas produce images of 300 to 400 MB. Increasing the depth to 16-bits would double their size making them difficult to manipulate. Such large image files pose practical challenges for subsequent processing and storing. This is further addressed in Sec. 2.5 on display of mosaics. The fluorescence images are captured with a fast DAQ card (PCI-6115 and Labview, National Instruments, Austin TX). The captured image strips are stitched with custom software developed by our group.

2.4.

Stitching of Image Strips

A stitching algorithm was written in Matlab (R2011a, Mathworks, Inc.) to automatically register image strips in the order in which they are collected. The stitching algorithm has three parts: a registration algorithm, an intensity fall-off correction, and blending in the overlap region between the two strips.

The strip-scanning method, as compared with the traditional mosaicing method, simplifies registration of images, as each image needs to be registered to only one neighbor instead of two. Strips are registered pair-wise using a phase correlation method chosen for speed and ease of implementation, as it is easily computed by fast Fourier transform (FFT). Computing the phase correlation between the two images produces a 2-D matrix of correlation values where the coordinate of each value in the correlation matrix corresponds to a possible offset between the two images. The correlation value is maximal at the coordinate at which the same features in the two images overlap. But, because the correlation of two rectangular functions (such as two images) is a triangle function, the correlation is biased toward solutions near zero offset between the two images (the peak of the triangle). The bias is removed in the $x$ direction by dividing the phase correlation matrix by a triangle function. The removal of the bias emphasizes the noise near the edges of the correlation matrix, where solutions of maximal offset between the two images lie. Solutions of maximal offset correspond to no overlap between the images. Because we expect the images to overlap by an amount that is set by the stage motors, we can exclude solutions outside of the expected range. Therefore coordinates in the noisy region near the edges of the correlation matrix are excluded from the search region. In the $y$ direction, we expect solutions near zero offset, and therefore the removal of the bias was determined to be unnecessary along the $y$ direction. The solution for $x$ offset and $y$ offset between the two images is found by determining the maximum of the correlation matrix after the bias has been removed and the edges of the matrix have been excluded from the search region.

Performing a 2-D FFT on very large images requires a large amount of memory. In practice, only a small section of the strip is required to perform registration, as long as the chosen $y$ length of the section is greater than the $y$ offset between the two strips. The chosen strip section must contain features to register. In our setup, the tissue lies in the center of the glass window. Hence, a section that is 10 to 30% the length of the strip is chosen from the center region of each strip. This also serves the purpose of restricting the possible offset solutions to those that we know are most likely to be a good match. The search region is further restricted to coordinates near that expected from our knowledge of the stage motion. Generally, an apodizing window function is applied to an image before performing a phase-correlation algorithm to reduce high-frequency edge effects. However, in this case, the illumination fall-off across the strips serves this purpose.

The illumination fall-off is corrected after the registration algorithm is applied. Because the optical scanning through the objective lens is in one dimension only, the illumination fall-off is along only the $x$ direction of each strip. The fall-off is corrected by averaging each strip along its length, normalizing, and dividing each line in the strip by the normalized average fall-off profile of the strip.

In the overlap region between strips, the strips are blended by a weighted average of the overlapping pixels determined by the pixel distance to the edge of the strip. Pixels close to the edge of the strip are weighted less than pixels farther away. The result is a seamless mosaic, with more weight given to pixels near the center of the strip and less weight given to pixels near the edges of the strip.

2.5.

Display of Mosaics

The mosaics are displayed in 8-bits on a large monitor with $2500\times 1600\mathrm{pixels}$ (30-in. flat-screen Dell 222–7175, with a GeForce 8800 GTS video card).When histopathology is examined with a bright field microscope, using, typically, a $2\times$, 0.08NA objective lens with white light illumination, the observed field of view (FOV) is 10 mm with $\sim 4\text{-}\mu \mathrm{m}$ resolution. This FOV consists of approximately $2500\times 2500\mathrm{pixels}$ assuming one pixel per one optical resolution point. Our display matches this standard $2\times$-view of histopathology. Zooming allows one to view submosaics at full resolution, mimicking the pathologist’s ability to switch to higher magnifications when necessary. Our Matlab program can easily manage the manipulation and display of mosaics of FOV $10\times 10\mathrm{mm}$. However, for larger mosaics, the program exhausts allowed allocated memory in our current system. (Potential future implementation at the bedside in, say, diverse and low-resource settings may require alternative practical and lower-cost approaches. One approach, for example, is scaling large mosaics to smaller size, while maintaining pathology-equivalent $2500\times 2500\mathrm{pixelation}$, which results in final size of 3 to 4 MB. This approach has been validated for adequate clinical quality and sensitivity and specificity by our Mohs surgeons.^18,19

3.1.

Skin Tissue from Mohs Surgery

Figure 5 shows a strip mosaic of a skin excision from Mohs surgery. The measured time for the $7.7\times 7.7{\mathrm{mm}}^2$ mosaic, consisting of 24-strip images, was about 3 min. This mosaic was created before we implemented simultaneous stitching with the acquisition. The strips were stitched after acquisition was completed, adding approximately an extra 96 s to complete the mosaic.

Nuclear detail and morphology is seen in the magnified insets A, B, and C. The morphologic features in the mosaics compare well to those in the corresponding pathology (Fig. 6) for both basal cell carcinomas and normal skin. However, a perfect one-to-one correlation of individual features is not always expected (nor desired by Mohs surgeons), because tissue is pliant and thus susceptible to distortions introduced during the fixturing for mosaicing or processing for frozen sections. As a result, the pathology is usually of a 5-μm-thin tissue section that may be adjacent to, but not exactly of, the imaged surface.

Figure 7 is also a strip mosaic of a skin excision from Mohs surgery. This mosaic is $6.8\text{-}\times \text{-}8{\mathrm{mm}}^2$ consisting of 17-strip images. The image strips were stitched simultaneously with acquisition, and the mosaic was completed in 55 s [The simultaneous stitching eliminating the extra time that was otherwise required for some of our earlier mosaics (Fig. 8)]. The nuclear detail and morphology is seen in the magnified insets A, B, and C. The morphologic features in the mosaics compare well to corresponding pathology (Fig. 8).

3.2.

Breast Tissue from Breast Cancer Surgery

Figure 9 is a strip mosaic of breast tissue from a lumpectomy. The actual size of the tissue is about $2.5\times 3.5{\mathrm{cm}}^2$. A step-shaped artifact can be seen at the upper and lower edges of the mosaic (Fig. 9). This was due to a malfunction in the tissue translation stage. [The malfunction manifested as a loss of position in the scan along the strip ($y$ direction) while stepping in the orthogonal direction ($x$ direction.)] Thirty 3.5-cm-long image strips were acquired in 13 min and stitched after acquisition. For this mosaic, we can only report the acquisition time due to hardware limitation in the present implementation. The scan of a such large area requires about 60 image strips. Our stitching program (written in Matlab) runs out of allowed allocated memory during attempts to stitch all the 2.5-cm long strips. Each strip is a factor of 2.5 longer than our usual 10-mm strip. In this mosaic we first stitched four 2.5-cm strips and used ImageJ to merge 4-strip images together to create the mosaic. In the future, this hurdle may be overcome by implementing a graphic processing unit (GPU) dedicated for stitching and display. Since the strips are being stitched during acquisition we expect the full $2.5\times 3.5{\mathrm{cm}}^2$ mosaic could be produced in 13 min with a dedicated GPU (plus the time to scan and stitch the last strip).

Within the mosaic, regions of interest can be identified. The central bright-appearing region is tumor and the outlying darker-appearing regions are primarily fat and connective tissue. Upon closer inspection, cellular and increased nuclear density is visible. For example, inset B of Fig. 9 shows a duct. A normal duct is made up of a double layer of cells surrounding an empty lumen. In this case, the normal growth of the epithelial cells has been disrupted and while the overall shape resembles a duct, the cellular architecture shows that this is no longer a typical cellular pattern. Inset C of Fig. 9 shows proliferation of cell growth in a duct. Instead of the normal double-cell layer, cells have grown inward to the lumen. The cells are contained within the boundaries of the duct, suggesting a ductal carcinoma in situ. Inset A of Fig. 9 shows invasive carcinoma in which the cancer cells have invaded the surrounding fat cells. The fat cells are large and appear dark. The tumor cells are small with bright nuclei.