2.1.

Structured Illumination—Conventional Approach

Structured illumination is achieved by projecting high frequency, spatially patterned light onto a specimen. The typical setup for SIM and for all experimentation throughout this paper is sketched in Fig. 1. The pattern lies on a plane conjugate to both the CCD and a discrete plane of focus at the specimen. As a result, light scattered from the in-focus plane is modulated, separating it from the out-of-focus light. Separating these two components allows for the removal of unwanted light from regions above and below the plane of interest. This is accomplished by measuring a total of three images, typically with phases 0, 120, and 240 deg, and then processing them with the differencing scheme:²

Fig. 1

Layout of the experimental setup. The dashed lines represent planes that are conjugate to one another. This focuses a pattern from the DMD onto a discrete plane of the specimen (after the objective) and relays this back onto the camera.

When selecting the frequency of the modulation pattern, there is a trade-off between sectioning depth and sectioning resolution. A high-frequency pattern will produce higher sectioning resolution as it quickly blurs away from the conjugate plane. However, it becomes difficult to resolve at depth, limiting the absolute sectioning depth. Using a lower frequency pattern will result in poorer sectioning resolution but good sectioning depth. The absolute value of the resolutions will depend on the numerical aperture (NA) of the system. However, for this paper, we use a single-modulation pattern of $40\text{cycles}/\text{image}$ for all depths and samples to simplify comparison across all scenarios. This particular value was selected to achieve a reasonable sectioning resolution ($\sim 2\mu \mathrm{m}$) and sectioning depth ($\sim 30\mu \mathrm{m}$). At these sizes, we will be able to section at biologically relevant sizes and depth, specifically when considering skin cells and at depths located near the junction of the epidermis and dermis.¹ Also, because we have used only a single modulation frequency, data acquisition time and processing will remain constant for all targets and depths.

2.2.

Spatial Frequency Domain Imaging—Using Hilbert Transform

To achieve single-image sectioning, we extend the spatial frequency domain imaging (SFDI) work done by Nadeau et al.¹² Similar to SIM, SFDI functions by modulating an image with a known frequency and phase. SFDI works to separate the absorption coefficient (${\mu }_{\mathrm{a}}$) and reduced scattering coefficient (${\mu }_{\mathrm{s}}^{\prime }$) of a material, which have differing sensitivities to the spatial frequency of the projected light. As a result, ${\mu }_{\mathrm{a}}$ and ${\mu }_{\mathrm{s}}^{\prime }$ can be decoupled using the DC and AC portions of the signal, respectively.^13–15 In SIM, we use the same principal to decouple in-focus and out-of-focus light.

Conventional SFDI requires three phases to be measured along with one image without modulation, for a total of four images. Recent advancements in signal processing and SFDI have produced a method of demodulating an image of unknown phase, frequency, and angle using the 2-D Hilbert transform. Developed by Larkin et al.,^10,11 a spiral function is applied to demodulate a 2-D fringe pattern of unknown frequency and phase. In the two-dimensional case, the Hilbert transform is applied using the spiral function, where $u$ and $v$ are positional indices within the frequency domain:

2.3.

One-Dimensional Simulation

Using a one-dimensional simulation, we describe how the in-focus light (AC) and out-of-focus light (DC) components from a signal can be isolated from one another. For the simulation, the “Hilbert” function from Mathworks MATLAB v2015a is used but serves as a suitable stand-in for the spiral function when applied to the two-dimensional cases going forward.

First, synthetic data are generated to demonstrate the algorithm. Here, the synthetic data represent a spatially varying signal in one direction, a simplified version of the two-dimensional images, which will be considered later on. This is the signal that will lie in the conjugate plane, just beyond the objective in Fig. 1. A random signal is constructed with a mean of 0.5, which represents the signal we hope to extract from a given focal plane (shown in green in Fig. 2). A modulation pattern is constructed as $M(x)=0.25\cos (2\pi f_x)+0.5$, which is the patterned light that will be projected by the digital micromirror device (DMD). This modulation pattern represents an irradiance pattern projected onto the sample, which has a mean of 0.5 and ranges from 0.25 to 0.75 (shown by the dashed blue line) and multiplied by the in-focus signal ($R_{\mathrm{IF}}$). Independently, a second low spatial frequency, random pattern is created to represent the out-of-focus light ($R_{\mathrm{OF}}$, shown in red). This signal is generated with low spatial frequency, as it represents the background scattered light from regions outside the conjugate plane.

Fig. 2

All data in raw samples (unitless counts). (a) The low frequency out-of-focus and high frequency in-focus signals that are combined with the modulation pattern, as described in Eq. (3). (b) Power spectral density of the measured signal before and after the DC component is removed. We can see the modulation pattern present at $\sim 150\text{cycles}/\text{image}$. (c) Time-series data of the signals before (black) and after signal subtraction (yellow). (d) The recovered in-focus portion of the signal using the Hilbert transform as compared to the original signal.

To simulate some error, we couple about 2% modulation into the out-of-focus light to visualize how small errors, such as imperfect focusing, will impact the final reconstruction. At the extremes, 100% of the modulation pattern would exist in the out-of-focus signal, making it impossible to decouple the two signals, thus, producing no sectioning at all, whereas as an error of 0% would produce a perfect reconstruction, with infinitely small sectioning resolution. Here, the small error is used to ensure that our simulation accurately models these imperfections. The 2% value is intended to show a small amount of error expected in real-world data, while still demonstrating the accuracy of the simulation/technique. The in-focus and out-of-focus signals are then added together to construct the simulated signal, shown in black. The construction of the signal is thus

The modulated in-focus portion of the signal is recovered by subtracting the combined in-focus and out-of-focus signals, without modulation:

The unmodulated portion, $R(x)$, can be independently measured without projecting a modulation pattern [simulated here, $R(x)=0.5*[R_{\mathrm{OF}}(x)+R_{\mathrm{IF}}(x)]$] or estimated by filtering out the modulation pattern, as we will do later in this paper. In Eq. (4), we are left with the modulated in-focus portion of light. The in-focus section can then be reconstructed using the “Hilbert” function to demodulate the signal as follows:

We have now successfully decoupled the in-focus light from the background. To show the accuracy of the technique, Fig. 2(d) shows an estimation of the in-focus light compared to the original signal. We note that there is some loss in accuracy due to modulation from some of the out-of-focus regions. This type of error will manifest itself as a loss of contrast, particularly in regions of high-spatial frequency. In the above simulation, $R(x)$ was known and subtracted from the modulated signal. This can be achieved by measuring the signal twice: one measurement with modulation and one without. However, similar results can also be achieved with only one measurement, as will be shown in the next section. Rather than trying to measure the unmodulated signal, it is estimated by filtering out the modulated portion of the image. In both scenarios, this unmodulated portion is then subtracted from the modulated measurement.

2.4.

Structured Illumination—Single Image Approach

By projecting a high-frequency modulation pattern under planar illumination, at a plane conjugate to the CCD, all components of this plane are spatially shifted outside the broadband signal. This scattered out-of-focus light remains centered in the baseband of the spatial frequency domain due to the pattern being blurred outside of the focal plane. In Sec. 3.2, a method of decoupling the in-focus and out-of-focus signals, which can readily be adapted to this type of 2-D application, is described. Our algorithm, outlined in Fig. 3, describes the process required to separate the modulated and unmodulated images, producing a high-contrast sectioned image. In the interest of developing an optical sectioning system with the least number of samples required, we have extended this work to function using only a single sample. Using a single sample will prove to be extremely useful in real-time biological imaging. Specifically, motion artifacts can create constraints on any method that requires multiple samples of data for reconstruction. As long as there is minimal movement within the integration time of the frame, a single sample will suffice in producing high-quality sectioned images in vivo. Also, in comparison to the three-phase SIM, there is no need to finely align multiple phases of data, making our method particularly robust at depth.

Fig. 3

Flow chart outlining the single-image sectioning process.

In the 1-D simulation, we directly measure the broadband signal without modulation. With this new method, only a single image, which contains both the in-focus and out-of-focus components and the modulation pattern, is measured. Using a combination of a low-pass filter and a notch filter, which are defined based on the frequency of the modulation pattern, the low frequency information from the out-of-focus light is estimated, as shown in Fig. 4. It should be noted that, for these experiments, a generic Gaussian filter is applied to eliminate the modulation pattern. However, the selection of this filter will differ depending on the frequency and angle of the modulation pattern applied, i.e., the region of data that will be filtered within the frequency domain. For our experimental data, the frequency ($40\text{cycles}/\text{image}$) and angle (30 deg) are known ahead of time and can be tuned to filter out the first orders of the modulation pattern. From multiple experimental datasets, we have empirically selected a Gaussian filter with a full width, half max, of 10 cycles, providing the best sectioning results. This estimated unmodulated image can then be subtracted from the image, leaving only the modulated in-focus signal. As we described in Sec. 3.2, we apply the Hilbert transform using the spiral function technique to remove the modulation pattern from the in-focus signal. The result is a sectioned image that has been demodulated. Depending on how well $R(x)$ is estimated, the success of the demodulation will vary; for instance, we may find some residual banding if the modulation pattern cannot be completely filtered out. However, it is shown that our method provides good sectioning over a wide variety of situations.

Fig. 4

(a) Frequency domain of a modulated image with AC pattern visible along the horizontal axis, scale bar is in dB. (b) Frequency domain of image after a notch and low-pass filter has been applied to estimate the DC signal.

2.5.

Experimental Setup

A 635-nm LED with a line width of 17 nm was used as the light source. The patterns were projected onto the specimen using a TI Lightcrafter DMD. The images were then captured using an Allied Vision Guppy PRO F-125 CCD camera. The objective has a nominal magnification of $10\times$ in air with a NA of 0.25. When combined with the tube lens, the system has an overall magnification of $9\times$. The configuration of the microscope is outlined in Fig. 1 and is used for all data captured throughout this paper.

3.1.

Single Image Results

We start by sectioning a business card, which contains multiple layers. To compare to the three-phase SIM, three total images are taken with phases of 0, 120, and 240 deg. Each image encompasses a $402\times 538\mu \mathrm{m}$ region, with a square wave modulation pattern of frequency $\sim 40\text{cycles}/\text{image}$ projected at an arbitrary angle of 30 deg. The images are processed using Eq. (1) to produce the demodulated AC signal. Then, only a single phase image is processed using the Hilbert technique developed above.

Figure 5 shows a comparison of the wide-field (a) versus the three image sectioning (b) and single-image sectioning (c). It is clear that both methods remove a great deal of light from the out-of-focus regions, isolating a single plane-of-interest. Additionally, the contrast is greatly improved in the remaining regions. There are some small artifacts in the single-image sectioning. Specifically, some banding remains from an imperfect estimation of the DC image. Additionally, there are some edge effects from the application of the Hilbert function. Taking the difference between the three-phase SIM and single-phase SIM, there are only small changes between images ($\sigma =0.14$, where the images are scaled from 0 to 1). However, on the whole, we produce a high-quality sectioned image, providing good isolation of a single plane, quite comparable to the three phase reconstruction. Furthermore, we have been able to produce this image with one-third the number of samples as required by traditional SIM, decreasing the measurement time from 150 to 50 ms. In this experiment, Fig. 6 quantifies the difference between the actual and estimated DC images, with the vast majority of pixels within a few percent ($\sigma =0.01$).

Fig. 5

Image of a business card at depth. (a) The widefield image with no sectioning is shown. (b) The three-phase SIM reconstruction is shown. (c) The single-image demodulation is shown.

Fig. 6

DC images shown of the business card from Fig. 5, representing the out-of-focus portions of the card. Both (a) and (b) are scaled from 0 to 1. (a) The actual DC image taken without modulation pattern present. (b) The estimated DC image by filtering as described above. (c) Difference between the two images. The color bar on (c) represents the normalized difference between the two.

3.2.

Image Results from Multiple Depths

By building a z-stack from 25 individually processed images, a full map has been built of each specimen across a depth of $25\mu \mathrm{m}$ at $1\text{-}\mu \mathrm{m}$ increments, as shown in Fig. 7. The same modulation frequency is used across all depths, where three phases are taken at each depth. These data can be used to verify the axial resolution and isolation of planes through the construction of a high depth-of-field (DoF) image. To show the extended DOF, we use a maximum intensity projection, which combines the pixels with the highest amplitude from each depth into a single image. This projection removes all scattered light and shows the entire specimen in focus along the $z$-axis, helping to visualize the two-dimensional structure without any occlusion from the layer above or below the focal plane.

For this data, we use the three-phase SIM as our ground truth and qualitatively compare it to our single image method. We should expect to see individual fibers existing on discrete planes, as well as continuity along each of the fibers. Reviewing Fig. 7, we see how well the single-image sectioning is able to section the image. It is clear that the fibers are well isolated from one another, on par with typical SIM methods. Figure 8 shows the structure of the fibers. Here, even the smaller fibers are retained providing good resolution and reconstruction of the card. When comparing the two height maps to one another, we find that all layers sectioned using the single-image technique are within $\pm 3\mu \mathrm{m}$. Similar to the single image above, the large DoF image matches within a few percent ($\sigma =0.14$, where the image amplitude is scaled from 0 to 1).

Fig. 7

Height map of a business card from 0 to $25\mu \mathrm{m}$ at $1\text{-}\mu \mathrm{m}$ increments. (a) Three-phase SIM and (b) single-image SIM.

Fig. 8

Maximum intensity projection of a business card from 0 to $25\mu \mathrm{m}$ at $1\text{-}\mu \mathrm{m}$ increments. The intensity for each image is normalized and mapped from 0 to 1. (a) Three-phase SIM and (b) single-image SIM.

3.3.

Sectioning Versus Depth

Additionally, we want to ensure that this method works at depth, even when the plane of interest is occluded by scattering layers above and below the plane of interest. To model skin imaging, we have built a phantom, as developed and demonstrated by Glazowski and Zavislan,¹⁶ for testing sectioning in highly scattering media. This phantom consists of a 1951 Air Force resolution chart at a depth of $30\mu \mathrm{m}$ below a piece of highly scattering ground glass. The space between the target and the glass has been impregnated with a gel ($n=1.33$) to simulate water. The objective lens has been increased to $20\times$ $\mathrm{NA}=0.4$; otherwise, all other components of the optical setup are the same, as described in Sec. 3.5.

As demonstrated in Fig. 9, the resolution chart was imaged using both the three-phase and single-phase SIM. In the widefield image (d), we see the structure present from the resolution chart below, but the contrast is low due to scattered light from the ground glass above. Reviewing the three-phase SIM, we notice that the sectioning is poor (e). This is likely due to small changes in the phases of the modulation patterns as it passes through multiple surfaces before reaching the focal plane.^7,8 Because three images are used, sectioning is highly dependent on how well each of the three phases overlaps at the focal plane. Any changes in phase are liable to produce artifacts at the intended target. This can be slightly mitigated using additional phases or multiple random patterns, but at the cost of additional samples.^9,17 For this experiment, the phase changes are likely due to differences in the optical path length from the rough surface of the ground glass above the target. However, with in vivo imaging, phase misalignment may be exacerbated by a host of factors, such as vibration, diffusion, and small changes in index of refraction, all of which can be overcome using our single-phase sectioning system.

Fig. 9

1951 Air Force resolution target $30\text{-}\mu \mathrm{m}$ below skin phantom. The chart displays group 6, elements 2 to 6, which encompasses resolutions of $\sim 7.0$ to $\sim 4.4\mu \mathrm{m}$. The top row shows the single-phase reconstruction on each of three phases, (a) 0 deg, (b) 120 deg, and (c) 240 deg, demonstrating the effectiveness regardless of modulation phase. (d) Widefield image with strong cluttering from out-of-focus regions. (e) Three-phase sectioning exhibiting major distortions due to local phase shifts from turbid media. (f) Average of all three single phase images (a)–(c), providing additional noise reduction.

As expected, the single-phase sectioning provides much better resolution as it is much less sensitive to phase. Because the spiral function used for the Hilbert transform does not require a-priori knowledge, small deviations in the frequency or angle of the pattern do not negatively impact the sectioning. This results in a much more robust method of sectioning at depth. Figures 9(a)–9(c) show good contrast at the target, isolating a single plane from the scattered light. We show that the processing works regardless of the phase, as long as the modulation pattern is present [(a) 0 deg; (b) 120 deg; (c) 240 deg]. By taking the mean of all three phases after they have been individually processed, we see that there is a further improvement in noise reduction and contrast [Fig. 9(f)]. Figure 10 shows the contrast improvement, by taking a cross-section of the image along group 6, element 4, with a line width of $5.52\mu \mathrm{m}$ [outlined in red in Fig. 9(d)]. There is a clear difference in the contrast of the signal as the scattered light from above the target has been removed. Note that there is a small trend along the resolution chart; this is due to slight nonuniformities in the lighting of our sample and is not related to the target itself. By calculating the relative contrast (max–min) for each cycle, there is a $2\times$ improvement in the vertical direction and a $5\times$ improvement in the horizontal direction. The three-phase sectioning has been left out as the bars are no longer resolvable at this depth.

Fig. 10

From Fig. 9, group 6, element 4, outlined in red, cross-sections along the (a) vertical bars and (b) horizontal bars are compared. The red line is taken from Fig. 9(d) and compared to the blue line taken from 9(f). This shows the contrast improvement and removal of DC signal using the single-phase sectioning.

A measurement of the resolution chart was also taken at the surface ($\text{depth}=0\mu \mathrm{m}$) to verify that the system is diffraction limited ($\text{Resolution}=1.22\lambda /\mathrm{NA}$). We find that the minimum resolution at the surface is $\sim 2.0\mu \mathrm{m}$, which agrees with our expectations. As we attempt to image the chart at depth, there will be considerable resolution loss due to the scattering of light above the target. In Fig. 9(d), it can be seen that, without any sectioning, the resolution varies quite a bit but, at best, has a resolution of about $5.5\mu \mathrm{m}$ (group 6, element 4, highlighted in red). Reviewing Fig. 9(f), it can be seen that the next element down (group 6, element 3) is now visible, having increased the lateral to $\sim 4.9\mu \mathrm{m}$. Therefore, at a depth of $30\mu \mathrm{m}$, we find there to be a $2.75x$ loss of resolution. However, after sectioning, this can be modestly improved.

3.4.

Three-Phase Structured Illumination Microscopy at Depth

To better understand why the three-phase sectioning suffers within a turbid media, we have imaged the square pattern on a mirror at the surface ($d=0\mu \mathrm{m}$) and within our skin phantom ($d=10\mu \mathrm{m}$; Fig. 11). Taking the cross-sections, how the relative phase and frequency begin to diverge can be seen. The amplitude from the surface measurements (left) extends from 0.1 to $1\times$, the total dynamic range of the camera, whereas at depth, the signal contrast is cut in half. As a result, the sectioning contrast of traditional SIM is greatly degraded, achieving a maximum amplitude of 0.5. Additionally, the phase of each signal is shifted slightly, creating a low frequency modulation, which does not actually exist on the mirror. As we attempt to image the pattern beyond $10\mu \mathrm{m}$, these errors are greatly exacerbated, rendering the final three-phase sectioned image nearly useless. The alignment of these patterns is critical for the success of the three-phase SIM. However, our single-phase technique is robust to small changes in frequency and phase and, therefore, produces high-quality sectioning in these scenarios.

Fig. 11

Cross-sections of a modulation pattern projected on a mirror at (a) $d=0\mu \mathrm{m}$ and (b) $d=10\mu \mathrm{m}$. All three phases are plotted as well as the sectioned signal using Eq. (1).

3.5.

Sectioning Biological Samples

To further validate the method, sectioning was applied to a number of biological samples, including onion cells and in vivo to the skin of a human arm. Again, a square modulation pattern of $40\text{cycles}/\text{image}$ is projected horizontally to modulate the image. Two layers of onion cells were placed directly on top of one another. The system was focused at the top layer of cells, as shown in Fig. 12. The widefield image [Fig. 12(a)] shows multiple cells aligned diagonally across the image. Scattered light from the cells below the surface can be seen in the image. The single-image sectioning [Fig. 12(c)] works in removing the scattered light from below. However, there are strong banding artifacts due to an imperfect demodulation. This imperfect demodulation comes as a result of data loss, where the patterned light goes to zero. This banding is exacerbated by choosing a low frequency modulation pattern. A higher frequency pattern would produce more uniform sectioning; however, it would decrease our sectioning depth. In clinical application, adjusting the frequency of the modulation pattern relative to the imaging depth might be worth considering. Adding in the other two phases ameliorates this issue by filling gaps of data loss. By processing the other two phases and adding them together [Fig. 12(d)], we see that the results are nearly identical to the three-phase sectioning [Fig. 12(b)]. In comparison to the traditional three-phase SIM, the results show that similar sectioning and resolution can be achieved through the methods developed above.

Fig. 12

Two layers of onion cells. This figure shows the top layer ($\text{depth}=0\mu \mathrm{m}$). Scale bar is $20\mu \mathrm{m}$. (a) Widefield image from adding all three phases. (b) Traditional three-phase sectioning. (c) Single-phase sectioning using Hilbert demodulation. (d) Three images sectioned using Hilbert demodulation then added together.

One particularly strong advantage of our method is the robustness of sectioning within a turbid media. Figure 13 shows the layer of onion cells about $5\mu \mathrm{m}$ below the cells, as shown in Fig. 12. The scattered light from the cells above can be seen clearly in the widefield image (A). Again, the single-image sectioning (C) produces high-quality sectioning with only minimal artifacts. When adding all three phases together (D), the advantages over the traditional SIM method (B) can be seen; first, many artifacts in traditional SIM can be seen around the edges, reducing total contrast within the image. Additionally, there is some residual modulation pattern within the image. After the sectioning is applied, the two layers are completely separated from one another. Given that onion cells have a thickness of $\sim 2$ to $3\mu \mathrm{m}$, this experiment demonstrates that our axial resolution is $\sim 3\mu \mathrm{m}$.

Fig. 13

Two layers of onion cells, this figure shows the bottom layer ($\text{depth}=5\mu \mathrm{m}$). Scale bar is $20\mu \mathrm{m}$. (a) Widefield image from adding all three phases. (b) Traditional three-phase sectioning. (c) Single-phase sectioning using Hilbert demodulation. (d) Three images sectioned using Hilbert demodulation then added together.

Finally, we apply our sectioning method directly to in vivo sectioning of human skin. The results are shown in Fig. 14. The widefield image shows very little detail as there is a great deal of scattering. Given the frame rate (21 Hz) of the camera, there is motion from frame to frame. The motion results from many factors including muscle twitches beneath the skin, as well as larger movements due to breathing and heartbeats. Due to all of this motion, the traditional three-phase imaging (B) is completely distorted. The differencing scheme relies on only the modulation changing phase but does not account for motion within the specimen itself. With the specimen changing location over the three frames, the result does not contain useful information. Reviewing Fig. 14(b), only highly reflective portions of the image are seen in the processed image. However, there is no indication of structure, such as nuclei, which would present themselves as dark circles within the tissue. This is an important detail as even minor changes in position of the pattern can result in major degradation of the sectioned image.

Fig. 14

Human skin cells imaged at a depth of $\sim 30\mu \mathrm{m}$. Scale bar is $20\mu \mathrm{m}$. All images are normalized and scaled from 0 to 1. (a) Widefield image from adding all three phases. (b) Traditional three-phase sectioning, greatly distorted by motion artifacts. (c) Single-phase sectioning using Hilbert demodulation. (d) Three images sectioned using Hilbert demodulation then added together. Arrows are superimposed on (c) and (d) to show the nucleus of the cells within the granular layer of the skin.

Applying the single-image processing (C) provides good sectioning and contrast of the cells below the surface. Here, we can see the granular layer of the skin, where the dark spots (shown by the superimposed arrows) point out the nucleus of the cells. The depth, dimension, and relative sizes appear to agree with those shown by Rajadhyaksha et al.⁴ Furthermore, the three phases can be added together (D) to increase the contrast of the image. For images that contain motion, the contrast is improved. As compared to the differencing method (B), there are only minor motion artifacts; otherwise, the method is much more robust to small changes in the pattern position.

To show the motion frame to frame, Fig. 15 plots an RGB map of the three images used in Fig. 14(d). At the bottom left edges of the structures within the image, the color is predominately red. The red represents the first frame in the stack and shows that only some of the cells are in this location for a moment. By the third frame (shown in blue), the cells have shifted toward the upper right corner. It can be seen that within those three frames, the subject moved a few microns diagonally, resulting in an imperfect reconstruction of the multiframe sectioning techniques.

Fig. 15

Three consecutive frames sectioned and plotted as an RGB image. Motion can be seen along the axis of the arrow superimposed on the image.