1.

Introduction

Digital holographic techniques^{1, 2, 3} have been applied for three-dimensional (3D) visualization and recognition of objects. ^{4, 5, 6, 7, 8, 9, 10, 11} Recently, 3D holographic-based microscopy has been applied to the 3D recognition of micro-objects.^{19, 20, 21} Single-exposure on-line (SEOL) digital holography¹⁰ can provide real-time sensing and recognition of moving, growing, and reproducing microorganisms. One of the advantages of digital holography is that it allows us to digitally produce an autofocused volumetric 3D image of microorganisms or micro-objects from a single hologram.

In particular, bacteria and other microorganisms have been currently analyzed by the traditional methods involving biochemical processing, which is not real time, may be very time-consuming, and requires special skills. Real-time sensing and recognition of bacteria or biological cells has many benefits. For example, it could be applied in biology, defense, and diagnosis of diseases.

Most of the research in the recognition and classification of microorganisms has been based on their colors, two-dimensional (2D) intensity images, shapes, aggregation size, and reaction time. ^{12, 13, 14, 15, 16, 17, 18} Recently, we addressed two approaches for real-time 3D sensing and recognition of biological microorganisms.^{19, 20, 21} One is 3D morphology-based recognition, and the other is shape-independent 3D recognition. In the first approach, a series of image recognition techniques are used to analyze 3D geometrical shapes of microorganisms and consists of magnitude and phase distributions including segmentation, feature extraction, and graph matching. For the second approach, a number of sample segment features are randomly extracted from a reconstructed 3D image of a microorganism. These samples are processed using cost function and statistical estimation and inference algorithms. A hypothesis test for the comparing of the parameters between two populations is then performed for a statistical decision on the basis of sampling distribution information.

In this paper, we present real-time volumetric 3D sensing and recognition of biological microorganisms using multivariate statistical estimation and inference algorithms based on digital holographic microscopy. The Fresnel transformation of biological microorganisms is recorded by SEOL holographic microscopy interfaced by computer. The complex-magnitude 3D image of the biological micro-object is reconstructed numerically. A number of serial section images of biological microorganisms are produced from the digital hologram. These section images can be digitally reconstructed along the longitudinal direction by inverse Fresnel transformation.

For the volumetric 3D recognition, the multivariate probability density function (MPDF) is calculated²² from the sample segment features of the volume image, and multivariate statistical tests are performed to compare the parameters of populations. Multivariate statistical analysis enables us to explore the joint relationship among multiple variables from the volumetric 3D image, determine the contribution of each variable in the presence of the other variables, and perform the number of possible hypotheses because of many statistical parameters,²² which can improve the discrimination performance between two data sets. We compare classification of microorganisms using a volumetric 3D image with one section image. The multivariate statistical approach is robust to changes in the shapes of the biological microorganisms.

Frameworks for the proposed approaches are presented in Fig. 1 . The following sections describe various stages of the proposed approach for the volumetric 3D sensing and recognition of biological microorganisms by using SEOL digital holography. In Sec. 2, the design procedure for volumetric 3D sensing and recognition of microorganisms is described. Statistical multivariate analysis and maximum likelihood estimation for the multivariate statistical inference are explained in Sec. 3. Preliminary experimental results using SEOL holographic microscopy and our conclusion are given in Sec. 4 and Sec. 5, respectively.

Fig. 1

Frameworks for volumetric 3D sensing and recognition of biological microorganisms.

2.

Proposed Design Procedure for Volumetric 3D Recognition of Microorganisms

In the following, we describe the design procedure for a volumetric 3D sensing and recognition system of microorganisms using digital holography.^{1, 2, 3} First, we record SEOL holograms of the microorganisms of interest, as shown in Fig. 2 . We then reconstruct the corresponding biological microorganism as a volume image that consists of $p$ section images from a SEOL digital hologram in the Fresnel domain (see Fig. 3 ). The intensity interference pattern of the biological microorganisms at the CCD plane can be represented as follows:

Fig. 2

Experimental setup for recording the SEOL digital hologram of a biological microorganism.

Fig. 3

Design procedure for real-time volumetric 3D sensing and recognition of biological microorganisms. The random sample vector is extracted in the reconstructed volume image from the SEOL digital hologram.

The 3D reconstruction of the original biological microorganism is performed computationally by using the following Fresnel propagation algorithm from the SEOL digital hologram:

Real-time sensing and recognition can be achieved by SEOL holography, which requires a single exposure for 3D recording and development in specialized hardware or parallel processing for the fast Fourier transformation (FFT), which is $O\left(N\right)=N\log 2N$ , where $N$ is the total number of pixels in the hologram.

For volumetric 3D recognition, we randomly select $n$ test pixel points in the segmented section image and extract $p\times n$ pixel values of each section image at the test pixel points. The image segmentation can be automatically performed by the following actions:

We repeat the preceding steps $m$ times to reduce statistical data fluctuation and obtain averaged statistical data and produce $m$ sample vectors consisting of $p\times n$ pixel values, where we denote the pixel values in the $m$ ’th sample vector as ${\mathbf{x}}_n^p$ .

We can extend the sample vectors with different specimens of the same microorganism. This process can be automatically performed on the computer after using the image segmentation algorithm. This idea can be efficiently used for sensing and classification of biological microorganisms such as bacteria and biological cells that are similar, unicellular, and branched in shape and profile.

To test our volumetric 3D sensing system, we similarly produce $m^{\prime }$ sample vectors corresponding to an unknown input sample to be classified. The vectors consist of $p^{\prime }\times n^{\prime }$ pixel values, where we denote the pixel values in the $m$ ’th sample vector as ${\mathbf{y}}_{n^{\prime }}^{p^{\prime }}$ .

3.

Statistical Multivariate Analysis and Maximum Likelihood Estimation

In the following, we apply multivariate statistical estimation and inference^{22, 23, 24} to the real-time volumetric 3D recognition system. From SEOL holographic microscopy, a number of serial section images of biological microorganisms can be produced. These section images are digitally reconstructed along the longitudinal direction by the Fresnel transformation and contain information about how the wave is diffracted. Therefore, we can obtain a number of diffraction patterns of a microorganism at different depths that provide us with information for recognizing biological microorganisms.

Using the histogram analysis of these section images, we assume that the distribution of the pixel values of each section image has approximately normal distribution.¹⁸ For volumetric 3D sensing and classification of biological microorganisms, multivariate normal distribution can be mathematically modeled on all the section images. The parameters of the joint probability density function are calculated by using the method of maximum likelihood estimation. One of the advantages of a multivariate statistical method is that the number of possible hypotheses is very large because of the large number of parameters.

Let $x^1\dots ,x^p$ be a $p$ -variate normal random variable with joint PDF given by:

For volumetric 3D sensing of biological microorganisms, we conduct multivariate statistical tests for the equality of the $p$ -dimensional mean vectors and covariance matrices [ $\frac{1}{2}p(p+1)$ parameters] of two populations. First, to compare the $p$ -dimensional mean vectors, we assume that two independent random samples ${\mathbf{x}}_1,\dots ,{\mathbf{x}}_{n_1}$ and ${\mathbf{y}}_1,\dots ,{\mathbf{y}}_{n_2}$ are drawn from $N_p({\mathbf{\mu }}_{\mathbf{x}},{\mathbf{k}}_{\mathbf{x}})$ and $N_p({\mathbf{\mu }}_{\mathbf{y}},{\mathbf{k}}_{\mathbf{y}})$ , which are statistical distributions of reference microorganisms and unknown input microorganisms, respectively. All three statistical parameters are unknown, and ${\mathbf{k}}_{\mathbf{x}}={\mathbf{k}}_{\mathbf{y}}=\mathbf{k}$ , where we use sample data accepted with statistical testing for equality of two covariance matrices. The likelihood ratio approach can be used for the statistical test to check the equality of the $p$ -dimensional mean vectors and covariance matrices between reference and unknown input populations. The likelihood function that is the product of the joint density of observations is defined^{22, 23} as:

The assumption of the equality of $p$ -dimensional mean vectors can be stated as a null hypothesis $H_0$ $({\mathbf{\mu }}_{\mathbf{x}}={\mathbf{\mu }}_{\mathbf{y}}=\mathbf{\mu })$ to be tested. Under $H_0$ , the likelihood function $L({\mathbf{\mu }}_{\mathbf{x}},{\mathbf{\mu }}_{\mathbf{y}},\mathbf{k})$ can be maximized by:

To compare covariance matrices, the assumption of equality of covariance matrices can be stated as a null hypothesis $H_0$ $({\mathbf{k}}_{\mathbf{x}}={\mathbf{k}}_{\mathbf{y}}=\mathbf{k})$ to be tested. Similarly, under $H_0$ , the likelihood function $L({\mathbf{\mu }}_{\mathbf{x}},{\mathbf{\mu }}_{\mathbf{y}},\mathbf{k})$ can be maximized by:

For the statistical decision about the equality of the covariance matrices between the two populations on the basis of the sampling distribution information, we can reject the null hypothesis $H_0$ $({\mathbf{k}}_{\mathbf{x}}={\mathbf{k}}_{\mathbf{y}}=\mathbf{k})$ if the statistics $M^{\prime }⩾{\chi }_{0.5p(p+1)}^2$ with critical values.

4.

Experimental Results

We show experimental results of the image formation of two biological microorganisms (oscillatoria cyanobacteria and diatom alga) using SEOL digital holography and identify them by their morphological characteristics and diffracted wave patterns with multivariate statistical estimation and inference.

4.1.

Volume Image of Microorganisms from SEOL Digital Hologram

In the following subsection, we present the visualization of microorganisms using SEOL digital holography. In the experiments, the microorganisms were around several $\mathrm{\mu }\mathrm{m}$ in size. They were recorded using a SEOL digital hologram with a CCD array of $2048\times 2048$ pixels and a pixel size of $9\mathrm{\mu }\mathrm{m}\times 9\mathrm{\mu }\mathrm{m}$ , where the specimen was sandwiched between two transparent cover slips.

Figure 4a and 4b shows the oscillatoria and diatom images reconstructed at a distance of $270\mathrm{mm}$ by the SEOL digital hologram, which was used to test our recognition system. Figure 5 shows the section images of diatom reconstructed from the single digital hologram at distances $d=285$ , 315, 345, and $375\mathrm{mm}$ , respectively. The volumetric 3D image of biological microorganisms can be obtained with these section images.

Fig. 4

The magnified intensity images at the distance $d=270\mathrm{mm}$ of microorganisms by using a $100\times$ microscope objective: (a) oscillatoria cyanobacteria and (b) diatom alga.

Fig. 5

Section images of diatom alga at distances: (a) $d=285\mathrm{mm}$ , (b) $d=315\mathrm{mm}$ , (c) $d=345\mathrm{mm}$ , and (d) $d=375\mathrm{mm}$ .

4.2.

Volumetric 3D Recognition by Multivariate Statistical Estimation and Inference

In the following subsection, we evaluate the performance of our real-time volumetric 3D recognition system using SEOL digital holography. First, we obtained eight section images at different planes along the longitudinal direction from the single SEOL hologram, where the section images were reconstructed at intervals of $5\mathrm{mm}$ from the distance $d=270\mathrm{mm}$ to $305\mathrm{mm}$ . The 16-variate random sample vector $[{\mathbf{x}}^1,\dots ,{\mathbf{x}}^{16}]$ was then selected from the real and imaginary parts in the eight section images; for example, ${\mathbf{x}}^1$ and ${\mathbf{x}}^2$ are random samples from the real part and the imaginary part in the first section image, respectively. Finally, we produced 100 trial sample vectors from the oscillatoria volume image as the reference data sets to reduce a statistical fluctuation, where we changed the size of trial sample vectors from 10 to 200 and removed the background of the reconstructed image by image segmentation²⁵ for more efficient recognition.

Similarly, we produced 100 trial sample vectors from the oscillatoria volume image as the true-class inputs and in the diatom volume image as the false-class inputs, respectively. In order to more practically measure the similarity or dissimilarity between two data sets (reference and input), we randomly selected the test pixel points from the same specimen as a reference for true-class 1 and from a different specimen of the same microorganism for true-class 2.

Figure 6 shows the experimental results of the Box M-test to measure the equality of the covariance matrices between the reference and the input sample vectors versus the sample size, where the 16-variate random sample vector was used and the percentage of correct matched sample vectors was calculated. For a statistical decision, we rejected the null hypothesis $H_0$ $({\mathbf{k}}_{\mathbf{x}}={\mathbf{k}}_{\mathbf{y}}=\mathbf{k})$ in the case that the statistic $M^{\prime }$ defined in Eq. 13 was greater than ${\chi }_{1∕\mathrm{10,000}136}^2$ , which is the critical value of chi-square ${\chi }^2$ distribution at level of significance 1/10,000. It is noted that the percentages of the correct matched sample vector by the decision rule for true-class 1 and 2 were around 100% and above 80%, respectively, while for the false-class, the percentage rapidly decreased as a sample size increases. True-class 2 had a slightly different statistical distribution compared to the reference. Therefore, the percentage of correctly matched sample patches went down slightly as the sample size increased.

Fig. 6

Experimental results of the Box M-test for the equality of the covariance matrices between the reference and input sample vectors versus the sample size. Sixteen-variate random sample vectors from the volumetric 3D image were tested.

To compare the classification of microorganisms using a volumetric 3D image with one section image, we similarly conducted the Box M-test using the bivariate data set that consists of the real and imaginary parts in the well-focused one-section image. Figure 7 shows the experimental results of the Box M-test to measure the equality of the covariance matrices between the reference and the input sample vectors versus the sample size, where a bivariate random sample vector was used and the percentage of correct matched sample vectors was calculated. For a statistical decision, we rejected the null hypothesis $H_0$ $({\mathbf{k}}_{\mathbf{x}}={\mathbf{k}}_{\mathbf{y}}=\mathbf{k})$ in the case that the statistic $M^{\prime }$ defined in Eq. 13 was greater than ${\chi }_{1∕\mathrm{10,000}3}^2$ , which is the critical value of chi-square ${\chi }^2$ distribution at level of significance 1/10,000. It is noted that the percentages of the correct matched sample data by the decision rule for true-class 1 and 2 were around 100%, while for the false-class, the percentages were above 60%. It is shown in Fig. 6 and Fig. 7 that the statistical discrimination performance can be improved by the volumetric 3D image.

Fig. 7

Experimental results of the Box M-test for the equality of the covariance matrices between the reference and input sample vectors versus the sample size. Two-variate random sample vectors from one section image (well-focused image) were tested.

The Box M-test to measure the equality of covariance matrices between the true-class and the false-class of the microorganism by using 16 variables gave rise to much better discrimination than by using 2 variables. We also measured the intercorrelations among each variable²² by:

Eq. 14

$$r={\left[\frac{\left({\sum }_{i=1}^p{\lambda }_i^2\right)-p}{p(p-1)}\right]}^{1∕2},$$

where $p$ is the number of variables and ${\lambda }_1,\dots ,{\lambda }_p$ are all the eigenvalues of a correlation matrix. If the $r$ in Eq. 14 is equal to 1, all of the variables are perfectly correlated. The experimental results are shown in Fig. 8 . It is noted that the measured value was around 0.3 for the oscillatoria bacteria and around 0.7 for the diatom alga. This indicates that there is a considerable intercorrelation among each variable in the multiple section images.

Fig. 8

Experimental results of measuring intercorrelation among a set of variables versus the sample size.

Figure 9 shows the results of the Hotelling ${\mathrm{T}}^2$ -test for the difference of the location vectors between the reference and input sample vectors, where we rejected the null hypothesis $H_0$ $({\mathbf{\mu }}_{\mathbf{x}}={\mathbf{\mu }}_{\mathbf{y}})$ in the case that the statistic Hotelling ${\mathrm{T}}^2$ defined in Eq. 8 was larger than $T_{161∕\mathrm{10,000},2n-2}^2$ , which is the critical value of the Hotelling ${\mathrm{T}}^2$ distribution at level of significance 1/10,000. We tested only the sample vectors accepted by the Box M-test for equality of two covariance matrices, since the assumption of ${\mathbf{k}}_{\mathbf{x}}={\mathbf{k}}_{\mathbf{y}}=\mathbf{k}$ is necessary for the Hotelling ${\mathrm{T}}^2$ -test. It is noted that the percentages of the correct matched sample vector by the decision rule for true-class 1 and 2 were around 100% and above 80%, while for the false-class, the percentage rapidly decreased as the sample size increased.

Fig. 9

Experimental results of the Hotelling ${\mathrm{T}}^2$ -test for the equality of the location vectors between the reference and input sample vectors versus the sample size. Sixteen-variate random sample vectors from the volumetric 3D image were tested.

We also conducted the Hotelling ${\mathrm{T}}^2$ -test using bivariate random sample data to compare with the results of the multivariate statistical test. Figure 10 shows the results of the Hotelling ${\mathrm{T}}^2$ -test for the difference of the mean vectors between the reference and input sample data, where we rejected the null hypothesis $H_0$ $({\mathbf{\mu }}_{\mathbf{x}}={\mathbf{\mu }}_{\mathbf{y}})$ in the case that the statistic Hotelling ${\mathrm{T}}^2$ defined in Eq. 8 was larger than $T_{21∕\mathrm{10,000},2n-2}^2$ , which is the critical value of the Hotelling ${\mathrm{T}}^2$ distribution at level of significance 1/10,000, and only the sample vectors accepted by the Box M-test were tested. It is noted that the percentages of the correct matched sample data by the decision rule for true-class 1 and 2 were around 100%, while for the false-class, the percentage rapidly decreased as the sample size increased. These experimental results indicate that there is a larger dissimilarity in the location parameter vectors between the oscillatoria bacteria and the diatom alga. It was more effective to use the Hotelling ${\mathrm{T}}^2$ -test to measure similarity or dissimilarity between the oscillatoria bacteria and the diatom alga.

Fig. 10

Experimental results of the Hotelling ${\mathrm{T}}^2$ -test for the equality of location vectors between the reference and input sample vectors versus the sample size. Two-variate random sample vectors from one section image (well-focused image) were tested.

5.

Conclusion

In conclusion, we have presented real-time volumetric 3D sensing and recognition of biological microorganisms using the multivariate statistical estimation and inference algorithms. We optically obtained the Fresnel diffraction intensity pattern of biological microorganisms by use of digital holographic microscopy and then digitally reconstructed many section images or diffraction patterns of biological microorganisms at different planes along the longitudinal direction from the digital hologram.

For volumetric 3D sensing and recognition, the test pixel points are randomly selected from the segmented section image. The parameters of the joint normal density functions are then calculated from the pixel values of all the section images according to the proposed design procedure. For the statistical decision on the basis of the multivariate sampling distribution information, multivariate hypothesis tests for the equality of the mean vectors and covariance matrices between two populations have been performed. It has been shown in experiments that the discrimination performance to distinguish bacteria or microorganisms can be increased by using a multivariate statistical distribution that consists of a number of section images instead of using only one section image. We have also measured a considerable intercorrelation among each variable from the volumetric 3D image, which must be a substantial factor for the recognition of biological microorganisms. It is obvious that the performance of the discrimination will be enhanced by multiple variables. Digital holography can produce a number of section images along a longitudinal direction. By using only one digital hologram, we can obtain many more variables than we need to classify the microorganisms.

The proposed method is independent of shape in recognizing biological microorganisms. Therefore, this technique will be more suitable to sensing and classification of biological microorganisms that look similar in terms of shape or profile.

We have shown promising experimental results for the recognition and classification of biological microorganisms by merely adopting the multivariate statistical method and inference algorithms, but a more advanced statistical decision rule can be applied.