2.1.

Overview of the Proposed Method

Figure 1 shows an overview of the proposed method. In an initial calibration stage, we capture the WFOV and NFOV images of the calibration pattern at predetermined $Z$ distances from 19 to 46 cm, at 1-cm steps. Then, the matrices of geometric transformation are calculated from the corresponding four points of each captured image according to the $Z$ distances. In the recognition stage, the WFOV and NFOV images of the user are captured simultaneously. The face and eye regions are detected in the WFOV image. Then, the iris region in the WFOV image is detected to estimate the $Z$ distance and mapped to the iris candidate area of the NFOV image. In detail, since the detected iris region in the WFOV image contains the size and position data of the iris region, we can estimate the $Z$ distance based on the iris size and anthropometric data. The mapping of the detected iris region to the iris candidate region of the NFOV image is done using a matrix of geometric transformation (${\mathbf{T}}_{\mathbf{Z}}$: the precalculated matrix corresponding to the estimated $Z$ distance). Then, the iris candidate region is redefined using the position of the corneal specular reflection (SR), and thereby iris localization is performed in the redefined iris region. Finally, iris recognition is conducted based on the iris code generated from the segmented iris region.

Fig. 1

Overall procedure of the proposed method.

2.2.

Proposed Capturing Device

Figure 2 shows the capturing device for acquiring the images of the WFOV and NFOV cameras simultaneously. It consists of a WFOV camera, two NFOV cameras, cold mirrors, and a near-infrared (NIR) illuminator (including 36 NIR light emitting diodes whose wavelength is 880 nm).¹¹ We used three universal serial bus cameras (Webcam C600 by Logitech Corp¹⁶) for the WFOV and two NFOV cameras, which can capture a $1600\times 1200\text{pixel}$ image. The WFOV camera captures the user’s face image, whereas the two NFOV cameras acquire both irises of the user. The NFOV cameras have an additional fixed focus zoom lens to capture magnified images of the iris. In order to reduce the processing time of iris recognition, the size of the captured iris image is reduced to $800\times 600\text{pixels}$. Based on the detected size and position of the iris region, we performed the iris code extraction in the original $1600\times 1200\text{pixel}$ NFOV image. Since a fixed focus zoom lens is used, our device meets the requirement of the resolution of the iris image. The average diameter of the iris captured by the proposed device is 180 to 280 pixels within a $Z$ distance operating range of 25 to 40 cm.¹¹ The $Z$ distance is the distance between the camera lens and the user’s eye. According to ISO/IEC 19794-6, an iris image in which the iris diameter is $>200\text{pixels}$ is regarded as good quality; 150 to 200 pixels is acceptable; and 100 to 150 pixels is marginal.^11,17 Based on this criterion, we can consider our iris image to be of acceptable or good quality in terms of iris diameter. The cold mirror has the characteristic of accepting NIR light while rejecting visible light. Therefore, the user can align his eye to the cold mirror according to his (visible light) reflected eye that he sees in the cold mirror, while his eye image illuminated by NIR light can be obtained by the NFOV camera inside the cold mirror. In order to remove the environmental visible light on the NFOV image, an additional NIR filter is attached to the NFOV cameras.

Fig. 2

The proposed capturing device.

2.3.

Estimating the Iris Region of NFOV Image by Using Geometric Transformation

This section describes a method of estimating the iris region of the NFOV image. The objective of this approach is to make the iris region detected in the WFOV image be mapped to the estimated iris region in the NFOV image. As shown in Fig. 3, we captured the images of the calibration pattern in order to obtain the matrices of geometric transformation in the calibration stage. The captured images are obtained at the predetermined $Z$ distances from 19 to 46 cm at 1-cm steps, and then the matrices are calculated from the corresponding four (manually detected) points of each captured image. These four points of the NFOV image [the four red and four blue points of Figs. 3(a) and 3(b), respectively] are the outermost points of the calibration pattern. As shown in Fig. 3(c), also the corresponding two pairs of four points of the WFOV image (the four red points and the four blue ones) are manually selected. Based on these points, the two transformation matrices (mapping function) between the region in the WFOV image [$(W_{x1},W_{y1})$, $(W_{x2},W_{y2})$, $(W_{x3},W_{y3})$, and $(W_{x4},W_{y4})$] and the region in the NFOV image [$(N_{x1},N_{y1})$, $(N_{x2},N_{y2})$, $(N_{x3},N_{y3})$, and $(N_{x4},N_{y4})$] are obtained at one predetermined $Z$ distance by using geometric transformation, as shown in Fig. 4 and Eq. (1).^18,28 The first matrix relates the region of Fig. 3(a) to the area using the four red points in Fig. 3(c), and the second one relates the region of Fig. 3(b) to the area using the four blue points in Fig. 3(c). The first matrix (${\mathbf{T}}_{\mathbf{Z}}$) is calculated by multiplying matrix $\mathbf{N}$ with the inverse matrix of $\mathbf{W}$, and the eight parameters $a$ to $h$ can be obtained.

Fig. 3

Calibration pattern used for computing geometric transformation matrix between the wide field of view (WFOV) and two narrow field of view (NFOV) image planes. (a) Captured image of the right NFOV camera of Fig. 2. (b) Captured image of the left NFOV camera of Fig. 2. (c) Captured WFOV image.

Fig. 4

Relation between the regions of the WFOV and NFOV image planes.

In the same way, the second matrix (${\mathbf{T}}_{\mathbf{z}}^{\prime }$) can be obtained by using the four blue points of Fig. 3(b) and the corresponding four blue points of Fig. 3(c).

After obtaining the matrices of ${\mathbf{T}}_{\mathbf{z}}$ according to the $Z$ distance in the calibration stage, the user’s iris position $(N_x^{\prime },N_y^{\prime })$ in the left NFOV image can be estimated using matrix ${\mathbf{T}}_{\mathbf{z}}$ and the iris position $(W_x^{\prime },W_y^{\prime })$ of the left eye in the WFOV image, as shown in Eq. (2).

In a similar way, the user’s iris position in the right NFOV image can be estimated using matrix ${\mathbf{T}}_{\mathbf{Z}}^{\prime }$ and the iris position of the right eye in WFOV image.

In general, the detected position of the iris in the WFOV image is not accurate because the image resolution of the eye region in the WFOV is low, as shown in Fig. 5. Therefore, we define the iris region in the WFOV image by the four points, as shown in Figs. 5(b) and 5(c), instead of using the one point of the iris center. Consequently, four points [$(W_{x1}^{\prime },W_{y1}^{\prime })$, $(W_{x2}^{\prime },W_{y2}^{\prime })$, $(W_{x3}^{\prime },W_{y3}^{\prime })$, and $(W_{x4}^{\prime },W_{y4}^{\prime })$] are determined from the left iris of the WFOV image; the other four points [$(W_{x1}^{\prime \prime },W_{y1}^{\prime \prime })$, $(W_{x2}^{\prime \prime },W_{y2}^{\prime \prime })$, $(W_{x3}^{\prime \prime },W_{y3}^{\prime \prime })$, and $(W_{x4}^{\prime \prime },W_{y4}^{\prime \prime })$] can be determined from the right iris in the WFOV image. With these two pairs of four points, matrices ${\mathbf{T}}_{\mathbf{Z}}$ and ${\mathbf{T}}_{\mathbf{Z}}^{\prime }$, and Eq. (2), two pairs of four points [$(N_{x1}^{\prime },N_{y1}^{\prime })$, $(N_{x2}^{\prime },N_{y2}^{\prime })$, $(N_{x3}^{\prime },N_{y3}^{\prime })$, $(N_{x4}^{\prime },N_{y4}^{\prime })$] and [$(N_{x1}^{\prime \prime },N_{y1}^{\prime \prime })$, $(N_{x2}^{\prime \prime },N_{y2}^{\prime \prime })$, $(N_{x3}^{\prime \prime },N_{y3}^{\prime \prime })$, $(N_{x4}^{\prime \prime },N_{y4}^{\prime \prime })$] in the left and right NFOV images are calculated as the iris regions, respectively.

Fig. 5

Result of face and iris detection in WFOV image. (a) The result in the WFOV image. (b) Magnified image showing the detected iris region and four points of the left eye [$(W_{x1}^{\prime },W_{y1}^{\prime })$, $(W_{x2}^{\prime },W_{y2}^{\prime })$, $(W_{x3}^{\prime },W_{y3}^{\prime })$, and $(W_{x4}^{\prime },W_{y4}^{\prime })$]. (c) Magnified image showing the detected iris region and four points of the right eye [$(W_{x1}^{\prime \prime },W_{y1}^{\prime \prime })$, $(W_{x2}^{\prime \prime },W_{y2}^{\prime \prime })$, $(W_{x3}^{\prime \prime },W_{y3}^{\prime \prime })$, and $(W_{x4}^{\prime \prime },W_{y4}^{\prime \prime })$].

Figure 5 shows the result of face and iris detection in the WFOV image. In the recognition stage, two iris regions in the WFOV image [$(W_{x1}^{\prime },W_{y1}^{\prime })$, $(W_{x2}^{\prime },W_{y2}^{\prime })$, $(W_{x3}^{\prime },W_{y3}^{\prime })$, $(W_{x4}^{\prime },W_{y4}^{\prime })$] and [$(W_{x1}^{\prime \prime },W_{y1}^{\prime \prime })$, $(W_{x2}^{\prime \prime },W_{y2}^{\prime \prime })$, $(W_{x3}^{\prime \prime },W_{y3}^{\prime \prime })$, $(W_{x4}^{\prime \prime },W_{y4}^{\prime \prime })$] are determined based on face and eye detection, as shown in Figs. 5(b) and 5(c).

First, we use the Adaboost algorithm to detect the face regions.^11,19 In order to reduce the effects of illumination variations, Retinex filtering is used for illumination normalization in the detected facial regions.^11,20 Then, the rapid eye detection (RED) algorithm is conducted to detect the iris region. It compares the intensity difference between the iris and its neighboring regions, which occurs because the iris region is usually darker.²¹ However, since only the approximate position and size of the iris region can be estimated by the RED algorithm, we perform CED to detect the iris position and size accurately.^27,29 The iris region is determined at the position where the maximum difference between the gray levels of the inner and outer boundary points of the circular template (whose radius is changeable) is obtained.²⁷ Consequently, we can determine the two iris regions of the left and right eyes in the WFOV image [$(W_{x1}^{\prime },W_{y1}^{\prime })$, $(W_{x2}^{\prime },W_{y2}^{\prime })$, $(W_{x3}^{\prime },W_{y3}^{\prime })$, $(W_{x4}^{\prime },W_{y4}^{\prime })$] and [$(W_{x1}^{\prime \prime },W_{y1}^{\prime \prime })$, $(W_{x2}^{\prime \prime },W_{y2}^{\prime \prime })$, $(W_{x3}^{\prime \prime },W_{y3}^{\prime \prime })$, $(W_{x4}^{\prime \prime },W_{y4}^{\prime \prime })$] based on the center position and radius of the iris detected by CED, as shown in Figs. 5(b) and 5(c). Then, in order to map the two iris regions in the WFOV image onto those in the left and right NFOV images, the matrices ${\mathbf{T}}_{\mathbf{Z}}$ and ${\mathbf{T}}_{\mathbf{Z}}^{\prime }$ of Eq. (1) should be selected corresponding to the $Z$ distance. However, it is difficult to estimate the $Z$ distance from the camera to the user using only one WFOV camera.

In previous research studies, Dong et al. used one WFOV camera to detect the face and estimate the $Z$ distance.¹⁴ After detecting the rectangular area of face, they used the width or height of the face area to estimate the $Z$ distance. However, this method has its limitations, because there are individual variations in facial size. To overcome this limitation, Lee et al. used the least squares regression method to enhance the accuracy of the $Z$ distance estimation by updating the model parameters for estimating the $Z$ distance.²³ However, their method requires user-dependent calibration at the initial stage to obtain the user-specific parameters.

Considering the limitations of the previous research studies, we propose a new method for estimating the $Z$ distance between the user’s eye and the camera lens based on the detected iris size in the WFOV image and anthropometric data of the human iris size, which does not require initial user calibration. The detailed explanations are as follows. Figure 6 shows a conventional camera optical model,²³ where $Z$ represents the distance between the camera lens and the object. $V$ is the $\mathrm{Z}$ distance between the image plane and the camera lens. $W$ and $w$ are the object sizes in the scene and image plane, respectively, and $f$ is a camera focal point. According to a conventional camera optical model,²³ we can obtain the relationship among $Z$, and $V$, $W$, $w$ as shown in Eq. (3):

Fig. 6

Camera optical model.²³

Since a lens of fixed focal length is used in our WFOV camera, $V$ is a constant value. Therefore, in the calibration stage, we can calculate $V$ by using the measured object size in the scene ($W$) and that captured in the image plane ($w$) with the measured $Z$ distance ($Z$) based on Eq. (4):

Consequently, with the calculated $V$, we can estimate the $Z$ distance between the user’s iris and the lens based on the iris diameter $w$ (detected by CED) in the WFOV image and anthropometric data of the human iris size $W$ by using Eq. (3).

Since the upper and lower iris regions are usually occluded by eyelids, we referred to the horizontal visible iris diameter (HVID) as the anthropometric data of the human iris size. Previous research studies have found the HVID.^24–26 Hall et al. measured the HVID, where the range of measured HVIDs was 9.26 to 13.22 mm,²⁴ and we used these values as the range of $W$. Consequently, we can obtain the minimum and maximum values of $Z$ distance according to the range of $W$. Since the transformation matrix [${\mathbf{T}}_{\mathbf{Z}}$ of Eq. (1)] from the area of the WFOV image to that of the NFOV image is defined according to $Z$ distance, multiple ${\mathbf{T}}_{\mathbf{Z}}$ values are determined according to the range of the estimated $Z$ distances, which produces multiple positions of iris area in the NFOV images, as shown in Fig. 7.

Fig. 7

Examples of the estimated iris positions in the left NFOV image according to the estimated $Z$ distance range and corresponding searching region.

Figure 7 shows examples of the estimated iris positions in the left NFOV image according to the estimated $Z$ distance range and corresponding searching region. The red and blue points in Fig. 7 show the mapped iris positions, which are calculated by transformation matrices [${\mathbf{T}}_{\mathbf{Z}}$ of Eq. (1)] based on the minimum and maximum $Z$ distances, respectively. We confirmed that the green points, which are mapped by the transformation matrix based on the ground-truth $Z$ distance, are included in the candidate searching region, which is defined by the blue points in the upper left and the red points in the lower right. As shown in Fig. 7, the searching region for iris localization can be reduced from the entire area of the NFOV image to the candidate searching region. The candidate searching region in the right NFOV image can be determined by the same method.

Based on the camera optical model, we can assume that the iris of 9.26 mm is projected in the WFOV image and its size in the image is $w$. In addition, the iris of 13.22 mm is projected in the WFOV image and its size in the image is assumed as $w^{\prime }$. Based on Eqs. (3) and (4), we can obtain the relationship of $w=V·W/Z$, and at the same $Z$ distance, the $w$ of the iris of 9.26 mm ($W$) is regarded as smaller than the $w^{\prime }$ of the iris of 13.22 mm ($W^{\prime }$). However, if the $w$ is same to $w^{\prime }$, we can think that the iris of 9.26 mm is closer to the camera (smaller $Z$ distance) than that of 13.22 mm based on the relationship of $w=V·W/Z$. In this research, because we do not know the actual size of iris of each user (between 9.26 and 13.22 mm), with one $w$ value measured in the WFOV image and Eq. (3), we calculate the minimum value of $Z$ distance (based on the iris size of 9.26 mm) and maximum one (based on the iris size of 13.22 mm), respectively. That is, because the iris [which is closer to camera (at the minimum value of $Z$ distance)] has the smaller size (9.26 mm) than that (13.22 mm) at the maximum value of $Z$ distance, we can think the size of iris of 9.26 mm in the NFOV image is similar to that of 13.22 mm. So, we do not use the bigger-sized searching region at the minimum $Z$ distance and define the estimated iris region as same size in Fig. 7.

2.4.

Iris Localization and Recognition

We use the candidate searching region of the NFOV image, as shown in Fig. 7, to further redefine the iris searching region by using the position of the SR in order to improve the performance of iris localization. As shown in Fig. 8, the SR is located near the center of the pupil because the NIR illuminator is positioned close to the NFOV cameras, as shown in Fig. 2, and the user aligns both eyes to the two NFOV cameras through the cold mirror.

Fig. 8

Examples of the detected specular reflection (SR) in the captured NFOV images according to $Z$ distance: (a) 25 cm, (b) 40 cm.

We applied the RED algorithm²¹ to detect the position of the SR. It compares the intensity difference between the SR and its neighboring regions, which occurs because the SR region is much brighter. After detecting the position of the SR, the final searching region for iris localization is redefined in consideration of the iris size, as shown in Fig. 9.

Fig. 9

Final searching region for iris localization.

Using the final searching region, we performed two CEDs to isolate the iris region.^22,27 In contrast with the CED for detecting iris area in the WFOV image (Sec. 2.3), both the iris and the pupil region are considered in the two CEDs.^22,27 Because the final searching region is much reduced, as shown by the red box in Fig. 9, the consequent searching range of the parameters of the two CEDs (the two radii of the iris and pupil, the two center positions of the iris and pupil) is also much decreased, which can enhance the accuracy and speed of detecting the iris region using the two CEDs.

After obtaining the center position and radius of both the iris and pupil regions, we detect upper and lower eyelids, and the eyelash region. The eyelid detection method extracts the candidate points of the eyelid using eyelid detecting masks and then detects an accurate eyelid region using a parabolic Hough transform.²⁷ In addition, eyelash detecting masks are used to detect the eyelashes.²⁷ Figure 10 shows examples of the detected iris, eyelid, and eyelash regions.

Fig. 10

Examples of the detected iris, eyelid, and eyelash regions. (a) Original images. (b) Result images.

Figure 10(b) shows the result image where the detections of iris, pupil, eyelid, and eyelashes are completed. Because our algorithm knows the detected positions of eyelashes through eyelash detection procedure,²⁷ all the detected positions are painted as white pixel (whose gray level is 255). That is, except for the detected iris region, all the other areas (such as pupil, eyelashes, eyelid, sclera, and skin) are painted as white pixel (whose gray level is 255) as shown in Fig. 10(b). Then, the image of Fig. 10(b) is transformed into that of Fig. 11, and our iris recognition system does not use the code bit (which is extracted from the areas of the white pixel) for matching because this code bit does not represent the feature of iris texture area.²⁷

Fig. 11

Example of the normalized image of the left eye of Fig. 10(b).

The segmented iris region is transformed into an image of polar coordinates and is normalized as an image consisting of 256 sectors and 8 tracks.²⁷ Figure 11 shows an example of the normalized image. Finally, an iris code is extracted from each sector and track based on a one-dimensional Gabor filter.²⁷ We use the hamming distance to calculate the dissimilarity between the enrolled iris code and the recognized code.^27,30