2.1.

Face Image Quality

In Ref. 4, the face image quality is given by the relation to the use of facial images with face biometric systems. The use of low quality face images in face recognition affects the performance of the system. Currently, it is common to use image quality assessment approaches to evaluate face quality before face recognition. It can make face images differently by either applying image enhancement methods to improve image quality, choosing different recognition systems depending on face quality, or recapturing face images. Therefore, it is necessary to assess face image quality before the recognition process.

There are many factors that can affect face image quality and the performance of biometric systems. It is important to take into account image quality attributes that influence face quality. Both image-based and modality-based face quality attributes are presented in Ref. 4. Since we do not investigate modality-based attributes, we only consider the following image-based face quality attributes: (1) image resolution and size, (2) noise, (3) illumination intensity, (4) image brightness, (5) image contrast, (6) focus, blur, and sharpness, and (7) color.⁸

Based on the aforementioned image-based attributes, several IQMs are proposed in Ref. 4, including a noise estimation IQM,⁹ which estimate an upper bound on the noise level from a single image based on a piecewise smooth image prior model and measured CCD camera response functions; a blur estimation IQM,¹⁰ which is based on the analysis of the spread of the edges in an image for blur estimation; and a blocking artifacts estimation IQM,¹¹ which analyze blocking artifacts as components residing across two neighboring blocks and use one-dimensional pixel vectors made of pixel rows or columns across two neighboring blocks for distortion estimation.

2.2.

Redefined Quality Attributes for Image-Based Biometric Samples

As introduced already, both image-based and modality-based quality attributes are considered in Ref. 4. Fingerprint, iris, or face images can be considered as different subspaces evoluted at different places within the natural image space. Thus, using image-based quality attributes makes it possible to develop multimodality biometric sample quality assessment method. Liu et al.⁵ suggest to employ five quality attributes when evaluating any kind of image-based biometric sample quality and they are based on the survey of state-of-the-art research works.^4,12–15 We apply four of them in this paper and these four image-based quality attributes and their definitions are given as

These are the most important image-based attributes for the evaluation of face image quality, and image-based multimodality biometric sample quality.

2.3.

No-Reference Image Quality Assessment in Face Biometrics

In general, there are two types of no-reference IQMs: distortion-specific IQMs and generalized IQMs. Different IQMs have their pros and cons. Distortion-specific IQMs may have better performance only measuring given distortion. On the other hand, generalized IQMs can assess different types of distortions; however, they may not perform as good as distortion-specific IQMs for certain distortion. In addition, some IQMs are natural scene statistics (NSS)-based metrics and some of them have been trained on image quality databases. NSS-based IQMs can better assess the quality of natural images, and trained IQMs have better performance on images that similar to trained databases. Yet for the other types of images (nonnatural images such as synthetic ones), the performance of these IQM schemes may decrease.

There are several existing studies using no-reference IQMs to assess face sample quality. Abaza et al.¹⁶ evaluated no-reference IQMs that can measure image quality factors in the context of face recognition. Then they proposed a face image quality index that combines multiple quality measures. Dutta et al.¹⁷ proposed a data-driven model to predict the performance of a face recognition system based on image quality features. They modeled the relationship between image-based quality features and recognition performance measures using a probability density function. Hua et al.¹⁸ investigated the impact of out-of-focus blur on face recognition performance. Fiche et al.¹⁹ introduced a blurred face recognition algorithm guided by a no-reference blur metric. From these studies we can see that no-reference IQMs can be helpful to assess the quality of face samples. The observed performance is comparable to some metrics proposed in Ref. 4 which are designed for specific face modality. However, the above-mentioned studies have two common shortages: (1) the image-based quality attributes in these studies do not cover all the five important attributes indicated in Liu et al.;⁵ (2) the databases used in these studies contain both image-based distortions and modality-based distortions. Due to these two shortcomings, the performance of studied no-reference IQMs could be affected.

3.1.

Face Image Database: ${\mathit{GC}}^2$ Multimodality Biometric Database

If we want to benchmark no-reference IQMs, it is recommended to use a standard database in order to compare the results directly.² There are many existing databases in the research field, such as the color FERET database,²⁰ the Yale face database,²¹ the AT&T face database (formerly the ORL database of faces),²² and so on. For more face databases information, we refer to the face recognition homepage.²³ The choice of an appropriate database to be used should be made accordingly to our purpose. Since we focus on image-based quality attributes, we need to choose a specific face database that only contains image-based distortions but not including modality-based degradations. All existing face databases contain both image-based and modality-based degradations. Therefore, we create a new multiple modality biometric database named “${\mathrm{GC}}^2$ Multimodality Biometric Database.”⁸ This database has three biometric modalities: face, contactless fingerprint, and visible wavelength iris. Three cameras are used for the acquisition: (1) a Lytro²⁴ first-generation light field camera (LFC) (11 megapixels), (2) a Google Nexus 5 embedded camera (8 megapixels), and (3) a Canon D700 with Canon EF 100 mm $f/2.8\mathrm{L}$ macrolens (18 megapixels). There are 50 subjects in the database. For the fingerprint modality, three fingers per hand and 15 sample images per finger per camera have been acquired. There are 13,500 fingerprint images in the database. For the iris modality, 15 iris samples per eye per camera have been acquired. There are 4500 iris images in the database. For the face modality, 2150 original face images are obtained in the database. In addition, we introduced different distortions to these original face images as described below. Therefore, totally 86,000 degraded face images are in the database. We only use the face modality in this paper. The acquisition is conducted in a normal office with normal luminance. The background is white and no modality-based distortions are contained in the database.

Since the face recognition application used in the paper only processes grayscale face images, we consider the four image-based quality attributes introduced in Sec. 2.2. In order to obtain image-based distortions correlated to these four attributes, we need to artificially degrade face images in the database. Inspired by the techniques used in CID:IQ image quality database²⁵ and a similar study in biometric sample quality assessment,²⁶ we degrade face images into five degradation levels (one to five, from little degraded to highly degraded) for each distortion as the following (all image processing is conducted by using Matlab R2016 a):

Fig. 1

Degraded face samples for degradation level 5. The first column represents too high contrast face image (upper) and too low contrast face image (lower); the second column represents motion blurred face image (upper) and Gaussian blurred face image (lower); the third column represents high luminance face image (upper) and low luminance face image (lower), and the last column represents face image contains Poisson noise (upper) and JPEG compressed face image (lower).

For the example of face images in all five degradation levels, we refer the reader to Fig. 12 in Appendix A.

3.2.

No-Reference IQMs and Their Classification

Based on the survey and the availability of the source codes, we selected 13 no-reference IQMs for the performance evaluation. These IQMs have high correlation with the image-based quality attributes.⁵ We classify these IQMs into two categories: (1) distortion specific and (2) generalized purposes holistic IQMs. In each category, we separate IQMs into two groups: NSS-based and non-NSS-based IQMs. The classification of the selected IQMs is illustrated in Table 1. As mentioned in the introduction, we consider that face images evoluted in a subspace of the whole natural images space. Thus, the relevance of using NSS-based IQM methods can be investigated.

Table 1

Classification of the selected IQMs.

3.3.

Face Recognition System

The open source face recognition system used in this paper is “The PhD (Pretty helpful Development functions for) face recognition toolbox,”³⁹ which is a collection of MATLAB^® functions and scripts for face recognition. The toolbox was produced as a byproduct of Štruc and Pavešić’s⁴⁰ research work and is freely available for download. Three face feature extraction algorithms are used:

3.4.

Approaches for the Evaluation of Face Recognition System Performance

To evaluate the performance of face recognition systems, many measures exist. Among all of them, we can consider the histograms of comparison scores. They are obtained from the genuine (comparison between samples from the same subject) and imposter (comparison between samples from different subjects) comparisons for all image samples. In general, high quality biometric samples could generate relatively “good” genuine comparison scores (in our case, a score closer to 1 the more similar the two face samples), which are well separated from imposter comparison scores.¹³

An IQM is useful if it can at least give an ordered indication of an eventual performance.¹³ Rank-ordered detection error trade-off (DET) characteristics curve is one of the most commonly used and widely understood method used to evaluate the performance of quality assessment approaches. The DET curve used here plots false none match rate (FNMR) versus false match rate (FMR).

Grother and Tabassi¹³ proposed to use quality-bin-based approaches to evaluate the image quality assessment methods. They believe if a certain percentage of low quality samples are excluded from the dataset, the comparison score would become “better” (closer to 1 in our case) and the equal error rate (EER) (when FMR and FNMR are equal) would decrease. We use it as one of the methods to represent the performance of no-reference IQMs. Because the scale of the quality score for each IQM is different and the linearity of the score is unknown, thus, we omit the percentile low quality samples and keep 80%, 50%, and 20% of highest quality samples from each subject for each of the trial IQM.⁴³

4.1.

Histogram of the Comparison Scores and Their Mean Values

In order to evaluate the performance of the IQMs, we first plot the original comparison score by using GF + KFA recognition algorithm for three cameras in Fig. 2. The $x$ axis represents the score and the $y$ axis represents the quantity of the comparison. The line plots (red “- -” line for genuine comparison and magenta “:” line for imposter comparison) correspond to the fitted normal distributions. The mean value of the comparison score is given as well in Fig. 2. As mentioned before, high quality biometric samples could generate relatively “good” genuine comparison scores (in our case, a score closer to 1 the more similar the two face samples), which are well separated from imposter comparison scores.¹³ This can be observed from Figs. 2(a) and 2(c). However, the histograms of genuine and imposter scores as well as the mean values for smartphone are not well separated compared to LFC and reflex camera. It means that GF + KFA recognition algorithm cannot perform well on face images taken by smartphone in ${\mathrm{GC}}^2$ multimodality biometric database. This can be due to the perspective distortion of the wide angle lens from smartphone. The perspective distortion can affect the performance of face recognition and needs to be compensated.⁴⁴ In addition, the face recognition system might be sensitive to the perspective distortion, which might be the reason why the genuine and imposter scores are not very well separated for smartphone compared to the other two cameras.

Fig. 2

Comparison scores and their mean values for GF + KFA recognition algorithm for the three cameras.

Here, we only illustrate the interesting examples in Fig. 3. The histogram of the genuine comparison score when omitting low quality samples by using 12 selected no-reference IQMs and GF + KFA recognition algorithm for three cameras are shown in Appendix B (Figs. 13Fig. 14–15). For each subplot in the figure, the red continuous line represents the original comparison score (the same fitted red line in Fig. 2); the magenta “-.” line represents the comparison score when we omit 20% lowest quality face samples and keep the remaining 80% higher quality samples; the blue “:” line represents the comparison score when we omit 40% of the lowest quality face samples and keep the remaining 60% higher quality samples; and the green “- -” line represents the comparison score when we omit 60% lowest quality face samples and keep only 40% highest quality samples in the database for the experiment. According to Grother’s theory,¹³ we expect to observe the fitted line moves from left to right (mean comparison value becomes closer to 1) when we keep 80%, 60%, and 40% highest quality samples.

Fig. 3

Examples of comparison scores and their mean values with omitting low quality samples by using GF + KFA recognition algorithm. The $x$ axis represents the comparison score, and the $y$ axis represents the quantity of the genuine comparison.

In Fig. 3(a), by using the assessment results from ILNIQE2 (as well as BRISQUE and DCTSP in Fig. 13) for LFC (BLIINDS2, BRISQUE, ILNIQE2, and SSEQ for smartphone in Fig. 14; BIQI, BRISQUE, and DCTSP for reflex camera in Fig. 15) to omit low quality samples, we can observe the expected right shift for fitted lines (as well as the mean values). It means that these IQMs can assess face image quality and it is correlated with the performance of face recognition algorithm (GF + KFA). In Fig. 3(b), the mean values increase from keeping 80% to 60% highest quality samples by using AQIP metric for reflex camera; however, the values decrease when there is only 40% highest quality samples left. Similar observations can be found for the other two cameras. In Fig. 3(c), the mean values become lower and lower than the original after omitting more and more low quality samples. This means that JNBM has reversed correlation with the performance of GF + KFA recognition algorithm for LFC.

In addition, we plot the mean values with omitting low quality samples in Fig. 4 for three cameras in order to show the overall performance of IQMs. The $x$ axis represents the percentage of kept high quality samples and the $y$ axis represents the mean of comparison score. The red “- -” line represents the original mean of comparison score. Same findings can be obtained from Fig. 4. From the observation above, we can summarize that, based on mean comparison scores, only BRISQUE can assess face quality based on the performance of GF + KFA face recognition algorithm for all three cameras; DCTSP can assess face quality for LFC and reflex camera; ILNIQE2 can assess face quality for LFC and smartphone. The rest of the IQMs either can assess face quality for only one camera or have low ability to assess face quality based on the system performance. However, AQIP, CONTRAST, JNBM, and SH (for LFC); AQI, AQIP, CONTRAST, DCTSP, JNBM, and SH (for smartphone); AQI, CONTRAST, JNBM, and SH (for reflex camera) have reversed correlation with the performance of GF + KFA recognition algorithm according to the histogram and the mean of comparison scores.

Fig. 4

Tendency of mean comparison scores with omitting low quality samples for GF + KFA recognition algorithm.

4.2.

DET Curve and EER

As mentioned before, we also obtain EER as an indicator to examine the performance of IQMs. The DET curves with EER for data with and without omitting low quality face samples for three cameras by using all selected IQMs are given in Appendix C (see Figs. 16Fig. 17–18). Here, we only illustrate interesting examples in Fig. 5. For each subplot in Fig. 5, the red continuous line represents the original DET curve; the magenta “- -” line represents the DET curve when we keep 80% highest quality face samples; the blue “:” line represents the comparison score when we keep 60% highest quality face samples; and the green “-.” line represents the comparison score when we keep only 40% highest quality face samples in the database for the experiment. If a DET curve is closer to the top-right point, it means that this set of data leads to a higher face recognition performance. Meanwhile, the lower EER value the better the system performance.

Fig. 5

Examples of DET curves with EER for comparison score with and without omitting low quality samples by using GF + KFA recognition algorithm.

From Figs. 5(a) and 5(b) we can see that, DET curves shift closer to top-right point when we keep 80%, 60%, and 40% highest quality samples using the assessment results from SSEQ (as well as BIQI and ILNIQE2 in Fig. 16) and DCTSP (as well as AQIP and BLIINDS2 for reflex camera in Fig. 18) to omit low quality samples taken by LFC and reflex camera, respectively. It means that such IQMs can assess face image quality and it is correlated with the performance of face recognition algorithm (GF + KFA). However, although the DET curves have no obvious shift but the EER values have decreased by using the assessment results from ILNIQE2 (as well as AQIP, BLIINDS2, and SSEQ) when we omit low quality samples for smartphone [see Fig. 5(c)]. In Figs. 5(d) and 5(e), EER values decrease when keeping 80% and 60% highest quality samples, but a slightly increasing can be observed when only 40% highest quality samples left in the database. We cannot conclude that such IQMs have “bad” performance because there is low percentage of samples left in the database. We can observe from Fig. 5(f) that PWN for reflex camera makes the DET curves shift to bottom-left point when omitting low quality samples. This means that such IQMs have reversed correlation with the performance of GF + KFA recognition algorithm.

In addition, we plot the tendency of EER values with omitting low quality samples in Fig. 6 for three cameras in order to show the overall performance of IQMs. The $x$ axis represents the percentage of kept high quality samples and the $y$ axis represents the EER. The red “- -” line represents the original EER without omitting low quality samples. Same findings can be obtained from Fig. 6. From the observation above we can summarize that, based on DET curves and EER values, there is not a single IQM can assess face quality based on the performance of GF + KFA face recognition algorithm for all three cameras. However, ILNIQE2 and SSEQ can assess face quality for LFC and smartphone; AQIP and BLIINDS2 can assess face quality for smartphone and reflex camera. The rest of the IQMs either can assess face quality for only one camera or have low ability to assess face quality based on the system performance. However, SH (for all three cameras), CONTRAST, and JNBM (for LFC and reflex camera), and AQI and PWN (for smartphone and reflex camera) have reversed correlation with the performance of GF + KFA recognition algorithm.

Fig. 6

Tendency of EER values with and without omitting low quality samples for GF + KFA recognition algorithm.

We also use EER values for all three cameras by omitting lowest quality face sample one by one until only one highest quality face sample left from each subject as another indicator to assess the performance of selected IQMs. The full plots are shown in Appendix D (see Figs. 19Fig. 20–21). Here we only give the interesting examples that illustrate the change of EER values. The $x$ axis in Fig. 7 represents the number of omitted lowest quality samples unit. There are 40 units per captured sample image per subject (eight distortions in five levels). Each unit has 750 images (15 captured sample image per subject). The $y$ axis represents the EER value. Here, we only illustrate the first omitted 30 units (75% of the entire number if unite) of low quality samples because when only a small part (25%) of images left in the database the change of EER values cannot be trusted. If the EER value has a smooth decreasing tendency when we omit lowest quality samples one by one, it means that the IQM used for generating the quality scores can predict the face recognition algorithm well which represents the high performance of such IQM. In Fig. 7(a) we can see that by using the assessment results from ILNIQE2 (as well as BLIINDS2, BRISQUE, PWN, and SSEQ in Fig. 19) to omit one lowest quality sample (taken by LFC) each time, the EER curves have decreasing tendency (similar decreasing tendency for smartphone by using IQMs ILNIQE2, and AQIP, and DCTSP for reflex camera). However, in Fig. 7(b), the EER curve from BIQI for LFC camera has fluctuation all the time. But the overall trend of the curve is decreasing and the EER value became very low in the end. On the other hand, as shown in Fig. 7(c), the EER curve from DCTSP for smartphone has fluctuation all the time as well. But the overall trend of the curve is increasing and, in the end, the EER value became higher than the values in the very beginning of the curve. Finally, if we see Fig. 7(d), the EER values seem to increase when we use AQI to assess face image quality for reflex camera. From the observation above we can summarize that, based on EER values with omitting low quality face samples one by one until the best quality sample left for three cameras by using GF + KFA recognition algorithm, ILNIQE2 can assess face image quality for three cameras. The rest of the IQMs can either assess face quality for only one camera or have low ability to assess face quality based on the system performance. However, CONTRAST, JNBM, and SH have reversed correlation with the performance of GF + KFA recognition algorithm according to the results obtained from one by one omitted EER values.

Fig. 7

Examples of EER values with omitting low quality face samples one by one until the best quality sample left by using GF + KFA recognition algorithm.

4.3.

Spearman Correlation between IQMs

Finally, we compute the Spearman correlation of quality scores between IQMs, which have a better performance as discussed above (AQIP, BLIINDS2, ILNIQE2, and SSEQ). From the correlation coefficients, we can analyze whether each IQM gives the similar quality score to the same face sample image. The correlation coefficients are illustrated in Table 2. We show the coefficients for LFC in the first value in each cell, the coefficients for smartphone in the second value in each cell, and the last value in each cell for reflex camera. In Table 2 we can see that none of the IQM has high correlation with each other. It means that these IQMs were designed for different types of distortions and they cannot be replaced by other selected IQMs.

Table 2

Spearman correlation of quality scores between IQMs. The first value in each cell represents LFC, the second value represents smartphone, and the last value represents reflex camera.

4.4.

Performance Comparison between Selected IQMs and ISO-Proposed IQMs

As mentioned in Sec. 2.1, several IQMs are proposed in Ref. 4. Here we compare the performance of selected IQMs and ISO-proposed IQMs: ISO1,⁹ ISO2,¹⁰ and ISO3.¹¹ In Fig. 8 we illustrate the DET curves with EER for LFC by using ISO-proposed IQMs to omit low quality samples. From Fig. 8 we can see that only ISO1 can give expected DET shift and EER decrease when we omit low quality samples in the dataset. The other two IQMs seem to have reversed correlation with the performance of GF + KFA recognition algorithm. Similar observations can be found in Fig. 9, which represent EER values with omitting low quality face samples one by one until the best quality sample left for reflex camera by using ISO-proposed IQMs. Therefore, compared to the performance evaluation results for selected no-reference IQMs, we do see there are several metrics have better performance than ISO-proposed metrics.

Fig. 8

DET curves with EER for comparison score with and without omitting low quality samples for LFC by using ISO-proposed IQMs (GF + KFA).

Fig. 9

EER values with omitting low quality face samples one by one until the best quality sample left for reflex camera by using ISO-proposed IQMs (GF + KFA).

4.5.

Retraining ILNIQE2 on Face Database

From previous results, one deduced that performances of NSS-based IQM methods are competitive to predict the quality of face images guaranteeing a high level of performance of biometric systems. One specific metric, namely ILNIQE2 shows interesting results in terms of correlation between the provided quality scores and the performance results. Since this quality index has been trained on general purpose natural images, it would be interesting to investigate if results can be improved retraining it on face images. To perform the retraining, the color FERET database²⁰ has been selected, which has 269 subjects and there are two acquisition sessions for most of subjects. For each session, 11 different sample images were acquired which contain different face angles and expressions. We use 269 images [one sample image (the front face) per subject] from the FERET database to retrain the ILNIQE2 IQM. These 269 images are all high quality face images because the ILNIQE2 metric only requires pristine images for training. The retrained metric is then used to reconduct the experiment removing lowest quality samples one by one from each subject. The plots of EER values for three cameras are shown in Fig. 10. The blue thin lines represent the original ILNIQE2 method, and the red bold lines represent the retrained ILNIQE2 method. From Fig. 10 we can see that, after the retraining process, the overall performance of the IQM is improved because the red bold lines are under the blue thin lines. It means that the overall EER values from the retrained method are lower than the original method. In addition, the improvements for smartphone and reflex camera are greater than LFC, especially for reflex camera. By using the original ILNIQE2 to omit lowest quality samples from the database, the EER values are not smoothly decreasing, moreover, the EERs increase after 18 unites of lowest samples are removed for reflex camera. However, by using the retrained method, the line becomes smoother and has a decreasing tendency. Finally, the EERs reach “0” when 24 unites of lowest quality samples are omitted. The difference of EERs between the original and the retrained method for reflex camera is obvious.

Fig. 10

Comparison of EER by omitting lowest quality sample one by one using ILNIQE2 for each subject between the original method and the retrained method.

We would like to investigate if such improvement for reflex camera is due to the better prediction of retrained ILNIQE2 for all distortions or for some distortions. Therefore, we illustrate EER curves for single distortions in Fig. 11. Since there are eight distortions for each sample image, the total units become five instead of 40 as the case was in Fig. 10. Here we only illustrate the 80% (four units) of the EERs because when only a small part of images (e.g., 20%) left in the database, the EER is not convincing. From Fig. 11 we can see that the retraining process has little impact on high luminance and JPEG artifacts distortions. It reduces the performance of ILNIQE2 for low contrast and low luminance distortions because the average EERs are higher after retraining, but the EER curves still have a decreasing tendency. Furthermore, the retraining has a positive effect for high contrast, Gaussian blur, motion blur, and Poisson noise distortions. The EER curves for the latter three distortions have an increasing tendency for the original method; however, they have a decreasing trend after the retraining process. It is worth noting that all curves after retraining have a decreasing tendency.

Fig. 11

Comparison of EER between the original method and the retrained method for each distortion for reflex camera.