2.1.

Considered Contrast Enhancement Algorithms

Several methods for CE have been proposed within the past decades. Various modifications to conventional global histogram equalization have been proposed to counteract mean brightness shifts^11,12 leading to annoying artifacts and allowing smooth transitions to identity.^25,26 Also learning-based methods^33–37 as well as methods based on image decomposition^38,39 have been proposed. The decomposition often relies on color information, making the methods inapplicable to single channel image data. However, the scope of this work is limited to easily implemented algorithms, given in Table 1, that operate on single channel image data.

2.2.

Data Generation

For the training of models for TOD,¹ images of triangles are generated with varying contrasts, sizes, and four orientations, i.e., up, down, left, and right. Misalignment angles uniformly distributed in $[-15\mathrm{deg},15\mathrm{deg}]$ are added to the orientation angles of the triangles to make the models more robust to misalignment. Exceeding the maximum rotation angle 15 deg would lead to incorrect labeling because rotations of an equally sided triangle by 30 deg result in other labeled orientations due to the 120 deg rotational symmetry. This rotation is crucial when applying models on real camera data because some misalignment between the field of view of a camera and a target is unavoidable.

2.3.

Background Overlay

Background images are extracted from OpenImages V6⁸ as the random square crops. RGB images are converted to floating-point grayscale images. Mean ${\mu }_{\text{crop}}$ and standard error ${\sigma }_{\text{crop}}$ are calculated within these crops. Then, gray levels $I_{\text{triangle}+\text{overlay}}(x,y)$ of an image with single triangle and background overlay are calculated as

$c_{\text{background}}$ is a constant gray level over the entire image. $a_{\text{triangle}}$ is an offset value only added for pixels related to the triangle. In Eq. (1), the pixel values of the image crop $I_{\text{crop}}(x,y)$ are normalized by subtracting the mean ${\mu }_{\text{crop}}$ and dividing by the corrected error, and

2.4.

Degradations

Several image degradations representing typical camera effects are applied. Temporal noise is applied as uncorrelated additive Gaussian noise. Fixed pattern noise of a sensor is modeled as line- and column-based additive Gaussian noise. Linear motion blur on the triangle is applied to represent moving targets. Stabilization errors due to camera vibration are applied as linear motion blur and Gaussian blur on the triangle with a background overlay. Blur due to optical diffraction by circular apertures is applied by filters

A variety of physical parameters ($\lambda$, $f$, $D$, and $p$) can be realized by random sampling of $s$ from a uniform distribution in $[0.1,s_{\max }]$. $s_{\max }=10$ is chosen to limit the proportion of severely degraded images, which aggravate the model training due to little possible accuracy gains compared with statistical fluctuations. Optical diffraction blur is applied by spatial filtering with random 2D kernels of width and height $K=6s_{\max }$. The kernel size $K$ is limited due to lack of information beyond the borders of images of finite size. The scaling factor $s\le s_{\max }$ is therefore also limited because higher values lead to radial kernel profiles biased by clipping effects due to the limited kernel size. To reduce aliasing due to the oscillatory form of the Airy disk [Eq. (4)], larger kernels $f_{ij}$ of $f_{\mathrm{os}}K\times f_{\mathrm{os}}K$ pixels are generated with an oversampling factor $f_{\mathrm{os}}=8$. These extended kernels are downsampled by average pooling with this oversampling factor to give $K\times K$ kernels $g_{ij}$. Normalized filter kernels are then formed as

Several aliasing effects may occur due to small detector fill factors $ff<1$ (the ratio between the detector dimension and pitch) or different shapes of detector footprints, e.g., rhombic or circular. These effects can be realized by masking extended kernels of Airy disks $f_{ij}$ with the detector profile before average pooling. However, this option is not used in this work due to the rare use of nonsquare detectors, low signal-to-noise ratio for low fill factors,⁴¹ and faster image generation.

2.5.

Contrast Enhancement

The algorithms in Table 1 are applied on 50% of the degraded image with equal probabilities. Due to the complex and divergent control flow of some algorithms, these methods are implemented and calculated by a separate application on the CPU. In Fig. 1, pristine training examples, as well as degraded and ADSP processed ones, are shown. These training examples are generated online during training to have a practically infinite amount of training data immune to overfitting. However, the source of background images and the number of possible crops is large but finite.

Fig. 1

Training examples with associated labels: left (0), up (1), right (2), and down (3). (a) Pristine training examples, (b) training examples with natural background, degradations, and CE.

2.6.

Model Setup

A conventional CNN architecture shown in Fig. 2 is used for TOD classification on images of dimensions $2^n\times 2^n$. To facilitate the model training, the input image is normalized by linear shifting and scaling of pixel values to have a mean of 0 and a standard deviation of 1 over spatial dimensions. Uniform input images with a standard deviation of 0 are not scaled. Then, the normalized image is downsampled by a chain of building blocks until the spatial dimensions are reduced to 2. Each building block consists of two 2D convolutional layers with $3\times 3$ kernels and rectified linear unit activations (ReLUs) and a subsequent $2\times 2\max$ pooling layer. Hence, downsampling by a factor 2 is applied per block. The spatial dimensions are reduced, and the number of feature maps, given as

Fig. 2

CNN model architecture for TOD classification of $64\times 64$ grayscale images. A $64\times 64$ image is fed into a chain of blocks. Each block consisting of two 2D convolutional layers with ReLU activation and subsequent max pooling layer downsamples the spatial dimensions while increasing the number of feature maps. The last block is followed by two dense layers and a final softmax layer to provide probabilities for all directions (left, up, right, and down).

2.7.

Model Training

The models are trained and evaluated by Python 3.9/Tensorflow 2.8. For optimization, ADAM⁴² with a learning rate $\eta =0.001$ is used. Weights are initialized by He normal initialization.⁴³ Models are trained for $N=1000$ iterations. Despite slower training, techniques for acceleration of training, such as batch normalization,⁴⁴ weight normalization,⁴⁵ or adaptive gradient clipping,⁴⁶ were deliberately omitted to achieve smaller models with faster inference, which are compatible for running on edge TPUs.⁴⁷ The loss function is cross entropy.

In each iteration, new sample images of triangles with background overlays are generated on the fly during the training for data augmentation. Triangles have a random size, position, and arbitrary orientation angle in [0 deg, 360 deg]. In addition, these images are impaired by the prescribed degradations. 50% of the training images were enhanced with one of the 25 methods for CE with equal probabilities. Corresponding disjoint sets of background images are randomly chosen from the respective partition of the OpenImages V6 database. For evaluation of model performance over the degradation parameters, i.e., ${\mathrm{SNR}}_{\text{background}}$, ${\mathrm{SNR}}_{\text{noise}}$, and triangle size, images are generated in the same way as background images chosen from the test subset of the database.

The question arises how large the percentage of degraded and ADSP processed images in the training data should be to obtain acceptable accuracies on validation sets of pristine and degraded images. $64\times 64$ models with different percentages of degraded and processed images in the training data were trained and evaluated on different kinds of validation data. The respective accuracies on the validation data are shown in Fig. 3. It can be observed that models trained only with pristine images perform very bad on degraded and ADSP processed images with natural backgrounds. A slight increase of the percentage significantly raises the validation accuracies on degraded imagery. On the other hand, models trained with a high percentage of degraded images still perform very well on pristine images. Therefore, to make the best use of the model capacity for the image degradations and ADSP methods, all models mentioned below are trained with 100% degraded images, whereas ADSP is applied with 50% probability.

Fig. 3

Dependency of the validation accuracy on the composition of training sets. Horizontal: percentage of degraded images in the training data. Vertical: accuracies on different validation sets, each with $N=N_{\text{orientations}}·N_{\text{backgrounds}}·N_{\text{samples}}=400,000$ images, with $N_{\text{backgrounds}}=1000$, $N_{\text{orientations}}=4$, $N_{\text{samples}}=100$. “All degradations” include images enhanced with one of the algorithms in Table 1.

3.1.

Dependency of Accuracy on Background Variance

A trained $64\times 64$ model is validated on images of a fixed target, a centered triangle with a circumradius of $r=10\text{pixel}$. The triangle circumradius $r$ is converted to the often-used square root area $S=\sqrt{A}$¹ using the Pythagorean theorem:

1000 random crops of different background images from OpenImages V6⁸ were used. The background variance ${\sigma }_{\text{background}}$ of gray levels was varied to have different ${\mathrm{SNR}}_{\text{background}}\in [\mathrm{0,1},\mathrm{2,3},\mathrm{4,5},\mathrm{10,20}]\mathrm{dB}$. White Gaussian noise was added to have ${\mathrm{SNR}}_{\text{noise}}\in [\mathrm{0,5},10]\mathrm{dB}$, where

Fig. 4

(a) Accuracy over ${\mathrm{SNR}}_{\mathrm{background}}\hspace{0.17em}(\mathrm{dB})$: fixed triangle circumradius $r(\text{pixel})=10$ and varying ${\mathrm{SNR}}_{\text{noise}}(\mathrm{dB})$. (b) Example validation images: fixed ${\mathrm{SNR}}_{\text{noise}}=0\mathrm{dB}$, four orientations and eight levels of ${\mathrm{SNR}}_{\text{background}}\in [\mathrm{0,1},\mathrm{2,3},\mathrm{4,5},\mathrm{10,20}]\mathrm{dB}$.

Obviously, accuracies are about 100% for ${\mathrm{SNR}}_{\text{background}}\ge 5\mathrm{dB}$ and ${\mathrm{SNR}}_{\text{noise}}\ge 5\mathrm{dB}$. The accuracies drop monotonically with decreasing ${\mathrm{SNR}}_{\text{background}}$. A similar behavior can be observed for varying triangle sizes, as shown in Fig. 5. As expected, the accuracies also drop for decreasing triangle sizes. Further variation of the relative triangle position in the subpixel range shows high fluctuations of accuracies for low triangle circumradius $r=1\text{pixel}$. This finding is consistent with the problem of recognition near the resolution limit mentioned before.¹

Fig. 5

(a) Accuracy over ${\mathrm{SNR}}_{\text{background}}(\mathrm{dB})$: fixed ${\mathrm{SNR}}_{\text{noise}}(\mathrm{dB})=40$ and varying triangle circumradius $r$. (b) Example validation images: triangle circumradius $r(\text{pixel})=2$, four orientations, and eight levels of ${\mathrm{SNR}}_{\text{background}}(\mathrm{dB})\in [\mathrm{0,1},\mathrm{2,3},\mathrm{4,5},\mathrm{10,20}]$.

3.2.

Upscaling of Receptive Field

Models according to the CNN architecture shown in Fig. 2 were trained for various resolutions of receptive fields, i.e., $128\times 128$, $256\times 256$, $512\times 512$, and $1024\times 1024$. For $512\times 512$ and $1024\times 1024$, a reduction of learning rate $\eta ={10}^{-4}$ was required to achieve significant model improvements compared with the initial model states. Otherwise, model performances stagnated on average at 25% guessing rates. In Fig. 6, validation accuracies over ${\mathrm{SNR}}_{\text{background}}$ are shown for different sizes of receptive fields.

Fig. 6

Validation accuracy over ${\mathrm{SNR}}_{\text{background}}(\mathrm{dB})$ for different sizes of receptive fields and ${\mathrm{SNR}}_{\text{noise}}\in [\mathrm{0,5},20]\mathrm{dB}$ based on $N_{\text{total}}=N_{\text{direction}}·N_{\text{background}}=4000$ images containing a centered triangle with circumradius $r=10\text{pixel}$, superposed by $N_{\text{background}}=1000$ different backgrounds formed by random image crops of respective sizes from the OpenImages V6 test set and $N_{\text{direction}}=4$ triangle orientations. For increasing field size, the model performance decreases.

There is a general trend of decreasing accuracies for larger receptive fields. This may be due to the fact that larger receptive fields can contain more objects with high similarity to triangles. Furthermore, the growth factor $g=1.2$ for the number of feature maps per block $n_{\text{feature-}\text{maps}}$ may be insufficiently small to provide enough model capacity for an increasing range of triangle sizes to handle. A surprising fact is the lower accuracies for higher ${\mathrm{SNR}}_{\text{noise}}=20\mathrm{dB}$ compared with ${\mathrm{SNR}}_{\text{noise}}=5\mathrm{dB}$ and receptive fields of $256\times 256\text{pixels}$ and larger. This might indicate beneficial properties of Gaussian noise by suppressing structures in the background resembling the triangle target.

3.3.

Comparison of Methods for Contrast Enhancement

A trained $64\times 64$ model was validated by images processed with the 25 ADSP algorithms shown in Table 1. Many algorithms only operate on integer pixel values based on gray level distributions, which are widespread for natural images. Hence, to prevent saturation due to clipping, the pixel values are shifted to have a mean of half of the dynamic range, upscaled by a factor $2^8-1=255$ and converted to 8 bit. After ADSP processing, this shifting and upscaling is reverted, and pixel values are converted to floating point numbers.

In Fig. 7, differences of accuracies between single CE algorithms and the identity are shown for varying ${\mathrm{SNR}}_{\text{background}}$ and ${\mathrm{SNR}}_{\text{noise}}$. For convenience, the algorithms were ranked with respect to the maximum value, and only the top 10 algorithms are shown.

Fig. 7

Accuracy difference $\mathrm{\Delta }I(\%)=I_{\mathrm{CE}}(\%)-I_0(\%)$ for top 10 CE algorithms ranked with respect to the maximal value. Accuracies $I_{\mathrm{CE}}$ ($I_0$) for every data point are based on evaluations of the $64\times 64$ TOD model on $N_{\text{total}}=N_{\text{direction}}·N_{\text{background}}=4000$ artificial images with (without) CE containing a centered triangle with circumradius $r=10\text{pixel}$, $N_{\text{direction}}=4$ triangle orientations, and $N_{\text{background}}=1000$ different backgrounds formed by random image crops from the OpenImages V6⁸ test set. In addition, before CE, Gaussian noise was added: (a) ${\mathrm{SNR}}_{\text{noise}}=0\mathrm{dB}$, (b) ${\mathrm{SNR}}_{\text{noise}}=5\mathrm{dB}$, and (c) ${\mathrm{SNR}}_{\text{noise}}=20\mathrm{dB}$.

It can be observed that there are accuracy gains for low ${\mathrm{SNR}}_{\text{background}}<5\mathrm{dB}$, and the accuracy differences are quite similar for the top 10 algorithms. Because the accuracy is about 100% for ${\mathrm{SNR}}_{\text{background}}\ge 5\mathrm{dB}$ and ${\mathrm{SNR}}_{\text{noise}}\ge 5\mathrm{dB}$ without ADSP processing according to Fig. 4, no significant improvement by CE can be expected for these cases. By contrast, a severe degradation of model performance occurs for some CE algorithms and high ${\mathrm{SNR}}_{\text{noise}}=20\mathrm{dB}$.

To validate CE algorithms on a variety of degradations, different triangle parameters and degradation parameters were varied and uniformly distributed in the ranges given in Table 2.

Table 2

Boundaries for uniformly distributed triangle parameters and degradation parameters with the image dimensions I=64 and the dynamic range DR=255.

$N_{\text{target}}=1000$ random samples of triangles and degradation parameters were combined with $N_{\text{backgrounds}}=1000$ natural backgrounds as random crops from Open Images V6.⁸ This procedure was repeated $N_{\text{chunks}}=10$ times with varying random seeds, resulting in different triangles and background images. Model accuracies were calculated on $N_{\text{total}}=N_{\text{chunks}}{N}_{\text{targets}}{N}_{\text{backgrounds}}={10}^7$ images with $64\times 64\text{pixels}$. The same procedure was repeated, with each of the 25 CE algorithms from Table 1 being applied respectively as the final step. Compared with grid variation of individual triangle and degradation parameters, random sampling of many of these parameters allows for investigation of individual parameters by arbitrary parameter cuts, whereas other parameters are widely distributed. This gives better insights on possible fluctuations on model performances when those parameters are unknown.

As shown in Fig. 8, accuracy differences between each of the 25 CE algorithms and identity were calculated for parameter cuts of the triangle circumradius $r$ and the signal-to-background ratio ${\mathrm{SNR}}_{\text{background}}$. Only accuracies on images with values in $r\in [\mathrm{1,3}]$ pixel (left), $r\in [\mathrm{14,16}]\text{pixel}$ (right), ${\mathrm{SNR}}_{\text{background}}\in [\mathrm{0,2}]\mathrm{dB}$ (left), and $\mathrm{SNR}\in [\mathrm{18,20}]\mathrm{dB}$ (right) were selected. The interquartile ranges (IRQ), shown as orange boxes, contain values between the 25%-percentile and the 75%-percentile. The IRQs are extended by whiskers by 1.5IRQ at both sides at maximum, but they are limited by the respective minimal and maximal values in the data. Outliers are shown as circle markers.

Fig. 8

Whisker-box plots of accuracy differences of the $64\times 64$ model between 25 CE algorithms and identity based on varying cuts for (a) triangle circumradius $r\in [\mathrm{1,3}]\text{pixel}$, (b) $r\in [14,16]\text{pixel}$, ${\mathrm{SNR}}_{\text{background}}\in [\mathrm{0,2}]\mathrm{dB}$ (left), and ${\mathrm{SNR}}_{\text{background}}\in [\mathrm{18,20}]\mathrm{dB}$ (right). Insets show examples of centered triangles for $r$, ${\mathrm{SNR}}_{\text{background}}$ in the corresponding ranges with ${\mathrm{SNR}}_{\text{noise}}=\infty$. For each algorithm, means (green triangle marker), medians (red lines), IQRs (orange boxes) of accuracy differences for the counts in the respective parameter intervals are shown (further details in the text). At high ${\mathrm{SNR}}_{\text{background}}$ and low triangle circumradius $r$, the 25 CE algorithms lead to significant impairment of the model accuracies.

Obviously, a high ${\mathrm{SNR}}_{\text{background}}$ and a low triangle circumradius $r$ lead to significant impairment of model accuracies by most of the 25 CE algorithms. Accuracy differences at high triangle circumradius $r$ and high ${\mathrm{SNR}}_{\text{background}}$ (right bottom) show small IRQs, as the accuracy is often saturated at 100% for high ${\mathrm{SNR}}_{\text{noise}}$. Hence, accuracies for low ${\mathrm{SNR}}_{\text{noise}}$ are rendered as outliers. For high $r\in [\mathrm{14,16}]\text{pixel}$ and low ${\mathrm{SNR}}_{\text{background}}=[\mathrm{0,2}]\mathrm{dB}$ (left bottom) only, some CE algorithms show accuracy gains. Also, monotonic transitions of accuracy differences for varying $r$ and ${\mathrm{SNR}}_{\text{background}}$ were observed. The reason for the significant impairment at high ${\mathrm{SNR}}_{\text{background}}$ could be due to the fact that a narrow gray level distribution of background values leads to excessive enhancement of the background by most CE algorithms, resulting in textures with a low dynamic range, steep edges, and a high similarity to the triangle to be discriminated. On the other hand, a large triangle reduces the number of background pixels and hence their contribution to the gray level distribution of the entire image. Most of the investigated CE algorithms depend on the image gray level distribution.

3.4.

Generalization on Background Images of Different Image Databases

To investigate the ability of the TOD model to generalize to a larger variety of background images, the trained $64\times 64$ model was validated on artificial images with a single centered triangle with a fixed circumradius of $r=10\text{pixel}$ superposed with background images resulting from random crops of images from different image databases. Examples of such image crops with $64\times 64\text{pixels}$ are shown in Fig. 9 for different image databases: Pascal VOC,⁴⁸ ILSVRC2012,⁴⁹ FLIR ADAS,⁵⁰ OpenImages V6,⁸ Stanford dogs,⁵¹ Oxford flowers 102,⁵² Caltech 101,⁵³ and Gaussian noise.

Fig. 9

Examples of $64\times 64$ random crops of different image databases used as backgrounds: Pascal VOC,⁴⁸ ILSVRC2012,⁴⁹ FLIR ADAS,⁵⁰ OpenImages V6,⁸ Stanford dogs,⁵¹ Oxford flowers 102,⁵² Caltech 101,⁵³ and Gaussian noise. Image are converted to single-channel data by averaging over channels of RGB data. For clarity, example images are shown using the colormap “viridis,” and pixel values are scaled to have a mean that equals the center of the dynamic range and a standard deviation of one-fourth of the dynamic range. Compared with other image databases, FLIR ADAS contains images with a high noise content.

In Fig. 10, the model accuracies over $N=N_{\text{direction}}·N_{\text{background}}$ images and different ${\mathrm{SNR}}_{\text{background}}$ are shown, with $N_{\text{direction}}=4$ triangle orientations and $N_{\text{background}}=1000$ different backgrounds from several image databases. In addition, the generated artificial images are impaired by Gaussian noise with a high noise level ${\mathrm{SNR}}_{\text{noise}}=0\mathrm{dB}$ (left) and a low noise level ${\mathrm{SNR}}_{\text{noise}}=20\mathrm{dB}$. It can be observed that model accuracies are very similar for most of the image databases. In contrast, background images of Gaussian noise yield significantly better accuracies than those of the image databases. Images of FLIR ADAS⁵⁰ show accuracies between those of Gaussian noise and the image databases, which may be due to the relatively high noise content in the FLIR ADAS images. This fact indicates that structured backgrounds from the image databases have a higher degradational effect on the triangle recognition than Gaussian noise for equal standard deviations of pixel value fluctuations.

Fig. 10

Model accuracies over ${\mathrm{SNR}}_{\text{background}}(\mathrm{dB})$. For each data point, $N=N_{\text{direction}}·N_{\text{background}}$ artificial images are generated with $N_{\text{direction}}=4$ triangle orientations and $N_{\text{background}}=1000$ different backgrounds used from random crops of different image databases. Gaussian noise is added with (a) ${\mathrm{SNR}}_{\text{noise}}=0\mathrm{dB}$ and (b) ${\mathrm{SNR}}_{\text{noise}}=20\mathrm{dB}$.

The qualitative behavior of the model accuracy is similar for different databases when applying the methods for CE. The same is true for the ranges of ${\mathrm{SNR}}_{\text{background}}$ values, for which CE yields an improvement in accuracy. For convenience, only an example for applying CLAHE is shown in Fig. 11.

Fig. 11

Model accuracy differences $\mathrm{\Delta }I=I_{\mathrm{CLAHE}}-I_{\text{Identity}}$ over ${\mathrm{SNR}}_{\text{background}(\mathrm{dB})}$ with accuracies $I_{\mathrm{CLAHE}}$ based on images enhanced by CLAHE and accuracies $I_{\text{Identity}}$ for images without CE. For each data point, $N=N_{\text{direction}}·N_{\text{background}}$ artificial images are generated with $N_{\text{direction}}=4$ triangle orientations and $N_{\text{background}}=1000$ different backgrounds used from random crops of different image databases. Gaussian noise is added with (a) ${\mathrm{SNR}}_{\text{noise}}=0\mathrm{dB}$ and (b) ${\mathrm{SNR}}_{\text{noise}}=20\mathrm{dB}$.

3.5.

Variation of Model Size

The used architectures so far were an arbitrary initial choice. One might ask if similar/better accuracies could have been achieved by smaller/larger models. To answer this question, further models were trained based on the default configuration (Sec. 2.6) with modifications of single parameters, i.e., the initial number of filters $N_0$, the growth factor $g$, the dense layer size $L$, and the number of dense layers. In Fig. 12, validation accuracies for varying initial number of filters $N_0$, the growth factor $g$, the dense layer size $L$, and the number of extra dense layers in addition to the final dense layer with four units are shown. Models are validated on images with $N_{\text{orientation}}=4$ orientations, $N_{\text{background}}=1000$, and $N_{\text{sample}}=100$ samples, resulting in $N_{\text{total}}=N_{\text{orientations}}·N_{\text{background}}·N_{\text{sample}}=400,000$ images. Accuracies denoted with “all degradations” contain images enhanced by one of the CE algorithms (Table 1) with equal probabilities. Furthermore, validation accuracies are compared for different activations, i.e., leaky ReLU,⁵⁴ exponential linear unit (ELU),⁵⁵ Gaussian error linear unit (GELU),⁵⁶ scaled exponential linear unit (SELU),⁵⁷ APLU,⁵⁸ tanh, sigmoid,⁵⁹ softplus,⁵⁹ softsign,⁵⁹ and swish.⁶⁰ It can be observed that reductions of the dense layer size as low as $L=22$, the growth factor $g=0.7$, and the number of filters $N_0=10$ result in comparable accuracies compared with the default configuration. Even for a varying number of extra dense layers with $L=1024$ units in addition to the final dense layer with four units, there are only slight variations in accuracies. From the 11 investigated activations, the ReLU⁵⁹ from the default configuration (Sec. 2.6) performs very well compared with most other activations. adaptive piecewise linear unit (APLU) and sigmoid did not converge at all above the guessing rate of 25%.

Fig. 12

Model comparisons for (a) varying dense layer size, (b) growth factor $g$, (c) initial number of filters $N_0$, (d) the number of extra dense layers with $L=1024$ units, and (e) different activation functions. Accuracies on different validation sets, each with $N=N_{\text{orientations}}·N_{\text{backgrounds}}·N_{\text{samples}}=400,000$ images, with $N_{\text{backgrounds}}=1000$, $N_{\text{orientations}}=4$, $N_{\text{samples}}=100$. “All degradations” include CE with one of the algorithms in Table 1.

3.6.

Model Complexity

We did benchmarks of our trained TOD model for 64x64 pixels on our machine (a Ryzen 9 3900X processor with an NVIDIA GeForce RTX 2080Ti graphics card and 64GB RAM). Table 3 gives the average running time on an NVIDIA GeForce RTX2080Ti, as well as the number of parameters and floating point operations (FLOPs), which equals twice the number of multiply-accumulate computations (MACs).

Table 3

Model properties and benchmark results on an NVIDIA GeForce RTX 2080Ti.

The trained TOD models are smaller and faster compared with current machine vision backbones, which are also shown in Table 3. Faster model inference allows for a stronger focus on several image degradations. Compared with the classification of RGB images in the visible spectrum, the TOD models are applicable on single-channel data, and the four triangle classes are symmetric and balanced. Furthermore, the triangle shape and texture are independent of any spectral band, in contrast to many image databases in the visible band. This is crucial, e.g., for range performance assessment of imagers in several infrared spectral bands [long-wavelength infrared (LWIR), mid-wavelength infrared (MWIR), and short-wavelength infrared (SWIR)].

3.7.

Comparison with Other Image Quality Metrics

Different image quality metrics were proposed for assessment of methods for CE, such as absolute mean brightness error (AMBE),²⁰ discrete entropy,¹² measure of enhancement (EME) and EME based on entropy (EMEE),⁶⁵ QRCM,⁶⁶ UIQ,⁶⁷ EBCM,⁶⁸ and CII.²⁹ A more detailed overview of further image quality metrics and methods for CE can be found in another work.⁶⁹

However, most of these metrics were validated by subjective image quality assessments and may not correlate well with accuracies of models for TOD recognition or other machine vision tasks. To investigate some current nonreference metrics on images used in the evaluation of TOD models, these metrics were calculated for $64\times 64$ images with a centered triangle superposed by backgrounds taken from OpenImages V6 scaled to different ${\mathrm{SNR}}_{\text{background}}$ and impaired by Gaussian noise with different ${\mathrm{SNR}}_{\text{noise}}$. In addition, these images were enhanced by three CE methods, CLAHE,¹³ EHS,¹⁷ and SUACE,²⁸ which were among the top 10 algorithms in Fig. 7. In Fig. 13, values for nonreference image quality metrics EBCM,⁶⁸ EME, EMEE,⁶⁵ and entropy¹² over ${\mathrm{SNR}}_{\text{background}}$ are shown.

Fig. 13

Means of nonreference metric values $Q_0$ (EBCM,⁶⁸ EME,⁶⁵ EMEE,⁶⁵ and entropy,¹² left column) and metric differences $\mathrm{\Delta }{Q}_{\mathrm{CE}}=Q_{\mathrm{CE}}-Q_0$ for three CE methods, i.e., CLAHE,¹³ EHS,¹⁷ and SUACE²⁸ (right columns) over $N_{\text{background}}=1000$ images of $64\times 64\text{pixels}$ with a centered triangle with circumradius $r=10\text{pixel}$ for different ${\mathrm{SNR}}_{\text{background}}(\mathrm{dB})$ and ${\mathrm{SNR}}_{\text{noise}}(\mathrm{dB})$. $\mathrm{\Delta }{Q}_{\mathrm{CE}}=0$ is shown as a horizontal blue dashed line. Example images with centered triangles superposed with background and impaired by Gaussian noise with ${\mathrm{SNR}}_{\text{noise}}={\mathrm{SNR}}_{\text{background}}=3\mathrm{dB}$ (bottom), (left) without CE and (right) CE by the respective algorithm.

It can be observed that metric values $Q_0$ are high for low ${\mathrm{SNR}}_{\text{background}}$ and low ${\mathrm{SNR}}_{\text{noise}}$, representing high variances of background and Gaussian noise, respectively. The metric values $Q_0$ decrease monotonically with increasing ${\mathrm{SNR}}_{\text{background}}$ and ${\mathrm{SNR}}_{\text{noise}}$. The only exception is EBCM for ${\mathrm{SNR}}_{\text{noise}}=20\mathrm{dB}$, which we assume is due to the regularization of denominators in the algorithm. Therefore, low metric values represent conditions under which TOD accuracies are high. However, very similar metric values $Q_0$ could also be observed when the triangle was omitted. This indicates that these metrics are mainly determined by background for triangles with $r\le 10\text{pixel}$. Thus, the metrics are weakly or not at all interrelated with the TOD task performance if the triangle is very small. CEs by the three CE methods generally lead to positive shifts of the metric values $Q_{\mathrm{CE}}$ or in other words $\mathrm{\Delta }{Q}_{\mathrm{CE}}=Q_{\mathrm{CE}}-Q_0>0$, which can be interpreted as predominant enhancement of the background and noise, which aggravates TOD recognition. Similar results were found for the evaluation of the full-reference metrics AMBE,²⁰ CII,²⁹ QRCM,⁶⁶ and UIQ.⁶⁷ In summary, the metrics can be meaningful for the assessment of CE, whereas they cannot provide insights if the CE is beneficial for TOD recognition and possibly other classification tasks with small targets.