Demosaicking is a method to restore the mosaicked image generated by the color filter array to a full-color image. The performance of the demosaicking algorithm is crucial to the reconstruction of high-quality color images. However, current demosaicking algorithms inevitably cause zipper effects or false color artifacts in the process of demosaicking. We combine low-rank constraint based on spectral domain with image non-local similarity to correct the interpolation errors. Our method is applicable to various existing demosaicking algorithms and can further improve the quality of full color image reconstruction. Experimental results demonstrate that the proposed algorithm can improve the color peak signal to noise ratio of the reconstructed images.

Can we automatically learn discriminative embedding features from images when human-annotated labels are absent? The problem of unsupervised embedded learning remains a significant and open challenge in image and vision community. A joint online deep embedded clustering and hard samples mining framework are proposed to improve the representation ability of embedded learning. In addition, to enhance the discriminability of feature representations, a structure-level pair-based loss is introduced to take full advantage of structure correlation between all the mined hard samples. Finally, the quantitative results of exhaustive experiments on three benchmarks show that our proposed method performs better than existing state-of-the-art methods.

In recent years, deep learning has become prevalent to solve applications from multiple domains. Convolutional neural networks (CNNs) particularly have demonstrated state-of-the-art performance for the task of image classification. However, the decisions made by these networks are not transparent and cannot be directly interpreted by a human. Several approaches have been proposed to explain the reasoning behind a prediction made by a network. We propose a topology of grouping these methods based on their assumptions and implementations. We focus primarily on white box methods that leverage the information of the internal architecture of a network to explain its decision. Given the task of image classification and a trained CNN, our work aims to provide a comprehensive and detailed overview of a set of methods that can be used to create explanation maps for a particular image, which assign an importance score to each pixel of the image based on its contribution to the decision of the network. We also propose a further classification of the white box methods based on their implementations to enable better comparisons and help researchers find methods best suited for different scenarios.

Attention mechanism originates from the study of human visual behavior, which has been widely used in various fields of artificial intelligence in recent years and has become an important part of neural network structure. Many attention mechanism-based trackers have gained improved performance in both accuracy and robustness. However, these trackers cannot suppress the influence of background information and distractors accurately and do not enhance the target object information, which limits the performance of these trackers. We propose new Siamese point attention (SPA) networks for robust visual tracking. SPA networks learn position attention and channel attention jointly on two branch information. To construct point attention, each point on the template feature is used to calculate the similarity on the search feature. The similarity calculation is based on the local information of the target object, which can reduce the influence of background, deformation, and rotation factors. We can obtain the region of interest by calculating the position attention from point attention. Position attention is integrated into the calculation of channel attention to reduce the influence of irrelevant areas. In addition, we also propose the object attention, and integrate it into the classification and regression module to further enhance the semantic information of the target object and improve the tracking accuracy. Extensive experiments are also conducted on five benchmark datasets. The experiment results show that our method achieves state-of-the-art performance.

A simple and effective recognition system for indoor scenes is presented. The proposed system has two phases, namely, creation of mandatory and desirable objects and an indoor scene recognition system (ISRS). In the first phase, a list of probable objects and their classification, such as mandatory and desirable objects, for any generic scene is created based on real-time indoor environment clubbed with human knowledge on standard datasets. In the second phase, the proposed system contains four stages. In the first stage, the proposed ISRS identifies and recognizes the objects of the given key frame based on a simplified version of CNN architecture of YOLO v3. In the second stage, the identified objects are divided into two sets of mandatory and desirable objects with a simple dictionary look-up. In the third stage, the objects are identified to belong to a probable scene, and this technique is called scene-object identification. In the final stage, a binary scene representation (BSR) is proposed for each probable scene, and a final scene recognition is obtained with a new scene-score, obtained after converting the binary BSR into a decimal number. The effect of the proposed ISRS has been experimented with standard datasets and measured in terms of standard metrics, besides comparison with existing schemes. The results are encouraging.

Pixel-based change detection (PBCD) is imperfect because it lacks spatial correlation and can cause misdetection and salt and pepper noise. Comparatively, object-based change detection (OBCD) is dependent on the accuracy of the segmentation scale, where over-segmentation or under-segmentation of the image objects reduce accuracy. The fusion of PBCD and OBCD maps has great potential in dealing with spectral variability and texture complexity in very high-resolution (VHR) remote sensing images. It is difficult to solve the problem of uncertainty, which is caused by the inaccuracy of the multiple-change maps. Evidence theory based on Dempster–Shafer (DS) theory is an effective method for modeling uncertainty and taking advantage of multiple pieces of evidence. In this study, we proposed a scale-driven CD method incorporating DS evidence theory and majority voting rule to generate CD by combining multiscale OBCD results and PBCD results. Experiments carried out in four different regions using the Gaofen-2 imagery were used to test the proposed approach. We conducted numerous experiments to compare the proposed approach with prevalent CD approaches. Based on the results, the proposed approach achieves the best performance because it combines the benefits of pixel-based and object-based methods.

The data produced in today’s Internet and computer world are expanding their wings day by day. With the passage of time, storage and archiving of this data are becoming a significant problem. To overcome this problem, attempts have been made to reduce data sizes using compression methods. Therefore, compression algorithms have received great attention. In this study, two efficient encoding algorithms are presented and explained in a crystal-clear manner. In all compression algorithms, frequency modulation is used. In this way, the characters with the highest frequency after each character are determined and the Huffman encoding algorithm is applied to them. In this study, the compression ratio (CR) is 49.44%. Moreover, 30 randomly selected images in three different datasets consisting of USC-SIPI, UCID, and STARE databases have been used to evaluate the performance of the algorithms. Consequently, excellent results have been obtained in all test images according to well-known comparison algorithms such as the Huffman encoding algorithm, arithmetic coding algorithm, and LPHEA.

Generative adversarial network (GAN) for super-resolution (SR) has attracted enormous interest in recent years. It has been widely used to solve the single-image super-resolution (SISR) task and made superior performance. However, GAN is rarely used for video super-resolution (VSR). VSR aims to improve video resolution by exploiting the temporal continuity and spatial similarity of video sequence frames. We design a GAN with multi-feature discriminators and combine it with optical flow estimation compensation to construct an end-to-end VSR framework OFC-MFGAN. Optical flow estimation compensation makes use of temporal continuity and spatial similarity features of adjacent frames to provide rich detailed information for GAN. Multi-feature discriminators based on visual attention mechanism include the pixel discriminator, edge discriminator, gray discriminator, and color discriminator. GAN with multi-feature discriminators makes the data distribution and visually sensitive features (edge, texture, and color) of SR frames similar to high-resolution frames. OFC-MFGAN effectively integrates the time, space, and visually sensitive features of videos. Extensive experiments on public video datasets and surveillance videos show the effectiveness and robustness of the proposed method. Compared with several state-of-the-art VSR methods and SISR methods, the proposed method can not only recover prominent edges, clear textures, and realistic colors but also make a pleasant visual feeling and competitive perceptual index.

Sparse approximation is critical to the applications of signal or image processing, and it is conducive to estimate the sparse signals with the joint efforts of transformation analysis. A simultaneous Bayesian framework is extended for sparse approximation by structured shared support, and a simultaneous sparse learning algorithm of structured approximation is proposed with transformation analysis, which leads to the feasible solutions more sensibly. Then the improvements of sparse Bayesian learning and iterative reweighting are embedded into the framework to achieve rapid convergence and high efficiency with robustness. Furthermore, the iterative optimization and transformation analysis are embedded in the overall learning process to obtain the relative optima for sparse approximation. Finally, compared with conventional reweighting algorithms for simultaneous sparse models with l₁ and l₂, simulation results present the preponderance of the proposed approach to solve the sparse structure and iterative redundancy in processing sparse signals. The fact indicates that the proposed method will be effective to sparsely approximate the various signals and images, which does accurately analyze the target in optimal transformation. It is envisaged that the proposed model could be suitable for a wide range of data in sparse separation and signal denoising.

The input of the iris classification network based on deep learning has two forms: one is the coarsely located iris region of interest; the other is the normalized iris. To solve the problem of whether it is necessary to normalize the iris, experiments are carried out on the above two input forms, and the results show that the iris normalization is still the best choice. To adapt to the visual characteristics of the neural network, an iris normalization processing method is proposed: starting from 90 deg, the iris circle is mapped to polar coordinates and the normalized rectangle is cropped, rotated, and spliced. Experimental and visualization results show that the proposed iris normalization strategy has better results of iris recognition than other normalization methods.

Deep neural networks have been successfully used in image processing and have benefitted image compression technology. On this basis, deeper and more extensive use of compression algorithms in remote sensing applications was investigated in our study. As a general residual block does not effectively utilize the connections between channels and spatial regions in the reconstruction process, an attention mechanism codec framework based on the channel-spatial attention residual block was introduced. Consequently, the network can account for the interdependence between channels and enhance specific spatial areas. Simultaneously, the feature importance map makes the network focus on valuable information, especially for small objects and remote sensing images with many tiny details, which is crucial to describe their contours. Introducing the attention mechanism commonly used in target detection can effectively recover the missing detailed information for remote sensing compressed images. Moreover, to utilize the underlying information of the various scene categories and characterize the feature differences between scenes in remote sensing images, an adaptive scene-aware module was introduced to enhance the feature representation in remote sensing images of different scenes. The experimental results show that the proposed method has superior subjective and objective effects in remote sensing image compression tasks.

We present a fast block partitioning scheme for chrominance intra prediction of the Versatile Video Coding (VVC) standard. VVC adopted several innovative tools to improve the coding performance compared with its predecessor standard, enabling the efficient handling of emerging video content such as ultra-high definition (UHD). These coding performance improvements come at the cost of a significant increase in coding complexity, mainly due to the new block partitioning structure called quadtree with nested multi-type tree (QTMT), which can be applied separately for luminance and chrominance components. Thus, this work proposes a scheme focusing on chrominance intra coding, composed of chrominance splitting early termination based on luminance QTMT and fast chrominance split decision based on the sub-block variances. Experimental results demonstrated that the proposed scheme provides 60.03% and 8.18% of chrominance and total encoding time savings, respectively, with a negligible impact of 0.66% in the YUV-BDBR.

For a damaged image, the loss of pixel information can be roughly divided into two categories, random missing and non-random missing. The missing of an entire row or column of the image is a specific structural missing pattern that is extremely difficult to deal with. Although most of the existing methods have partially fixed this information missed problem, the diffusion-based methods tend to produce blur, the exemplar-based methods are prone to error filling, and the neural network-based methods are highly dependent on data, which cannot handle this special structural missing very well. Using the nonlocal self-similarity prior and the low-rank prior, we present multidirectional search and nonlocal low-rank tensor completion (MS-NLLRTC) algorithm based on the tensor ring (TR) decomposition and multidirectional search (MS). The MS method is a newly proposed method that can search similar patches much more diversified. Using MS method, we directly stack the similar patches into a three-dimensional similar tensor instead of pulling them into column vectors, then the similar tensor can be completed by TR decomposition. The optimization results can be obtained by leveraging the alternating direction method under the augmented Lagrangian multiplier framework. Moreover, we add a weighted nuclear norm to the tensor completion model (WNLLRTC), achieving a better inpainting performance. We also combine a noise removal method with WNLLRTC algorithm, which can handle image random missing and image noise removal simultaneously. Experimental results indicate that our proposed algorithms are competitive with some state-of-the-art inpainting algorithms in terms of both numerical evaluation and visual quality.

Low light image enhancement is a challenging task, and it has been a hot research topic. Inspired by retinex theory and U-Net network, we propose a U-Net-based multiscale feature preserving method for low light image enhancement, which can realize the extraction of low-level features and high-level semantic features. Before feature extraction, we carry out multiscale pre-extraction processing on the image to improve the feature extraction ability of the network. Considering the discontinuity between low-level features and high-level semantic features, we propose a spatial consistency method to maintain the global feature correlation. Finally, we propose a new multiscale structure calculation method, which greatly alleviates the phenomenon of uneven illumination and color deviation after enhancement and makes the enhancement results more consistent with human visual perception. Extensive experiments demonstrate that compared with other advanced enhancement methods, our method has better enhancement effect and can retain more details. The enhanced image not only has good visual perception but also is better than other methods in objective evaluation.

Pedestrian-related accidents are more frequent at night when visible (VI) cameras are inefficient. Compared with VI cameras, thermal cameras work better in this particular environment. Conversely, thermal images have several drawbacks, such as high noise, low-resolution, less detailed information, and susceptibility to ambient temperature. To overcome these shortcomings, an improved algorithm based on you only look once version 3 (YOLOv3) is proposed. First, the number and size of anchors are obtained using k-means++, which makes the shape of the anchors more suitable for detecting the target. Second, the attention module is added to the backbone network, which is helpful with extracting better feature maps from low-quality thermal images. Finally, the improved atrous spatial pyramid pooling module is added to the back of the backbone network to enable the extracted feature maps to contain more multi-scale information and context information. Experiments on the computer vision center-09 dataset show that the average precision is 86.1%, which is 3.5% higher than YOLOv3 and 0.8% higher than YOLOv4. The detection speed reaches 48 FPS. The results show that the improved algorithm has good accuracy and generalization.

We propose a photorealistic style transfer network to emphasize the natural effect of photorealistic image stylization. In general, distortion of the image content and lacking of details are two typical issues in the style transfer field. To this end, we design a framework employing the U-Net structure to maintain the rich spatial clues, with a multi-layer feature aggregation (MFA) method to simultaneously provide the details obtained by the shallow layers in the stylization processing. In particular, an encoder based on the dense block and a decoder form a symmetrical structure of U-Net are jointly staked to realize an effective feature extraction and image reconstruction. In addition, a transfer module based on MFA and “adaptive instance normalization” is inserted in the skip connection positions to achieve the stylization. Accordingly, the stylized image possesses the texture of a real photo and preserves rich content details without introducing any mask or postprocessing steps. The experimental results on public datasets demonstrate that our method achieves a more faithful structural similarity with a lower style loss, reflecting the effectiveness and merit of our approach.

Hyperspectral image (HSI) contains both spatial pattern and spectral information, which has been widely used in food safety, remote sensing, and medical detection. However, the acquisition of HSIs is usually costly due to the complicated apparatus for the acquisition of optical spectrum. Recently, it has been reported that HSI can be reconstructed from single RGB image using convolution neural network (CNN) algorithms. Compared with the traditional hyperspectral cameras, the method based on CNN algorithms is simple, portable, and low cost. In this study, we focused on the influence of the RGB camera spectral sensitivity (CSS) on the HSI. A xenon lamp incorporated with a monochromator was used as the standard light source to calibrate the CSS. And the experimental results show that the CSS plays a significant role in the reconstruction accuracy of an HSI. In addition, we proposed a new HSI reconstruction network where the dimensional structure of the original hyperspectral datacube was modified by 3D matrix transpose to improve the reconstruction accuracy.

In today’s era, digital data are being created and transmitted primarily in the form of images and videos. Storing such a huge number of images and transmitting them requires a lot of computer resources such as storage and bandwidth. So, instead of storing the image data as is, if we compress and store it, it saves a lot of resources. Image compression is the act of removing the maximum possible redundant data from an image and maintaining only the non-redundant data. To compress and decompress such big image data, a distributed environment with a map-reduce paradigm using Hadoop distributed file system and Apache Spark is used. In addition to these, Microsoft Azure cloud environment with infrastructure as a service is also used. Various setups such as a single system, 1 + 4 node cluster, 1 + 15 node cluster, and 1 + 18 node cluster cloud infrastructure are used to show the time comparisons among these setups with the self-created large image dataset. On these four self-made clusters, more than 100 million (109,670,400) images are compressed and decompressed; the execution times are compared with two of the traditional image compression methods: Lempel-Ziv-Welch (LZW) and Huffman coding. Both the LZW and Huffman coding are lossless image compression techniques. LZW removes both spatial and coding redundancies and whereas the Huffman coding removes only coding redundancy. These two compression techniques: LZW and Huffman are just placeholders, these can be replaced with any other compression technique for large image data. In our work, we have used compression ratio, average root mean square error (ARMSE), and average peak signal to noise ratios to validate that the compression and decompression process for each technique is exactly the same irrespective of the number of systems used, distributed or not.

Multifocus image fusion technology uses a mathematical model to integrate multifocus areas to obtain full-focus, clear images. The fusion method based on convolutional sparse representation (CSR) trains and “learns” translation-invariant filters, thereby addressing the missing signal infrastructure and high redundancy of the patch-based method. Convolutional dictionary learning and CSR rely on the alternating-direction method of multipliers and ignore model matching between the training and testing phases, leading to convergence difficulties due to the tricky parameter tuning. The block proximal extrapolated gradient method using the majorization and gradient-based restarting scheme (reG-BPEG-M) adopts the driving force coefficient formula and adaptive restart rule to solve the model mismatch problem. We introduce reG-BPEG-M into multifocus image fusion to update filters and sparse code using two-block and multiblock schemes. Compared with other state-of-the-art fusion methods, our strategy reduces model mismatch and improves the convergence of fusion for gray and color multifocus images.

We propose a method that uses the visibility feature extraction module and support vector machine (VFEM SEM) classification algorithm for visibility estimation. First, five statistical image features are extracted to fit a multivariate Gaussian (MVG) model; then, the visibility indexes and Mahalanobis distance are calculated from the MVG model and the reference samples, which are selected from the highest and lowest visibility levels, respectively. Finally, the SVM classifier is trained for visibility classification. Due to the lack of real scene foggy datasets, we collect a dataset called foggy highway from highway surveillance scenes. To evaluate the proposed algorithm’s performance fairly and objectively, we synthesize the foggy virtual KITTI image dataset with continuous annotations using the atmospheric scattering model and use another public synthetic dataset called “foggy road sign images.” Comprehensive experiments are conducted under three datasets. The experimental results show that the proposed method has achieved better classification accuracy compared with the other traditional image processing methods. Moreover, compared with deep learning-based methods, the proposed method has also achieved competitive performance and requires much less computational resources and image data. All of these advantages make the proposed algorithm more suitable for mobile device deployment.

High-resolution (HR) synthetic aperture radar (SAR) images play an important role in people’s daily life and military applications. However, due to the interference of speckle noise, the texture details of the SAR images become relatively blurred. The fine texture details can be reconstructed by increasing the resolution of the SAR images. Generative adversarial networks achieve high performance in image super-resolution (SR) reconstruction, but the existing generative adversarial networks only pay attention to the discrimination of HR images without that of the low-resolution (LR) images. If the reconstructed HR image is sufficiently realistic, the LR image obtained from downsampled super-resolved images should also be the same as the original LR image. To take advantage of the LR image, an SAR image SR reconstruction algorithm based on cross-resolution discrimination (CRD) using teacher–student network is proposed. First, the teacher discriminator network (TD-Net) discriminates the HR images, which enriches the reconstructed HR images with more high-frequency texture details. Second, the student discriminator network (SD-Net) discriminates the LR images, which enables the reconstructed HR images to be accurately downsampled to the original LR image. Finally, the TD-Net guides the training of the SD-Net by transmitting distillation knowledge to the SD-Net, which further improves the discriminative performance of the SD-Net. Experiments on the SAR image dataset demonstrate that the performance of the proposed CRD algorithm is better than other algorithms when both the objective evaluations and subjective effects are considered.

The problem of distribution of force, having the frictional contact between the passive object (manipulating) and active fingers (end-effector) of Delta robots, has been discussed. Computation of force components acting on the object’s points, given the wrench acting on the object and the contact points with respect to the body frame (estimated from the vision sensor) using combined parallel robot architecture, is performed. A pre-calibrated vision sensor was used for the pose estimation of the object. The proposed architecture design is intended to carry heavy payload objects for catching in the real-time scenario.

To effectively detect the surface cracks of subway tunnels, an automatic tunnel crack detection system based on machine vision is presented. Aiming at the problems of environmental complexity and low contrast in subway tunnels, the image texture feature is first enhanced by the methods of frequency domain filtering and spatial differencing. Then, depending on the characteristics of the tunnel cracks in question, the crack propagation method is used to extract the complete cracks. Finally, broken cracks are connected during processing, and the method of combining projection and threshold is used to determine the crack types. At the same time, characteristics such as the length, width, and area of the cracks are obtained. The experimental results show that the presented methods can effectively extract complete cracks in complex tunnel environments. The identification error of tunnel crack parameters meets the actual engineering requirements.

Recently, the long short-term memory network (LSTM) and attention mechanism have greatly boosted the research of video-based action recognition. For this task, feature extraction especially temporal feature extraction is essential. However, most studies focus on improving the temporal feature extraction ability of the model, ignoring the lack of temporal information in the input. To alleviate the issue above, we propose multi-views reinforced LSTM (MR-LSTM). First, we propose an innovative feature extractor named multi-views temporal feature extractor (MTFE) to extract multi-views temporal features from RGB frames in different views. Secondly, we propose multi-views reinforced attention (MRA) mechanism, which utilizes multi-views features to enrich the temporal information in the input of LSTM. MTFE and MRA mechanisms alleviate the lack of temporal information in the input of LSTM. Equipped with the modules above, LSTM can extract more discriminative temporal features. Finally, we propose non-fair fusion strategy to obtain more discriminative fusion features that are beneficial for classification. The ablation experiment demonstrates the effectiveness of all proposed modules. In comprehensive experiments on UCF101 and HMDB51 datasets, our architecture performs competitively against state-of-the-art methods.

The segmentation of a point cloud on the roof plane is of great significance to the reconstruction of building models. However, the traditional segmentation methods segment the aerial point cloud of the roof, which cannot fully express the geometric structure of the roof, whereas the deep learning-based methods have problems such as too much manual annotation and training time. In this work, a plane segmentation method for a building roof based on the PointNet network combined with the random sample consensus (RANSAC) algorithm is proposed to directly segment the whole point cloud of the building, but it is not limited to the point cloud of the roof. With the proposed framework, the roof part is extracted from the building by an improved PointNet network, and then the roof semantic point cloud is segmented by the RANSAC algorithm to complete the roof extraction. Based on the experimental results gained from multiple building point clouds, it is shown that the proposed method achieves the segmentation of a roof on most multi-plane roof building point clouds and that it has strong practical value.

Image registration algorithm is the process of establishing a mapping between image in a sequence and another overlapping one. Sufficient overlap ratio in the frames of the sequence is a necessary condition to find that mapping. Image mosaic is one of the important applications for which image registration may be used. Also, object tracking is another interesting image processing task that deals with finding mapping relation between successive images. Local features such as points are generally used to simplify images processing through taking significant pixels in the image and applying some operations for getting information about the whole image. Several techniques were proposed for finding corresponding features between images; among which descriptors-based matching is the most efficient approach. The suggested floating-points descriptors such as scale invariant feature transform and speed up robust feature are generally good in term of performance but require a long time; in most cases, they imply refinement methods for removing false matches, thus increasing the time cost. Therefore, we propose a fuzzy matching method that combines two binary descriptors, which are local binary patterns and Fast REtinA key point (FREAK) for image registration purpose. Our algorithms are tested on Matlab for different real scenes and satisfying results were obtained. To make program execution simple and to easily do numerical comparisons, a graphical interface has been implemented on LabVIEW platform.

Image quality assessment (IQA) is widely used in image transmission, processing, and storage, which has important application value. To obtain an IQA model with the excellent performance, combining with the image content features and human visual system characteristics, based on the contrast definition in visual psychology, an IQA method and its mathematical model are proposed. In this method, first based on the contrast definition, combining with the visual characteristics, the contrast between the distorted image and the reference image is used to describe the difference between them; further, based on the difference, a definition method for image quality is proposed. Second, based on the gray and gradient co-occurrence matrix, a concept, which is called as the image gray-gradient expectation (GGE), is proposed, and its calculation method is also illustrated. And based on the GGE values and local contrast of image, a method for describing the image content and its visual perception is proposed. Finally, based on the image content features and the proposed definition method of image quality, an IQA method and its mathematical model are proposed. Further, they were tested using the 142 reference images and 7220 distorted images in the LIVE, CSIQ, TID2008, TID2013, CIDIQ, and IVC databases. And the results were compared with those of seven existing typical IQA models in terms of accuracy, complexity, and generalization performance. These experimental results show that, in the six databases, the IQA accuracy Pearson linear correlation coefficient of the proposed model can all be more than 0.7892 and reach 0.9638 highest, whose comprehensive efficiency is better than ones of the seven existing IQA models. These analysis and comparison show that the proposed model is an excellent IQA model on performance.

The dense connection is a powerful technique to build wider and deeper convolution neural networks (CNNs) for handling several computer vision tasks. Despite the excellent performance, it consumes numerous parameters and produces a large weight model file. We studied the distribution of convolution layers and proposed a hard parameter sharing approach known as convolution pool (CP) for compressing dense-connection-based image restoration CNN models. CP is used to reallocate the parameters to specific convolution layers to ensure that some can be shared in different layers. We design a set of dense-connection-based baselines for three typical image restoration tasks, including image denoising, super-resolution, and JPEG deblocking, to validate the performance of the proposed method. Moreover, we comprehensively analyze the potential problems by introducing CP, including group convolution, dilated convolution, and modeling efficiency. Experimental results demonstrate that the proposed method can efficiently achieve an impressive compression rate with negligible performance reduction.

Video quality assessment (VQA) is an important technique in video service systems. In recent years, the development of deep learning has provided further possibilities for VQA. A no-reference VQA (NR-VQA) method that combines the attention mechanism and human visual perception is proposed for in-the-wild videos. First, a deep network consisting of a convolutional neural network and attention mechanism is constructed to extract depth perception features for frame-level images, and global covariance pooling is integrated into the downsampled features to extract the second-order information of the features. Second, a Transformer network is used for temporal modeling to learn the long-term dependence of the perceptual quality prediction. Finally, a temporal weighting strategy for visual perception is used for weighted summation of the frame-level scores to obtain the final video quality scores. The results of experiments on three public user-generated content authentic distorted video databases, namely KoNViD-1k, CVD2014, and LIVE-VQC, demonstrate that the proposed method can achieve effective quality assessment in authentic distortion and outperforms other partially recent NR-VQA methods.

A large amount of training data (including samples and labels) are required to achieve the task of object detection. However, the construction of databases generally has several problems. The samples are not balanced. In some scenarios, very few or even no samples can be collected; manual collection and labeling are expensive. Obtaining samples from public object detection data sets is simple, but the categories that can be obtained are limited and cannot meet the needs of other detection tasks. Therefore, we propose a method to automatically construct a flexible object detection database and complete automatic labeling. Our main advantage lies in the proposed synthetic region screening and adaptive feature fusion algorithm, which generates sample images and corresponding annotation files that are similar to the real collection. Our algorithm has a stronger and more realistic synthesis ability for three-dimensional objects than other data synthesis methods. Our method improves the system’s object detection accuracy in the following ways: (1) it synthesizes a wider range of images for certain categories with few samples in the data set to solve the problem of sample imbalance; (2) it adds new categories to the public data set to meet the needs of rapid deployment of special object detection tasks; and (3) it fuses different foregrounds and backgrounds to enrich the diversity of database samples. To effectively evaluate the proposed method, we conducted experiments separately, and the experimental results proved that our method is superior to other existing data synthesis and data enhancement methods. Combining the generated samples with the real data set, the object detection accuracy value increased from 9.47% to 58.92%. Extending the objects of the self-collected data set to the public data set and automatically generating high-quality annotations, the detection accuracy of extended objects reached more than 50%.

To address the tracking challenges such as weak feature expression ability of small targets and susceptibility to interference by similar objects in complex backgrounds, we use the principle of feature enhancement in the field of small target detection to redesign the backbone network of the siamese network tracker and propose a small target tracking algorithm based on a bidirectional feature pyramid fusion framework. The algorithm first constructs a deep feature pyramid with semantic and contextual information in the second half of the backbone network using successive deconvolution; then, considering that tracking also has a strong dependence on shallow information, we construct a shallow feature pyramid with location and spatial structure information in the first half of the backbone network as a complement to the deep feature pyramid, which is used to repair small target internal structure details and enhance its localization ability. Finally, the features processed from the deep and shallow pyramids are merged to construct a bidirectional pyramid fusion framework, while a self-attention mechanism is introduced to treat each type of information after fusion in a targeted manner, and the asymmetric convolution is used to lighten the fusion framework. The algorithm in this paper achieves a more advanced performance compared with existing algorithms for experiments on four publicly available datasets, GOT-10k, LaSOT, UAV123, and DTB70.

Facial expression recognition, which aims at recognizing the human expression information by cameras, has attracted more interest thanks to its significance and potential application. However, the existing expression recognition methods are based on convolution and pooling stack to expand the receptive field, ignoring the interaction of the pixel in the feature maps. To solve this problem, we propose a disentangled-region non-local (DRNL) neural network, which can capture the long-range dependencies directly by calculating the interaction between the pixel and the region, not limited to the adjacent pixels, to maintain more information. At the same time, we decouple the DRNL block into two terms to extract clearer visual features, one of which is a whitened paired term to model the relationship between pixels and regions, and the other is a unary term to represent the saliency information of each pixel. We evaluate the proposed network on two public datasets, including the real-world affective faces database (RAFDB) and the static facial expressions in the wild (SFEW), and show the performance using the visualization method. The accuracy rates of our method achieve 90.450% on RAFDB and 63.855% on SFEW, respectively. Abundant experiments demonstrate state-of-the-art performance by comparing with the previous methods.