Most of the methods surveyed for detecting moving objects through image processing in general purpose processors (GPPs) presented an approach that did not allow successful implementation in field programmable gate array (FPGA). The correct classification of the pixels in these methods is directly related to the need for many resources in terms of memory due to the complex mechanism to get the reference image. Additionally, almost all the methods analyzed used only conventional techniques of digital image processing (DIP). Thus, we propose an approach that combines conventional techniques of DIP with fuzzy integral in a parallel processing aiming to improve the detection of moving vehicles. Hence, the proposed decision-making system seeks not to compromise results in terms of pixel classification even by implementing a simple modeling mechanism. Results in terms of occupied resource, maximum operating frequency, and pixel classification are verified in low-cost FPGA-based implementation. The proposed system processes the images in real-time and presents better pixel classification than just fuzzy decision-making.

Underwater images suffer from various degradation factors, such as blur, haze, color degradation, and marine snow. Marine snow is a type of noise, caused mostly by biological particles that fall into the ocean bottom, and which impedes proper object detection in underwater vision. A method for real-time marine snow removal from underwater color and monochrome video is presented. It is based on the proposed marine snow model, spatiotemporal patch analysis, and three-dimensional median filtering. The method was evaluated on a number of real underwater sequences endowed with the hand-annotated ground-truth data which were made available from the Internet. As shown by the experiments, the method attains high accuracy and performs in real time.

Illumination estimation is an important part of color constancy. It is known that estimating the scene illumination color is an ill-posed problem, and no method can be considered as universal. To select an optimal method for a particular image or scene, some combination methods based on characteristic similarity were proposed. These methods found that the images having similar characteristics can use the same method as an optimal technique. Although the combination methods based on image characteristics have worked well in some scenes, the accuracy of these methods is limited to the accuracy of the set of the unitary method. However, we assume that images with similar characteristics have similar scene illumination color and propose an illumination estimation method which is based on image characteristics. According to the characteristics of each image, we search K images whose characteristics are similar to the image from the image dataset, and the standard illumination color of the K selected images is known. Then, we use the weighted average method to estimate the new image’s illumination color by combining the standard illumination color of the K selected images. The experimental results show that the proposed method outperforms other state-of-the-art methods.

A low-resolution depth map can be upsampled through the guidance from the registered high-resolution color image. This type of method is so-called guided depth map upsampling. Among the existing methods based on Markov random field (MRF), either data-driven or model-based prior is adopted to construct the regularization term. The data-driven prior can implicitly reveal the relation between color-depth image pair by training on external data. The model-based prior provides the anisotropic smoothness constraint guided by high-resolution color image. These types of priors can complement each other to solve the ambiguity in guided depth map upsampling. An MRF-based approach is proposed that takes both of them into account to regularize the depth map. Based on analysis sparse coding, the data-driven prior is defined by joint cosparsity on the vectors transformed from color-depth patches using the pair of learned operators. It is based on the assumption that the cosupports of such bimodal image structures computed by the operators are aligned. The edge inconsistency measurement is explicitly calculated, which is embedded into the model-based prior. It can significantly mitigate texture-copying artifacts. The experimental results on Middlebury datasets demonstrate the validity of the proposed method that outperforms seven state-of-the-art approaches.

We propose a random-forest scheme, namely joint maximum purity forest (JMPF), for classification, clustering, and regression tasks. In the JMPF scheme, the original feature space is transformed into a compactly preclustered feature space, via a trained rotation matrix. The rotation matrix is obtained through an iterative quantization process, where the input data, inclined to different classes, is clustered to the respective vertices of the feature space with maximum purity. In the feature space, orthogonal hyperplanes, which are employed at the split nodes of the decision trees in a random forest, can effectively tackle the clustering problems. We evaluated our proposed method on public benchmark datasets for regression and classification tasks, and experiments showed that JMPF remarkably outperforms other state-of-the-art random-forest-based approaches. Furthermore, we applied JMPF to image super-resolution (SR) specifically because the transformed, compact features are more discriminative to the clustering-regression scheme. Experimental results on several public datasets also showed that the JMPF-based image SR scheme is consistently superior to recent state-of-the-art image SR algorithms.

The paper presents an original hybrid image reconstruction algorithm ART-TVS for few-views computed tomography of strongly absorbing media. It is based on the well-known algebraic reconstruction technique (ART), regularization of interim results through minimization of the total variation norm (TV-regularization), and a method of adaptive segmentation, which is a modernization of the known region growing algorithm. It is shown that the ART-TVS algorithm does not give stripe artifacts even if the number of views is very small (eight or less). ART-TVS reconstruction results for two numerical models of metal shells are compared with those obtained with the ART-TV algorithm (ART with TV-regularization and without adaptive segmentation), the iterative Potts minimization algorithm (IPMA), and our MART-AP algorithm (multiplicative ART with a priori information) we developed earlier for few-views discrete tomography. It is shown that ART-TVS outperforms ART-TV and IPMA and is comparable with MART-AP in reconstruction accuracy. Also, ART-TVS converges markedly faster than IRMA in cases where strongly underdetermined systems are treated. The algorithm we propose also demonstrates quite satisfactory resistance to projection data noise that is inherent in tomography of strongly absorbing media.

Convolutional neural networks have been recently shown to have the highest accuracy for single-image super-resolution reconstruction. A multibranch prediction and selection network that can gradually reconstruct robust images in multiple scales is proposed. This endeavor is achieved through a network structure with two interacting subnetworks: one is a deep cascaded, multibranch prediction network (DCMBPN) and another is a deep block-selection network (DBSN). In particular, in each cascade, DCMBPN predicts multiple reconstructed images progressively with its special multibranch and cascaded structure. DBSN then adaptively selects the predicted confident blocks from these reconstructed images. Our method does not require traditional interpolation methods to upsample the image as a preprocessing step. It, thus, greatly reduces the computational complexity. We use Euclidean and perception loss functions in each branch to obtain two high-quality reconstructions. In addition, for the cascade structure, our network can achieve reconstructions in different scales, such as 1.5  ×  , 2  ×  , 2.5  ×  , 3  ×  , 3.5  ×  , and 4  ×  . Extensive quantitative and qualitative evaluations on benchmark datasets show that the proposed algorithm performs favorably against the state-of-the-art methods in terms of accuracy and visual improvement.

Designing effective local descriptors is crucial for many computer vision tasks such as image matching and patch verification. We propose a convolutional neural network (CNN)-based local descriptor named DHNet with a considerate sampling strategy and a dedicated loss function. By considerate sampling, both the closest nonmatching sample and the farther matching sample can be obtained for effectively training a discriminative model. In addition, an improved triplet loss is designed by adding a constraint that limits the absolute distance for the closest nonmatching pair. Based on hard samples and the constraint, our lightweight CNN can quickly generate local descriptors with enhanced intraclass compactness and interclass separation. Experimental results show that our method significantly outperforms the state-of-the-art methods in terms of strong discrimination ability, as evidenced by a considerable performance improvement on several benchmarks.

Visual saliency is used to guide the coding tree unit (CTU)-level bit allocation process in high-efficiency video coding (HEVC) to improve the visual quality. At first, a saliency detection algorithm is proposed. With the detected saliency map, the distortion of each CTU is weighted by the corresponding saliency, so that the distortion of the salient areas (SAs) is more critical. Then, the optimal bit allocation problem constrained by the picture-level target bits and minimum weighted distortion is built. Numerical method is used to solve the bit allocation problem. Finally, bits reallocation and the rate distortion optimization parameters clipping methods are presented to tackle the bits error accumulation problem. Experiment results show that quality gaining in SAs is up to 0.8658 dB and the gaining of saliency weighted peak signal-to-noise ratio is up to 1.0318 dB.

A learning-based multibranch deep probability network is proposed to estimate the illuminated color of the light source in a scene, commonly referred to as color constancy. The method consists of two coupled subnetworks, which are the deep multibranch illumination estimating network (DMBEN) and deep probability computing network (DPN). The one branch of DMBEN estimates the global illuminant through pooling layer and fully connected layer, whereas the other branch is built as an end-to-end residuals network (Res-net) to evaluate the local illumination. The other adjoint subnetwork DPN separately computes the probabilities that results of DMBEN are similar to the ground truth, then determines the better estimation according to the two probabilities under a new criterion. The results of extensive experiments on Color Checker and NUS 8-Camera datasets show that the proposed approach is superior to the state-of-the-art methods both in efficiency and effectiveness.

Many existing methods effectively remove impulse noise from images, but they usually perform well only in a certain noise level (i.e., either low or high contamination). An impulse noise detection technique that is based on mean shift filtering and an effective vector filtering method that is based on channel suppression processing are proposed. The proposed denoising method excellently suppresses impulse noise in color images and performs well in both low and medium noise contamination cases. First, the noise detection method is performed to divide image pixels into noisy and possibly noise-free ones. Then, for the noisy pixels detected, according to the contamination levels, different denoising strategies are employed. For slightly corrupted images, the proposed channel-suppressed vector filter is performed, whereas for medially and highly contaminated images, a total variation denoising technique is applied, followed by a further impulse noise detection and channel-suppressed filtering. Finally, for the possible noise-free pixels, fine noise detection and channel-suppressed filtering are performed. Extensive simulation results exhibit the validity of the proposed solution by showing clear performance improvements over other widely used color image filtering methods.

Two-dimensional discriminant locality preserving projection (2DDLPP) is an effective method for image feature extraction. However, original 2DDLPP is based solely on the Euclidean distance, which is sensitive to noises and illumination changes in images. To overcome this drawback, we propose a method named nuclear norm-based two-dimensional discriminant locality preserving projection (NN2DDLPP). In NN2DDLPP, two optimal neighbor graphs are first built. Then the nuclear norm-based between-class scatter and within-class scatter are defined. Finally, in order to obtain an optimal projection matrix, the ratio of between-class scatter to within-class scatter is maximized. Using nuclear norm metric and labeled information, NN2DDLPP can both efficiently extract the discriminative features and improve the robustness to illumination changes and noises. Experiments carried out on several different face image databases validate that NN2DDLPP is efficacious for face recognition and better than other related works.

The second version of the high-efficiency video coding standard introduced a scalable extension, which is called scalable high-efficiency video coding. It is necessary to design appropriate rate controllers for this extension based on the specific application requirements. According to delay and buffering constraints, rate control algorithms are divided into constant bit rate and variable bit rate (VBR) algorithms. Our focus is on high-delay applications like streaming and broadcast for which VBR controllers are suitable. Our proposed VBR controller utilizes a fuzzy logic-based algorithm to control long-term average bit rate while preventing buffer overflow or underflow. Targeting perceptual quality, the proposed algorithm has been implemented at the group of pictures level to minimize unnecessary fluctuations of quantization parameter and consequently minimizing video quality fluctuations. Experimental results show a high performance in terms of rate control and quality of the compressed video for the proposed algorithm.

This paper proposes a saliency detection method based on the aggregated Wasserstein distance. A multidimensional Gaussian mixture model is used to model the superpixels, whereby the color information of different color spaces is combined. To overcome the lack of the closed-form solution for the Gaussian mixture model, we employ the aggregated Wasserstein distance to measure the perceptual similarity between different superpixels. The saliency value is then calculated from two aspects. First, the global saliency is computed through all the superpixels in the image using the aggregated Wasserstein distance. Second, the local saliency is computed in a lower range with the same measure. Finally, a saliency map is obtained by combining the two types of saliencies, and is filtered by spectral clustering all the superpixels. The experimental results show that the proposed method outperforms 11 recent exact algorithms on three widely used open datasets.

The detection of foreign substances in transparent-bottled liquid is a key procedure in production. An improved algorithm based on the combination of guided filter and visual background extractor (ViBe) is proposed for the detection of foreign substances in transparent-bottled turbid liquid on the market. In the stage of image preprocessing, the image enhancement is carried out by the guided filter and then the liquid region is detected by the Hough transform. Finally, the area of the connected region is determined for the image processed by the ViBe algorithm to remove the tiny noise in detection images. The simulation results show that the proposed algorithm can not only detect foreign substances in transparent liquid but also detect foreign substances in the turbid liquid.

Perceptual quality assessment of a three-dimensional (3-D) image is one of the most important tasks in various applications such as 3-D image coding, processing, enhancement, and monitoring system. But objective quality assessment of 3-D images is still a challenging task. Especially, blind quality assessment of 3-D images encounters an arduous challenge due to lack of prior information about the original images. To solve this problem, we propose a blind 3-D images’ quality evaluator by simulating binocular rivalry and orientation responses of the human visual system. As a main technical contribution of this research, both the low- and high-level binocular rivalry responses (BRR), as well as binocular orientation-tuned (BOT) responses, are considered for blind quality assessment of 3-D images. Specifically, the self-similarity of the BRR and BOT responses is extracted from the distorted 3-D images, which will change in the presence of distortions. Subsequently, all quality-aware features are mapped to subjective quality scores of the distorted 3-D images via using support vector regression. The performance of our algorithm is evaluated over two popular LIVE 3-D phase I and phase II databases and shown to be competitive with the state-of-the-art algorithms.

We performed a comprehensive study of hand–eye calibration approaches for augmented reality (AR) using endoscopes, aiming to find the approaches that yield the best performance and to reveal the mechanism that makes these approaches successful. The two unknown values in this calibration problem are the hand–eye transformation between an endoscope and the endoscope-attached optotracked marker and the transformation between a calibration board with a checker pattern and the board-attached marker. We classify possible approaches to solving hand–eye transformation as direct, simultaneous, and sequential. The effect of the translation components of transformations on an approach’s accuracy is theoretically analyzed using error equations derived from the approaches and demonstrated using both synthetic and real data. We found that sequential approaches performed the best when the magnitude of the translation of hand–eye transformation was larger than that between the board and its marker, which is the general case in implementing AR using endoscopes. In addition, this approach is less sensitive to noise and the number of calibration poses than others. Our results and analyses provide guidance for choosing an optimal hand–eye calibration solution for AR using endoscopes.

We report the results of a set of psychophysical experiments that measure the perceptual strengths of videos with different combinations of blockiness, blurriness, and packet-loss artifacts. Participants were instructed to search each video for impairments and rate the strength of their individual features (artifacts). A repeated-measure analysis of variance (RM-ANOVA) performed on the data showed that artifact physical strengths have a significant effect on annoyance judgments. We tested a weighted Minkowski model, a support vector regression model, and a linear model on the experimental data. We found that all these models give a good description of the relation between individual artifact perceptual strengths and the overall annoyance. In the other words, all models presented a very good correlation with the experimental data, showing that annoyance can be modeled as a multidimensional function of the individual artifact perceptual strengths. Additionally, results show that there are interactions among artifact signals.

The requirements of distance measurement have increased with the development of auto vehicle driving. Traditional methods for distance estimation require the complex calibration from intrinsic and external parameters of the camera. Recent methods based on the neural network structure mainly measure the relative depth of whole images. We adopt monocular vision with instance segmentation and camera focal length to detect the absolute distance of front cars from in-car cameras. First, we extract the location of the cars from the object detection network. Second, the location of cars is sent to the vehicle classification network and instance segmentation network to obtain the type of the cars and their mask value. Here, we use a model trained by the CompCars dataset to classify car types, and we train a new instance segmentation model using the Cityscapes dataset to obtain each car’s mask. Third, in accordance with the camera imaging principle, the absolute distance of cars in the images is calculated based on the relationship between the size information of different car types and their mask values. The proposed method is examined with the KITTI dataset, and the experiment shows that its results can be close to the ground truth. Moreover, the proposed method uses the instance segmentation network to reduce complexity of the depth estimation process and it can still generate satisfactory results even when cars are partly occluded.

The Viola–Jones face detection algorithm was (and still is) a quite popular face detector. In spite of the numerous face detection techniques that have been recently presented, there are many research works that are still based on the Viola–Jones algorithm because of its simplicity. We study the influence of a set of blind preprocessing methods on the face detection rate using the Viola–Jones algorithm. We focus on two aspects of improvement, specifically badly illuminated faces and blurred faces. Many methods for lighting invariant and deblurring are used in order to improve the detection accuracy. We want to avoid using blind preprocessing methods that may obstruct the face detector. To that end, we perform two sets of experiments. The first set is performed to avoid any blind preprocessing method that may hurt the face detector. The second set is performed to study the effect of the selected preprocessing methods on images that suffer from hard conditions. We present two manners of applying the preprocessing method to the image prior to being used by the Viola–Jones face detector. Five different datasets are used to draw a coherent conclusion about the potential improvement caused by using prior enhanced images. The results demonstrate that some of the preprocessing methods may hurt the accuracy of the Viola–Jones face detection algorithm. However, other preprocessing methods have an evident positive impact on the accuracy of the face detector. Overall, we recommend three simple and fast blind photometric normalization methods as a preprocessing step in order to improve the accuracy of the pretrained Viola–Jones face detector.

Relative attributes have a more detailed and accurate description than previous binary ones. We propose to utilize the acquired attribute-correlated local regions of image for learning deep relative attributes. Different from previous works, which usually discover the spatial extent of the corresponding attribute based on the ranking list of all the images in the image set, we first classify the images according to the presence or absence of each provided attribute. Then, we sort the images in the classified image sets using a semisupervised method and learn the most relevant regions corresponding to a specific attribute. The learned local regions in two classified image sets are integrated to obtain the final result. The images and localized regions are then fed into the pretrained convolutional neural network model for feature extraction. Therefore, the concatenation of the high-level global feature and intermediate local feature is adopted to predict the relative attributes. We show that the proposed method produces a competitive performance compared with the state of the art in relative attribute prediction on three public benchmarks.

This paper presents an algorithm to improve images with hazing effects. Usually, the dehazing methods based on the dark channel prior make use of two different stages to compute the transmission map of the input image. The stages are the transmission map estimation and a transmission map refinement. However, the main disadvantage of these strategies is the trade-off between accurate restoration and computational time. The proposed method uses a multilayer perceptron to compute the transmission map directly from the minimum channel and a contrast stretching technique to improve the dynamic range of the restored image. The multilayer perceptron is trained in terms of mean squared error using a training set of 80 images. To evaluate the restoration quality, the metrics of peak signal-to-noise ratio (PSNR) and the structural similarity (SSIM) index are used. The experimental results have proven that the proposed method achieves superior performance in terms of restoration quality (PSNR = 18.77, SSIM index = 0.8454) compared with nine state-of-the-art dehazing methods. In addition, based on the average computational time achieved by the proposed method (0.52 s using a test set of 46 images), it can be highly suitable for real-time applications.

Recently, some steganographic schemes, which take into account the interaction among adjacent modifications, have emerged to tackle the issue of nonadditive embedding and exhibited higher security when compared with the original additive schemes. However, the performances of these schemes are volatile and very sensitive to both the heuristic parameters and the choice of initial additive distortion functions. We proposed a model-based steganographic scheme, which incorporates the adjacent embedding information without relying on any heuristic parameters. Following the spirit of prior arts, the proposed scheme divides the cover image into several interleaved sublattices and embeds messages on each sublattice sequentially. Adjacent modifications are utilized as a priori knowledge to optimize the asymmetric change probabilities of modifying each pixel by +1 and −1. Experimental results show that the proposed scheme can rival or outperform the prior arts and, moreover, maintains a stable security performance, irrespective of any initial additive distortion or any steganalysis detector.

The local binary pattern (LBP) is a simple yet efficient texture operator, and the completed local binary pattern (CLBP) is a completed modeling for LBP that has been adopted in many texture classification methods. However, existing CLBP operators are sensitive to noise and they cannot extract the regional structure information efficiently. To overcome these disadvantages, we propose a circular regional mean completed local binary pattern (CRMCLBP) by introducing a circular regional mean operator to modify the traditional CLBP. We also present two encoding schemes for CRMCLBP. The proposed CRMCLBP not only achieves rotation invariance and completed representation capability but also has high robustness to image noise. In order to evaluate the performance, we compare the CRMCLBP with recent state-of-the-art methods by extensive experiments on two popular texture databases including Outex database and Columbia-Utrecht reflection and texture database. Excellent experimental results demonstrate that the proposed CRMCLBP is comparable with recent state-of-the-art texture descriptors and superior to other approaches for robustness.

We propose an approach for automatically colorizing grayscale images using fully convolutional networks (FCNs). In contrast to traditional colorization methods, our approach operates only on grayscale images without any manual assistance. We first build an end-to-end deep learning network based on an FCN. Global, midlevel, and local features are extracted from the network and fused to construct each deconvolutional layer. To ensure color consistency, a low-frequency regularization term is presented to maintain the coherence between neighboring pixels. We then present an improved level-set method, which we apply to the output of the FCN to repair color bleeding caused by the rough segmentation performed by the FCN. To evaluate our approach, we compare the objective image quality resulting from our method with the results of other methods by assessing the peak signal-to-noise ratio, the mean squared error, the structural similarity index (SSIM), and the multiscale SSIM (MS-SSIM). In addition, we design a Turing test to evaluate the subjective image quality. The results show that our colorized images more closely resemble the ground-truth images and are more robust than those produced via other methods.

A discriminative deep transfer metric learning method called DDTML is proposed for cross-scenario person re-identification (Re-ID). To develop the Re-ID model in a new scenario, a large number of pairwise cross-camera-view person images are deemed necessary. However, this work is very expensive due to both monetary cost and labeling time. In order to solve this problem, a DDTML for cross-scenario Re-ID is proposed using the transferring data in other scenarios to help build a Re-ID model in a new scenario. Specifically, to measure distribution difference across scenarios, a maximum mean discrepancy based on class distribution called MMDCD is proposed by embedding the discriminative information of data into the concept of the maximum mean discrepancy. Unlike most metric learning methods, which usually learn a linear distance to project data into the feature space, DDTML uses a deep neural network to develop the multilayers nonlinear transformations for learning the nonlinear distance metric, while DDTML transfers discriminative information from the source domain to the target domain. By bedding the MMDCD criteria, DDTML minimizes the distribution divergence between the source domain and the target domain. Experimental results on widely used Re-ID datasets show the effectiveness of the proposed classifiers.

Many approaches for effective object tracking have been proposed in the literature, among which the sparse representation has been a successful method for finding the best candidate with the minimal reconstruction error. We propose a generic object tracking algorithm using the sparse representation of the target’s local patches. Sparse codes are adopted as a confidence measure to avoid drifting from the target during tracking. Experiments demonstrate that mentioned confidence measure can specify appearance change of the target accurately. Furthermore, given this measure, we propose a double search scheme for tracking targets. By using this approach, the proposed tracker can work with fewer particles than its rivals and as a result does not suffer from the extracomputation of redundant particles in a scene with no particular challenge. Moreover, a simple yet effective online template update approach is adopted in order to overcome the challenges such as abrupt illumination variation, the manifestation of occlusion, blurriness, or sudden movements of the target. Both quantitative and qualitative evaluations on a challenging dataset demonstrate a favorable performance compared to several state-of-the-art algorithms in terms of accuracy and robustness.

The effective use of context information can significantly improve the effect of object detection. This paper proposes a method to exploit the context of a hard example searched by online hard example mining for improving the detection effect for people in crowded scenes. As shown by these experiments, this method can improve the effectiveness of faster R-CNN networks on people detection with a smaller convolutional neural network model. Because smaller convolutional neural network model is used, both the running memory consumption and the computing time can be reduced. Hence, our method can be implemented more easily on embedded devices.

We present a practical, robust, and effective pipeline to compute a high-resolution (HR) image of the corneal endothelium starting from a low-resolution (LR) video sequence obtained with a general purpose slit lamp biomicroscope. An image quality typical of dedicated and more expensive confocal microscopes is achieved via software magnification by exploiting information redundancy in the video sequence. In particular, the HR image is generated from the best LR frames, obtained by identifying the most suitable endothelium video subsequence using a support vector machine-based learning approach, followed by a robust graph-based frame registration. Results on long, real sequences show that the proposed approach is fast and produces better quality images than both classical multiframe super-resolution approaches and commercial state-of-the-art mosaicing software. Only low-cost equipment is required that makes the proposed method a valid diagnostic tool and an affordable resource for medical practice in both developed and developing countries.

This paper proposes an automatic target recognition of synthetic aperture radar image by joint classification of multiresolution representations with discrimination analysis. The multiresolution representations are generated to better convey the inner discriminability of the original image. Considering the extended operating conditions (EOCs) in practical applications, it is preferred that those resolutions, which make little contributions to correct target recognition, are not used. Accordingly, the discrimination analysis is performed to select the resolutions with notably high discriminability for the present operating conditions based on their reliability levels calculated from sparse representation-based classification. Afterward, the selected resolutions are jointly classified based on joint sparse representation for target recognition. Finally, extensive experiments are conducted on the moving and stationary target acquisition and recognition dataset under both the standard operating condition and various EOCs. The results validate the effectiveness and robustness of the proposed method.

Enhancing the spatial resolution of images has always been a hotspot in digital imaging. An adaptive multiframe image super-resolution (SR) algorithm has been proposed, which suppresses noise while preserving edges simultaneously. Based upon the maximum-a-posteriori (MAP) concept, the objective function of the SR algorithm consists of a regularization term and a data error term. The proposed adaptive algorithm introduces a set of weighting coefficients, which control the contribution between the regularization term and data error term in each of the estimated high-resolution pixels. The employed coefficients are defined according to the information of neighbors of the estimated pixel. Our proposed method is robust to the Gaussian noise and its destructive effect on image quality. Visual evaluation and numerical results in both of the real and synthetic images show that the performance of the proposed method is better than the other methods.

The traditional frame accumulation technology can effectively reduce the random noise and improve the signal-to-noise ratio in the image. A frame accumulation technology with superimposed sawtooth-shaped-function can reduce the quantization noise and increase the grayscale resolution. However, in the case of continuous operating environment, it is not suitable because constant amplitude light used must be turned off for taking pure and shadowy images, which are generated by superimposed shaped light. To solve this problem, we propose an improved method named dual optical signals with different periods (DOSDP) to improve grayscale resolution of an image. The method does not need to capture signals twice. In other situations, where it is not appropriate to directly superimpose a shaped function optical signal to an object, a piece of glass is installed on the light path between the camera and the object so that the superimposed light can enter the camera through the glass reflection. Furthermore, our method allows superimposed signals to be nonlinear. Experimental results show that DOSDP method can effectively improve quality of the image.

Recent studies have shown that spatial information is necessary for hyperspectral image classification, but how to appropriately integrate the distinct spectral and spatial information is still an open issue. Meanwhile, the illposed classification problem due to the limited labeled samples also results in deterioration of recognition performance. An integration of the labeled and unlabeled samples for class discriminant information through semisupervised learning is one of the promising solutions. An integration method of combining different kinds of spectral–spatial features through a semisupervised tritraining learning scheme is proposed. In the method, the dimensionality reduction process is first conducted for a hyperspectral image to reduce the data amount. Subsequently, an extended attribute profile (EAP) is used for texture extraction and spectral–spatial feature stacking. Three different attributions including area, moment of inertia, and diagonal of the box bounding the region are calculated based on the reduced image. Finally, the three kinds of concatenated features are used as inputs of tritraining with three supervised classifiers. These classifiers are trained and refined by the labeled and unlabeled samples in the tritraining process, and an improvement in the final classification accuracy is achieved. In tritraining, to address the problem of the invalid estimation of the classifier error due to the limited labeled samples, a technique based on the posterior probabilities of samples is adopted to estimate the error. Experiments on three benchmark hyperspectral datasets indicate that the proposed method can effectively integrate the information from the spectra, texture, labeled, and unlabeled samples for land cover classification.

We aim to improve the negative-accelerated sampling framework and construct a more reasonable and effective active sampling framework by introducing the technique of reinforcement learning. Compared with traditional uncertainty-based active sampling strategy, the proposed sample selection framework consists of both a certainty metric and sample postprocessing for more precise evaluation. The certainty metric is measured by the visual classifying model, and the postprocessing module is implemented by the Q-learning algorithm to construct a compact training set for the visual module to further improve the effectiveness and efficiency of classification. Meanwhile, the parameters of the whole sampling framework are calculated adaptively instead of being set manually to improve the adaptiveness of the whole framework. Experimental results on real-world datasets show the effectiveness of the proposed framework.

Considering that the rectangular windows that are used by many super-resolution (SR) methods are not suitable for the arc regions and the running time is long, we propose an image and video SR reconstruction scheme, by which rich texture details can be better maintained and the efficiency is higher as compared with those of some state-of-the-art SR methods. In our approach, we do not use the conventional rectangle windows and interpolation technique in the Cartesian coordinates, instead, we adopt the nonlinear interpolation function with an interpolation window in the polar coordinates to perform the image and video SR reconstruction. The basic idea is first to interpolate every pixel’s intensity by the Thiele–Newton’s rational function and the Newton–Thiele’s rational function both in the polar coordinates, respectively, to get two magnified images, second to set different balance factors for different image patches according to whether they are texture patches or flat patches, and then to add these balanced patches to the two magnified image patches to get final result. We demonstrate the performance of the proposed algorithm in producing high-quality resolution as compared with the state-of-the-art methods, and its application to video sequences without algorithmic modification. Experimental results show that the proposed method achieves much better results than other methods in terms of visual effect, running time, and peak signal-to-noise ratio.

Due to the limits of frame rate sampling like time-of-flight and transmission bandwidth constraints of depth video, depth frame rate up-conversion is imperative. Most researches interpolate the depth frames using the motion information of the texture images, which require well-rectified images. They usually find dense correspondences to synthesize an in-between frame. Estimating the correspondences is sensitive to occlusion, disocclusion, and color or luminance changes. This paper proposes a phase-based method to interpolate depth frames without explicitly finding correspondences. Combining top-down phase difference preservation and bottom-up phase difference correction, varying amounts of motions are well modeled in a complex-steerable pyramid decomposition. The pixel amplitude is also shifted and interpolated based on the phase information. Finally, a depth value correction is employed to correct the inaccurate depth value. Experimental results show that the depth video interpolated by our proposed approach provides better subjective and objective qualities.

Recent studies have used deep residual convolutional neural networks (CNNs) for JPEG compression artifact reduction. This study proposes a scalable CNN called S-Net. Our approach effectively adjusts the network scale dynamically in a multitask system for real-time operation with little performance loss. It offers a simple and direct technique to evaluate the performance gains obtained with increasing network depth, and it is helpful for removing redundant network layers to maximize the network efficiency. We implement our architecture using the Keras framework with the TensorFlow backend on an NVIDIA K80 GPU server. We train our models on the DIV2K dataset and evaluate their performance on public benchmark datasets. To validate the generality and universality of the proposed method, we created and utilized a new dataset, called WIN143, for over-processed images evaluation. Experimental results indicate that our proposed approach outperforms other CNN-based methods and achieves state-of-the-art performance.

Camera calibration is a key step in three-dimensional (3-D) reconstruction; however, the calibration accuracy and stability often suffer from complicated illumination in real applications. The camera calibration method using the checkerboard pattern is improved for high-precision calibration under complicated illumination. First, an improved Harris corner detector based on color constancy, as well as a subpixel optimization method based on prior knowledge of the checkerboard, are proposed to overcome the influence of bad lighting. Second, a checkerboard pattern with a central circle point is designed for better reference point location for uneven illumination. The checkerboard identification by Delaunay triangulation is also improved for the situation of discontinuity caused by over-saturation. Finally, the camera parameter optimization process is improved to reduce the calibration error produced by complicated illumination. Experimental results show that the improvements achieve more accurate and stable calibration results in complicated lighting conditions compared with traditional methods.

In video coding, the Lagrange-based rate distortion optimization (RDO) plays important roles in improving coding efficiency. We proposed a method to improve the RDO of stereoscopic video coding. It uses two independent perception-based distortion metrics for the left and right views, as the distortion in the RDO. The distortion metric of the right view is to measure binocular perceptual distortion of the two views. The distortion metric of the left view is to measure two-dimensional (2-D) perceptual distortion of the left view. Both distortion metrics are developed based on the structural similarity index metric (SSIM) that is weighted by visual sensitivity. A binocular just-noticeable distortion (JND) model is proposed to measure binocular visual sensitivity for the right view. The commonly used 2-D JND models are adopted for the left view. In addition, a 1/SSIM-based rate-distortion (R-D) model is improved based the proposed distortion metrics. Based on it, the Lagrange multiplier is then derived according to the R-D theory. The experiments demonstrate that the proposed method has better performance than other perceptual RDO methods. Subjective evaluations also show that it can achieve better perceptual quality with less bit rate.

Human action recognition is one of the raison d’être for doing human–computer interaction research, as it is highly vital in meeting the demands of modern society, such as automatic video surveillance for security, patient monitoring for recovery, content-based video retrieval, etc. In line with this, deep learning systems are fast becoming the defacto standard for object recognition, video understanding, and pattern recognition due to their inherent powerful feature learning ability from vast amount of data. It makes sense to capitalize on its great success and to further improve it for the complex task of action recognition. One of the contributions in this paper is an effective and yet simple method for encoding the spatiotemporal information from skeleton sequences into what we call temporal kinematic images. In the input encoding scheme, we embed various geometric relational features derived from the skeleton sequence in the form of our proposed skeletal optical flows (SOFs). SOFs collectively represent the variations of kinetic energy, angles between limbs, and pair-wise displacements between joints over consecutive frames of skeleton data, as color variations in the temporal kinematic images. Another contribution is our convolutional neural network with a correctness-vigilant regularizer. It is employed to exploit the discriminative features from the temporal kinematic image for human action recognition. Lastly, we additionally investigated an adaptive label smoothing technique employed toward the end of training iterations. Empirical results show that the efficiency of the proposed method is superior to existing works in terms of the generalizability of the generated model, training convergence speed, and the resulting classification accuracy on nine popular benchmarking datasets, such as MHAD, MSR Activity 3D, HDM05, MSR Daily Activity 3D, and the latest challenging databases, such as UTKinect-Action, NTU RGB+D, Northwestern-UCLA, UWA3DII, and SBU Kinect Interaction datasets.

Deep neural networks have been applied to a wide range of problems in recent years. Convolutional neural network is applied to the problem of determining the depth from a single camera image (monocular depth). Eight different networks are designed to perform depth estimation, each of them suitable for a feature level. Networks with different pooling sizes determine different feature levels. After designing a set of networks, these models may be combined into a single network topology using graph optimization techniques. This “semiparallel deep neural network (SPDNN)” eliminates duplicated common network layers and can be further optimized by retraining to achieve an improved model compared to the individual topologies. Four SPDNN models are trained and have been evaluated at two stages on the KITTI dataset. The ground truth images in the first part of the experiment are provided by the benchmark, and for the second part, the ground truth images are the depth map results from applying a state-of-the-art stereo matching method. The results of this evaluation demonstrate that using postprocessing techniques to refine the target of the network increases the accuracy of depth estimation on individual mono images. The second evaluation shows that using segmentation data alongside the original data as the input can improve the depth estimation results to a point where performance is comparable with stereo depth estimation. The computational time is also discussed in this study.

To preserve more useful information in visible and infrared images and improve the quality of the fused image, a method based on the nonsubsampled shearlet transform (NSST) and fuzzy C-means clustering is proposed. First, the source images are decomposed by NSST so as to get their own low- and high-frequency subbands. Second, the low-frequency subbands are divided into the infrared target part and the background part by fuzzy C-means clustering while different fusion rules are applied to the infrared target part and background part, respectively. Then, a choose-max fusion rule based on the sum-modified Laplacian of source images and local energy of coefficient is proposed to integrate the high-frequency subbands. Finally, the fused image is obtained by inverse NSST. The comparison experiment with the other three state-of-the-art fusion methods shows that the proposed method has good subjective visual effects and superior objective evaluations.

At present, in the field of person reidentification (re-id), the commonly used supervised learning algorithms require a large amount of labeled samples, which is not conducive to the model promotion. On the other hand, the accuracy of unsupervised learning algorithms is lower than supervised algorithms due to the lack of discriminant information. To address these issues, we make use of a small amount of labeled samples to add discriminant information in the basic dictionary learning. Moreover, the sparse coefficients of dictionary learning are decomposed into a projection problem of the original features, and the projection matrix is trained by labeled samples, which is transformed into a metric learning problem. It thus integrates the advantages of the two methods through combining dictionary learning and metric learning. After the data are trained, a projection matrix is used to project the unlabeled features into a feature subspace and the labels of the samples are reconstructed. The semisupervised learning problem is then transformed to a supervised learning problem with a graph regularization term. Experiments on different public pedestrian datasets, such as VIPeR, PRID, iLIDS, and CUHK01, show that the recognition accuracy of our method is better than some other existing person re-id methods.

We focus on the challenging cross-view action recognition problem. The key to this problem is to find the correspondence between two different views, which is realized in two stages. First, we construct a dual-codebook for the two views, which contains one codebook for each view. Each codeword in one codebook has a corresponding codeword in the other codebook, whereas traditional methods implement independent codebooks for the views. We propose an effective coclustering algorithm based on seminonnegative matrix factorization to derive the dual-codebook. Additionally, to represent actions in one view, unlike most other methods using the codebook of that view only, we also exploit the codebook-specific information from the other view. Thus, we construct mapped-codebooks via codebook transformation, complementing the codebook-to-codebook correspondence within the dual-codebook. In the second stage, observing that the temporal relationship between action segments within an action is view invariant, we further propose a hierarchical transfer framework based on a temporal structure that can effectively capture such action-segment temporal relationship at multiple timescales, which is more discriminative than the usual video-level transfer strategy. Extensive experimental results on the INRIA xmas motion acquisition sequences and West Virginia University datasets demonstrate superiority of the proposed method compared with state-of-the-art approaches.

Image style transfer processing is an important part of image processing, which processes image information such as the color, silhouette, and line to other image styles by computer. It has made great progress in recent years, with the development of deep learning. This paper designs a deep architecture to generate highly stylized images based on convolutional neural networks and generative adversarial networks (GANs). In addition, we construct an image style similarity measure model, which can discriminate whether the generated style image is similar to the real ones. Experimental results show that the generated style images can achieve good visual results compared with related image style transformation algorithms. Generally, we propose an efficient real-time image style transfer model to generate highly stylized images.

A discrete trinion Fourier transform (DTFT) is proposed and computation of the DTFT is derived from the discrete Fourier transform. A block-based color image watermarking scheme is designed based on the proposed DTFT. The watermark bits are embedded by modifying the DTFT coefficients of each block by means of the quantization index modulation method. Finally, assessment of this watermarking approach is performed in terms of watermark capacity, invisibility, and robustness. A comparison with a quaternion discrete Fourier transform is also provided. Experimental results show that the proposed method is not only more robust in most standard attack situations, but also has superior time efficiency.