3.1.

Mathematical Framework

Several mathematical methods have been compared to obtain a reliable classification of material type, such as:

These transforms are commonly employed in signal processing tasks. Each of them has particular properties or characteristics that can aid the material analysis process. Since the material to be recognized is examined by analyzing several sequences of frames over time, it is important to use transforms that are able to extract significant features from signals.

In our tests, we have taken advantage of the fast Fourier transform (FFT), i.e., a faster alternative to discrete Fourier transform (DFT). In general, the FFT is a powerful tool for pattern recognition. It is commonly employed to extract invariant features²⁴ because of its important properties; for example, a shift in the time domain does not involve any change in the amplitude spectrum of the image. Good predictive accuracies in material recognition are expected since the frequency domain representation might provide more useful information than the time domain one. Moreover, a low processing time should be required for computing this transform.

The Hilbert transform extends a differentiable real signal into the Gauss plane. This transform adds information to the Fourier analysis because it introduces the conjugate harmonic of a given signal. Usually, it is used to handle nonstationary processes or signals for which the Fourier spectral analysis is not often suited. In our algorithm, the discrete version is exploited.

The DCT is similar to the DFT since they both decompose a discrete-time signal into a sum of scaled and shifted basis functions. However, the DCT uses only cosine functions as its kernel. The DCT is widely used in signal processing and data compression applications because of its high-compression degree of spectrum. One of the most relevant properties^25,26 is the noise high-frequency isolation in a small number of coefficients compared to other transforms such as the DFT. Average recognition rates are expected since the high-compression level of this transform might negatively affect the informative content of input signals.

The KLT is a representation of orthogonal functions. It has different expansion bases that depend on the stochastic process, and their coefficients are random variables. The kernels employed in this representation are defined by the covariance function of the process. Although the KLT has a high computational complexity, it is suitable to obtain the best bases for linear decorrelation of signals and energy compression. Hence, good predictive rates should be achieved by employing this mathematical tool.

Finally, the last employed transform is the chirp-z-transform (CZT), which can be considered a generalized case of DFT. In fact, the CZT samples the Z-plane along spiral arcs, which correspond to straight lines in the Laplace-plane. The kernel of this transform is a complex number.

All listed mathematical domains have been used and compared to extract features suitable for the material recognition method. Different materials having different physical characteristics have been examined. Particularly, the material target under investigation has been fastened on a panel and then placed in front of the acquisition system. Several working conditions have been taken into account. For instance, the shutter value (or exposure time), the position (or distance), and the angle (or heading) of panels with respect to the sensor have been varied.

It is necessary to emphasize that only a portion of the scene has been considered. Specifically, a region of interest (RoI) of the panel has been extracted, as reported in Fig. 1. The target has been centered in the middle of field-of-view (FoV) of the ToF sensor. In this way, the distortion effects due to the curvature of lenses have been consistently reduced. Nevertheless, a preliminary step aimed to rectify the depth images has been performed to compensate the distortion effects. In this regard, the camera calibration toolbox for MATLAB,²⁷ along with the well-known notation introduced by Heikkilä,²⁸ has been exploited to get a faithful reconstruction of scenes without distortions.

Fig. 1

Example of the extracted RoI of $20\times 25\text{pixels}$ (see cyan rectangle) of a wooden panel during an experiment. The 3-D point clouds mapped according to both (a) distances and (b) IR reflectivity are shown.

At this stage, a punctual analysis of the panel is performed. In other words, every pixel of the extracted RoI is evaluated over time. A sequence of $n$ frames is acquired for every material. Thus, each pixel has $n$ values of 3-D coordinates together with the related intensity levels

Eq. (1) represents exactly how the data linked to a pixel of 2-D coordinates $(u,v)$ is arranged. In this regard, the vector $p(u,v)$ contains both the distance variations $s_k$ of the pixel and the associated intensity values $i_k$. Distance fluctuations are computed using Eq. (2). Specifically, the value of distance $d_k$ is evaluated by means of Eq. (3), whereas $\overline{d}$ represents the average value of all $n$ distances related to the pixel under investigation. The terms $X_k$, $Y_k$, and $Z_k$ represent the 3-D coordinates of the pixel $(u,v)$ at the time $k$.

Subtracting the average distance from the set of measures fulfills a precautionary step aimed at reducing the possibility that material recognition might be achieved by the classifier by considering the evaluated distance instead of its fluctuations. Artificial biases introduced by the experiment itself in the material recognition process are therefore drastically decreased.

Once the input vector has been collected, the different transformation domains listed before have been computed. In this way, one or more features are associated to each pixel of the RoI.

3.2.

Description of Decision Tree

As stated before, the aim of this paper is to provide a method able to discern the category of materials by analyzing the data returned from a ToF camera.

Material recognition is achieved by exploiting a decision tree as a model of classification. In brief, a decision tree is a predictive machine-learning model able to provide an output value by evaluating numerous attribute values of the available data. It can be considered a treelike graph that has nodes and branches. Specifically, the internal nodes denote the different attributes, whereas the branches between nodes specify the possible values these attributes can have. Finally, the terminal nodes identify the final value or the classification output of the dependent variable.

Regarding this specific case, the attributes of the decision tree are the coefficients of the various computed features together with other parameters that characterize the performed experiment. Further information will be provided in Sec. 4.

In our tests, we employed the open source software Weka,²⁹ a collection of machine-learning algorithms, for fulfilling material recognition. Specifically, we chose the J48 classifier, a high-performance algorithm suitable for large datasets, which is a popular open-source implementation of the C4.5 decision tree (Ref. 30) classifier that is available in Weka. The employed classifier supplies an easy interpretation model and it is suitable for datasets, which are heavily affected by noise. Although other classifiers (e.g., RF) provide comparable predictive accuracies with respect to J48, they require higher computational processing. Moreover, preference was given to deterministic algorithms for repeatability reasons.

3.3.

Data Arrangement

Section 3.1 reports the main domains of analysis for associating features to a single pixel under examination. However, material recognition is achieved by considering the evaluation of a set of pixels, i.e., those pixels belonging to the considered RoI.

As previously stated, a decision tree is designed to provide the category of material under examination. Therefore, the data of interest have to be properly arranged to be managed by the decision graph for material classification. In other words, starting from the computed features for a set of materials and considering other parameters tied to the typology of experiments, a suitable representation has been obtained and stored for processing using Weka. Specifically, the file has been organized according to the attribute-relation file format (ARFF).

The header of an ARFF file is reported in Table 1. The coefficients of the features are arranged along the columns, then other parameters such as position, angle, active brightness, and shutter are organized in the same way. The position and the angle values represent the pose of the panel to be examined with respect to the ToF sensor. Further details will be provided in the following section. The active brightness is a boolean parameter of the camera that enables improvement of the quality of acquisition. In fact, when this parameter is disabled, all measurements returned by the photoreceivers are considered good, even if some reflected light is captured by the photosensors. Therefore, it is preferable to enable this modality to obtain more accurate measurements. The shutter value is the time duration in which the camera chip is exposed to IR light. This value is expressed in milliseconds (ms).

Table 1

ARFF file organization. The coefficients of features along with the other parameters are arranged along the columns. In contrast, each pixel of the selected RoI is arranged along the rows.

The last column represents the expected output value, which is identified by a cardinal number. In this regard, a specific number is assigned to represent each material to be classified (see Fig. 2 for more clarity).

Fig. 2

The materials employed in our tests are in the order: iron, fir wood, plywood, plastic, polystyrene, reflective surface, white fabric, dark fabric, aluminum, and glass.

Finally, it is worth noting that each row of the file refers to the single pixel belonging to the extracted RoI. Furthermore, as shown in Table 1, more than one feature transform can be employed. In this regard, some tests will show how the classification is affected.

It is important to underline that the data given from the acquisitions have been split into two categories: the training and the validation set. Specifically, 20% of the entire dataset has been reserved to represent the validation set. In this way, the validity of the method has been tested by analyzing the predictive accuracies of recognition.

4.1.

Eight Materials with Fixed Panel Poses

In this experiment, all materials except glass and fir wood have been considered. The position and the orientation of the panel on which the materials are fastened have been held fixed to 1 m and 0 deg, respectively. Furthermore, a sequence of 300 frames has been acquired.

Figure 4 shows the predictive accuracies given from the analysis of the eight materials. These accuracy rates have been obtained by considering the confusion matrices returned by Weka. Three different shutters have been tested. As shown by the bar graph, the shutter value heavily affects the likelihood of correctly recognizing the material type. In this regard, the DCT and HILBERT transforms seem less stable than the others. In contrast, the remaining transforms ensure higher recognition rates for all the reported shutter speeds.

Fig. 4

Three different integration times have been evaluated for each employed feature. Note how the predictive accuracy consistently decreases for DCT and HILBERT transforms when an increase of shutter value occurs.

The FOURIER, CHIRP, and KL transforms provide more reliable classification of the presented materials. Although the related recognition rates are comparable, the FOURIER transform requires less time to be computed with respect to the others. This aspect should be taken into account when real-time requirements for material recognition need to be addressed for specific applications.

As the bar graph shows, the KLT ensures high recognition rates. Such a transform was commonly employed in previous works^34,35 for its effectiveness in feature extraction. As already stated, this transform is a representation of a linear combination of orthogonal functions such as the Fourier series, but the number of coefficients is variable. The Karhunen–Loève expansion is better suited, even though it requires more time to be computed with respect to the chirp and Fourier transforms. Therefore, the Fourier-based feature represents a good compromise between classification accuracy and computational time required.

In contrast, the DCT-based feature does not provide stable predictive accuracies for the considered shutter speeds. In this regard, the high-frequency noise that affects a signal is often isolated in a small number of coefficients. Since in our approach we take advantage of both the distance fluctuations and the intensity information for extracting a feature, it is fairly probable that such informative content is not preserved when the DCT transform is computed. Therefore, loss of information might occur when this transform is used.

Nevertheless, the classification rates obtained employing a unique feature do not compare favorably against previous methods. Therefore, experiments have been performed by providing the J48-based classifier with features extracted simultaneously by several transforms.

Hence, all the possible pair combinations of the presented transforms have been considered to enhance the predictive accuracies. Similar to the case of unique transform previously discussed, the features employing the DCT transform appear to be more affected by shutter value variations. As a matter of fact, a consistent decrease of predictive rates is observable from Fig. 5.

Fig. 5

The bar graph shows how the combination of transforms allows an increase of recognition rates of materials despite shutter changes.

Conversely, the features extracted by using combinations of the FOURIER, CHIRP, and KL transforms ensure good predictive accuracies. Such results are still coherent with the ones obtained by employing a unique transform for extracting a feature of interest. In this regard, such features are less affected by the analyzed different shutter values.

Table 3 reports some important metrics useful in measuring the performance of the predictor. Among the computed features, only the most relevant has been proposed in this table. Moreover, a shutter of only 20 ms has been taken into account since that seems to ensure higher recognition rates.

Table 3

The true positive (TP), true negative (TN), false positive (FP), false negative (FN), precision, recall, and F-measures are reported for each material and extracted feature combination.

As shown by the $F$-measure (i.e., the harmonic mean between precision and recall), the materials made of plastic and reflective surfaces can be predicted with good performance by means of KL-based features and CHIRP–KL-based features. In contrast, the dark fabric sample presents the worst case of this classification task due to its high absorption property. Other materials such as aluminum and iron show average $F$-measure values since the considered shutter causes sensor saturation. Moreover, considering the average rank of the $F$-measure, the FOURIER–KLT combination and CHIRP–KLT show the best results. In general, if the KL score is low, most of the descriptors perform weakly.

Some of presented features enable recognition of the material type with good accuracy. However, the computational time required to calculate them has to be taken into account when dealing with real-time applications. For the sake of completeness, Table 4 reports the computational times needed to compute the features of interest.

Table 4

Computational times obtained to compute the features of interest. All the transforms have been obtained using the software MATLAB. These outcomes refer to the analysis of one pixel belonging to plywood over time.

The values in bold benchmark the elapsed time for computing the features that have shown the best results in terms of predictive accuracy. As observable, the FOURIER-based feature requires the least time to be computed with respect to the others. In fact, the FFT has been employed to obtain such features. In contrast, the features based on the KLT need more time to be extracted. In this regard, one order of magnitude higher than the FOURIER- and CHIRP-based features is evident from this table.

4.2.

Four Materials with Different Panel Poses

In this case, a subset of considered materials has been examined since several panel poses have been taken into account, as shown in Table 5. This has been necessary for limiting the number of material/distance/orientation combinations to be acquired. Specifically, the plywood, the plastic panel, the reflective surface, and the glass surface have been analyzed since these materials are commonly present in indoor environments.

Table 5

List of parameter values employed during our experiments.

It is worth highlighting that the glass surface has been introduced in this experiment because it is more challenging for it to be acquired by the ToF camera, like the reflective surface. Therefore, its analysis could be of more interest than those of other material types.

Table 5 reports the different attribute values used during the experiment. In other words, several acquisitions have been performed considering the combinations of values listed in the table.

It is worth noting that the position and shutter values are related. In other words, when the panel is placed close to the ToF sensor, small shutter values are needed to get reliable depth measures. In contrast, the more the position increases its value, the more the shutter can be increased. Therefore, by considering a short distance between the panel and the acquisition system, small values of the shutter have to be chosen, thus enabling the avoidance of unwanted saturations of photosensors.

As shown in Fig. 6, an average increase of 10% in prediction accuracy has been obtained with respect to results reported in Fig. 4, since there are fewer material types that need to be considered. In this case, all features appear to be stable against shutter variations. The features based on FOURIER, CHIRP, and KL transforms show better predictive rates, although of a small margin.

Fig. 6

The bar graph reports the predictive accuracies by considering several poses of panel and different integration times. The highest recognition percentages are achieved with a shutter value of 20 ms.

By combining the transforms, an increase of predictive rates has been achieved, as shown in Fig. 7. Such outcomes prove once more that features extracted from multiple domains are better suited to be interpreted by the classifier.

Fig. 7

The combination of transforms involves an increase of predictive accuracies of about 10% with respect to those computed by considering a unique domain of transform.

This is consistent with the results obtained in the previous experiment. Moreover, a shutter value of 20 ms provides the highest predictive rates among those considered. In fact, this value seems to be the most suitable for the considered distance range and the materials used during the tests.

Table 6 reports the main metrics obtained by limiting the experiment to four material types only using a shutter value of 20 ms. As highlighted, very high $F$-measure values have been achieved by using the FOURIER–CHIRP-based features and FOURIER–KL-based features. Conversely, lower predictive rates have been obtained when features based on unique transforms have been used. In fact, as already stated, the combination of features enables enhancement of the predictive rates for material classification.

Table 6

The main metrics for evaluating the classifier have been reported for the case of analysis. A shutter value of 20 ms has been considered to collect the data presented here.

The results of the second experiment, therefore, confirm that the descriptors based on FOURIER–KL and CHIRP–KL provide the most effective material classification.

Table 6 shows that the predictive accuracies of materials made of plastic are considerably lower than others. Specifically, the reported data refer to a shutter value of 20 ms. In this regard, saturations of photosensitive receivers occur when the plastic panel has been acquired by considering this exposure time. Therefore, meaningless measurements have been returned by the ToF camera. In contrast, the other materials have not involved any photosensor saturations, thus providing more reliable measurements. Additionally, the measurement fluctuations obtained by analyzing these materials appear to be more distinctive and meaningful by the classifier. As a consequence, they have been better classified, as shown in the table. Further experiments will be conducted to better investigate these material categories.

In Fig. 8, the recognition rates are shown, taking into account the position changes. As already highlighted, the features that ensure the highest scores are those based on a combination of FOURIER, CHIRP, and KL transforms.

Fig. 8

The recognition rates of variations of distance from the ToF sensor and the examined panels are reported. The features that ensure the highest and most reliable scores are still those based on the combination of FOURIER, CHIRP, and KL transforms.

The predictive accuracies related to short distances are slightly lower with respect to other positions, although by a small margin. This is probably due to the shutter values employed for these positions. Specifically, only integration times of 3, 5, 10, and 15 ms have been considered for such positions. However, it has been demonstrated that a shutter value of 20 ms ensures the highest recognition scores among those remaining. As a consequence, the scores of Fig. 8 are indirectly affected by the shutter value.

Similar to cases of analysis of shutter and position changes, the correlation between the angle variation and the predictive accuracy has been investigated. In this regard, Fig. 9 shows the obtained recognition rates by varying the angle value. It is important to underline that only the positive angles have been reported since the negative ones present very similar scores.

Fig. 9

Here, the predictive accuracies are shown by considering different orientations between the panel and the acquisition system.

The most stable features against angle variations are still those obtained in previous experiments, namely the FOURIER–CHIRP, FOURIER–KL, and CHIRP–KL. High predictive accuracies have been achieved in this case as well.

In summary, it is possible to state that the combination of features enables enhancement of the prediction rates. Moreover, three of these features can be considered more stable than others in material classification, even when different shutter values and panel poses are taken into account.

4.3.

Eight Materials by Introducing Another Wooden Panel in the Dataset

The presented experiments have shown the efficiency of the proposed methodology for recognizing the material type. In this paper, the most common material categories have been considered. However, several objects belonging to the same category can be found when an environment is explored. For example, there are several surface typologies made of iron, wood, plastic, or fabric having different surface smoothness or reflective properties.

Nevertheless, it is unthinkable to classify all the existing materials. In this regard, a large dataset should be used to enhance the recognition rate of material. In fact, a wide set of objects made of the same material has to be considered during the training stage for each category.

In this experiment, another wooden panel made of fir, having different characteristics from the others (plywood), has been analyzed. In this regard, the fir panel has a lighter color with respect to plywood and is also less smooth.

At first, data related to the fir panel have not been included in the training step. Therefore, the material category should be identified by the built classifier using only the data related to the other eight panels (similar to tests reported in Sec. 4.1). However, a low-predictive accuracy ($\sim 20\%$) has been reached. As a consequence, the fir panel has not always been identified as wood. This likely happens because the diverse roughness of the two wooden panels could affect category recognition. Conversely, by considering the data of the fir panel in the training stage and classifying the panel as wood, a significant increase of the recognition rate is achieved.

Figure 10 reports the predictive accuracies by considering the fir panel in the computation as well. As observable, a slight increase of scores has been obtained in comparison with the outcomes reported in Fig. 4. The improvement of rates is probably due to a better representation of the class “wood.” In other words, the more data used in the training step, the more the classifier is able to represent the real world.

Fig. 10

Here, the recognition rates for each feature are reported by considering eight materials. Specifically, two typologies of wood (fir wood and plywood) have been taken into account to evaluate the accuracy of the classifier. The features based on FOURIER, CHIRP, and KL show more stable results against shutter value changes.

The combination of features enables higher recognition rates (see Fig. 11). The features based on the combination of FOURIER, CHIRP, and KL show more stable scores with respect to those that significantly decrease the predictive accuracies when the shutter changes its value. Hence, such outcomes prove once more that these features are sufficiently distinctive and reliable for achieving category classification.

Fig. 11

The features based on FOURIER, CHIRP, and KL ensure the highest predictive accuracies. Moreover, these features are more reliable against shutter variations with respect to the remaining ones.

The metrics related to this experiment are reported in Table 7. As observable from the table, the features computed from a combination of transforms ensure the highest classification rates. By comparing these results with those of Table 3, it is possible to note that similar prediction scores have been achieved. Nevertheless, the scores referring to the category of wood are considerably improved. Specifically, the $F$-measure of wood is increased by about 30% for all the considered features. This enhancement is due to a better representation of the class under investigation.

Table 7

Here, the main metrics related to this test are shown. Very high recognition scores for materials made of wood have been achieved in comparison with Table 3. Other predictive accuracies related to other categories are still comparable with those of that table.

4.4.

Discussion

Many works have been proposed in recent years for solving the problem of material recognition. Many of the proposed methodologies are essentially based on color and texture analysis of 2-D images. Very few works tackle this topic by exploiting 3-D information. In this regard, our method is able to classify the material typology by exploiting both the 3-D information and the intensity levels returned by the ToF camera.

Although the method proposed in this paper is directly comparable with just a few works, and, even then, it works on different datasets, a comparison table is show in Table 8, with papers grouped according to the employed processing method.

Table 8

Overall recognition rates of discussed papers.

Works in Refs. 1, 5, and 6 employed high-resolution 2-D cameras to acquire images with respect to the low resolution of our 3-D ToF sensor. In Ref. 6, an acquisition system based on a concave parabolic mirror and a beam splitter have been used to obtain the reflectance disks of materials. Different material typologies have been analyzed. In this regard, only Ref. 6 investigated similar materials. In contrast, Refs. 1 and 5 employed natural and building materials, respectively.

The work in Ref. 2 presents average scores even though several materials are examined. However, its methodology is applied to recognize natural materials, while this work is more concerned with structured indoor environments.

Finally, Refs. 14, 15, and 16 refer to the analysis of 3-D information for achieving material recognition. Among the discussed works, these are the closest to our approach since they handled 3-D data to extract features. Furthermore, analogous material categories (wood, plastic, fabric, and paper) have been investigated. It is observable that a slightly lower accuracy rate has been achieved since in our case, challenging materials such as a reflective surface and glass have been considered.

Part of this advantage might also be dependent on the sensor’s performance difference. In fact, the absolute accuracy (or the maximum systematic error on the distance measurements) and the repeatability ($1\sigma$) of SR4000³⁶ are equal to $\pm 10$ and 4 mm, respectively. Conversely, the absolute accuracy and the repeatability ($1\sigma$) of Fotonic E70 are $\pm 20$ and 7 mm. Hence, our acquisition system used to collect the data is less accurate than the other one.