2.1.

Overview

Forest pest damage has obvious visual and dynamical features, and they are usually characterized by defoliation and discoloration. Visual detection of an infested stand is not straightforward in many cases, particularly in large and dispersed forests. Video surveillance technology has been called to address this issue mainly due to two reasons: first, DVS can visualize the forests continually to check the health of woods beyond our own eyes; second, DVS has a wider vision and can monitor extensive forest areas and record these data in real time. In addition, to our knowledge, the application of video surveillance technology for forest insect defoliation and discoloration is rarely reported. This paper focuses on the integrated monitoring framework and explores a new application which will be very helpful for further study.

Compared with previous methods, the video-based method can provide continual and real-time monitoring, and the video data can be stored for a long time. Video surveillance techniques have many attractive attributes, but the most prominent are the real-time monitoring capability and the immediate availability of the electronic signal for both visual interpretation and digital processing.¹⁶ Visual images can convey more memorable and meaningful information than written words, numbers, or other types of media.¹⁷ Visualization refers to any technique for creating images, diagrams, or animations to communicate a message.¹⁸ The visualization of forest scene is essential for forestry departments to detect disturbances and make decisions for pest control. With the development of computer hardware and software, it is possible to visualize complex natural phenomena, such as a three-dimensional (3-D) scene or landscape. Visualization means to represent data in an intuitionistic and perceptible form that can be sensed by people’s vision. The 3-D visualization for forest insect defoliation and discoloration monitoring is supported by a geographic information system (GIS) and is characterized by multiperspective views, depending on the relationships with observers from different directions, distances, or locations.

In the proposed method, satellite remote sensing monitoring is mainly used to identify the susceptible areas of forest insect infestations in order to provide support to make appropriate monitoring strategies that reduce the impact of disturbances.¹⁹ The remote sensing sources used to identify the susceptibility areas should be high-resolution images, because forest insect infestation is often studied at the individual tree scale.²⁰ DVS is used to capture video images to integrate with GIS, to provide intuitionistic display for the command and control center, and to obtain the geographic coordinates of insect damage for field survey. Then ground survey by human walkthrough is mainly responsible for conducting field verification and to simultaneously provide complementary details that video images lack. In addition, for those forest areas difficult to access, satellite imagery and video images are often the only means of gathering information. The main framework diagram is shown in Fig. 1. A detailed description of the above-mentioned procedures will be given in the following part of the article.

Fig. 1

Framework diagram of the proposed comprehensive monitoring method for forest insect defoliation and discoloration.

2.2.

Video Positioning of Forest Insect Damage

For the system implementation, we used a high-definition CCD camera, digital PTZ, zoom lens, protective slab, decoder, watchtower, and power supply facilities. In the process of video surveillance, PTZ can control the camera to rotate at the horizontal (pan) and vertical planes (tilt) and to change the level of magnification (zoom). The camera, fixed on the PTZ, is responsible for capturing video images. The PTZ can control the camera to rotate smoothly in the horizontal direction and in the vertical direction, to guarantee complete coverage. Briefly, the front-end equipment and the monitoring center are connected through a wireless network.

The geographical position of forest insect damage is traditionally determined by comparing the video scene with a corresponding map. There are some unavoidable subjective factors that affect the accuracy. To improve the accuracy, we designed an automatic positioning algorithm for forest insect damage based on digital elevation model (DEM) and returned parameters from the PTZ camera. The principle of the disaster positioning is to build a ray emitting from the monitoring camera according to the orientation parameters returned from the PTZ. From the perspective of geometry, the process of the disaster positioning is to get the intersection point of the DEM and the ray. The DEM cannot be directly expressed by a mathematical function. Thus, in a discrete space, disaster positioning can be seen as a process of finding the optimal solution regarding the constraint satisfaction problem. To improve the efficiency of the algorithm, the viewshed analysis was used to generate the viewshed layer, which could reduce the search space of the optimal solution. To generate the ray, the Bresenham algorithm²¹ was used which adopts incremental computation and is a classical algorithm to generate a line in computer graphics. Thus, for each column, just check the sign of an error item, and then the pixel of this column is selected. This algorithm does not include the operation of real numbers, so it can generate a line quickly.

Figure 2 illustrates the spatial model for disaster positioning, where $M$ represents the monitoring point and $P$ is the disaster point. $\alpha$ and $\beta$ are the azimuth angle and pitch angle, respectively. $\theta$ is a vertical angle in the direction of $\alpha$. $L$ is a ray from the monitoring point. Based on this model, the flow chart of the positioning algorithm is shown in Fig. 3, where $H$ represents the height from the camera to the ground. ($X_{\mathrm{c}}$, $Y_{\mathrm{c}}$, $Z_{\mathrm{c}}$) are the projective coordinates of the monitoring point.

Fig. 2

Spatial model for forest insect damage positioning. The meaning of parameters: $M$, monitoring point; $P$, disaster point; $L$, a ray from $M$; $\alpha$, the azimuth angle of $P$; $\beta$, the pitch angle of $P$; $\theta$, a vertical angle in the direction of $\alpha$.

Fig. 3

Flow chart of positioning forest insect damage. The meaning of parameters: ($X_{\mathrm{c}}$, $Y_{\mathrm{c}}$, $Z_{\mathrm{c}}$), the projective coordinates of the monitoring point; $H$, the height from the camera to the ground; $D_i$ ($i=1\dots n$), the discrete points of the ray $L$; ${\theta }_{Dk}$ ($k=1\dots m$), a vertical angle in the direction of $\alpha$.

2.3.

Integration of GIS and Video Surveillance

The hardware architecture depicting the system context is shown in Fig. 4. For real-world image capture, several PTZ cameras are utilized. A PTZ camera has three degrees of freedom because of its ability to rotate in the horizontal (pan) and vertical planes (tilt) and to change the level of magnification (zoom).

Fig. 4

Hardware structure of distant video monitoring.

Three servers are deployed at the server-side and are connected to the Internet with a speed of $10\mathrm{GB}/\mathrm{s}$. The video server uses the D-Link DNS-726-4 to provide the load balancing service and is utilized to distribute front-end video streams and to provide load balancing service for an increasing number of users. The data server can provide basic spatial data service and pest information storage service for the application server. The DVS system is deployed on the application server using a middleware of Internet Information Services 7.0. As shown in Fig. 5, the software structure of the DVS system consists of two-dimensional (2-D) GIS, 3-D GIS, video control, disaster positioning, and communication control. In comparison to previous research, our method integrates 2-D GIS and 3-D GIS synergistically. This can make the monitoring process more intuitionistic.

Fig. 5

Software structure of distant video monitoring.

Functionalities of the 2-D GIS component are related to 2-D visualization, layer management, geospatial data access, feature querying, and identification. Furthermore, the 2-D GIS can navigate monitoring points and aid disaster positioning. In addition, the 2-D GIS can serve as an eagle-eye map of the 3-D GIS component. The 3-D GIS component is responsible for the visualization of the DEM and 2-D geospatial data of the forest area. The 2-D geospatial data are displayed as a texture that is laid over the virtual terrain. Moreover, the disaster points can be added and displayed in the 3-D GIS context.

The video control component is responsible for controlling the camera to pan, tilt, and zoom as well as for setting the window size and the bitrate of the camera video. The communication control component is responsible for the acquisition of the camera’s orientation parameters as well as for communications between different components. The component for disaster positioning is responsible for calculating the geographic coordinates of the forest pest disasters.

2.4.

Field Survey

In order to evaluate the method, ground survey by human is used to collect field data for on-site verification. The field survey is used for both forward validation and reversed validation (Fig. 6). Forward validation is used to assess the accuracy of damage detection using video surveillance. Reversed validation is used to evaluate the completeness of damage detection. Due to the limited memory and screen size of the embedded equipment, the massive spatial data need to be preprocessed, such as establishing the image pyramid and segmenting the vector data.²² In the system, the ShapeFile format is used for storing all the vector data because it uses less storage space. The pyramid images are managed in FGI format, which consists of a master file (.fgi), a secondary file (.aux), and a coordinate file (.txt). After downloading the processed data into the embedded GIS, field surveys can be conducted to collect the spatial data, attribute information of the pest disasters, and upload the results to the server in time.

Fig. 6

Flow chart of field surveys for validation.

Field verification is aided by smartphones or personal digital assistant (PDA) with GPS embedded (Fig. 7). After the geographical coordinates of the forest pest disasters are obtained, the thematic map of disasters can be output and imported into PDA, and afterward the foresters are able to conduct on-site verification following the optimal path planned in advance. In addition, to guarantee the reality and authenticity of the field data, the proposed approach can provide supervision and inspection to enable managers to avoid fake data to some extent.

Fig. 7

Field survey for data acquisition.

3.1.

Application Effect

The DVS system is implemented in Microsoft Visual Studio 2010, as a Visual C# project. Browser/Server structure is utilized in this system. The main user interface is composed of three components: 2-D view, 3-D view, and video view (Fig. 10). Besides these views, the application interface contains the disaster information table (bottom-left) to record the disaster information including the monitoring point names, the latitude and longitude of disasters, damage degree, and date, which is simultaneously stored in the database. The first component, 2-D view, provides the overall perspective of the forest area based on the forestry data and other geographic spatial data. The component is implemented purely by the web pattern. Basic GIS functions related to the view are map zooming, map panning, object identification, and layer management. Users can choose to display the selected layers of geospatial data according to actual needs.

The second component, 3-D view, is responsible for the generation and visualization of the virtual terrain. The component is implemented based on OpenSceneGraph, which is an open source 3-D graphic engine. At the data layer, Geospatial Data Abstraction Library 1.8.0 is used to analyze the spatial data. At the network layer, the library of libcurl 7.21.3 is used to transmit the Web Map Service terrain/image pyramid models. At the function layer, the terrain rendering library of libMini 10.2 is utilized to implement 3-D terrain visualization. Functions for the 3-D view include roaming, zooming, and visibility analysis. Visibility analysis is used to determine whether a given object located on a terrain is visible from a viewpoint and how much of the object is visible.²³ In addition, when the coordinates of disaster points are obtained, they can be simultaneously added to the 3-D scene.

Finally, the third component, video view, displays the on-site video of a selected surveillance camera. With this view, it is feasible and flexible to manipulate the camera by changing the parameters of pan, tilt, and zoom, or utilizing the mouse as a virtual joystick (Fig. 8).

Fig. 8

Video monitoring in application mode that displays video control.

During the process of scanning, when finding dubious phenomena, the PTZ parameters can be adjusted to clearly observe and analyze the video images to detect forest pest disasters. Then, combined with DEM and the angles of azimuth and pitch of the PTZ camera, the geographical coordinates of the disasters can be obtained through the positioning algorithm. Then the disaster information is stored into the disaster database. Users can query disaster by single field or multifield, and the query results can be simultaneously visualized in the 2-D view. In addition, the thematic map can be conveniently output and printed, to aid forestry surveyors to conduct field verification. Furthermore, the 2-D view, 3-D view, and video view can interact with each other. For example, when the user selects a monitoring point, the 3-D view and the video view will transform accordingly. Thus, the user can monitor the on-site situation in real time to detect forest pest disasters timely and accurately. In addition, the thematic map can be generated and printed, providing great convenience for the foresters to conduct on-site verification.

For on-site verification, the system for ground surveys (Fig. 9) is used, which includes the mobile terminal and the server terminal. The disaster information is transmitted to the system for ground surveys by extensible markup language (XML). The mobile terminal is developed by VC++. NET, and MSXML3.0 is used to parse the XML schema documents. The server terminal is developed by VC++6.0, using data access object to operate the database and MSXML6.0 to establish the dynamic form. Web services are used in the server terminal to receive the field survey data from the mobile terminal. The embedded system is connected with the server through a general packet radio service and the survey results are uploaded using these services. The implementation of the system for ground surveys relies on the previous work summarized in Ref. 22.

Fig. 9

Video monitoring integrated with two-dimensional geographic information system (GIS) and three-dimensional GIS.

3.2.

Experiment Results and Validation

To evaluate the availability of the positioning algorithm, accuracy verification was conducted. First, differential GPS was used to measure the geographic coordinates of simulated disaster points. Combined with local three parameters, the geographic coordinates were transformed into projective coordinates. Next, the simulated disaster points were positioned through the proposed video positioning method. Finally, these two measurement results were compared. The projective coordinates and elevation of the simulated monitoring point are (547,366, 4,466,675, 1420).

From Table 1, we can see that the maximum error for $X$ is 49 m and for $Y$ is 97 m. The minimum discrepancy for $X$ is 21 m and for $Y$ is 45 m. These discrepancies are acceptable for forest pest damage monitoring.²⁴

Table 1

Video monitoring positioning, GPS measuring, and their discrepancies.

Field verification is often considered the preferred method for evaluating insect damage detection derived from video remote sensing. In order to better assess the potential of the proposed approach, two forest areas are selected for field validation. Jinghai County is located in southwestern Tianjin (Fig. 10), with an area of $\sim 1476{\mathrm{km}}^2$. Longitude ranges from 116°42′E to 117°12′E, and latitude ranges from 38°35′N to 39°04′N. The vegetation types of this area are mainly broadleaved deciduous forest. The Hyphantria cunea damage in this area is representative of this city and was, therefore, selected as the test site for this study. Qianshan County is located in southwestern Anhui (Fig. 10), with an area of $\sim 1686{\mathrm{km}}^2$. Longitude ranges from 116°14′E to 116°41′E, and latitude ranges from 30°27′N to 31°04′N. The vegetation types of this region are mainly pinewood and mixed deciduous forest. Planted forests account for 88.4% of the whole county, resulting in single forest form and poor biodiversity, which facilitates the occurrence and spread of insect pests. The Dendrolimus punctatus walker damage in this region is representative of the province, so it was selected as one experimental area. Both Hyphantria cunea and Dendrolimus punctatus walker occur with three generations in a year.

Fig. 10

Forest areas for field validation. One is in Jinghai County of Tianjin, and the other is in Qianshan County of Anhui Province.

To quantitatively evaluate the proposed method, we employ the forward detection ratio (fDR) and reversed detection ratio (rDR) as the assessment principles, which are defined as

According to the different stages of insects, we selected sampling points (Fig. 11) according to the visual features of discoloration and defoliation (Fig. 12). Then, field surveys were carried out to validate the detection results. The data are obtained from the local forest bureau. To further demonstrate the method, we also carried out a reversed field validation task. We selected 20 sampling points by field surveys, respectively, in the two areas and validated them by video monitoring. Tables 2 and 3 show the field validation results.

Fig. 11

Sampling points for accuracy validation: (a) test site in Jinghai County and (b) test site in Qianshan County.

Fig. 12

Pictures from field surveys showing forest insect defoliation and discoloration.

Table 2

Accuracy validation of site 1.

Table 3

Accuracy validation of site 2.

3.3.

Comparative Experiment

Here, an assessment of accuracy was conducted for the detection and positioning of forest insect damage. Currently, the identification of forest insect damage by video cameras is usually conducted by a forest fire video monitoring system. Due to low definition of the camera, it is difficult to identify forest insect damage. In our method, we selected a high-definition camera, so the identification rate is improved, as shown in Table 4. The fDR and rDR of the current method are 0.79 and 0.85 in site 1, respectively. The fDR and rDR of the proposed method are 0.84 and 0.92 in site 1. The fDR and rDR of the current method are 0.81 and 0.76 in site 2. The fDR and rDR of the current method are 0.87 and 0.82 in site 2. Experiments show that the accuracy of the proposed method is higher than the current method.

Table 4

Comparison of the forest insect damage detection method by current method and proposed method.

Traditionally, the geographical position of forest insect damage is determined by comparing the video scene with the corresponding map. There are some unavoidable subjective factors that affect the accuracy. We utilized the traditional method to determine the position of the simulated disaster points, and the comparison results are shown in Table 5. The overall accuracy is calculated as $\text{Overall}=\sqrt{{(\mathrm{\Delta }X)}^2+{(\mathrm{\Delta }Y)}^2+{(\mathrm{\Delta }Z)}^2}$. Experiments show that the proposed positioning method has better overall accuracy than the traditional method. In addition, the proposed method is automatic and avoids the subjectivity of observers to some extent.

Table 5

Comparison of the positioning method by traditional method and proposed method.