CN108564109A - A kind of Remote Sensing Target detection method based on deep learning - Google Patents

Abstract

The Remote Sensing Target detection method based on deep learning that the present invention relates to a kind of, including：Associated data set is built using remote sensing images：After remote sensing images are classified and are marked, image data set and the class label generated by markers work；It builds based on the panchromatic sharpening model for generating confrontation network；The target detection model based on depth convolutional neural networks is built, end-to-end training is carried out to model by the methods of backpropagation and stochastic gradient descent；End-to-end test is carried out to the model built.The present invention has the advantages that accuracy is high.

Description

A remote sensing image target detection method based on deep learning

technical field

The invention relates to the fields of remote sensing image processing, deep learning, pattern recognition, etc., and in particular to a method based on deep learning and using a generative confrontation network to perform panchromatic sharpening processing and target detection on spectral images.

Background technique

Due to the limitation of signal transmission band and imaging sensor storage, most remote sensing satellites only provide multispectral (MSI) images with high spectral resolution, and panchromatic (PAN) images with high spatial resolution. Utilizing the complementary advantages of the two images, they are fused into a fused remote sensing image with clear spatial details and rich spectral information. This fusion technology is also known as panchromatic sharpening technology.

At present, the mainstream panchromatic sharpening methods in the field of remote sensing include component replacement method and multi-scale analysis method. The component replacement method mainly transforms the spectral image in the color space domain through principal component analysis, Schmidt orthogonalization, and intensity, hue, and saturation transformations, and replaces the spatial information channel of the multispectral image with a panchromatic image , the fused image is obtained by the inverse transformation.

Multi-scale analysis method refers to the method of decomposing the source image into a sequence of decomposition coefficients by using multi-resolution analysis tools based on wavelet transform, Laplace pyramid and multi-scale geometric analysis, and then passing these decomposition coefficients through a fusion criterion Merge into the decomposition coefficients of the fused image, and finally obtain the fused image through the inverse transformation of the multi-resolution analysis tool.

In recent years, based on the emergence of large-scale data and the development of deep neural networks, deep learning methods have become an important research direction in the field of machine learning. Based on the idea of deep learning technology and game theory, Generative Adversarial Networks (GAN), which can generate high-dimensional samples from low-dimensional features, can be introduced for the pan-color sharpening process. GAN consists of a generative network model and a discriminative network model. constituted. The generative model can help generate relevant sample data, while the discriminative model can judge the authenticity of the sample. The two are trained at the same time, and the generative model is continuously strengthened. Through continuous iteration, the generated samples are getting closer and closer to the real samples.

As an important application of pattern recognition in the field of remote sensing, multi-type and multi-scale target detection and recognition based on remote sensing images is a key technology in the fields of geographic survey, military reconnaissance and precision strikes. How to improve the accuracy of target detection has also been It is a research hotspot and difficulty in the field of remote sensing applications, and has important military and civilian values. With the rapid development of high-resolution remote sensing technology, large-scale high-resolution remote sensing image data sets can be built, which provides the possibility for the development of more intelligent remote sensing image target detection systems. Extracting effective features of targets from massive data has become a remote sensing image application. key technologies.

Traditional detection algorithms are almost all based on a given feature to complete the classification and detection work. The extracted features and the detection model are two important factors that determine the detection effect, and play a vital role in the model. This requires strict requirements on the input features and finding a detection model that matches the features.

However, the above requirements are undoubtedly complex and time-consuming, and strongly depend on professional knowledge and the characteristics of the data itself. In addition, it is difficult to learn an effective classification model from large-scale data to fully exploit the interaction between large-scale data. associated.

With the development of deep learning methods, it becomes possible to use raw data as input to realize an end-to-end learning process. The deep artificial neural network has a strong feature learning ability. The feature data learned by the deep learning model is more representative of the original data. The target detection model based on deep learning technology trained on large-scale data can extract its richness. Intrinsic information, which is beneficial to visualization and classification problem processing. Therefore, based on the convolutional neural network, it is possible to design a method that can automatically learn features, obtain the most effective deep features by learning a large amount of data itself, and fully mine the relationship between data by establishing a relatively complex network structure. .

Contents of the invention

The purpose of the present invention is to provide a remote sensing image target detection method based on deep learning. The present invention is applied to the generative confrontation network model of panchromatic sharpening, which can realize the expansion of information content in remote sensing images; deep convolution applied to target detection The neural network model has higher accuracy, better real-time performance and stronger robustness. In order to realize the above-mentioned purpose of the invention, the present invention adopts following technical scheme:

A remote sensing image target detection method based on deep learning, comprising the following steps:

1) Construct related datasets using remote sensing images: After classifying and labeling remote sensing images, image datasets and category labels generated through labeling work are divided into training sets and test sets for subsequent network training and testing;

2) Build a panchromatic sharpening model based on the generative confrontation network: GAN's generative network G can learn the mapping from the random noise vector z and the image x in the data set described in 1 to the generated sample image y, namely G: {x,z}→y, the generative model adopts the U-Net structure with added jump connections, and is divided into two parts: the encoding layer and the decoding layer. For each encoding layer, the length and width of the feature map are halved, and the number of feature layers is increased by half. , for each decoding layer, the length and width of the feature map are doubled, the number of feature layers is doubled, and the corresponding coding layer is concatenated through channels, and then deconvolution processing; based on the convolutional neural network CNN for classification, the design Discriminant network model, which is designed to contain a concatenated layer and four convolutional layers;

3) Building a target detection model based on a deep convolutional neural network: According to the four steps of target detection algorithm candidate area generation, feature extraction, classification, and position refinement, the above steps are unified into a deep network framework, and the GPU Internal parallel operation, feature extraction uses the residual network ResNet as the basic classification network, which includes several convolutional layers and linear unit ReLU, and designs the region generation network structure. On the extracted feature map, all possible candidate frames are discriminated. By sharing convolution, the marginal cost of calculating the proposal box is reduced, and the model is trained end-to-end by backpropagation and stochastic gradient descent;

4) End-to-end testing of the built model: based on the data set built in step 1, train the target detection model and test the model.

Compared with the prior art, the promotion and advantages of the present invention are:

1. Different from the idea of all existing remote sensing image target detection methods, the present invention innovatively proposes a method based on cascading deep learning methods that first performs pan-color sharpening processing and then performs target detection. In the process of panchromatic sharpening, GAN can use deep convolutional neural network to extract high-dimensional deep features hidden in large-scale data, and its structure can also minimize the information loss in the convolution process. Remote sensing images after panchromatic sharpening have clear spatial details and rich spectral information. Based on the abundance of information, the improvement of spatial resolution has more practical significance for the detection of small targets. It is more efficient to directly perform end-to-end detection on images through the trained model, and the time and calculation redundancy is lower.

2. Different from the existing traditional methods of remote sensing image target detection, the present invention innovatively proposes a target detection network model based on a deep convolutional neural network. Compared with manual feature extraction such as HOG features, the features can be directly represented by the feature map obtained after the convolutional neural network in the data. Since the convolution operation has translation and no deformation, the feature map not only contains the category information of the object, but also contains The location information of the object, so the classification result of the feature and the location regression have better accuracy and stronger universality. The invention adopts the region generation network, and the region recommendation is also completed in the network, and the whole process from feature extraction to final detection is completed in one network, the speed is improved higher, and the fitting related problems are solved at the same time.

Description of drawings

Figure 1 is a flowchart of the experiments required for the present invention.

Figure 2 is a schematic diagram of the generative network structure for panchromatic sharpening.

Fig. 3 is a panchromatic sharpening effect diagram of a remote sensing image, (a) panchromatic remote sensing image data; (b) spectral remote sensing image data; (c) fusion remote sensing image data; (d) fusion remote sensing image data according to the present invention.

Figure 4 is a structural diagram of the region generation network.

Detailed ways

In order to make the technical solution of the present invention clearer, the specific implementation manners of the present invention will be further described below. As shown in Figure 1, the present invention is concretely realized according to the following steps:

1. Construct a large-scale remote sensing image dataset

The present invention selects remote sensing image collections such as SpaceNet on AWS, NWPU VHR-10, and US Geological Survey USGS published on the Internet to construct a data set for detection tasks.

The NWPU VHR-10 dataset is a publicly available ten-category geospatial object detection dataset. These ten categories of items are aircraft, ships, oil storage tanks, ports and bridges, etc., including high-resolution images and label files of objects in the pictures and their annotations.

SpaceNet is a large-scale remote sensing image data set hosted on Amazon's AWS cloud service platform. It was jointly completed by DigitalGlobe, CosmiQ Works and NVIDIA. It contains an online repository of satellite images and marked training data. It is a publicly released high-resolution High-rate, satellite imagery data platform dedicated to training machine learning algorithms. In addition, the present invention also combines relevant remote sensing data from the Chinese Academy of Sciences geospatial data cloud platform, the United States Geological Survey (USGS) and Google to build the required data sets for training and testing.

The image data in the above data set is divided into training set and test set according to the ratio of 4:1. After the present invention classifies and marks them, the present invention produces image data sets and labels according to the format of the PASCAL VOC challenge for subsequent network training and testing.

2. Build a panchromatic sharpening model based on generative confrontation network

Based on the relevant remote sensing database constructed in 1, a generative adversarial neural network for panchromatic sharpening of remote sensing images is built and trained. This step is to provide remote sensing image data with high spatial and spectral resolution for subsequent detection. The generative confrontation network for pan-color sharpening consists of two networks, a generative network and a discriminative network. The generative network and the discriminative network are usually composed of multi-layer networks including convolutional and/or fully connected layers. Through multiple experimental tests on the network structure with excellent effect, the present invention constructs a convolutional neural network based on the U-Net network as a generation network.

Use the U-Net architecture of the full convolution structure to build a generation network, and build a discriminant network architecture with different sizes of receptive fields. By using the convolution kernel in the U-Net network to achieve downsampling, it can not only reduce the redundancy of the operation, but also extract the abstract features of the target to a certain extent; use a variety of different convolution kernels to convolve the image operation, the responses on different kernels can be obtained as features of the image. A new image matrix is obtained after the input image matrix undergoes a convolution kernel (kernal) operation, that is, a feature map.

The units connected thereafter can keep the scale of the feature map unchanged. In addition, the pooling layer in the network is replaced by a convolutional layer with a constant feature map scale; the fully connected layer in the network is deleted, and the deconvolution layer is used to realize the image. The upsampling, here can process the features output by the shallow convolution layer and the deep convolution layer to improve the accuracy of feature extraction.

Input the random noise signal z vector and the panchromatic image x in the database into the generation network as described above, and use the image data y generated by the generation network as the input of the discriminant network, that is, G: {x,z}→y, after training , the generated samples cannot be discriminated as false by the discriminative network model. The discriminative network model D, after training, can complete the classification problem of discriminating and generating samples as well as possible.

The training target of GAN can be expressed by the following loss function formula, where x is the existing input image, y is the output sample image, and z is the random noise vector:

Both the generative model and the discriminative model adopt the structure of convolutional layer-batch normalization-linear unit, and GAN processes the details such as high-frequency structural information in the image. During the training process, the generative model minimizes the target, while the discriminative model to maximize

G ^* ＝argmin _G max _D L _cGAN (G,D)

During the generation of the final sample image, the input panchromatic image and the output fused image have the same underlying structure, sharing the positions of salient edges. In order to enable the generative model to obtain this information, the generative model adds jump connections and adopts the overall structure of U-Net.

U-Net is a fully convolutional structure, which is based on the traditional encoder-decoder architecture, adding skip links between the corresponding layers of the encoding module and the decoding module (layers with feature maps of the same size) .

The U-Net network is divided into two parts, the encoding layer (eight layers in total) and the decoding layer (eight layers in total). For each encoding layer, the length and width of the feature map are halved, and the number of feature layers is increased by half. layer, the length and width of the feature map are doubled, and the number of feature layers is doubled, that is, it is also connected with the corresponding coding layer through channels, and then deconvolution processing is performed.

A mirror operation is performed around the input image, the number of convolutional layers is designed to be 20, 4 times of downsampling, and 4 times of upsampling. Specifically, for an n-layer network, the present invention adds a jump connection between each i-th layer and n-i-th layer, and connects all channels in the i-th layer to the n-i-th layer.

Based on the convolutional neural network (CNN) for classification, a discriminative network model is designed. The training order of the discriminative network is before the generative network, and the discriminative model can actually act as its loss function for the generative model, so the discriminator is more fully trained than the generator to provide the correct target for the convergence of the generator. The network is designed with one concatenated layer and four convolutional layers. The CNN with reduced parameter design only classifies the authenticity of each block in the generated fused image, runs the network convolutionally on the image, and averages all the responses to provide the final output of D.

3. Build a target detection model based on a deep convolutional neural network. Based on deep learning, the present invention can learn more useful features from large-scale training data by constructing a neural network model with multiple hidden layers, thereby finally improving the accuracy of classification or prediction. In order to realize the high precision and high adaptability of remote sensing target detection, here, the present invention adopts the target detection network structure of basic feature extraction network + area generation network + classification network to construct the detection network model.

The implementation steps of the detection network algorithm based on deep network designed by the present invention are as follows:

(1) Input the remote sensing image after panchromatic sharpening;

(2) Input the whole picture into the convolutional neural network for feature extraction;

(3) Use RPN to generate suggestion windows, and generate 300 suggestion windows for each picture;

(4) Map the suggestion window to the last layer of convolutional feature map of the convolutional neural network;

(5) Generate a fixed-size feature map for each region of interest through the pooling layer;

(6) Joint training of classification probability and position regression using detection classification probability and detection bounding box regression.

Among them, the feature extraction network adopts the residual network structure (ResNet), and through the residual network, the deepening of the network structure and the significant improvement of the classification effect are realized. Compared with the traditional convolutional neural network such as VGG, the complexity of the residual network is reduced, and the required parameter reduction can be made deeper, and there will be no problem of gradient dispersion. The deep convolutional residual network is to learn the mapping from input to (output-input), thereby obtaining prior information that the output is composed of input parts.

First construct a 18-layer and a 34-layer residual network, and insert shortcuts on the simple network, which can greatly reduce the amount of calculation. By introducing a shortcut connection between the output and the input, instead of the simple stacked network in the traditional method, the problem of gradient disappearance in the deep network is solved.

The Region Proposal Network (RPN) can discriminate all possible candidate frames that are currently relatively sparse on the feature map extracted by ResNet. Using the mapping mechanism of SPP-Net, the region generation network maps back to the original image from the convolutional layer according to the one-to-one corresponding points, and trains the network according to the design of different fixed initial scales. Negative labels, so that it learns whether there is a target object in it.

In order to reduce the computational complexity of the region generation network, based on the deep network, the convolution calculation results can be shared, the scale change, scale change and sampling method can be fixed, and then the target candidate region, that is, the candidate window of the feature, can be obtained. First, 9 candidate windows are generated according to the scale and aspect ratio, and a fixed-size window is used to slide on the last feature map of the convolution. Each window will output features of a fixed size dimension, and each window is suitable for the 9 candidates. Perform regression coordinates and classification.

The objective function of the region generation network is the sum of classification and regression losses. Classification uses cross entropy, regression uses stable Smooth L1, and its formula can be expressed as:

The overall loss function is specifically:

The loss function is divided into two parts, corresponding to the two branches of the region generation network, that is, the classification error of the target or not and the regression error of the detection frame, where Using a smooth L1 function, it is easier to adjust the learning rate than the error of the L2 form. For the lettering of the detection frame, only the candidate windows that are judged to have targets are considered, and the coordinates marked by them are used as the target of the letter. In addition, when calculating the error of the detection frame, instead of comparing the coordinates of the four corners, it is t _x , t _Y , t _W , t _H , as described below, the calculation method of the specific four dimensions:

t _X ＝(xx _a )/w _a , t _Y ＝(yy _a )/h _a ,

t _W ＝log(w/w _a ), t _h ＝log(h/h _a ),

t _X ^* ＝(x ^* -x _a )/w _a ,t _Y ^* ＝(y ^* -y _a )/h _a ,

t _W ^* ＝log(w ^* /w _a ), t _h ^* ＝log(h ^* /h _a ),

During testing, the region of interest (ROI) pooling layer obtains a list of candidate ROIs from the region generation network, gets all the features through the convolutional layer, and performs subsequent classification and regression. By sharing the convolutional layers that generate proposal windows with the region generation network and the detection network, sharing between generated candidates and detections can be achieved.

The training process of the above network adopts a four-step training method. In the first step, the network is generated by training the region separately, and the network parameters are loaded from the pre-trained model; in the second step, the detection network is trained separately, and the output candidate region of the region generation network in the first step is as input to the detection network. The region generation network outputs a candidate frame, intercepts the original image through the candidate frame, and passes the intercepted image through several convolution-pooling operations, and then outputs two branches through ROI pooling, which are the detection and classification probabilities of the target classification (Softmax Loss) and detection frame regression (Smooth L1 Loss). The third step is to train the region generation network again. At this time, the parameters of the common part of the network are fixed, and only the parameters of the unique part of the region generation network are updated; finally, according to the results of the region generation network, the detection network structure is fine-tuned again, and the parameters of the public part are fixed. Only parameters that are unique to the detection network framework are updated.

4. Cascade the network for end-to-end testing. Input the panchromatic remote sensing image in the test set of the database, and perform panchromatic sharpening processing on it according to the trained generation confrontation network. Then input it into the constructed detection network model, and evaluate its detection results. For the remote sensing images in the database, on the test data set constructed in the content of the invention 1, a data set based on the PASCAL VOC competition format was constructed according to the spectral images before and after fusion, and it was verified that the remote sensing image after panchromatic sharpening has better detection Results: Using the traditional image processing and classification method and the deep detection network constructed in this paper to conduct detection experiments, the five types of military and civilian targets such as aircraft, ships, oil storage tanks, bridges, and ports in the picture are detected, and the detection effect is significantly better than that of traditional algorithms. promote.

The detection and evaluation methods are as follows:

The number of all pictures tested by the system is ALL, and the number of images in the picture set 1 that the system recognizes that there are five types of targets to be detected is recorded as F, including those that originally had no target but recognized a target, and those that originally had a target. The number of correctly identified pictures is recorded as FP and FN respectively, then F=FP+FN; the number of images in the picture set 2 that the system recognizes that no target exists is recorded as T, including those that were originally recognized correctly without a target, and The number of pictures that originally had a target but were not recognized are denoted as TP and TN respectively, then T=TP+TN. According to the actual identification needs, the system defines the following indicators:

Claims (1)

1. a kind of Remote Sensing Target detection method based on deep learning, includes the following steps：

1) remote sensing images are utilized to build associated data set：After remote sensing images are classified and are marked, image data set and process The class label that markers work generates, and training set and test set are divided, it is used for subsequent network training and test；

2) it builds based on the panchromatic sharpening model for generating confrontation network：The generation network G of GAN be can learn from random noise to The image x in data set described in z and 1 is measured, to the mapping of the sample image y generated, i.e. G：{ x, z } → y generates model and takes The U-Net structures for redirecting connection are increased, coding layer and decoding layer two parts are divided into, often encode one layer, characteristic pattern length and width subtract Half, the feature number of plies increases half, often decodes one layer, the length and width of characteristic pattern double, and feature number of plies increase doubles and corresponding volume Code layer is concatenated by channel, then carries out deconvolution processing；Based on the convolutional neural networks CNN for classification, design differentiates net Network model, the network are designed to containing there are one concatenation layer and four layers of convolutional layers；

3) the target detection model based on depth convolutional neural networks is built：It is generated according to the candidate region of algorithm of target detection, Feature extraction, classification, four steps of position refine, within above-mentioned steps unification to a depth network frame, in GPU Concurrent operation, feature extraction is using residual error network ResNet as basic sorter network, wherein including several convolutional layers and linear list First ReLU, design section generate network structure and differentiate to all possible candidate frame on the characteristic pattern extracted, lead to Shared convolution is crossed, the marginal cost for calculating Suggestion box is reduced, model is carried out by backpropagation and stochastic gradient descent method End-to-end training；

4) end-to-end test is carried out to the model built：Based on the data set built in step 1, training objective detection model And test model.