CN108171663A - The image completion system for the convolutional neural networks that feature based figure arest neighbors is replaced - Google Patents

Abstract

The image filling system of the convolutional neural network based on the nearest neighbor replacement of the feature map belongs to the field of image filling technology, and solves the problem that the existing image filling method cannot quickly obtain the filling image with consistent overall semantics and good clarity. The system: the generating network encodes the image to be filled and then decodes it to obtain the filled image. The decoder of the generation network includes N deconvolution layers. For any M deconvolution layers in the first deconvolution layer to the N-1th deconvolution layer, the generation network is based on the output of each deconvolution layer The result and the output result of the convolutional layer corresponding to the deconvolution layer, and the feature map nearest neighbor replacement method is used to obtain the additional feature map, and the output result of each deconvolution layer, the convolution layer corresponding to the deconvolution layer The output of the convolution layer and the additional feature map are used as the input object of the next deconvolution layer. The discriminative network is used to judge whether the filled image is the real image corresponding to the image to be filled.

Description

Image Filling System Based on Convolutional Neural Networks with Feature Map Nearest Neighbor Replacement

technical field

The invention relates to an image filling system, which belongs to the technical field of image filling.

Background technique

Image filling is a basic problem in the field of computer vision and image processing, which is mainly used to repair and reconstruct damaged images or remove redundant objects in images.

Existing image filling methods mainly include diffusion-based image filling methods, sample-based image filling methods and deep learning-based image filling methods.

The basic idea of the diffusion-based image filling method is to diffuse the image information at the edge of the area to be filled to the inside of the area to be filled in units of pixels. When the area to be filled is small, the structure is simple, and the texture is single, the image filling method can better complete the image filling task. However, when the area to be filled is large, the clarity of the filled image obtained by using this image filling method is poor.

The basic idea of the sample-based image filling method is: take the image block as a unit, and gradually fill from the known area of the image to the area to be filled. Each time an image block is filled, the area to be filled is filled with the image block in the known area of the image that is most similar to the edge image block of the area to be filled. Compared with the diffusion-based image filling method, the sample-based image filling method obtains better texture and higher definition of the filled image. However, since the sample-based image filling method uses similar image blocks in the known area of the image to gradually replace the unknown image blocks in the area to be filled, it is impossible to obtain an overall semantically consistent filled image using this image filling method.

The image filling method based on deep learning mainly refers to the application of deep neural network to the field of image filling. At present, some scholars have proposed to use the encoder-decoder network to fill in the missing image in the middle area. However, this image filling method is only suitable for 128*128 RGB images. Although the filled image obtained by this image filling method can meet the requirement of overall semantic consistency, the clarity of the filled image is poor. In response to this problem, some scholars try to clear fill large images by using multi-scale iterative updates. However, although the filled image obtained by this image filling method has overall semantic consistency and good clarity, its speed is extremely slow. Under the running environment of the Titan X graphics card, it takes tens of seconds to several minutes to fill a 256*256 RGB image.

Contents of the invention

In order to solve the problem that the existing image filling method cannot quickly obtain a filling image with consistent overall semantics and good definition, the present invention proposes an image filling system based on a convolutional neural network with feature map nearest neighbor replacement.

The image filling system of the convolutional neural network based on the feature map nearest neighbor replacement of the present invention includes a generation network and a discrimination network;

The generation network includes an encoder and a decoder, the encoder includes N convolutional layers, and the decoder includes N deconvolutional layers, N≥2;

The generation network obtains the filled image by first encoding and then decoding the image to be filled;

For any M deconvolution layers in the first deconvolution layer to the N-1th deconvolution layer, the generation network is based on the output result of each deconvolution layer and the convolution layer corresponding to the deconvolution layer. Output the result, and use the feature map nearest neighbor replacement method to obtain an additional feature map, and use the output result of each deconvolution layer, the output result of the convolution layer corresponding to the deconvolution layer and the obtained additional feature map as The input object of the next deconvolution layer;

1≤M≤N-1;

The discriminative network is used to judge whether the filled image is the real image corresponding to the image to be filled, and then constrains the weight learning of the generating network.

Preferably, the encoder includes a convolutional layer E ₁ to a convolutional layer E ₈ , and the decoder includes a deconvolutional layer D ₁ to a deconvolutional layer D ₈ ;

The image to be filled is the input object of the convolutional layer _E1 ;

For the convolutional layer E ₁ ~ convolutional layer E ₈ , the output result of the former is used as the input object of the latter after batch normalization and Leaky ReLU function activation in turn;

The output result of the convolutional layer E ₈ is used as the input object of the deconvolutional layer D ₁ after being sequentially activated by batch normalization and Leaky ReLU function;

The output result of the deconvolution layer _D1 is activated by the ReLU function as the first input object of the deconvolution layer _D2 ;

For the deconvolution layer D ₂ ~ deconvolution layer D ₈ , the output of the former is the first input object of the latter after being activated by the ReLU function and batch normalized in turn;

The second input objects of deconvolution layer D ₂ to deconvolution layer D ₈ are the output results of convolution layer E ₇ to convolution layer E ₁ after batch normalization and Leaky ReLU function activation in sequence;

The output result of the deconvolution layer D ₈ activated by the Tanh function is a filled image;

The convolutional layer E ₁ is used to perform 64 convolution operations of 4*4 with a step size of 2 on the input object;

The convolutional layer E ₂ is used to perform 128 convolution operations of 4*4 with a step size of 2 on the input object;

The convolutional layer E ₃ is used to perform 256 convolution operations of 4*4 with a step size of 2 on the input object;

The convolutional layers E ₄ to E ₈ are all used to perform 512 convolution operations of 4*4 with a step size of 2 on the input object;

The deconvolution layer D ₁ ~ deconvolution layer D ₄ are all used to perform 512 deconvolution operations of 4*4 with a step size of 2 on the input object;

The deconvolution layer D ₅ is used to perform 256 4*4 deconvolution operations with a step size of 2 on the input object;

The deconvolution layer D ₆ is used to perform 128 deconvolution operations of 4*4 with a step size of 2 on the input object;

The deconvolution layer D ₇ is used to perform 64 deconvolution operations of 4*4 with a step size of 2 on the input object;

The deconvolution layer D ₈ is used to perform three 4*4 deconvolution operations with a step size of 2 on the input object;

The generation network is based on the output results of the deconvolution layer _D5 and the output results of the convolution layer _E3 , and adopts the method of feature map nearest neighbor replacement to obtain additional feature maps, and uses the additional feature maps as the output of the deconvolution layer _D6 . The third input object.

Preferably, the generation network is based on the output result of the deconvolution layer _D5 and the output result of the convolution layer _E3 , and the specific process of obtaining the additional feature map by means of feature map nearest neighbor replacement is as follows:

Select a feature map to be assigned with a feature value of 0, which has the same number of channels and the same space size as the output feature map of the deconvolution layer _D5 and the output feature map of the convolution layer _E3 ;

Calculate the masked area of the output feature map of the deconvolution layer _D5 and the non-masked area of the output feature map of the convolutional layer _E3 , and simultaneously cut the masked area and the non-masked area into multiple a feature block;

A plurality of feature blocks are cuboids with a size of C*h*w, wherein C, h and w are respectively the number of channels of the output feature map of the deconvolution layer _D5 , the length of the cuboid and the width of the cuboid;

For each feature block p ₁ in the masked area, select the feature block p ₂ closest to the feature block p ₁ among the multiple feature blocks in the non-masked area;

Select the region to be assigned in the feature map to be assigned, the region to be assigned is consistent with the position of the feature block _p1 in the output feature map of the deconvolution layer _D5 ;

Assign the feature value of the feature block _p2 to the area to be assigned.

Preferably, the cosine distance between the feature block p ₂ and the feature block p ₁ is the shortest.

Preferably, the calculation method of the mask area and the non-mask area of the output feature map is:

Given a mask image to replace the image to be filled, the mask image has the same size as the image to be filled, the number of channels is 1, and the feature value is 0 or 1;

0 indicates that the corresponding position of the feature point on the image to be filled is not a point to be filled;

1 indicates that the corresponding position of the feature point on the image to be filled is the point to be filled;

calculating the mask area and non-mask area of the feature map of the mask image through a convolutional network, the convolutional network includes a first convolutional layer to a third convolutional layer;

The mask image is the input object of the first convolutional layer;

For the first convolutional layer to the third convolutional layer, the output of the former is the input object of the latter;

The first convolutional layer to the third convolutional layer are used to perform a 4*4 convolution operation with a step size of 2 on the input object;

The output of the third convolutional layer is the feature map of the mask image, which has a size of 32*32 and a channel of 1;

For the feature map of the mask image, when one of the feature values is greater than the set threshold, the feature point is determined to be a mask point, otherwise, the feature point is determined to be a non-mask point;

The mask area of the feature map of the mask image is a set of mask points, and the non-mask area of the feature map of the mask image is a set of non-mask points;

The masked area of the output feature map is equal to the masked area of the feature map of the mask image, and the unmasked area of the output feature map is equal to the non-masked area of the feature map of the mask image.

Preferably, the generation network is trained in a guided loss constraint, and the specific method of guiding the loss constraint is to perform feature extraction on the real image and the input image in any convolution layer or deconvolution layer during the training process of the generation network. similar constraints;

The input image is a masked real image.

Preferably, the specific way to generate the network for training is as follows:

Input the target image I ^gt to the generation network, calculate the mask area of the feature map of the l-th layer, and obtain (Φ _l (I ^gt )) _y information;

Input the image I to be filled into the generation network, calculate the mask area of the feature map of the L1 layer, and obtain (Φ _L1 (I)) _y information;

At this point define the bootstrap loss constraint L _g :

In the formula, Ω is the mask area, L is the total number of layers of the generated network, y is any coordinate point in the mask area, Φ _Ll (I) is when the input object is an image to be filled, The feature map output by the layer, (Φ _Ll (I)) _y is the information of y in the mask area of the output feature map of the Ll layer, Φ _l (I ^gt ) is when the input object is the target image, the generation network The feature map output by the layer, (Φ _l (I ^gt )) _y is the information of y in the mask area of the output feature map of the l-th layer.

Preferably, the discriminant network includes a convolutional layer E ₉ to a convolutional layer E ₁₃ ;

The input object of the convolutional layer E ₉ is a filled image;

The output result of the convolutional layer E ₉ is activated by the Leaky ReLU function as the input object of the convolutional layer E ₁₀ ;

For the convolutional layer E ₁₀ ~ convolutional layer E ₁₃ , the output result of the former is used as the input object of the latter after batch normalization and Leaky ReLU function activation in turn;

The output result of the convolutional layer E ₁₃ after successive batch normalization and Sigmoid function activation is the output result of the discriminant network;

The convolutional layer E ₉ is used to perform 64 convolution operations of 4*4 with a step size of 2 on the input object;

The convolutional layer E ₁₀ is used to perform 128 convolution operations of 4*4 with a step size of 2 on the input object;

The convolutional layer E ₁₁ is used to perform 256 convolution operations of 4*4 with a step size of 2 on the input object;

The convolutional layer E ₁₂ is used to perform 512 4*4 convolution operations with a step size of 1 on the input object;

The convolutional layer E ₁₃ is used to perform a 4*4 convolution operation with a step size of 1 on the input object.

Preferably, the filled image is a 256*256 RGB image, the output result of the convolutional layer _E13 has a space size of 64*64, and a channel of 1.

Preferably, the image filling system uses Adam optimization algorithm for end-to-end training.

The image filling system of the convolutional neural network based on the nearest neighbor replacement of the feature map according to the present invention takes the image to be filled as its input object, and performs the nearest neighbor replacement of the feature map through the intermediate output of the decoding part of the generation network, so that a forward Propagation results in a filled image with overall semantic consistency and good clarity. Compared with existing image filling methods, the image filling system can obtain filled images more quickly because it only needs to perform one forward pass.

Description of drawings

In the following, the image filling system of the convolutional neural network based on feature map nearest neighbor replacement according to the present invention will be described in more detail based on the embodiments and with reference to the accompanying drawings, wherein:

Fig. 1 is the structural block diagram of the generation network that embodiment mentions;

Fig. 2 is the block diagram of the discriminant network mentioned in the embodiment;

Figure 3 is any missing image to be filled;

Figure 4 is the filled image obtained after inputting any missing image to be filled into the generation network;

Figure 5 is an image to be filled with a missing center;

Figure 6 shows the filled image obtained after inputting the image to be filled with missing center into the generation network.

Detailed ways

The image filling system based on the convolutional neural network with feature map nearest neighbor replacement according to the present invention will be further described below with reference to the accompanying drawings.

Embodiment: The present embodiment will be described in detail below with reference to FIGS. 1 to 6 .

The image filling system based on the convolutional neural network of feature map nearest neighbor replacement described in this embodiment includes a generation network and a discrimination network;

The generation network includes an encoder and a decoder, the encoder includes N convolutional layers, and the decoder includes N deconvolutional layers, N≥2;

The generation network obtains the filled image by first encoding and then decoding the image to be filled;

For any M deconvolution layers in the first deconvolution layer to the N-1th deconvolution layer, the generation network is based on the output result of each deconvolution layer and the convolution layer corresponding to the deconvolution layer. Output the result, and use the feature map nearest neighbor replacement method to obtain an additional feature map, and use the output result of each deconvolution layer, the output result of the convolution layer corresponding to the deconvolution layer and the obtained additional feature map as The input object of the next deconvolution layer;

1≤M≤N-1;

The discriminative network is used to judge whether the filled image is the real image corresponding to the image to be filled, and then constrains the weight learning of the generating network.

The encoder in this embodiment includes a convolutional layer E ₁ to a convolutional layer E ₈ , and the decoder includes a deconvolutional layer D ₁ to a deconvolutional layer D ₈ ;

The image to be filled is the input object of the convolutional layer _E1 ;

For the convolutional layer E ₁ ~ convolutional layer E ₈ , the output result of the former is used as the input object of the latter after batch normalization and Leaky ReLU function activation in turn;

The output result of the convolutional layer E ₈ is used as the input object of the deconvolutional layer D ₁ after being sequentially activated by batch normalization and Leaky ReLU function;

The output result of the deconvolution layer _D1 is activated by the ReLU function as the first input object of the deconvolution layer _D2 ;

For the deconvolution layer D ₂ ~ deconvolution layer D ₈ , the output of the former is the first input object of the latter after being activated by the ReLU function and batch normalized in turn;

The second input objects of deconvolution layer D ₂ to deconvolution layer D ₈ are the output results of convolution layer E ₇ to convolution layer E ₁ after batch normalization and Leaky ReLU function activation in sequence;

The output result of the deconvolution layer D ₈ activated by the Tanh function is a filled image;

The convolutional layer E ₁ is used to perform 64 convolution operations of 4*4 with a step size of 2 on the input object;

The convolutional layer E ₂ is used to perform 128 convolution operations of 4*4 with a step size of 2 on the input object;

The convolutional layer E ₃ is used to perform 256 convolution operations of 4*4 with a step size of 2 on the input object;

The convolutional layers E ₄ to E ₈ are all used to perform 512 convolution operations of 4*4 with a step size of 2 on the input object;

The deconvolution layer D ₁ ~ deconvolution layer D ₄ are all used to perform 512 deconvolution operations of 4*4 with a step size of 2 on the input object;

The deconvolution layer D ₅ is used to perform 256 4*4 deconvolution operations with a step size of 2 on the input object;

The deconvolution layer D ₆ is used to perform 128 deconvolution operations of 4*4 with a step size of 2 on the input object;

The deconvolution layer D ₇ is used to perform 64 deconvolution operations of 4*4 with a step size of 2 on the input object;

The deconvolution layer D ₈ is used to perform three 4*4 deconvolution operations with a step size of 2 on the input object;

The generation network is based on the output results of the deconvolution layer _D5 and the output results of the convolution layer _E3 , and adopts the method of feature map nearest neighbor replacement to obtain additional feature maps, and uses the additional feature maps as the output of the deconvolution layer _D6 . The third input object.

The generation network of this embodiment is based on the output result of the deconvolution layer _D5 and the output result of the convolution layer _E3 , and the specific process of obtaining additional feature maps by means of feature map nearest neighbor replacement is as follows:

Select a feature map to be assigned with a feature value of 0, which has the same number of channels and the same space size as the output feature map of the deconvolution layer _D5 and the output feature map of the convolution layer _E3 ;

Calculate the masked area of the output feature map of the deconvolution layer _D5 and the non-masked area of the output feature map of the convolutional layer _E3 , and simultaneously cut the masked area and the non-masked area into multiple a feature block;

A plurality of feature blocks are cuboids with a size of C*h*w, wherein C, h and w are respectively the number of channels of the output feature map of the deconvolution layer _D5 , the length of the cuboid and the width of the cuboid;

For each feature block p ₁ in the masked area, select the feature block p ₂ closest to the feature block p ₁ among the multiple feature blocks in the non-masked area;

Select the region to be assigned in the feature map to be assigned, the region to be assigned is consistent with the position of the feature block _p1 in the output feature map of the deconvolution layer _D5 ;

Assign the feature value of the feature block _p2 to the area to be assigned.

The calculation method of the masked area and non-masked area of the output feature map is:

Given a mask image to replace the image to be filled, the mask image has the same size as the image to be filled, the number of channels is 1, and the feature value is 0 or 1;

0 indicates that the corresponding position of the feature point on the image to be filled is not a point to be filled;

1 indicates that the corresponding position of the feature point on the image to be filled is the point to be filled;

calculating the mask area and non-mask area of the feature map of the mask image through a convolutional network, the convolutional network includes a first convolutional layer to a third convolutional layer;

The mask image is the input object of the first convolutional layer;

For the first convolutional layer to the third convolutional layer, the output of the former is the input object of the latter;

The first convolutional layer to the third convolutional layer are used to perform a 4*4 convolution operation with a step size of 2 on the input object;

The output of the third convolutional layer is the feature map of the mask image, which has a size of 32*32 and a channel of 1;

For the feature map of the mask image, when one of the feature values is greater than the set threshold, the feature point is determined to be a mask point, otherwise, the feature point is determined to be a non-mask point;

The mask area of the feature map of the mask image is a set of mask points, and the non-mask area of the feature map of the mask image is a set of non-mask points;

The masked area of the output feature map is equal to the masked area of the feature map of the mask image, and the unmasked area of the output feature map is equal to the non-masked area of the feature map of the mask image.

The generation network of this embodiment is trained in a guided loss constraint. The specific method of the guidance loss constraint is to perform feature similarity between the real image and the input image in any convolution layer or deconvolution layer during the generation network training process. constraint;

The input image is a masked real image.

The specific way for the generation network of this embodiment to train is as follows:

Input the target image Igt to the generation network, calculate the mask area of the feature map of the l-th layer, and obtain (Φ _l (I ^gt )) _y information;

Input the image I to be filled into the generation network, calculate the mask area of the feature map of the L1 layer, and obtain (Φ _L1 (I)) _y information;

At this point define the bootstrap loss constraint L _g :

In the formula, Ω is the mask area, L is the total number of layers of the generated network, y is any coordinate point in the mask area, Φ _Ll (I) is when the input object is an image to be filled, The feature map output by the layer, (Φ _Ll (I)) _y is the information of y in the mask area of the output feature map of the Ll layer, Φ _l (I ^gt ) is when the input object is the target image, the generation network The feature map output by the layer, (Φ _l (I ^gt )) _y is the information of y in the mask area of the output feature map of the l-th layer.

In addition, the image I to be filled is denoted as Φ(I; W) through the generative network, and W is the parameter of the generative network model. Define reconstruction loss

For each (Φ _Ll (I)) _y , its distance from (Φ _l (I)) _x is calculated as follows:

x is any coordinate point in the non-mask area, (Φ _l (I)) _x is the information of x in the non-mask area of the output feature map of the l-th layer, is the unmasked area.

The distance metric formula is as follows:

After finding the closest point x ^* (y), replace with x ^* (y) in the same plane as y in the area is the additional feature map to be input to the next deconvolution layer.

That is:

The discriminant network in this embodiment includes convolutional layers E ₉ to convolutional layers E ₁₃ ;

The input object of the convolutional layer E ₉ is a filled image;

The output result of the convolutional layer E ₉ is activated by the Leaky ReLU function as the input object of the convolutional layer E ₁₀ ;

For the convolutional layer E ₁₀ ~ convolutional layer E ₁₃ , the output result of the former is used as the input object of the latter after batch normalization and Leaky ReLU function activation in turn;

The output result of the convolutional layer E ₁₃ after successive batch normalization and Sigmoid function activation is the output result of the discriminant network;

The convolutional layer E ₉ is used to perform 64 convolution operations of 4*4 with a step size of 2 on the input object;

The convolutional layer E ₁₀ is used to perform 128 convolution operations of 4*4 with a step size of 2 on the input object;

The convolutional layer E ₁₁ is used to perform 256 convolution operations of 4*4 with a step size of 2 on the input object;

The convolutional layer E ₁₂ is used to perform 512 4*4 convolution operations with a step size of 1 on the input object;

The convolutional layer E ₁₃ is used to perform a 4*4 convolution operation with a step size of 1 on the input object.

The filled image is a 256*256 RGB image, the output result of the convolutional layer E ₁₃ has a spatial size of 64*64 and a channel of 1.

The input of the discriminant network is Φ(I; W) or I ^gt of the output of the generator network, and the generator network and the discriminant network are trained against each other, and the confrontation loss L _adv is generated at this time:

In the formula, p _data (I ^gt ) is the distribution of the real image, p _miss (I) is the distribution of the input image, and D( ) represents the probability prediction of the discriminant network for the image input into the discriminant network from p _data (I ^gt ) , log is a logarithmic function, I ^gt is the target image, and I is the image to be filled.

Therefore, when training the generative network, the total loss is L:

where _λg and _λadv are both hyperparameters.

Figure 3 shows any missing image to be filled, and Figure 4 shows the filled image obtained after inputting any missing image to be filled into the generation network. Comparing Figure 3 with Figure 4, it can be seen that the image filling system based on the convolutional neural network based on the feature map nearest neighbor replacement described in this embodiment is suitable for filling any missing image to be filled, and can obtain a better filling effect .

Figure 5 is the image to be filled with missing center, and Figure 6 is the filled image obtained after inputting the image to be filled with missing center into the generation network. Comparing Figure 5 with Figure 6, it can be seen that the image filling system based on the convolutional neural network based on the nearest neighbor replacement of the feature map described in this embodiment is suitable for filling the image to be filled with a missing center, and can obtain a better filling effect .

Through simulation experiments, the image filling system based on the convolutional neural network with feature map nearest neighbor replacement described in this embodiment takes about 80 ms for a 256*256 RGB image. Compared with the existing image filling method, which takes tens of seconds to several minutes, the image filling system of this embodiment has a significant improvement in filling speed.

The image filling system of the convolutional neural network based on the feature map nearest neighbor replacement described in this embodiment uses the Adam optimization algorithm for end-to-end training.

Although the invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It is therefore to be understood that numerous modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the invention as defined by the appended claims. It shall be understood that different dependent claims and features described herein may be combined in a different way than that described in the original claims. It will also be appreciated that features described in connection with individual embodiments can be used in other described embodiments.

Claims (10)

1. the image completion system for the convolutional neural networks that feature based figure arest neighbors is replaced, which is characterized in that described image is filled out Charging system includes generation network and differentiates network；

Generate network include encoder and decoder, encoder include N number of convolutional layer, decoder include N number of warp lamination, N >= 2；

Generation network fills decoded mode after image first encodes by treating, and obtains having been filled with image；

For the arbitrary M warp lamination in the first warp lamination~N-1 warp laminations, generation network is based on each deconvolution The output of the output result convolutional layer corresponding with the warp lamination of layer by the way of the replacement of characteristic pattern arest neighbors as a result, and obtained To supplementary features figure, and by the output result of each warp lamination, the output result of the corresponding convolutional layer of warp lamination and obtain The supplementary features figure arrived is collectively as the input object of next warp lamination；

Network is differentiated for judging to have been filled with whether image is the corresponding true picture of image to be filled, and then to generation network Weight study is constrained.

2. the image completion system for the convolutional neural networks that feature based figure arest neighbors as described in claim 1 is replaced, special Sign is that encoder includes convolutional layer E₁~convolutional layer E₈, decoder include warp lamination D₁~warp lamination D₈；

Image to be filled is convolutional layer E₁Input object；

For convolutional layer E₁~convolutional layer E₈, the former output result is successively through batch standardization and the activation of Leaky ReLU functions Afterwards, as the input object of the latter；

Convolutional layer E₈Output result successively through batch standardization and Leaky ReLU functions activation after, as warp lamination D₁'s Input object；

Warp lamination D₁Output result through ReLU functions activation after be used as warp lamination D₂The first input object；

For warp lamination D₂~warp lamination D₈, the former output result successively through ReLU functions activate and batch standardization after, The first input object as the latter；

Warp lamination D₂~warp lamination D₈The second input object be followed successively by convolutional layer E₇~convolutional layer E₁Successively through batch specification Change and the output result after the activation of Leaky ReLU functions；

Warp lamination D after the activation of Tanh functions₈Output result to have been filled with image；

Convolutional layer E₁For the convolution operation for carrying out 64 4*4 to input object, step-length is 2；

Convolutional layer E₂For the convolution operation for carrying out 128 4*4 to input object, step-length is 2；

Convolutional layer E₃For the convolution operation for carrying out 256 4*4 to input object, step-length is 2；

Convolutional layer E₄~convolutional layer E₈It is used to the convolution operation for 512 4*4 being carried out to input object, step-length is 2；

Warp lamination D₁~warp lamination D₄The deconvolution for being used to carry out 512 4*4 to input object, step-length is 2 operates；

Warp lamination D₅For carrying out 256 4*4 to input object, step-length is operated for 2 deconvolution；

Warp lamination D₆For carrying out 128 4*4 to input object, step-length is operated for 2 deconvolution；

Warp lamination D₇For carrying out 64 4*4 to input object, step-length is operated for 2 deconvolution；

Warp lamination D₈For carrying out 3 4*4 to input object, step-length is operated for 2 deconvolution；

It generates network and is based on warp lamination D₅Output result and convolutional layer E₃Output as a result, and being replaced using characteristic pattern arest neighbors The mode changed obtains supplementary features figure, and using the supplementary features figure as warp lamination D₆Third input object.

3. the image completion system for the convolutional neural networks that feature based figure arest neighbors as claimed in claim 2 is replaced, special Sign is that generation network is based on warp lamination D₅Output result and convolutional layer E₃Output as a result, and use characteristic pattern arest neighbors The detailed process that the mode of replacement obtains supplementary features figure is：

It is 0 to treat assignment characteristic pattern to choose a characteristic value, this feature figure and warp lamination D₅Output characteristic pattern and convolutional layer E₃Output characteristic pattern have equal port number and identical space size；

Warp lamination D is calculated₅The output masked areas of characteristic pattern and convolutional layer E₃Output characteristic pattern unmasked areas Domain, and the masked areas and the unmasked areas are cut into multiple characteristic blocks simultaneously；

Multiple characteristic blocks are cuboid, size C*h*w, wherein, C, h and w are respectively warp lamination D₅Output characteristic pattern Port number, the length of cuboid and the width of cuboid；

For each characteristic block p in the masked areas₁, choose in multiple characteristic blocks of the unmasked areas with characteristic block p₁Closest characteristic block p₂；

It chooses and treats to treat assignment region in assignment characteristic pattern, this treats assignment region and characteristic block p₁It is special in the output of warp lamination D5 Levy the position consistency in figure；

By characteristic block p₂Characteristic value assign described in treat assignment region.

4. the image completion system for the convolutional neural networks that feature based figure arest neighbors as claimed in claim 3 is replaced, special Sign is, characteristic block p₂With characteristic block p₁COS distance it is nearest.

5. the image completion system for the convolutional neural networks that feature based figure arest neighbors as claimed in claim 4 is replaced, special Sign is that the calculation of masked areas and unmasked areas for exporting characteristic pattern is：

A width mask image is given to substitute image to be filled, mask image is identical with the size of image to be filled, and port number is 1, characteristic value is 0 or 1；

0 represents that corresponding position of this feature point on image to be filled is non-point to be filled；

1 represents that corresponding position of this feature point on image to be filled is point to be filled；

The masked areas of the characteristic pattern of mask image and unmasked areas are calculated by convolutional network, which includes the One convolutional layer~third convolutional layer；

Mask image is the input object of the first convolutional layer；

For the first convolutional layer~third convolutional layer, the former output result is the input object of the latter；

First convolutional layer~third convolutional layer is used to the convolution operation for carrying out 1 4*4 to input object, step-length is 2；

Characteristic pattern of the output result of third convolutional layer for mask image, size 32*32, channel 1；

For the characteristic pattern of mask image, when one characteristic value is more than the threshold value of setting, judgement this feature point is mask point, Otherwise, it is determined that this feature point is unmasked point；

The masked areas of the characteristic pattern of mask image is the set of mask point, and the unmasked areas of the characteristic pattern of mask image is non- The set of mask point；

The masked areas for exporting characteristic pattern is equal with the masked areas of the characteristic pattern of mask image, exports the unmasked areas of characteristic pattern Domain is equal with the unmasked areas of the characteristic pattern of mask image.

6. the image completion system for the convolutional neural networks that feature based figure arest neighbors as claimed in claim 5 is replaced, special Sign is that generation network is trained by the way of Loss constraint is guided, and guides the concrete mode of Loss constraint to generate It is similar about to true picture and input picture progress feature in arbitrary convolutional layer or warp lamination during network training Beam；

Input picture is the true picture through masking operations.

7. the image completion system for the convolutional neural networks that feature based figure arest neighbors as claimed in claim 6 is replaced, special Sign is that the concrete mode that generation network is trained is：

By target image I^gtGeneration network is input to, calculates the masked areas of l layers of characteristic pattern, and obtains (Φ_l(I^gt))_yLetter Breath；

Image I to be filled is input to generation network, calculates the masked areas of L-l layers of characteristic pattern, and obtains (Φ_L-l(I))_y Information；

Definition guiding Loss constraint L at this time_g：

In formula, Ω is masks area, and L makes a living into total number of plies of network, and y is any coordinate points in masks area, Φ_L-l(I) it is When input object is image to be filled, the characteristic pattern of generation network output at L-l layers, (Φ_L-l(I))_yIt is L-l layers Export the information of y in the masked areas of characteristic pattern, Φ_l(I^gt) when to be input object be target image, generation network is defeated at l layers The characteristic pattern gone out, (Φ_l(I^gt))_yInformation for y in the masked areas of l layers of output characteristic pattern.

8. the image completion system for the convolutional neural networks that feature based figure arest neighbors as claimed in claim 7 is replaced, special Sign is, differentiates that network includes convolutional layer E₉~convolutional layer E₁₃；

Convolutional layer E₉Input object to have been filled with image；

Convolutional layer E₉Output result through Leaky ReLU functions activation after, as convolutional layer E₁₀Input object；

For convolutional layer E₁₀~convolutional layer E₁₃, the former output result is successively through batch standardization and the activation of Leaky ReLU functions Afterwards, as the input object of the latter；

Convolutional layer E after batch standardization and the activation of Sigmoid functions successively₁₃Output result be differentiate network output knot Fruit；

Convolutional layer E₉For the convolution operation for carrying out 64 4*4 to input object, step-length is 2；

Convolutional layer E₁₀For the convolution operation for carrying out 128 4*4 to input object, step-length is 2；

Convolutional layer E₁₁For the convolution operation for carrying out 256 4*4 to input object, step-length is 2；

Convolutional layer E₁₂For the convolution operation for carrying out 512 4*4 to input object, step-length is 1；

Convolutional layer E₁₃For the convolution operation for carrying out 1 4*4 to input object, step-length is 1.

9. the image completion system for the convolutional neural networks that feature based figure arest neighbors as claimed in claim 8 is replaced, special Sign is, has been filled with the RGB image that image is 256*256, convolutional layer E₁₃Output result space size for 64*64, channel It is 1.

10. the image completion system for the convolutional neural networks that feature based figure arest neighbors as claimed in claim 9 is replaced, special Sign is that described image fill system carries out end-to-end training using Adam optimization algorithms.