EP3678059A1 - Bildverarbeitungsverfahren, bildverarbeitungsvorrichtung und verfahren zum trainieren eines neuronalen netzes - Google Patents

Bildverarbeitungsverfahren, bildverarbeitungsvorrichtung und verfahren zum trainieren eines neuronalen netzes Download PDF

Info

Publication number
EP3678059A1
EP3678059A1 EP18850585.3A EP18850585A EP3678059A1 EP 3678059 A1 EP3678059 A1 EP 3678059A1 EP 18850585 A EP18850585 A EP 18850585A EP 3678059 A1 EP3678059 A1 EP 3678059A1
Authority
EP
European Patent Office
Prior art keywords
training
image
convolutional
neural network
data information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP18850585.3A
Other languages
English (en)
French (fr)
Other versions
EP3678059B1 (de
EP3678059A4 (de
Inventor
Hanwen Liu
Pablo Navarrete Michelini
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
BOE Technology Group Co Ltd
Original Assignee
BOE Technology Group Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by BOE Technology Group Co Ltd filed Critical BOE Technology Group Co Ltd
Publication of EP3678059A1 publication Critical patent/EP3678059A1/de
Publication of EP3678059A4 publication Critical patent/EP3678059A4/de
Application granted granted Critical
Publication of EP3678059B1 publication Critical patent/EP3678059B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/04—Context-preserving transformations, e.g. by using an importance map
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—Two-dimensional [2D] image generation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/776—Validation; Performance evaluation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions

Definitions

  • the embodiments of the present disclosure relate to a training method of a neural network for implementing image style transfer, an image processing method for implementing image style transfer based on a neural network and an image processing device.
  • deep learning techniques based on an artificial neural network have made great progress in fields such as object classification, text processing, engine recommendation, image search, face recognition, age and speech recognition, human-machine dialogue, emotional computing and so on.
  • the deep learning techniques With the deepening of an artificial neural network structure and the improvement of algorithms, the deep learning techniques have made breakthrough progress in the field of human-like data perception, and the deep learning techniques can be used to describe image content, recognize objects in complex environments in an image, perform speech recognition in noisy environments, and the like.
  • the deep learning techniques can also be used to solve a problem of image generation and fusion.
  • At least one embodiment of the present disclosure provides a training method of a neural network for implementing image style transfer, comprising: acquiring a first training input image and a second training input image; inputting the first training input image to the neural network; performing a style transfer process on the first training input image by using the neural network, so as to obtain a training output image; based on the first training input image, the second training input image and the training output image, calculating a loss value of parameters of the neural network through a loss function; and modifying the parameters of the neural network according to the loss value, in a case where the loss function satisfies a predetermined condition, obtaining a trained neural network, and in a case where the loss function does not satisfy the predetermined condition, continuing to input the first training input image and the second training input image so as to repeatedly perform the above training process, the loss function comprising a weight-bias-ratio loss function.
  • the neural network comprises a first convolutional neural network
  • the first convolutional neural network comprises a plurality of first convolutional kernels and a plurality of biases
  • the parameters of the neural network comprise the plurality of first convolutional kernels and the plurality of biases
  • calculating the loss value of the parameters of the neural network through the loss function comprises: calculating a weight-bias-ratio loss value of the parameters of the neural network by the weight-bias-ratio loss function according to the plurality of first convolutional kernels and the plurality of biases, and the loss value comprises the weight-bias-ratio loss value.
  • modifying the parameters of the neural network according to the loss value comprises: adjusting a ratio between the plurality of first convolutional kernels and the plurality of biases according to the weight-bias-ratio loss value.
  • the neural network further comprises a first conversion matrix
  • the first training input image has a first training input color channel, a second training input color channel and a third training input color channel
  • inputting the first training input image to the neural network comprises: converting the first training input image into a first training intermediate image by the first conversion matrix; and inputting the first training intermediate image to the neural network;
  • converting the first training input image into the first training intermediate image by the first conversion matrix comprises: converting, by the first conversion matrix, data information of the first training input color channel, data information of the second training input color channel and data information of the third training input color channel of the first training input image into data information of a first training luminance channel, data information of a first training color difference channel and data information of a second training color difference channel of the first training intermediate image.
  • the first convolutional neural network comprises a first sub-network, a second sub-network and a third sub-network
  • performing the style transfer process on the first training input image by using the neural network so as to obtain the training output image comprises: performing the style transfer process on the data information of the first training luminance channel of the first training intermediate image, the data information of the first training color difference channel of the first training intermediate image, and the data information of the second training color difference channel of the first training intermediate image by using the first sub-network, the second sub-network and the third sub-network respectively, so as to generate data information of a second training luminance channel of a second training intermediate image, data information of a third training color difference channel of the second training intermediate image, and data information of a fourth training color difference channel of the second training intermediate image; converting the second training intermediate image into the training output image, where the training output image is an image in a RGB format, and the second training intermediate image is an image in a YUV format
  • the neural network further comprises a second conversion matrix
  • converting the second training intermediate image into the training output image comprises: converting, by the second conversion matrix, the data information of the second training luminance channel of the second training intermediate image, the data information of the third training color difference channel of the second training intermediate image and the data information of the fourth training color difference channel of the second training intermediate image into data information of a first training output color channel of the training output image, data information of a second training output color channel of the training output image and data information of a third training output color channel of the training output image.
  • the first sub-network comprises a first set of first convolutional layers
  • the second sub-network comprises a second set of first convolutional layers
  • the third sub-network comprises a third set of first convolutional layers
  • the second sub-network comprises a first standard up-sampling layer and a first standard down-sampling layer
  • the third sub-network comprises a second standard up-sampling layer and a second standard down-sampling layer
  • the first standard down-sampling layer is used to replace an initial first convolutional layer in the second set of first convolutional layers of the second sub-network
  • the first standard up-sampling layer is used to replace a last first convolutional layer in the second set of first convolutional layers of the second sub-network
  • the second standard down-sampling layer is used to replace an initial first convolutional layer in the third set of first convolutional layers of the third sub-network
  • the second standard up-sampling layer is
  • the loss function further comprises a content loss function, calculating the loss value of the parameters of the neural network through the loss function based on the first training input image, the second training input image and the training output image further comprises: extracting a first training input feature of the first training input image and extracting a first training output feature of the training output image by an analysis network; and calculating a content loss value of the parameters of the neural network by the content loss function, according to the first training input feature and the first training output feature, and the loss value comprises the content loss value.
  • the loss function further comprises a style loss function, calculating the loss value of the parameters of the neural network through the loss function based on the first training input image, the second training input image and the training output image further comprises: extracting a second training input feature of the second training input image and extracting a second training output feature of the training output image by the analysis network; and calculating a style loss value of the parameters of the neural network by the style loss function, according to the second training input feature and the second training output feature, and the loss value comprises the style loss value.
  • both the first training input feature and the first training output feature are content features
  • both the second training input feature and the second training output feature are style features
  • the analysis network comprises a second convolutional neural network
  • the second convolutional neural network comprises a plurality of second convolutional layers sequentially connected and a plurality of second pooling layers interposed between adjacent second convolutional layers, each of the plurality of second convolutional layers is used for extracting the first training input feature, the first training output feature, the second training output feature and/or the second training input feature
  • a l -th second convolutional layer has N l second convolutional kernels
  • the l -th second convolutional layer is used for generating and outputting N l first training feature images of the first training input image, N l second training feature images and N l third training feature images of the training output image, and N l fourth training feature images of the second training input image
  • the N l convolutional kernels are in one-to-one correspondence to the N l first training feature images, to the N l second training feature images, to the N l third training feature images and to the N
  • At least one embodiment of the present disclosure further provides an image processing method for implementing image style transfer based on a neural network
  • the neural network is a neural network obtained by training according to the training method according to any one of the above embodiments
  • the image processing method comprises: acquiring a first image; inputting the first image to the neural network; and performing a style transfer process on the first image by using the neural network to generate a second image.
  • the neural network further comprises a first conversion matrix
  • the first image has a first input color channel, a second input color channel and a third input color channel
  • inputting the first image to the neural network comprises: converting the first image into a first intermediate image by the first conversion matrix; and inputting the first intermediate image into the neural network; where converting the first image into the first intermediate image by the first conversion matrix comprises: converting, by the first conversion matrix, data information of the first input color channel of the first image, data information of the second input color channel of the first image and data information of the third input color channel of the first image into data information of a first luminance channel of the first intermediate image, data information of a first color difference channel of the first intermediate image and data information of a second color difference channel of the first intermediate image.
  • the neural network further comprises a first convolutional neural network
  • the first convolutional neural network comprises a first sub-network, a second sub-network and a third sub-network
  • performing a style transfer process on the first image by using the neural network to generate a second image comprises: performing the style transfer processing on the data information of the first luminance channel of the first intermediate image, the data information of the first color difference channel of the first intermediate image and the data information of the second color difference channel of the first intermediate image by using the first sub-network, the second sub-network and the third sub-network, respectively, so as to generate data information of a second luminance channel of a second intermediate image, data information of a third color difference channel of the second intermediate image and data information of a fourth color difference channel of the second intermediate image; converting the second intermediate image into the second image, the second image being an image in a RGB format, and the second intermediate image being an image in a YUV format.
  • the neural network further comprises a second conversion matrix
  • converting the second intermediate image into the second image comprises: converting, by the second conversion matrix, the data information of the second luminance channel of the second intermediate image, the data information of the third color difference channel of the second intermediate image and the data information of the fourth color difference channel of the second intermediate image into data information of a first output color channel of the second image, data information of a second output color channel of the second image and data information of a third output color channel of the second image.
  • At least one embodiment of the present disclosure further provides an image processing device, comprising: an image acquisition module, configured to acquire a first image; an image processing module, comprising a neural network obtained by training according to the training method according to any one of the above embodiments, the image processing module being used for performing a style transfer process on the first image by using the neural network to generate a second image.
  • the image processing device further comprises a first conversion module
  • the first image has a first input color channel, a second input color channel and a third input color channel
  • the first conversion module is used for converting data information of the first input color channel of the first image, data information of the second input color channel of the first image and data information of the third input color channel of the first image into data information of a first luminance channel of a first intermediate image, data information of a first color difference channel of the first intermediate image, and data information of a second color difference channel of the first intermediate image
  • the neural network comprises a first sub-network, a second sub-network and a third sub-network, the first sub-network, the second sub-network and the third sub-network are respectively used for processing the data information of the first luminance channel of the first intermediate image, the data information of the first color difference channel of the first intermediate image, and the data information of the second color difference channel of the first intermediate image, so as to generate data information of a second luminance
  • the image processing device further comprises a second conversion module, the second conversion module is used to convert the data information of the second luminance channel of the second intermediate image, the data information of the third color difference channel of the second intermediate image, and the data information of the fourth color difference channel of the second intermediate image into data information of a first output color channel of the second image, data information of a second output color channel of the second image and data information of a third output color channel of the second image.
  • the second conversion module is used to convert the data information of the second luminance channel of the second intermediate image, the data information of the third color difference channel of the second intermediate image, and the data information of the fourth color difference channel of the second intermediate image into data information of a first output color channel of the second image, data information of a second output color channel of the second image and data information of a third output color channel of the second image.
  • the first sub-network comprises a first set of first convolutional layers
  • the second sub-network comprises a second set of first convolutional layers
  • the third sub-network comprises a third set of first convolutional layers
  • the second sub-network comprises a first standard up-sampling layer and a first standard down-sampling layer
  • the third sub-network comprises a second standard up-sampling layer and a second standard down-sampling layer
  • the first standard down-sampling layer is used to replace an initial first convolutional layer in the second set of first convolutional layers of the second sub-network
  • the first standard up-sampling layer is used to replace a last first convolutional layer in the second set of first convolutional layers of the second sub-network
  • the second standard down-sampling layer is used to replace an initial first convolutional layer in the third set of first convolutional layers of the third sub-network, the second standard up-sampling layer
  • At least one embodiment of the present disclosure further provides an image processing device, comprising: a memory for storing non-temporary computer-readable instructions; and a processor for executing the computer-readable instructions, the training method according to any one of the above embodiments being performed while the computer-readable instructions are executed by the processor.
  • the artistic style transfer of pictures with aid of a deep neural network is a technology that has emerged with development of deep learning technology. For example, based on a style given in a reference image, an input image is processed to obtain an output image that at least partially reflects the style.
  • the reference image is a work of a certain artist, and then the output image may show some style features of the artist's painting while retaining an original content, even making people mistake it as the artist's work.
  • This style transfer effect cannot be achieved by traditional image processing effects (such as various filters provided by instagram corporation, etc.).
  • At least one embodiment of the present disclosure provides a training method of a neural network for implementing image style transfer, an image processing method for implementing image style transfer based on a neural network and an image processing device.
  • a weight-bias-ratio (L1) loss function as or in a loss function used in a process of training neural network
  • the role of an activation function in the neural network is played better, more optimized convolutional parameters are obtained, and effects of image style transfer and image fusion are improved.
  • the processing effect, processing speed and other aspects are all taken into account at the same time, and an application prospect is better and wider.
  • CNN convolutional neural network
  • CNN mainly simplifies the complexity of neural network models and reduces the number of weights by local perception field and weight sharing.
  • an application scope of the CNN is no longer limited to just the field of image recognition, and the CNN may also be applied to face recognition, character recognition, animal classification, image processing and other fields.
  • FIG. 1 is a schematic diagram of a convolutional neural network.
  • the convolutional neural network may be used for image processing, the convolutional neural network uses images as an input and an output, and replaces weights of scalar by convolutional kernels.
  • FIG. 1 shows merely a convolutional neural network with a three-layer structure, and embodiments of the present disclosure are not limited thereto.
  • the convolutional neural network includes an input layer 101, a hidden layer 102 and an output layer 103.
  • the input layer 101 has four inputs
  • the hidden layer 102 has three outputs
  • the output layer 103 has two outputs.
  • the convolutional neural network outputs two images.
  • the four inputs of the input layer 101 may be four images, or four features of one image.
  • the three outputs of the hidden layer 102 may be feature images of an image inputted through the input layer 101.
  • a convolutional layer has a weight w ij k and a bias b i k .
  • the weight w ij k represents a convolutional kernel
  • the bias b i k is a scalar superimposed on an output of the convolutional layer, where k indicates a label of the input layer 101, i and j are labels of a unit of the input layer 101 and a unit of the hidden layer 102, respectively.
  • a first convolutional layer 201 includes a first set of convolutional kernels ( w ij 1 in FIG. 1 ) and a first set of biases ( b i 1 in FIG. 1 ).
  • a second convolutional layer 202 includes a second set of convolutional kernels ( w ij 2 in FIG. 1 ) and a second set of biases ( b i 2 in FIG. 1 ).
  • each convolutional layer includes dozens or hundreds of convolutional kernels, and if the convolutional neural network is a deep convolutional neural network, then the convolutional neural network may include at least five convolutional layers.
  • the convolutional neural network further includes a first activation layer 203 and a second activation layer 204.
  • the first activation layer 203 is arranged behind the first convolutional layer 201
  • the second activation layer 204 is arranged behind the second convolutional layer 202.
  • the activation layer e.g., the first activation layer 203 and the second activation layer 204 include an activation function, the activation function is used for introducing nonlinear factors into the convolutional neural network, so that the convolutional neural network can solve more complex problems better.
  • the activation function may include a Rectified Linear Unit (ReLU) function, a S-type function (sigmoid function), a hyperbolic tangent function (tanh function), or the like.
  • the ReLU function is an unsaturated nonlinear function
  • the Sigmoid function and the tanh function are saturated nonlinear functions.
  • the activation layer may be a single layer of the convolutional neural network alone, or the activation layer may also be included in a convolutional layer (for example, the first convolutional layer 201 may include the first activation layer 203, and the second convolutional layer 202 may include the second activation layer 204).
  • the first convolutional layer 201 firstly, several convolutional kernels w ij 1 in the first set of convolutional kernels and several biases b i 1 in the first set of biases are applied to each of the inputs, so as to obtain outputs of the first convolutional layer 201; afterwards, the outputs of the first convolutional layer 201 can be processed by the first activation layer 203, so as to obtain outputs of the first activation layer 203.
  • the second convolutional layer 202 firstly, several convolutional kernels w ij 2 in the second set of convolutional kernels and several biases b i 2 in the second set of biases are applied to the outputs of the first activation layer 203 inputted to the second convolutional layer 202, so as to obtain outputs of the second convolutional layer 202; afterwards, the outputs of the second convolutional layer 202 may be processed by the second activation layer 204, so as to obtain outputs of the second activation layer 204.
  • an output of the first convolutional layer 201 may be a result that is obtained by applying the convolutional kernels w ij 1 to an input of the first convolutional layer 201 and then adding it to a bias b i 1
  • an output of the second convolutional layer 202 may be a result that is obtained by applying the convolutional kernels w ij 2 to an output of the first activation layer 203 and then adding it to a bias b i 2 .
  • the convolutional neural network Before performing an image processing by a convolutional neural network, the convolutional neural network needs to be trained. After training, convolutional kernels and biases of the convolutional neural network remain unchanged during image processing. In the training process, each convolutional kernel and each bias are adjusted by a plurality of input/output sample images and optimization algorithms, so as to obtain an optimized convolutional neural network model.
  • FIG.2A is an exemplary model diagram of a convolutional neural network
  • FIG. 2B is a schematic diagram of a working process of a convolutional neural network.
  • a class identifier of the input image is output after the input image is processed through several processing steps (such as each level in FIG. 2A ) in sequence.
  • the constituent parts of the convolutional neural network may include a plurality of convolutional layers, a plurality of pooling layers and a fully connected layer.
  • a complete convolutional neural network may be composed of these three layers in a superposing manner. For example, FIG.
  • the processing procedure at each level may include performing convolution and sub-sampling on an input image.
  • the processing procedure at each level may further include a local constant normalization (LCN) processing and so on.
  • LCN local constant normalization
  • a convolutional layer is a core layer of a convolutional neural network.
  • the convolutional layer can apply several convolutional kernels to an input image, so as to extract features of various types of the input image.
  • Each of the convolutional kernels can extract a feature of one type.
  • the convolutional kernels are usually initialized in a form of a random fractional matrix, and in the training process of the convolutional neural network, the convolutional kernels will go through learning to obtain reasonable weights.
  • a result obtained by applying one convolutional kernel to an input image is called as a feature map, and the number of feature maps is equal to the number of convolutional kernels.
  • Each of the feature maps is composed of some neurons arranged in a matrix, and neurons of the same feature map share weights, and the shared weights are convolutional kernels.
  • a feature map outputted from a convolutional layer at a level may be input to a convolutional layer at the adjacent next level, and is processed again to obtain a new feature map.
  • a convolutional layer of a first level may output a first feature map, and the first feature map is input to a convolutional layer of a second level to be processed again, so as to obtain a second feature map.
  • a convolutional layer can use different convolutional kernels to convolute data of a local receptive field of an input image, and a convoluting result is inputted into an activation layer, and the activation layer performs calculation according to a corresponding activation function so as to obtain feature information of the input image.
  • a pooling layer is between adjacent convolutional layers, and is a form of down-sampling.
  • the pooling layer can be used to reduce the scale of an input image, simplify the computational complexity, and reduce the phenomenon of over-fitting to a certain extent; on the other hand, the pooling layer can also be used to perform feature compression, so as to extract main features of the input image.
  • the pooling layer can reduce the sizes of the feature images, but does not change the number of feature images.
  • the input image is sampled by a filter having a size of 6 ⁇ 6, and then an output image having a size of 2 ⁇ 2 can be obtained, and this means that 36 pixels in the input image are merged into one pixel in the output image.
  • the last pooling layer or convolutional layer may be connected to one or more fully connected layers, the fully connected layer is used to connect all the extracted features.
  • An output of the fully connected layer is a one-dimensional matrix, that is, a vector.
  • FIG. 3 is a schematic structural view of another convolutional neural network.
  • an output of the last convolutional layer that is, an no-th convolutional layer
  • the planarization layer can convert a feature image (a 2D image) into a vector (ID).
  • the output (i.e., the 1D vector) of the planarization layer is input to a fully connected layer.
  • the fully connected layer may have the same structure as the convolutional neural network, but the difference between the fully connected layer and the convolutional neural network is that the fully connected layer uses different scalar values to replace convolutional kernels.
  • the output of the last convolutional layer may also be input to a homogenization layer (AVG).
  • AVG homogenization layer
  • the homogenization layer is used for performing an average operation on the output, that is, an average value of a feature image is used to represent an output image, and therefore, a 2D feature image is converted into a scalar.
  • the convolutional neural network includes a homogenization layer, then it may not include a planarization layer.
  • the homogenization layer or the fully connected layer may be connected to a classifier, the classifier can perform a classification operation according to the extracted features, and an output of the classifier may act as the final output of the convolutional neural network, i.e., a category label that represents a category of an image.
  • the classifier may be a support vector machine (SVM) classifier, a softmax classifier, a k-nearest neighbor (KNN) rule classifier, etc.
  • SVM support vector machine
  • KNN k-nearest neighbor
  • the convolutional neural network includes a softmax classifier
  • the softmax classifier is a generator of a logical function and can compress a K-dimensional vector z containing any real number into a K-dimensional vector ⁇ ( z ).
  • each category label in the K-dimensional vector z is given a certain prediction probability, and the category label with the maximum prediction probability is selected as the identifier or category of the input image.
  • FIG. 4A is a flowchart of a training method of a neural network for implementing image style transfer provided by an embodiment of the present disclosure
  • FIG. 4B is a schematic block diagram of a training module of a neural network during a training process provided by an embodiment of the present disclosure.
  • the training method of a neural network for implementing image style transfer includes:
  • a training module of the neural network includes a neural network, an analysis network, and a loss function 30.
  • the neural network includes a first convolutional neural network 10.
  • the first convolution neural network 10 is used for performing a style transfer process on the inputted first training input image, so as to generate a training output image.
  • the loss function 30 is used to calculate the loss value of the parameters of the first convolutional neural network 10, and to modify the parameters of the first convolutional neural network 10 according to the loss value.
  • the loss function 30 includes a weight-bias-ratio (L1) loss function.
  • the predetermined condition corresponds to minimization of the loss value of the neural network under the condition of a certain number of first training input image inputs.
  • the predetermined condition is that the number of training times or the number of the training period of the neural network reaches a predetermined number.
  • the above example is merely used to exemplarily illustrate the training process of the neural network.
  • Those skilled in the art should know that, in the training phase, a large number of sample images need to be used to train the neural network; at the same time, in the training process of each of the sample images, multiple repeated iterations may be included to modify the parameters of the neural network.
  • the training phase further includes fine-tuning the parameters of the neural network, so as to obtain more optimized parameters.
  • initial parameters of the first convolutional neural network 10 may be random numbers, and for example, the random numbers conform to Gaussian distribution.
  • the initial parameters of the first convolutional neural network 10 may also adopt trained parameters in an image database such as ImageNet or the like. Embodiments of the present disclosure do not limit this.
  • the loss function 30 can calculate the loss value based on the initial parameters of the first convolutional neural network 10.
  • the loss value may be back-propagated layer-by-layer from an output layer of the first convolutional neural network 10 to an input layer, so that the loss value can be shared by all units of each layer of the first convolutional neural network 10, thereby obtaining an error value of each unit.
  • the error value is the basis for modifying parameters of each unit.
  • the error value may be expressed as a partial derivative of the loss value to each parameters of each unit.
  • the training module of the neural network may further include an optimization function (not shown), the optimization function can calculate the error values of the parameters of the first convolutional neural network 10 according to the loss value, and parameters of the first convolutional neural network 10 are adjusted according to the error values.
  • the optimization function can calculate the error values of the parameters of the first convolutional neural network 10 according to the loss value, and parameters of the first convolutional neural network 10 are adjusted according to the error values.
  • the optimization function can adopt a stochastic gradient descent (SGD) algorithm, a batch gradient descent (BGD) algorithm or the like to calculate the error values of the parameters of the first convolutional neural network 10.
  • SGD stochastic gradient descent
  • BGD batch gradient descent
  • the first training input image may be an image of various types.
  • the first training input image may be an image taken by a digital camera or a mobile phone, and may be a person image, an animal or plant image, a landscape image, etc.
  • the role of an activation function in the first convolutional neural network can be fully played, more optimized parameters of the first convolutional neural network are obtained, and effects of image style transfer and image fusion are improved.
  • L1 loss function weight-bias-ratio loss function
  • FIG. 5A is a schematic structural view of a first convolutional neural network provided by an embodiment of the present disclosure
  • FIG. 5B is a schematic structural view of another first convolutional neural network provided by an embodiment of the present disclosure.
  • the first convolutional neural network 10 includes a first sub-network 101, a second sub-network 102 and a third sub-network 103.
  • the first sub-network 101 includes a first set of first convolutional layers
  • the second sub-network 102 includes a second set of first convolutional layers
  • the third sub-network 103 includes a third set of first convolutional layers.
  • the first set of first convolutional layers includes F1 first convolutional layers
  • the second set of first convolutional layers also includes F2 first convolutional layers
  • the third set of first convolutional layers also includes F3 first convolutional layers.
  • the first sub-network 101 includes a first convolutional layer C11, a first convolutional layer C12 and a first convolutional layer C1n 1 ;
  • the second sub-network 102 includes a first convolutional layer C21, a first convolutional layer C22 and a first convolutional layer C2n 1 ;
  • the third sub-network 103 includes a first convolutional layer C31, a first convolutional layer C32 and a first convolutional layer C3n 1 .
  • the first sub-network 101 further includes a plurality of first pooling layers and a plurality of composite layers
  • the second sub-network 102 further includes a plurality of first pooling layers and a plurality of composite layers
  • the third sub-network 103 further includes a plurality of first pooling layers and a plurality of composite layers.
  • the first sub-network 101 includes a first pooling layer P11, a first pooling layer P12 and a first pooling layer P1n 1 ;
  • the second sub-network 102 includes a first pooling layer P21, a first pooling layer P22 and a first pooling layer P2n 1 ;
  • the third sub-network 103 includes a first pooling layer P31, a first pooling layer P32 and a first pooling layer P3n 1 .
  • the first sub-network 101 includes a composite layer M11, a composite layer M12 and a composite layer M1n 1 ;
  • the second sub-network 102 includes a composite layer M21, a composite layer M22 and a composite layer M2n 1 ;
  • the third sub-network 103 includes a composite layer M31, a composite layer M32 and a composite layer M3n 1 .
  • the first sub-network 101, the second sub-network 102 and the third sub-network 103 have the same structure.
  • detailed descriptions will be given by taking the first sub-network 101 as an example.
  • a plurality of first convolutional layers are connected sequentially; a first pooling layer is between some adjacent first convolutional layers, and a composite layer is between other adjacent first convolutional layers.
  • the first sub-network 101 includes n 1 first convolutional layers, n 1 is a positive integer, and is typically greater than 3, or even hundreds.
  • the low-level convolutional layer is used to extract the lower-order features (e.g., points, edges, etc.) of a first image; as the level increases, the high-level convolutional layer can extract the higher-order features (e.g., straight lines, bends, triangles, etc.) of the first image.
  • the higher-order features can be obtained by combining lower-order features.
  • the number of the plurality of first pooling layers is the same as the number of the plurality of composite layers, and down-sampling factors of the plurality of first pooling layers are in one-to-one correspondence to up-sampling factors of the plurality of composite layers, so that the first image and a second image have the same size.
  • the first sub-network 101 may include n 1 /2 first pooling layers and n 1 /2 composite layers, where n 1 is a positive even number.
  • the first sub-network 101 may include two first pooling layers and two composite layers, and down-sampling factors of the two first pooling layers are 1/x1 and 1/x2 respectively, then up-sampling factors of the two composite layers are x1 and x2 respectively, x1 and x2 are positive integers.
  • the number of the plurality of first pooling layers and the number of the plurality of composite layers may also be different, provided that the first training input image and the training output image that is processed by the first convolutional neural network 10 can have the same size.
  • the plurality of first pooling layers are located at lower levels of the first sub-network 101, and the plurality of composite layers are located at higher levels of the first sub-network 101. That is, firstly, the first sub-network 101 performs down-sampling on an inputted first image by the first pooling layers, and then performs up-sampling on the first image by the composite layers.
  • the first convolutional neural network 10 includes a plurality of first convolutional kernels and a plurality of biases
  • the plurality of first convolutional kernels are first convolutional kernels and biases that are included in all the first convolutional layers of the first convolutional neural network 10
  • the parameters of the neural network may include the plurality of first convolutional kernels and the plurality of biases.
  • the loss function 30 is used to adjust parameters of each first convolutional layer in the first convolutional neural network 10. It should be noted that, parameters of all layers in the first convolutional neural network 10 can be adjusted by the loss function 30.
  • the structure of the first sub-network 101, the structure of the second sub-network 102, and the structure of the third sub-network 103 may also be different to each other.
  • the second sub-network 102 some of the first pooling layers or the composite layers, or the like can be omitted.
  • the first convolutional neural network 10 includes a first sub-network 101', a second sub-network 102' and a third sub-network 103'.
  • the first sub-network 101' has the same structure as the first sub-network 101 shown in FIG. 5A .
  • the second sub-network 102' may include a first standard up-sampling layer SU1 and a first standard down-sampling layer SD1
  • the third sub-network 103' may include a second standard up-sampling layer SU2 and a second standard down-sampling layer SD2.
  • the first standard down-sampling layer SD1 is used to replace an initial first convolutional layer in the second set of first convolutional layers of the second sub-network 102'
  • the first standard up-sampling layer SU1 is used to replace a last first convolutional layer in the second set of first convolutional layers of the second sub-network 102'
  • the second standard down-sampling layer SD2 is used to replace an initial first convolutional layer in the third set of first convolutional layers of the third sub-network 103'
  • the second standard up-sampling layer SU2 is used to replace a last first convolutional layer in the third set of first convolutional layers of the third sub-network 103'.
  • the first standard down-sampling layer SD1 and the second standard down-sampling layer SD2 are used to reduce the resolution of an inputted first training input image, so as to increase the speed of image processing.
  • the first standard down-sampling layer SD1 and the second standard down-sampling layer SD2 are used to reduce values of respective dimensions of a feature image, thereby reducing the data amount of the feature image.
  • the first standard up-sampling layer SU1 and the second standard up-sampling layer SU2 are used to increase values of the respective dimensions of a feature image, thereby increasing the data amount of the feature image.
  • the first standard down-sampling layer SD1 and the second standard down-sampling layer SD2 may adopt a maximum merging (max pooling) method, an average value merging (average pooling) method, a strided convolution method or other down-sampling methods.
  • the first standard up-sampling layer SU1 and the second standard up-sampling layer SU2 may adopt a strided transposed convolution methods or other up-sampling methods.
  • the first standard down-sampling layer SD1 may further be used to replace an initial first pooling layer or composite layer in the second sub-network 102'
  • the first standard up-sampling layer SU1 may further be used to replace a last first pooling layer or composite layer in the second sub-network 102'
  • the second standard down-sampling layer SD2 may further be used to replace an initial first pooling layer or composite layer in the third sub-network 103'
  • the second standard up-sampling layer SU2 may further be used to replace a last first pooling layer or composite layer in the third sub-network 103'.
  • the first standard down-sampling layer SD1 replaces the initial first convolutional layer and the initial first pooling layer
  • the first standard up-sampling layer SU1 replaces the last first convolutional layer and the last composite layer. Therefore, a network model size of the second sub-network 102' is reduced.
  • a network model size of the third sub-network 103' is also reduced.
  • a network model size of the first convolutional neural network 10 is reduced.
  • the second sub-network 102 'and the third sub-network 103' may have the same structure. That is to say, the first standard down-sampling layer SD1 is the same as the second standard down-sampling layer SD2, and the first standard up-sampling layer SU1 is also the same as the second standard up-sampling layer SU2.
  • the structure of the second sub-network 102 'and the structure of the third sub-network 103' may be also different. Embodiments of the present disclosure are not limited thereto.
  • a convolutional kernel is used to determine how an input image is processed, and a bias is used to determine whether the output of the convolutional kernel is input to a next level or not. Therefore, in an activation layer of the convolutional neural network, the bias can be figuratively compared to "switch", which is used to determine that the convolutional kernel is "opened” or "closed”. For different input images, different convolutional kernels can be "opened” or "closed”, so as to achieve various effects.
  • a bias needs to have a relatively large absolute value, so as to play the role of "switch" more effectively.
  • the L1 loss function is used to adjust the ratio between the plurality of first convolutional kernels and the plurality of biases of a first convolutional layer, so as to enhance the effect of an activation function of the first convolutional neural network 10.
  • the step S40 may include calculating a weight-bias-ratio loss value of the parameters of the neural network by the weight-bias-ratio loss function according to the plurality of first convolutional kernels and the plurality of biases.
  • the value of L L1 may represent the weight-bias-ratio (L1) loss value of the parameters of the first convolutional neural network 10, and the loss value of the neural network includes the L1 loss value.
  • the step S50 may include adjusting a ratio between the plurality of first convolutional kernels and the plurality of biases according to the weight-bias-ratio loss value.
  • the L1 loss value needs to be adjusted repeatedly, so as to make W as small as possible and B as large as possible.
  • C w is the number of first convolutional kernels possessed by the first convolutional layer
  • w represents a value of each convolutional kernel of the first convolutional layer.
  • a convolutional kernel is a matrix of 2 ⁇ 2, and w denotes a sum of respective elements of the matrix.
  • an activation function in the first convolutional neural network 10 can adopt a ReLU function.
  • the activation function in the first convolutional neural network 10 can also adopt a sigmoid function, a tanh function or the like.
  • in 1 w ⁇ in 0 + b
  • in 0 represents a pixel matrix of a first training input image that is input to, for example, a first convolutional layer C11
  • in 1 represents a pixel matrix of a feature image that is output by the first convolutional layer C11 after processing in 0
  • w represents values of convolutional kernels in the first convolutional layer C11
  • b represents values of biases in the first convolutional layer C11.
  • the first pooling layer may be a down-sampling layer of the first convolutional neural network 10, and is used to reduce values of respective dimensions of a feature image, thereby reducing the data amount of the feature image.
  • the composite layer is an up-sampling layer of the first convolutional neural network 10, and is used to increase the values of respective dimensions of the feature image, thereby increasing the data amount of the feature image.
  • the first pooling layer can use various down-sampling methods to down-sample a feature image.
  • the down-sampling methods include, but are not limited to: a maximum merging method, an average merging method, a random merging method, an under-sampling method (decimation, for example, selecting fixed pixels), a demultiplexing output method (demuxout, for splitting an input image into a plurality of smaller images), and so on.
  • the composite layer can use interpolation algorithm or the like to achieve up-sampling.
  • the interpolation algorithm may include, for example, an interpolation algorithm, a Bicubic Interprolation algorithm, and so on.
  • FIG. 6A is a schematic diagram of a composite layer provided by an embodiment of the present disclosure
  • FIG. 6B is a schematic diagram of another composite layer provided by an embodiment of the present disclosure.
  • the composite layer adopts a pixel difference method to implement the up-sampling.
  • An up-sampling factor of 2 ⁇ 2 is used in the composite layer, so that four input feature images (i.e., INPUT 4n, INPUT 4n+1, INPUT 4n+2, INPUT 4n+3 in FIG. 6A ) can be combined to obtain one output feature image (i.e., OUTPUT n in FIG. 6A ) with a fixed pixel order.
  • a composite layer in the first sub-network 101 acquires a first number of input feature images that are input, and interleaves and rearranges pixel values of these input feature images so as to generate the same first number of output feature images.
  • the number of the output feature images is not changed, but a size of each of the output feature images is increased by a corresponding multiple.
  • more data information is added through the composite layer adopting various permutations and combinations, and these combinations can give all possible up-sampling combinations.
  • the selection can be made from the up-sampling combinations by an activation layer.
  • a composite layer adopts a pixel value interleaved rearrangement method to achieve the up-sampling.
  • the composite layer also uses an up-sampling factor of 2 ⁇ 2, that is, every four input feature images (i.e., INPUT 4n, INPUT 4n+1, INPUT 4n+2, INPUT 4n+3 in FIG. 6B ) are taken as a group, and their pixel values are interleaved to generate four output feature images (i.e., OUTPUT 4n, OUTPUT 4n+1, OUTPUT 4n+2, OUTPUT 4n+3 in FIG. 6B ).
  • the number of input feature images is the same as the number of output feature images obtained after processing the input feature images by the composite layer, while a size of each output feature image is increased to be four times that of an input feature image, that is, the output feature image has a pixel number that is four times that of an input feature image.
  • the first sub-network 101 further includes a plurality of instance normalization layers
  • the second sub-network 102 further includes a plurality of instance normalization layers
  • the third sub-network 103 further includes a plurality of instance normalization layers.
  • the first sub-network 101 includes an instance normalization layer N11, an instance normalization layer N12 and an instance normalization layer N1n 1
  • the second sub-network 102 includes an instance normalization layer N21, an instance normalization layer N22 and an instance normalization layer N2n 1
  • the third sub-network 103 includes the instance normalization layer N31, the instance normalization layer N32 and the instance normalization layer N3n 1
  • the first sub-network 101 may include n 1 instance normalization layers
  • the second sub-network 102 also includes n 1 instance normalization layers
  • the third sub-network 103 also includes n 1 instance normalization layers.
  • the first sub-network 101 may include n 1 instance normalization layers, i.e. the instance normalization layers are in one-to-one correspondence to the first convolutional layers.
  • the present disclosure is not limited to this, several first convolutional layers (e.g., two or three first convolutional layers, etc.) may also correspond to one instance normalization layer, thus the number of instance normalization layers is less than that of the first convolutional layers.
  • the embodiments of the present disclosure do not specifically limit this.
  • the first standard down-sampling SD1 layer may further replace an initial instance normalization layer
  • the first standard up-sampling layer SU1 may further replace a last instance normalization layer.
  • the second standard down-sampling SD2 and the second standard up-sampling layer SU2 may also replace instance normalization layers.
  • the instance normalization layer is used to normalize a feature image output from a first convolutional layer, so that gray values of pixels of the feature image changes within a predetermined range, thereby simplifying generation and improving the quality of style transfer.
  • the predetermined range may be [-1, 1].
  • the instance normalization layer normalizes a feature image according to the mean value and variance of each feature image itself.
  • the instance normalization layer can be used to standardize a single image.
  • a size of a mini-batch gradient descent method is T
  • a quantity of feature images output from a first convolutional layer is C
  • each feature image is a matrix of H rows and W columns
  • a model of the feature image is represented as (T, C, W, H).
  • y tijk represents a result obtained after x tijk is processed by the instance normalization layer.
  • ⁇ is a very small integer, so as to avoid the denominator being 0.
  • the analysis network may include a second convolutional neural network 20.
  • the second convolutional neural network 20 is configured to receive the first training input image, the second training input image and the training output image, and to respectively generate and output a first training input feature of the first training input image, a first training output feature and a second training output feature of the training output image and a second training input feature of the second training input image.
  • the second training input image is a style image.
  • the second training input image may be a famous painting of various art masters (such as Monet, Van Gogh, Picasso, etc.).
  • the present disclosure is not limited to this, for example, the second training input image may also be an ink-and-wash painting, a sketch, etc.
  • the training output image is an image formed after the first convolutional neural network 10 performs a style transfer on the first training input image.
  • the training output image includes content features of the first training input image and style features of the second training input image.
  • the first training input image, the second training input image and the training output image all have the same size.
  • both the first training input feature and the first training output feature are content features
  • both the second training input feature and the second training output feature are style features
  • the content features denote the distribution of objects in the image over the whole image
  • the style features denote the relationship between different feature images in different layers of the convolutional neural network.
  • the content features include the content information of an image
  • the style features may include the texture information of the image, the color information of the image, and so on.
  • the texture information for example, represents the correlation among feature images, and is independent of location.
  • a feature image in the convolutional neural network may be a one-dimensional matrix
  • a Gram matrix may be used to measure the correlation degree of respective vectors in the one-dimensional matrix.
  • the Gram matrix can be introduced into the convolutional neural network, and is used for calculating style features of the image.
  • FIG. 7 is a schematic structural view of a second convolutional neural network provided by an embodiment of the present disclosure.
  • the second convolutional neural network 20 includes a plurality of second convolutional layers (e.g., a second convolutional layer C41, a second convolutional layer C42, a second convolutional layer C43, and a second convolutional layer C4n 2 ) sequentially connected and a plurality of second pooling layers (e.g., a second pooling layer P41, a second pooling layer P42, and a second pooling layer P43) interposed between adjacent second convolutional layers.
  • a plurality of second convolutional layers e.g., a second convolutional layer C41, a second convolutional layer C42, a second convolutional layer C43, and a second convolutional layer C4n 2
  • second pooling layers e.g., a second pooling layer P41, a second pooling layer P42, and a second pooling layer P43
  • Each of the second convolutional layers is used for extracting and outputting a first training input feature, a first training output feature, a second training output feature and/or a second training input feature.
  • the second convolutional neural network 20 includes n 2 second convolutional layers and n 2 second pooling layers, where n 2 is a positive integer.
  • each of the second convolutional layers includes a plurality of second convolutional kernels.
  • the second convolutional kernels can be used to extract content features and/or style features of an input image.
  • the input image shown in FIG. 7 may be the first training input image, the second training input image and the training output image.
  • the second convolutional neural network 20 may employ a convolutional neural network model that has been trained.
  • parameters e.g., second convolutional kernels, etc.
  • the second convolutional neural network 20 may adopt AlexNet, GoogleNet, VGG, Deep Residual Learning or other neural network models to implement extraction of the content features and the style features of the input image.
  • VGG neural network is a kind of deep convolutional neural network, is developed by a Visual Geometry Group of Oxford University, and has been widely used in the field of visual recognition.
  • the VGG network may include 19 layers, and some of them may be normalized.
  • the second convolutional neural network 20 may employ a deep neural network capable of classifying images.
  • an input image is processed by a plurality of second convolutional layers and a plurality of second pooling layers, so that features are extracted.
  • An output of each of the second convolutional layers is a feature image of the input image.
  • the second pooling layer can reduce the resolution of the feature image and pass the feature image to a second convolutional layer of the next level.
  • a plurality of second convolutional layers may output a plurality of feature images, and the plurality of feature images can characterize features of different levels of the input image (e.g. textures, edges, objects, etc.).
  • the feature image is input to a planarization layer, the planarization layer converts the feature image into a vector and passes the vector to a fully connected layer and a classifier.
  • the classifier layer may include a softmax classifier, and the softmax classifier may output the probabilities that the input image belongs to respective category labels, a label with the greatest probability will be a final output of the second convolutional neural network 20.
  • the second convolutional neural network 20 implements the image classification.
  • a l -th second convolutional layer has N l second convolutional kernels
  • the l -th second convolutional layer is used for generating and outputting N l first training feature images of the first training input image, N l second training feature images and N l third training feature images of the training output image, and N l fourth training feature images of the second training input image
  • the N l second convolutional kernels are in one-to-one correspondence to the N l first training feature images, to the N l second training feature images, to the N l third training feature images and to the N l fourth training feature images, respectively.
  • one second convolutional kernel is used to generate and output one first training feature image, one second training feature image, one third training feature image or one fourth training feature image. If the N l second convolutional kernels are different from each other, then the N l first training feature images are different from each other, the N l second training feature images are different from each other, the N l third training feature images are different from each other, and the N l fourth training feature images are also different from each other.
  • / is a positive integer.
  • / represents a label of a second convolutional layer in the second convolutional neural network 20.
  • the second convolutional neural network 20 has n2 second convolutional layers, and then a value range of l is [1, n2].
  • l is [1, n2].
  • the present disclosre is not limited to this, and according to the actual application requirements, l can also be other values.
  • the N l first training feature images, the N l second training feature images, the N l third training feature images and the N l fourth training feature images all have the same size.
  • the first training feature images and the third training feature images are content feature images
  • the second training feature images and the fourth training feature images are style feature images. That is, some second convolutional layers in the second convolution neural network 20 may be used to extract content features of the first training input image so as to obtain the first training feature images, other second convolutional layers in the second convolutional neural network 20 are used to extract style features and content features of the training output image so as to obtain the second training feature images and the third training feature images, respectively, still some other second convolutional layers in the second convolutional neural network 20 are further used to extract style features of the second training input image so as to obtain the fourth training feature images.
  • second convolutional layers for extracting content features and second convolutional layers for style features may be the same, and may also be different.
  • the loss function 30 may further include a content loss function and a style loss function.
  • the content loss function is used to describe the difference in content between the first training input image and the training output image
  • the style loss function is used to describe the difference in style between the second training input image and the training output image.
  • the content loss function is used to calculate, based on the first training input feature of the first training input image and the first training output feature of the training output image, a content loss value of the parameters of the first convolutional neural network 10.
  • the style loss function is used to calculate, based on the second training output feature of the training output image and the second training input feature of the second training input image, a style loss value of the parameters of the first convolutional neural network 10.
  • the step S40 may further include extracting a first training input feature of the first training input image and extracting a first training output feature of the training output image by an analysis network; and according to the first training input feature and the first training output feature, calculating the content loss value of the parameters of the neural network by the content loss function.
  • the loss value of neural network also includes the content loss value.
  • the step S50 may further include adjusting values of the plurality of first convolutional kernels and values of the plurality of biases according to the content loss value.
  • the step S40 may further include extracting a second training input feature of the second training input image and a second training output feature of the training output image by the analysis network; according to the second training output feature and the second training input feature, calculating the style loss value of the parameters of the neural network by the style loss function.
  • the loss value of neural network includes the style loss value.
  • the step S50 may further include adjusting the values of the plurality of first convolutional kernels and the values of the plurality of biases according to style loss value.
  • a content loss function of the l -th convolutional layer is expressed as a formula (4):
  • C l 1 2 ⁇ S 1 ⁇ ij F ij l ⁇ P ij l 2
  • C l represents the content loss function
  • F ij l represents a value of a j-th position in a first training feature image corresponding to an i-th second convolutional kernel in the l -th second convolutional layer
  • P ij l represents a value of a j-th position in a third training feature image corresponding to the i-th second convolutional kernel in the l -th second convolutional layer
  • S 1 is a constant.
  • L content represents the total content loss function
  • Q1 is a positive integer
  • w 1 l represents a weight of C 1 .
  • E l represents the style loss function
  • M l represents a size of the fourth training feature image
  • a ij l represents a value of a j-th position in a Gram matrix of a second training feature image corresponding to the i-th second convolutional kernel in the l -th second convolutional layer
  • G ij l represents a value of a j-th position in a Gram matrix of a fourth training feature image corresponding to the i-th second convolutional kernel in the l -th second convolutional layer
  • S 2 is a constant.
  • the size of the second training feature image is the same as the size of the fourth training feature image, and therefore, M l also represents the size of the second training feature image.
  • L style represents the total style loss function
  • Q2 is a positive integer, and denotes the number of second convolutional layers for extracting and outputting the second training input feature and the second training output feature
  • w 2 l denotes a weight of E l .
  • Q1 and Q2 may be the same, and may also be different.
  • the style features may be extracted by a plurality of second convolutional layers (e.g., two or three second convolutional layers, etc.), while the content features may be extracted by one second convolutional layer.
  • the style features may also be extracted by one second convolutional layer.
  • the content features may also be extracted by a plurality of second convolutional layers. The embodiments of the present disclosure do not specifically limit this.
  • the loss value of the parameters of the first convolutional neural network 10 includes the L1 loss value, the content loss value and the style loss value. That is to say, a value of L total represents the loss value of the parameters of the first convolutional neural network 10.
  • the neural network may include a first conversion matrix 50 and a second conversion matrix 51.
  • the first training input image has a first training input color channel, a second training input color channel and a third training input color channel.
  • the step S20 may include converting the first training input image into a first training intermediate image by the first conversion matrix; and inputting the first training intermediate image to the neural network.
  • converting the first training input image into the first training intermediate image by the first conversion matrix includes: converting, by the first conversion matrix, data information of the first training input color channel, data information of the second training input color channel and data information of the third training input color channel of the first training input image into data information of a first training luminance channel, data information of a first training color difference channel and data information of a second training color difference channel of the first training intermediate image.
  • the step S30 may include: performing the style transfer process on the data information of the first training luminance channel of the first training intermediate image, the data information of the first training color difference channel of the first training intermediate image, and the data information of the second training color difference channel of the first training intermediate image by using the first sub-network 101, the second sub-network 102 and the third sub-network 103 respectively, so as to generate data information of a second training luminance channel of a second training intermediate image, data information of a third training color difference channel of the second training intermediate image, and data information of a fourth training color difference channel of the second training intermediate image; and converting the second training intermediate image into the training output image.
  • the first training input image is firstly down-sampled by a first standard down-sampling layer SD1, so as to reduce the amount of data information of the first training color difference channel of the first training intermediate image; next, the data information of the first training color difference channel is processed by a plurality of first convolutional layers; after that, the data information of the first training color difference channel is up-sampled by a first standard up-sampling layer SU1, so as to increase the amount of data information of the first training color difference channel; finally, the data information of the first training color difference channel is transmitted to the second conversion matrix 51.
  • the third sub-network 103' is used to process the data information of the second training color difference channel of the first training intermediate image.
  • converting the second training intermediate image into the training output image includes: receiving and converting, by the second conversion matrix 51, the data information of the second training luminance channel, the data information of the third training color difference channel and the data information of the fourth training color difference channel of the second training intermediate image processed by the first convolutional neural network 10, so as to obtain data information of a first training output color channel of the training output image, data information of a second training output color channel of the training output image and data information of a third training output color channel of the training output image.
  • the first training input image is an image in a RGB format
  • the first training intermediate image is an image in a YUV format
  • the first training input color channel, the second training input color channel and the third training input color channel are a red (R) channel, a green (G) channel and a blue (B) channel, respectively.
  • the first training luminance channel, the first training color difference channel and the second training color difference channel are a Y channel, a U channel and a V channel, respectively.
  • the Y channel, the U channel and the V channel are separated from each other; Y represents brightness, U and V represents color difference, and U and V are two components constituting color.
  • the training output image is an image in a RGB format
  • the second training intermediate image is an image in a YUV format
  • the first training output color channel, the second training output color channel and the third training output color channel are a red (R) channel, a green (G) channel and a blue (B) channel, respectively.
  • the second training luminance channel, the third training color difference channel and the fourth training color difference channel are a Y channel, a U channel and a V channel, respectively.
  • FIG. 8 is a schematic flowchart of an image processing method for implementing image style transfer based on a neural network, provided by an embodiment of the present disclosure.
  • the neural network is a neural network trained according to the training method provided by any one of the above embodiments.
  • the image processing method provided by an embodiment of the present disclosure may include the following steps:
  • the image processing method provided by the embodiment of the present disclosure can perform a style transfer process on the first image by a neural network, so that not only a better style transfer effect can be achieved, but also the processing speed can be improved, thereby having a better and wider application prospect.
  • the first image may be an image of various types.
  • the first image may be a person image, an animal or plant image, a landscape image, or the like.
  • the first image may be obtained by an image acquisition device.
  • the image acquisition device may be, for example, a camera of a smartphone, a camera of a tablet, a camera of a personal computer, a lens for a digital camera, or even a webcam.
  • the second image is an image formed by performing style transfer on the first image through the first convolutional neural network.
  • the second image includes content features of the first image and style features of a style image.
  • the style features are determined and unchanged.
  • a painting of Picasso e.g., named as "Dream”
  • the style of the second image obtained by processing the first image by the trained neural network is the style of the Picasso's painting (named as "Dream”).
  • the style image may be the second training input image in an embodiment of the training method of the neural network.
  • FIG. 9A is an exemplary diagram of a first image
  • FIG. 9B is a schematic diagram of a style image
  • FIG. 9C is a schematic diagram of a second image obtained through an image processing method provided by an embodiment of the present disclosure.
  • the first image may be a photograph taken by a user.
  • the style image may be an Ukiyo-e engraving, and for example, the picture shown in FIG. 9B is The Great Wave of Kanagawa.
  • the image shown in FIG. 9C is a second image obtained by the image processing method provided in an embodiment of the present disclosure, and the second image has content features of the first image and style features of the style image.
  • the first image and the second image may have the same resolution, and may also have different resolutions.
  • the first image and the second image are of the same size.
  • the neural network further includes a first conversion matrix 50, and the first image has a first input color channel, a second input color channel, and a third input color channel.
  • the step S101 may include: converting the first image into a first intermediate image by the first conversion matrix; and inputting the first intermediate image into the neural network.
  • the first image is an image in a RGB format
  • the first intermediate image is an image in a YUV format
  • the first input color channel, the second input color channel and the third input color channel are a red (R) channel, a green (G) channel and a blue (B) channel, respectively.
  • the first luminance channel, the first color difference channel and the second color difference channel are a Y channel, a U channel and a V channel, respectively.
  • Y represents brightness
  • U and V represent color difference
  • U and V are two components constituting color.
  • converting the first image to the first intermediate image by the first conversion matrix may include: converting, by the first conversion matrix, data information of the first input color channel of the first image, data information of the second input color channel of the first image and data information of the third input color channel of the first image into data information of a first luminance channel of the first intermediate image, data information of a first color difference channel of the first intermediate image and data information of a second color difference channel of the first intermediate image.
  • Y1, U1 and V1 respectively represent luminance information
  • first chrominance information and second chrominance information of the first intermediate image namely, Y1, U1 and V1 respectively represent data of the Y channel, data of the U channel and data of the V channel of the first intermediate image.
  • R1, G1 and B1 respectively represent the red information, the green information and the blue information of the first image, that is, R1, G1 and B1 respectively represent the data of the R channel, the data of the G channel and the data of the B channel of the first image.
  • the neural network includes a first convolutional neural network.
  • the first convolutional neural network may include a first sub-network, a second sub-network and a third sub-network.
  • the first sub-network is used for processing the data information of the Y channel
  • the second sub-network is used for processing the data information of the U channel
  • the third sub-network is used for processing the data information of the V channel.
  • the image processing method provided by an embodiment of the present disclosure may separately process different channels of the first intermediate image, thereby reducing the model of generating neural network, further improving the processing speed, and increasing the flexibility of image processing.
  • the first sub-network includes a first set of first convolutional layers
  • the second sub-network includes a second set of first convolutional layers
  • the third sub-network includes a third set of first convolutional layers.
  • the first sub-network 101 further includes a plurality of first pooling layers, a plurality of composite layers and a plurality of instance normalization layers
  • the second sub-network 102 further includes a plurality of first pooling layers, a plurality of composite layers and a plurality of instance normalization layers
  • the third sub-network 103 further includes a plurality of first pooling layers, a plurality of composite layers, and a plurality of instance normalization layers.
  • the YUV format may include YUV444, YUV420, YUV422 and other formats.
  • the main difference among YUV444, YUV420, YUV422 and other formats is the sampling mode and storage mode of data of the U channel and the V channel.
  • a data stream for storing or processing the four pixel points is: Y0U0V0 Y1U1V1 Y2U2V2 Y3U3V3
  • the mapped pixel points are represented respectively as: [Y0U0V0][Y1U1V1][Y2U2V2][Y3U3V3]
  • the mapped pixel points are original pixel points.
  • the YUV420 format indicates that only one kind of chromaticity information (first chromaticity information U or second chrominance information V) is in each row of pixels, and the first chromaticity information U or the second chrominance information V is sampled and stored at a frequency of 1/2. During image processing, different chromaticity information of adjacent rows is processed.
  • a data stream for storing or processing the eight pixel points is:
  • the mapped pixel points are represented as:
  • FIG. 10A is a schematic structural diagram of a neural network provided by an embodiment of the present disclosure
  • FIG. 10B is a schematic structural diagram of another neural network provided by an embodiment of the present disclosure.
  • the first intermediate image is in an image format of YUV 444.
  • the structures of a first sub-network, a second sub-network and a third sub-network may be the same, and may also be different.
  • a first sub-network 101, a second sub-network 102 and a third sub-network 103 have the same structure. Descriptions will be given below by taking the first sub-network 101 as an example.
  • the first sub-network 101 includes five first convolutional layers (a first convolutional layer C11, a first convolutional layer C12, a first convolutional layer C13, a first convolutional layer C14 and a first convolutional layer C15), two first pooling layers (a first pooling layer P11 and a first pooling layer P12), two composite layers (a composite layer M11 and a composite layer M12) and four instance normalization layers (an instance normalization layer N11, an instance normalization layer N12, an instance normalization layer N13 and an instance normalization layer N14).
  • the step S102 may include: performing the style transfer processing on the first luminance channel, the first color difference channel and the second color difference channel of the first intermediate image by using the first sub-network, the second sub-network and the third sub-network, respectively, so as to generate a second intermediate image, the second intermediate image comprising data information of a second luminance channel, data information of a third color difference channel and data information of a fourth color difference channel; and converting the second intermediate image into the second image.
  • the first intermediate image is in an image format of YUV 420.
  • the first image may be processed on the YUV color space so that the first intermediate image is in the image format of YUV420. Therefore, the size of U channel and the size of V channel are reduced, and furthermore, the number of convolutional kernels in the generating neural network is reduced.
  • the second sub-network includes a first standard up-sampling layer and a first standard down-sampling layer.
  • the first standard down-sampling layer is used to replace an initial first convolutional layer, an initial first pooling layer, and an initial instance normalization layer of the second set of first convolutional layers in the second sub-network of the first convolutional neural network.
  • the first standard up-sampling layer is used to replace a last first convolutional layer, a last composite layer and a last instance normalization layer of the second set of first convolutional layers in the second sub-network of the first convolutional neural network.
  • the third sub-network includes a second standard up-sampling layer and a second standard down-sampling layer.
  • the second standard down-sampling layer is used to replace an initial first convolutional layer, an initial first pooling layer and an initial instance normalization layer of the third set of first convolutional layers in the third sub-network of the first convolutional neural network.
  • the second standard up-sampling layer is used to replace a last first convolutional layer, a last composite layer and a last instance normalization layer of the third set of first convolutional layers in the third sub-network of the first convolutional neural network.
  • a first sub-network 101' may be identical in structure to the first sub-network 101 in the first example.
  • the second sub-network 102' and the third sub-network 103' have the same structure. Descriptions will be given below by taking the second sub-network 102' as an example.
  • the second sub-network 102' includes one first standard down-sampling layer SD1, one first standard up-sampling layer SU1, three first convolutional layers (a first convolutional layer C22, a first convolutional layer C23 and a first convolutional layer C24), one first pooling layer (a first pooling layer P22), one composite layer (a composite layer M21) and two instance normalization layers (an instance normalization layer N22 and an instance normalization layer N23).
  • the first standard down-sampling layer SD1 replaces the initial first convolutional layer, the initial first pooling layer, and the initial instance normalization layer
  • the first standard up-sampling layer SU1 replaces the last first convolutional layer, the last first pooling layer and the last instance normalization layer.
  • the first standard down-sampling layer SD1 replaces the initial first convolutional layer C21, the initial first pooling layer P21 and the initial instance normalization layer N21; and the first standard up-sampling layer SU1 replaces the last first convolutional layer C25, the last composite layer M22 and the last instance normalization layer N24.
  • a first convolutional layer for processing the highest resolution of the U channel is omitted in the second sub-network 102', and a first convolutional layer for processing the highest resolution of the V channel is omitted in the third sub-network 103'.
  • the image processing method can further increases the image processing speed.
  • the first standard down-sampling layer SD1 and the first standard up-sampling layer SU1 are respectively used for down-sampling and up-sampling the data information of U channel.
  • the second standard down-sampling layer SD2 and the second standard up-sampling layer SU2 are respectively used for down-sampling and up-sampling the data information of V channel.
  • the structures and sampling factors of the first standard down-sampling layer SD1 and the second standard down-sampling layer SD2 may be the same or different.
  • the structures and sampling factors of the first standard up-sampling layer SU1 and the second standard up-sampling layer SU2 may be the same or different. There are no specific limitations on this.
  • the structures and down-sampling factors of the first standard down-sampling layer SD1 and the second standard down-sampling layer SD2 are the same, the structures and up-sampling factors of the first standard up-sampling layer SU1 and the second standard up-sampling layer SU2 are also the same.
  • the second sub-network 102' and the third sub-network 103' have the same structure.
  • the step S102 includes: processing a first luminance channel, a first color difference channel and a second color difference channel of the first intermediate image by using the first sub-network 101', the second sub-network 102' and the third sub-network 103' respectively, so as to generate a second intermediate image, the second intermediate image including data information of a second luminance channel, data information of a third color difference channel and data information of a fourth color difference channel; and converting the second intermediate image into a second image.
  • the second image is an image in a RGB format
  • the second intermediate image is an image in a YUV format
  • the first output color channel, the second output color channel and the third output color channel are a red (R) channel, a green (G) channel and a blue (B) channel, respectively.
  • the second luminance channel, the third color difference channel and the fourth color difference channel are a Y channel, a U channel and a V channel, respectively.
  • the neural network may further include a second conversion matrix 51.
  • Converting the second intermediate image into the second image includes: converting, by the second conversion matrix, the data information of the second luminance channel of the second intermediate image, the data information of the third color difference channel of the second intermediate image and the data information of the fourth color difference channel of the second intermediate image into data information of a first output color channel of the second image, data information of a second output color channel of the second image and data information of a third output color channel of the second image.
  • the second intermediate image is an image obtained after the first intermediate image is processed by the neural network.
  • the neural network includes an instance normalization layer, then a range of gray values of pixels of the second intermediate image is [-1, 1], and when performing the operation of converting the second intermediate image into the second image, the method further includes: converting the data information of the Y channel, the data information of the U channel and the data information of the V channel of the second intermediate image into data information of the R channel, data information of the G channel and data information of the B channel of the second image; and converting the gray values of the pixels of the second image to be within a range of [0, 255].
  • the neural network may not include a planarization layer, a fully connected layer, and a classifier.
  • FIG. 11 is an exemplary block diagram of an image processing device provided by an embodiment of the present disclosure.
  • the image processing device provided in an embodiment of the present disclosure includes an image acquisition module 78 and an image processing module 80.
  • the image acquisition module 78 is configured to acquire a first image
  • the image processing module 80 is configured to perform a style transfer process on the first image, so as to generate a second image.
  • the image processing module 80 may include the neural network that is obtained by training according to the training method as described in any one of the above embodiments.
  • the image processing device provided by an embodiment of the disclosure performs a style transfer process on the first image through the neural network, so as to improve effects of the image style transfer and image fusion, and the processing effect, the processing speed and other aspects are all taken into account at the same time, thereby having a better and wider application prospect.
  • the image acquisition module 78 may include a memory that stores the first image.
  • the image acquisition module 78 may also include one or more cameras, so as to obtain the first image.
  • the image acquisition module 705 may be hardware, software, firmware, and any feasible combination thereof.
  • the image processing device further includes a first conversion module 82.
  • the first conversion module 82 includes a first conversion matrix 50 in the neural network in an embodiment of the image processing method as described above.
  • the first conversion matrix 50 is used to convert the first image into a first intermediate image.
  • the first image has a first input color channel, a second input color channel and a third input color channel
  • the first intermediate image has a first luminance channel, a first color difference channel and a second color difference channel.
  • the first conversion matrix 50 may be used to convert data information of the first input color channel, data information of the second input color channel and data information of the third input color channel of the first image into data information of the first luminance channel, data information of the first color difference channel and data information of the second color difference channel of the first intermediate image.
  • the neural network may include a first convolutional neural network
  • the first convolutional neural network may include a first sub-network, a second sub-network, and a third sub-network.
  • the first sub-network, the second sub-network and the third sub-network are respectively used for processing the first luminance channel, the first color difference channel and the second color difference channel of the first intermediate image, so as to generate data information of a second luminance channel of a second intermediate image, data information of a third color difference channel of the second intermediate image and data information of a fourth color difference channel of the second intermediate image.
  • the first sub-network includes a first set of first convolutional layers
  • the second sub-network includes a second set of first convolutional layers
  • the third sub-network includes a third set of first convolutional layers.
  • the second sub-network includes a first standard up-sampling layer and a first standard down-sampling layer
  • the third sub-network includes a second standard up-sampling layer and a second standard down-sampling layer.
  • the first standard down-sampling layer is used to replace the initial first convolutional layer in the second set of first convolutional layers of the second sub-network
  • the first standard up-sampling layer is used to replace the last first convolutional layer in the second set of first convolutional layers of the second sub-network.
  • the second standard down-sampling layer is used to replace the initial first convolutional layers in the third set of first convolutional layers of the third sub-network
  • the second standard up-sampling layer is used to replace the last first convolutional layer in the third set of first convolutional layers of the third sub-network.
  • the image processing device further includes a second conversion module 84.
  • the second conversion module 84 includes a second conversion matrix 51 in the neural network described above.
  • the second conversion matrix 51 may be used to convert the second intermediate image into a second image.
  • the second conversion module 84 is used to convert data information of a second luminance channel, data information of a third color difference channel and data information of a fourth color difference channel of the second intermediate image into data information of a first output color channel of the second image, data information of a second output color channel of the second image and data information of a third output color channel of the second image.
  • the first conversion matrix 50 is used for receiving the first image, and converting the first image into the first intermediate image.
  • the first intermediate image outputted from the first conversion matrix 50 is input into the image processing module 80.
  • the image processing module 80 is used to perform a style transfer process on the first intermediate image, so as to generate the second intermediate image.
  • the second conversion matrix 51 is used for receiving the second intermediate image outputted from the image processing module 80, and converting the second intermediate image into the second image and then outputting the second image.
  • the second image is used as a final output of the image processing device.
  • the second image has content features of the first image and style features of another image (e.g., an oil painting of a master of art, etc.).
  • FIG. 12 is a schematic block diagram of another image processing device provided by an embodiment of the present disclosure. At least one embodiment of the present disclosure provides an image processing device, as shown in FIG. 12 , the image processing device includes a processor 70 and a memory 75. It should be noted that the components of the image processing device shown in FIG. 12 are merely exemplary and are not limited, and according to actual application needs, the image processing device can also have other components.
  • processor 70 and the memory 75 can be communicated with each other directly or indirectly.
  • the processor 70, the memory 75 and other components may be connected and communicated with each other through a network.
  • the network may comprise a wireless network, a wired network, and/or any combination of the wireless network and the wired network.
  • the network may comprise a local area network, the Internet, a telecommunication network, Internet of things based on the Internet and/or the telecommunication network, and/or any combination of the above networks, and the like.
  • the wired network may communicate by means such as a twisted pair, coaxial cable or optical fiber transmission.
  • the wireless network may communicate by means such as 3G/4G/5G mobile communication networks, Bluetooth, Zigbee or WiFi.
  • the present disclosure does not limit types and functions of the network herein.
  • the processor 70 may control other components in the image processing device to perform a desired function.
  • the processor 70 may be a central processing unit (CPU), a tensor processing unit (TPU), a graphics processing unit (GPU), or other forms of device having data processing capabilities and/or program execution capabilities.
  • the central processing unit (CPU) may be an X86, ARM architecture, or the like.
  • the GPU can be separately and directly integrated into a motherboard, or may be built into a north bridge chip of the motherboard.
  • the GPU can also be built into the central processing unit (CPU).
  • the memory 75 may comprise an arbitrary combination of one or more computer program products.
  • the computer program products may comprise various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory.
  • the volatile memory may comprise, for example, a random access memory (RAM) and/or a cache or the like.
  • the non-volatile memory may comprise, for example, a read only memory (ROM), a hard disk, an erasable programmable read only memory (EPROM), a portable compact disc-read only memory (CD-ROM), a USB memory, a flash memory, and the like.
  • one or more computer instructions may be stored on the memory 75, and the processor 70 can execute the computer instructions, so as to achieve various functions.
  • a variety of application programs and various data, such as style images, various data used and/or generated by the application programs and so on, may also be stored in a computer-readable storage medium.
  • At least one embodiment of the present disclosure further provides a storage medium.
  • one or more computer instructions may be stored on the storage medium.
  • Some computer instructions stored on the storage medium may be, for example, instructions for implementing one or more steps of the above described image processing method.
  • Some other computer instructions stored on the storage medium may be, for example, instructions for implementing one or more steps of the above described training method of the neural network.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)
EP18850585.3A 2017-08-29 2018-08-16 Bildverarbeitungsverfahren, bildverarbeitungsvorrichtung und verfahren zum trainieren eines neuronalen netzes Active EP3678059B1 (de)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201710757133.7A CN109426858B (zh) 2017-08-29 2017-08-29 神经网络、训练方法、图像处理方法及图像处理装置
PCT/CN2018/100833 WO2019042139A1 (zh) 2017-08-29 2018-08-16 图像处理方法、图像处理装置、神经网络的训练方法

Publications (3)

Publication Number Publication Date
EP3678059A1 true EP3678059A1 (de) 2020-07-08
EP3678059A4 EP3678059A4 (de) 2021-05-26
EP3678059B1 EP3678059B1 (de) 2023-06-07

Family

ID=65503575

Family Applications (1)

Application Number Title Priority Date Filing Date
EP18850585.3A Active EP3678059B1 (de) 2017-08-29 2018-08-16 Bildverarbeitungsverfahren, bildverarbeitungsvorrichtung und verfahren zum trainieren eines neuronalen netzes

Country Status (4)

Country Link
US (1) US11461639B2 (de)
EP (1) EP3678059B1 (de)
CN (1) CN109426858B (de)
WO (1) WO2019042139A1 (de)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113240573A (zh) * 2020-10-26 2021-08-10 杭州火烧云科技有限公司 一种针对千万级像素数字图像的基于局部和全局并行学习的风格变换方法及系统
TWI876866B (zh) * 2024-02-05 2025-03-11 大陸商上海柏愈科技有限公司 人臉紋理識別模型建立方法

Families Citing this family (161)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11132529B2 (en) 2016-11-16 2021-09-28 Ventana Medical Systems, Inc. Convolutional neural networks for locating objects of interest in images of biological samples
CN109426858B (zh) * 2017-08-29 2021-04-06 京东方科技集团股份有限公司 神经网络、训练方法、图像处理方法及图像处理装置
US10664718B1 (en) 2017-09-11 2020-05-26 Apple Inc. Real-time adjustment of hybrid DNN style transfer networks
WO2019066794A1 (en) 2017-09-27 2019-04-04 Google Llc END-TO-END NETWORK MODEL FOR HIGH-RESOLUTION IMAGE SEGMENTATION
CN109754357B (zh) 2018-01-26 2021-09-21 京东方科技集团股份有限公司 图像处理方法、处理装置以及处理设备
US10991150B2 (en) * 2018-05-09 2021-04-27 Massachusetts Institute Of Technology View generation from a single image using fully convolutional neural networks
KR102096388B1 (ko) * 2018-06-05 2020-04-06 네이버 주식회사 모바일 환경에서 실시간 추론이 가능한 ｄｎｎ 구성을 위한 최적화 기법
US10963748B1 (en) * 2018-08-31 2021-03-30 Snap Inc. Generative neural network distillation
US12050461B2 (en) * 2018-09-07 2024-07-30 DoorDash, Inc. Video system with frame synthesis
CN110956575B (zh) 2018-09-26 2022-04-12 京东方科技集团股份有限公司 转变图像风格的方法和装置、卷积神经网络处理器
CN109360261B (zh) * 2018-09-28 2019-12-31 北京达佳互联信息技术有限公司 图像处理方法、装置、电子设备及存储介质
KR102640234B1 (ko) * 2018-09-28 2024-02-23 삼성전자주식회사 디스플레이 장치의 제어 방법 및 그에 따른 디스플레이 장치
CN109191382B (zh) 2018-10-18 2023-12-05 京东方科技集团股份有限公司 图像处理方法、装置、电子设备及计算机可读存储介质
CN111127304B (zh) * 2018-10-31 2024-02-20 微软技术许可有限责任公司 跨域图像转换
US10748324B2 (en) * 2018-11-08 2020-08-18 Adobe Inc. Generating stylized-stroke images from source images utilizing style-transfer-neural networks with non-photorealistic-rendering
US11227075B2 (en) * 2019-01-25 2022-01-18 SWATCHBOOK, Inc. Product design, configuration and decision system using machine learning
US10769764B2 (en) * 2019-02-08 2020-09-08 Adobe Inc. Hierarchical scale matching and patch estimation for image style transfer with arbitrary resolution
US11308598B2 (en) * 2019-02-14 2022-04-19 Sharif University Of Technology Quality assessment of an image
CN109978840A (zh) * 2019-03-11 2019-07-05 太原理工大学 一种基于卷积神经网络的含纹理图像质量的判别方法
CN109934300B (zh) * 2019-03-21 2023-08-25 腾讯科技（深圳）有限公司 模型压缩方法、装置、计算机设备及存储介质
CN109978788B (zh) * 2019-03-25 2020-11-27 厦门美图之家科技有限公司 卷积神经网络生成方法、图像去马赛克方法及相关装置
US11494615B2 (en) * 2019-03-28 2022-11-08 Baidu Usa Llc Systems and methods for deep skip-gram network based text classification
CN110060508B (zh) * 2019-04-08 2020-11-20 武汉理工大学 一种用于内河桥区的船舶自动检测方法
CN110009044B (zh) * 2019-04-09 2021-09-03 北京七鑫易维信息技术有限公司 一种模型训练方法及装置、图像处理方法及装置
CN109993140B (zh) * 2019-04-09 2023-07-25 上海市刑事科学技术研究院 一种现场纹身取证手持装置及系统、控制方法
CN110188776B (zh) 2019-05-30 2025-09-23 京东方科技集团股份有限公司 图像处理方法及装置、神经网络的训练方法、存储介质
US11367163B2 (en) 2019-05-31 2022-06-21 Apple Inc. Enhanced image processing techniques for deep neural networks
CN110335206B (zh) * 2019-05-31 2023-06-09 平安科技（深圳）有限公司 智能滤镜方法、装置及计算机可读存储介质
CN112013962B (zh) * 2019-05-31 2022-11-04 南京理工大学 一种基于cnn神经网络的mrtd参数测试方法
RU2706891C1 (ru) * 2019-06-06 2019-11-21 Самсунг Электроникс Ко., Лтд. Способ формирования общей функции потерь для обучения сверточной нейронной сети для преобразования изображения в изображение с прорисованными деталями и система для преобразования изображения в изображение с прорисованными деталями
US11521011B2 (en) 2019-06-06 2022-12-06 Samsung Electronics Co., Ltd. Method and apparatus for training neural network model for enhancing image detail
CN112085041B (zh) * 2019-06-12 2024-07-12 北京地平线机器人技术研发有限公司 神经网络的训练方法、训练装置和电子设备
CN112101372B (zh) * 2019-06-17 2024-06-28 北京地平线机器人技术研发有限公司 确定图像特征的方法和装置
CN112115452A (zh) * 2019-06-20 2020-12-22 北京京东尚科信息技术有限公司 用于生成验证码图像的方法和装置
US10977528B1 (en) * 2019-06-25 2021-04-13 Amazon Technologies, Inc. Detecting similarity between images
CN110458906B (zh) * 2019-06-26 2024-03-15 广州大鱼创福科技有限公司 一种基于深度颜色迁移的医学图像着色方法
CN110288090B (zh) * 2019-06-28 2023-11-07 广东中星微电子有限公司 训练卷积神经网络的方法及装置、计算机设备和存储介质
CN110276411B (zh) * 2019-06-28 2022-11-18 腾讯科技（深圳）有限公司 图像分类方法、装置、设备、存储介质和医疗电子设备
CN110288607A (zh) * 2019-07-02 2019-09-27 数坤（北京）网络科技有限公司 分割网络的优化方法、系统和计算机可读存储介质
CN110309837B (zh) * 2019-07-05 2021-07-06 北京迈格威科技有限公司 基于卷积神经网络特征图的数据处理方法及图像处理方法
CN110378092B (zh) * 2019-07-26 2020-12-04 北京积加科技有限公司 身份识别系统及客户端、服务器和方法
CN111797881B (zh) * 2019-07-30 2024-06-28 华为技术有限公司 图像分类方法及装置
CN110473141B (zh) * 2019-08-02 2023-08-18 Oppo广东移动通信有限公司 图像处理方法、装置、存储介质及电子设备
US10956776B2 (en) * 2019-08-06 2021-03-23 Alibaba Group Holding Limited 2D convolutional accelerator that generates 3D results
CN110651277B (zh) * 2019-08-08 2023-08-01 京东方科技集团股份有限公司 计算机实现的方法、计算机实现的诊断方法、图像分类设备、以及计算机程序产品
CN112348161B (zh) * 2019-08-09 2024-08-16 北京地平线机器人技术研发有限公司 神经网络的训练方法、神经网络的训练装置和电子设备
CN110580726B (zh) * 2019-08-21 2022-10-04 中山大学 基于动态卷积网络的自然场景下人脸素描生成模型及方法
CN110675309A (zh) * 2019-08-28 2020-01-10 江苏大学 一种基于卷积神经网络和VGGNet16模型的图像风格转换方法
CN110544154B (zh) * 2019-08-30 2022-03-29 北京市商汤科技开发有限公司 服饰搭配方法及装置、电子设备和存储介质
CN110427922A (zh) * 2019-09-03 2019-11-08 陈�峰 一种基于机器视觉和卷积神经网络病虫害识别系统和方法
CN110555481B (zh) * 2019-09-06 2024-08-06 腾讯科技（深圳）有限公司 一种人像风格识别方法、装置和计算机可读存储介质
CN110728354B (zh) * 2019-09-11 2024-04-09 东南大学 一种基于改进的滑动式分组卷积神经网络的图像处理方法
US10943353B1 (en) 2019-09-11 2021-03-09 International Business Machines Corporation Handling untrainable conditions in a network architecture search
US11023783B2 (en) * 2019-09-11 2021-06-01 International Business Machines Corporation Network architecture search with global optimization
KR102916001B1 (ko) * 2019-09-16 2026-01-22 삼성전자주식회사 디스플레이 장치 및 이의 영상 처리 방법
CN111783779B (zh) * 2019-09-17 2023-12-05 北京沃东天骏信息技术有限公司 图像处理方法、装置和计算机可读存储介质
CN110796162B (zh) * 2019-09-18 2023-08-29 平安科技（深圳）有限公司 图像识别、训练识别模型的方法、相关设备及存储介质
CN110706220B (zh) * 2019-09-27 2023-04-18 贵州大学 一种胶囊内镜图像处理和分析方法
CN112580772B (zh) * 2019-09-30 2024-04-26 华为技术有限公司 卷积神经网络的压缩方法及装置
KR102623148B1 (ko) * 2019-10-15 2024-01-11 삼성전자주식회사 전자 장치 및 전자 장치의 제어 방법
CN110837830B (zh) * 2019-10-24 2023-06-23 上海兑观信息科技技术有限公司 一种基于时空卷积神经网络的图像字符识别方法
CN110766681B (zh) * 2019-10-28 2023-04-14 福建帝视信息科技有限公司 一种基于三重损失网络的竹条表面缺陷检测方法
CN112734864A (zh) * 2019-10-28 2021-04-30 天津大学青岛海洋技术研究院 一种用于灰度图像上色的三通路卷积神经网络结构
CN110826694B (zh) * 2019-10-30 2021-06-11 瀚博半导体(上海)有限公司 一种基于卷积神经网络的图像处理方法及装置
CN110969600A (zh) * 2019-11-12 2020-04-07 华北电力大学扬中智能电气研究中心 一种产品缺陷检测方法、装置、电子设备及存储介质
CN110866866B (zh) * 2019-11-14 2023-10-17 腾讯科技（深圳）有限公司 图像仿色处理方法、装置、电子设备及存储介质
CN110874826B (zh) * 2019-11-18 2020-07-31 北京邮电大学 应用于离子束精准镀膜的工件图像去雾化方法及装置
CN110909258B (zh) * 2019-11-22 2023-09-29 上海喜马拉雅科技有限公司 一种信息推荐方法、装置、设备及存储介质
WO2021106174A1 (ja) * 2019-11-29 2021-06-03 オリンパス株式会社 画像処理方法、学習装置及び画像処理装置
CN111062329B (zh) * 2019-12-18 2023-05-30 中山大学 基于增广网络的无监督行人重识别方法
CN111191704B (zh) * 2019-12-24 2023-05-02 天津师范大学 一种基于任务图卷积网络的地基云分类方法
CN111126494B (zh) * 2019-12-25 2023-09-26 中国科学院自动化研究所 基于各向异性卷积的图像分类方法及系统
US11080834B2 (en) * 2019-12-26 2021-08-03 Ping An Technology (Shenzhen) Co., Ltd. Image processing method and electronic device
CN113052756A (zh) * 2019-12-27 2021-06-29 武汉Tcl集团工业研究院有限公司 一种图像处理方法、智能终端及存储介质
KR102537207B1 (ko) * 2019-12-30 2023-05-25 포항공과대학교 산학협력단 머신 러닝에 기반한 이미지 처리 방법 및 장치
CN113128494B (zh) * 2019-12-30 2024-06-28 华为技术有限公司 识别图像中文本的方法、装置和系统
CN113076966B (zh) * 2020-01-06 2023-06-13 字节跳动有限公司 图像处理方法及装置、神经网络的训练方法、存储介质
CN113139893B (zh) * 2020-01-20 2023-10-03 北京达佳互联信息技术有限公司 图像翻译模型的构建方法和装置、图像翻译方法和装置
CN111291866B (zh) * 2020-01-22 2024-03-26 上海商汤临港智能科技有限公司 神经网络的生成、图像处理、智能行驶控制方法及装置
CN113159081B (zh) * 2020-01-23 2024-06-21 华为技术有限公司 一种图像处理方法以及相关设备
CN111275128B (zh) * 2020-02-13 2023-08-25 平安科技（深圳）有限公司 图像识别模型训练方法及系统和图像识别方法
CN113269213B (zh) * 2020-02-17 2023-06-30 百度在线网络技术（北京）有限公司 训练集的获取方法、装置及电子设备
CN111340188B (zh) * 2020-02-21 2024-06-28 北京集创北方科技股份有限公司 图像处理方法及装置、电子设备和可读存储介质
CN111415333B (zh) * 2020-03-05 2023-12-01 北京深睿博联科技有限责任公司 乳腺x射线影像反对称生成分析模型训练方法和装置
CN115244569A (zh) * 2020-03-09 2022-10-25 华为技术有限公司 一种基于神经网络的图像处理方法及装置
CN111401247B (zh) * 2020-03-17 2023-07-28 杭州小影创新科技股份有限公司 一种基于级联卷积神经网络的人像分割方法
CN111523561A (zh) * 2020-03-19 2020-08-11 深圳市彬讯科技有限公司 图像风格识别方法、装置、计算机设备及存储介质
CN111652262A (zh) * 2020-03-19 2020-09-11 深圳市彬讯科技有限公司 图像物体识别方法、装置、计算机设备及存储介质
CN113496159B (zh) * 2020-03-20 2022-12-23 昆明理工大学 一种多尺度卷积与动态权重代价函数的烟尘目标分割方法
CN115335852B (zh) * 2020-03-31 2026-03-27 华为技术有限公司 一种基于神经网络的图像处理方法及装置
CN111563263B (zh) * 2020-04-16 2023-04-25 绍兴聚量数据技术有限公司 一种任意图像风格迁移的无载体信息隐藏方法
US11501107B2 (en) 2020-05-07 2022-11-15 Adobe Inc. Key-value memory network for predicting time-series metrics of target entities
CN113630576A (zh) * 2020-05-08 2021-11-09 辛纳普蒂克斯公司 自适应视频流式传输系统和方法
US20210357730A1 (en) * 2020-05-12 2021-11-18 Alibaba Group Holding Limited Multi-size convolutional layer background
CN111598858B (zh) * 2020-05-12 2023-05-16 上海大学 一种基于迁移学习的橡胶手套的检测方法及系统
CN111860585A (zh) * 2020-06-11 2020-10-30 南京简凡科技有限公司 一种基于超声影像人工智能算法的脂肪肝自动检测方法
CN111523521B (zh) * 2020-06-18 2023-04-07 西安电子科技大学 一种双支路融合多尺度注意神经网络的遥感图像分类方法
CN111652830A (zh) * 2020-06-28 2020-09-11 Oppo广东移动通信有限公司 图像处理方法及装置、计算机可读介质及终端设备
CN111968127B (zh) * 2020-07-06 2021-08-27 中国科学院计算技术研究所 一种基于全切片病理图像的癌灶区域识别方法及系统
CN111814165B (zh) * 2020-07-07 2024-01-26 重庆大学 一种基于深度神经网络中间层的图像隐私保护方法
CN111885280B (zh) * 2020-07-17 2021-04-13 电子科技大学 一种混合卷积神经网络视频编码环路滤波方法
CN111862039A (zh) * 2020-07-20 2020-10-30 中山西尼视觉科技有限公司 快速编带视觉检测方法
CN111914854A (zh) * 2020-07-21 2020-11-10 同济大学 一种基于神经网络的喷雾特征提取方法、设备及存储介质
CN111986275B (zh) * 2020-07-31 2024-06-11 广州嘉尔日用制品有限公司 一种多模态半色调图像的逆半色调化方法
CN111986075B (zh) * 2020-08-12 2022-08-09 兰州交通大学 一种目标边缘清晰化的风格迁移方法
CN111950643B (zh) * 2020-08-18 2022-06-28 创新奇智(上海)科技有限公司 图像分类模型训练方法、图像分类方法及对应装置
CN112084948A (zh) * 2020-09-10 2020-12-15 深圳市迈航信息技术有限公司 采用人脸数据的方式来提高安检安全的方法以及系统
US11477464B2 (en) 2020-09-16 2022-10-18 Qualcomm Incorporated End-to-end neural network based video coding
CN112258592B (zh) * 2020-09-17 2024-11-01 深圳市捷顺科技实业股份有限公司 一种人脸可见光图的生成方法及相关装置
US12387132B1 (en) * 2020-09-29 2025-08-12 Amazon Technologies, Inc. Orchestration for building and executing machine learning pipelines on graph data
CN112258381B (zh) * 2020-09-29 2024-02-09 北京达佳互联信息技术有限公司 模型训练方法、图像处理方法、装置、设备及存储介质
CN112115920B (zh) * 2020-09-30 2024-07-26 广州市行动者科技有限责任公司 生成单目标检测神经网络的方法和装置
DE102020212515A1 (de) * 2020-10-02 2022-04-07 Robert Bosch Gesellschaft mit beschränkter Haftung Verfahren und Vorrichtung zum Trainieren eines maschinellen Lernsystems
CN112241941B (zh) * 2020-10-20 2024-03-22 北京字跳网络技术有限公司 获取图像的方法、装置、设备和计算机可读介质
CN114444554B (zh) * 2020-10-30 2025-05-09 中移(上海)信息通信科技有限公司 手写数字识别方法、装置、电子设备及计算机存储介质
CN114511075B (zh) * 2020-11-17 2026-03-06 罗伯特·博世有限公司 卷积神经网络的训练方法和系统
WO2022110027A1 (en) * 2020-11-27 2022-06-02 Boe Technology Group Co., Ltd. Computer-implemented image-processing method, image-enhancing convolutional neural network, and computer product
US20220172064A1 (en) * 2020-12-02 2022-06-02 Htc Corporation Machine learning method and machine learning device for eliminating spurious correlation
JP7815247B2 (ja) * 2020-12-10 2026-02-17 クゥアルコム・インコーポレイテッド ニューラルネットワークベースのビデオコーディングのためのフロントエンドアーキテクチャ
CN112580623B (zh) * 2020-12-25 2023-07-25 北京百度网讯科技有限公司 图像生成方法、模型训练方法、相关装置及电子设备
US11785068B2 (en) * 2020-12-31 2023-10-10 Synaptics Incorporated Artificial intelligence image frame processing systems and methods
CN112464916B (zh) * 2020-12-31 2023-09-19 上海齐感电子信息科技有限公司 人脸识别方法及其模型训练方法
CN112802160B (zh) * 2021-01-12 2023-10-17 西北大学 一种基于u-gat-it改进的秦腔角色卡通化风格迁移的方法
CN112767361B (zh) * 2021-01-22 2024-04-09 重庆邮电大学 基于轻量级残差U-net的反射光铁谱图像分割方法
CN112802063A (zh) * 2021-02-03 2021-05-14 阳光电源股份有限公司 卫星云图的预测方法、装置以及计算机可读存储介质
CN113033744B (zh) * 2021-03-16 2022-05-24 北京航空航天大学 一种艺术风格qr码的生成方法及系统
CN113139653B (zh) * 2021-03-18 2024-06-14 有米科技股份有限公司 用于图像哈希求解的神经网络训练方法及装置
KR20220132301A (ko) * 2021-03-23 2022-09-30 삼성전자주식회사 신경망 처리 장치를 포함하는 애플리케이션 프로세서, 및 이의 동작 방법
CN112950460A (zh) * 2021-03-26 2021-06-11 成都理工大学 一种用于图像风格迁移的技术
CN113011567B (zh) * 2021-03-31 2023-01-31 深圳精智达技术股份有限公司 一种卷积神经网络模型的训练方法及装置
CN113191208B (zh) * 2021-04-09 2022-10-21 湖北工业大学 一种用于遥感图像实例分割的特征提取方法和计算机设备
CN112990370B (zh) * 2021-04-26 2021-09-10 腾讯科技（深圳）有限公司 图像数据的处理方法和装置、存储介质及电子设备
CN113191944B (zh) * 2021-05-31 2023-11-07 大连民族大学 一种多通道图像内容特征融合风格迁移方法及系统
CN113379635B (zh) * 2021-06-18 2025-05-02 北京小米移动软件有限公司 图像处理方法及装置，模型训练方法、装置及存储介质
CN113486752B (zh) * 2021-06-29 2023-06-16 吉林大学 基于心电信号的情感识别方法及系统
CN113591756B (zh) * 2021-08-06 2024-06-28 南京空天宇航科技有限公司 基于异质信息交互卷积网络的车道线检测方法
TWI779824B (zh) * 2021-09-10 2022-10-01 瑞昱半導體股份有限公司 卷積神經網路的圖像處理方法與系統
CN115965840A (zh) * 2021-10-11 2023-04-14 北京字节跳动网络技术有限公司 图像风格迁移和模型训练方法、装置、设备和介质
GB2612775A (en) * 2021-11-10 2023-05-17 Sony Interactive Entertainment Inc System and method for generating assets
CN116152871A (zh) * 2021-11-18 2023-05-23 北京眼神智能科技有限公司 卷积神经网络训练和人脸检测方法、装置、介质及设备
ES3023747T3 (en) * 2022-01-28 2025-06-03 Fujitsu Ltd Kernel transfer
CN114550102A (zh) * 2022-03-01 2022-05-27 上海中通吉网络技术有限公司 货物堆积检测方法、装置、设备和系统
CN114612778A (zh) * 2022-03-14 2022-06-10 成都唐源电气股份有限公司 一种基于向量神经网络的接触网鸟窝识别方法及模型
CN114693826A (zh) * 2022-03-31 2022-07-01 深圳万兴软件有限公司 基于边缘损失和风格损失的图像卡通化方法、装置、计算机设备及存储介质
CN114778555B (zh) * 2022-04-24 2025-07-01 中国矿业大学 一种索膜结构损伤智能检测和全寿命健康监测方法
CN115018154A (zh) * 2022-06-06 2022-09-06 邢台路桥建设总公司 一种黄土湿陷性预测方法
US12437526B2 (en) 2022-06-23 2025-10-07 Honeywell International Inc. Methods and systems for automated display verification
CN114820871B (zh) * 2022-06-29 2022-12-16 北京百度网讯科技有限公司 字体生成方法、模型的训练方法、装置、设备和介质
CN114915496B (zh) * 2022-07-11 2023-01-10 广州番禺职业技术学院 基于时间权重和深度神经网络的网络入侵检测方法和装置
CN116208356B (zh) * 2022-10-27 2023-09-29 浙江大学 一种基于深度学习的虚拟货币挖矿流量检测方法
CN115620869A (zh) * 2022-11-09 2023-01-17 武汉市武昌医院 一种基于混合视觉与患肢镜像虚拟的评估与训练系统
CN115908919A (zh) * 2022-11-18 2023-04-04 河南科技学院 一种基于统一多颜色模型学习的水下图像分类深度级联网络
CN115761339A (zh) * 2022-11-18 2023-03-07 北京字跳网络技术有限公司 图像处理方法、装置、设备、介质及程序产品
CN115641045A (zh) * 2022-12-23 2023-01-24 河南新飞电器集团有限公司 一种冷链运输远程监控系统
CN116310714A (zh) * 2023-02-22 2023-06-23 佛山科学技术学院 一种基于双梯度强化对比学习的小样本图像识别方法
CN116524245B (zh) * 2023-04-11 2026-01-02 中国人民解放军国防科技大学 结构化平面物体识别模型训练及识别方法、电子设备
CN116342578A (zh) * 2023-04-22 2023-06-27 武汉纺织大学 基于卷积神经网络的楚国纺织品服装元素迁移方法及系统
CN116721306B (zh) * 2023-05-24 2024-02-02 北京思想天下教育科技有限公司 基于大数据云平台的线上学习内容推荐系统
CN116416628A (zh) * 2023-06-06 2023-07-11 广州宏途数字科技有限公司 基于手写字体识别的方法及识别系统
CN117671395B (zh) * 2024-02-02 2024-04-26 南昌康德莱医疗科技有限公司 一种癌细胞种类识别装置
CN118247283B (zh) * 2024-05-29 2024-09-27 江苏濠汉信息技术有限公司 基于图像灰度共生矩阵的输电线路覆冰预警方法及系统

Family Cites Families (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9826208B2 (en) * 2013-06-26 2017-11-21 Nvidia Corporation Method and system for generating weights for use in white balancing an image
DE102015009981A1 (de) * 2015-07-31 2017-02-02 Eberhard Karls Universität Tübingen Verfahren und Vorrichtung zur Bildsynthese
EP3507773A1 (de) * 2016-09-02 2019-07-10 Artomatix Ltd. Systeme und verfahren zur bereitstellung von auf neuronalem konvolutionsnetzwerk basierender bildsynthese unter verwendung stabiler und steuerbarer parametrischer modelle, eines mehrskaligen syntheserahmens und neuartigen netzwerkarchitekturen
CN106782602B (zh) * 2016-12-01 2020-03-17 南京邮电大学 基于深度神经网络的语音情感识别方法
US10685057B1 (en) * 2016-12-30 2020-06-16 Shutterstock, Inc. Style modification of images in search results
CN106651766A (zh) * 2016-12-30 2017-05-10 深圳市唯特视科技有限公司 一种基于深度卷积神经网络的图像风格迁移方法
CN106847294B (zh) * 2017-01-17 2018-11-30 百度在线网络技术（北京）有限公司 基于人工智能的音频处理方法和装置
CN106709532B (zh) * 2017-01-25 2020-03-10 京东方科技集团股份有限公司 图像处理方法和装置
US10803378B2 (en) * 2017-03-15 2020-10-13 Samsung Electronics Co., Ltd System and method for designing efficient super resolution deep convolutional neural networks by cascade network training, cascade network trimming, and dilated convolutions
CN106952224B (zh) * 2017-03-30 2019-08-13 电子科技大学 一种基于卷积神经网络的图像风格转移方法
CN107038421A (zh) * 2017-04-17 2017-08-11 杭州电子科技大学 基于稀疏堆栈自编码的调制样式识别方法
US10565757B2 (en) * 2017-06-09 2020-02-18 Adobe Inc. Multimodal style-transfer network for applying style features from multi-resolution style exemplars to input images
US10318889B2 (en) * 2017-06-26 2019-06-11 Konica Minolta Laboratory U.S.A., Inc. Targeted data augmentation using neural style transfer
WO2019025909A1 (en) * 2017-08-01 2019-02-07 3M Innovative Properties Company NEURONAL STYLE TRANSFER FOR VARIETY AND IMAGE RECOGNITION
CN109426858B (zh) * 2017-08-29 2021-04-06 京东方科技集团股份有限公司 神经网络、训练方法、图像处理方法及图像处理装置
US10984286B2 (en) * 2018-02-02 2021-04-20 Nvidia Corporation Domain stylization using a neural network model
US10482565B1 (en) * 2018-02-12 2019-11-19 Snap Inc. Multistage neural network processing using a graphics processor
US10825132B2 (en) * 2018-02-20 2020-11-03 Element Ai Inc. Training method for convolutional neural networks for use in artistic style transfers for video
US10748324B2 (en) * 2018-11-08 2020-08-18 Adobe Inc. Generating stylized-stroke images from source images utilizing style-transfer-neural networks with non-photorealistic-rendering

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113240573A (zh) * 2020-10-26 2021-08-10 杭州火烧云科技有限公司 一种针对千万级像素数字图像的基于局部和全局并行学习的风格变换方法及系统
TWI876866B (zh) * 2024-02-05 2025-03-11 大陸商上海柏愈科技有限公司 人臉紋理識別模型建立方法

Also Published As

Publication number Publication date
US11461639B2 (en) 2022-10-04
EP3678059B1 (de) 2023-06-07
EP3678059A4 (de) 2021-05-26
CN109426858B (zh) 2021-04-06
CN109426858A (zh) 2019-03-05
WO2019042139A1 (zh) 2019-03-07
US20190220746A1 (en) 2019-07-18

Similar Documents

Publication Publication Date Title
EP3678059B1 (de) Bildverarbeitungsverfahren, bildverarbeitungsvorrichtung und verfahren zum trainieren eines neuronalen netzes
CN111767979B (zh) 神经网络的训练方法、图像处理方法、图像处理装置
CN110188776B (zh) 图像处理方法及装置、神经网络的训练方法、存储介质
JP7464752B2 (ja) 画像処理方法、装置、機器及びコンピュータプログラム
US11200638B2 (en) Image style transform methods and apparatuses, devices and storage media
CN109361934B (zh) 图像处理方法、装置、设备及存储介质
US20210358082A1 (en) Computer-implemented method using convolutional neural network, apparatus for generating composite image, and computer-program product
CN112534443B (zh) 图像处理设备及其操作方法
CN114586056A (zh) 图像处理方法及装置、设备、视频处理方法及存储介质
CN109191382A (zh) 图像处理方法、装置、电子设备及计算机可读存储介质
CN111724309B (zh) 图像处理方法及装置、神经网络的训练方法、存储介质
CN110956575B (zh) 转变图像风格的方法和装置、卷积神经网络处理器
CN112991171A (zh) 图像处理方法、装置、电子设备及存储介质
CN113205449A (zh) 表情迁移模型的训练方法及装置、表情迁移方法及装置
CN116912268B (zh) 一种皮肤病变图像分割方法、装置、设备及存储介质
US12175628B2 (en) Training method and apparatus for image processing model, and image processing method and apparatus
CN118608389A (zh) 实时动态超分辨率图像重建方法及其重建系统
CN109447897B (zh) 一种真实场景图像合成方法及系统
CN111696026A (zh) 基于l0正则项的可逆灰度图算法、计算设备
Fadaeddini et al. Data augmentation using fast converging CIELAB-GAN for efficient deep learning dataset generation
CN114627293A (zh) 基于多任务学习的人像抠图方法
EP4421750A1 (de) Avatarsteuerung
CN120783319B (zh) 障碍物检测方法、装置、设备、介质及产品
HK40025785B (en) Method, device and apparatus for processing image and computer readable storage medium
HK40025785A (en) Method, device and apparatus for processing image and computer readable storage medium

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20190329

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20210426

RIC1 Information provided on ipc code assigned before grant

Ipc: G06N 3/04 20060101AFI20210420BHEP

Ipc: G06T 11/00 20060101ALI20210420BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

INTG Intention to grant announced

Effective date: 20230309

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

Ref country code: AT

Ref legal event code: REF

Ref document number: 1577218

Country of ref document: AT

Kind code of ref document: T

Effective date: 20230615

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602018051531

Country of ref document: DE

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20230607

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230907

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1577218

Country of ref document: AT

Kind code of ref document: T

Effective date: 20230607

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230908

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231007

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231009

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231007

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602018051531

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230816

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230816

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230831

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20230831

26N No opposition filed

Effective date: 20240308

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20230907

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230816

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230907

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230907

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230831

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230816

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230831

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20180816

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20180816

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20250820

Year of fee payment: 8

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230607