EP4100914B1 - Bildverarbeitungsverfahren - Google Patents

Bildverarbeitungsverfahren Download PDF

Info

Publication number
EP4100914B1
EP4100914B1 EP21753231.6A EP21753231A EP4100914B1 EP 4100914 B1 EP4100914 B1 EP 4100914B1 EP 21753231 A EP21753231 A EP 21753231A EP 4100914 B1 EP4100914 B1 EP 4100914B1
Authority
EP
European Patent Office
Prior art keywords
image
filter
network
processed
rendering
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP21753231.6A
Other languages
English (en)
French (fr)
Other versions
EP4100914A4 (de
EP4100914A1 (de
EP4100914C0 (de
Inventor
Bin Sun
Yanlong Sun
Yong A
Wei Li
Lanlan Zhang
Zhezhu Jin
Yingying JIANG
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Samsung Electronics Co Ltd
Original Assignee
Samsung Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Samsung Electronics Co Ltd filed Critical Samsung Electronics Co Ltd
Publication of EP4100914A1 publication Critical patent/EP4100914A1/de
Publication of EP4100914A4 publication Critical patent/EP4100914A4/de
Application granted granted Critical
Publication of EP4100914B1 publication Critical patent/EP4100914B1/de
Publication of EP4100914C0 publication Critical patent/EP4100914C0/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/77—Retouching; Inpainting; Scratch removal
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
    • G06F18/24133—Distances to prototypes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration using local operators
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/761—Proximity, similarity or dissimilarity measures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/7715—Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/94—Hardware or software architectures specially adapted for image or video understanding
    • G06V10/95—Hardware or software architectures specially adapted for image or video understanding structured as a network, e.g. client-server architectures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20021—Dividing image into blocks, subimages or windows
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20092—Interactive image processing based on input by user
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face

Definitions

  • the present disclosure generally relates to a computer technology application field, an artificial intelligence field, and a digital image processing field, and more specifically, to an image processing method.
  • An image filter is a technology that performs processes, such as filtering, modifying, beautifying, and the like, on a color/hue/light of an image and the like through a computer technology and an image processing technology.
  • Image filter is an important image processing technology, which is widely used in fields of image editing, video editing, mobile camera photography and the like.
  • Related art image filter technology typically includes a preset filter method, a custom filter method of adjusting parameters setting, and a custom filter method of generating a filter by selecting a filter image.
  • a method of image processing includes: obtaining a first image; removing, using an image de-filter network, a filter effect applied to the first image to obtain a second image; obtaining, based on the first image and the second image, an image filter corresponding to the filter effect; and rendering a third image using the obtained image filter to output a fourth image.
  • the removing the filter effect may include: performing feature extraction on the first image, via cascaded feature extraction units of an encoder network of the image de-filter network, to generate an extracted feature map; and decoding the extracted feature map, via cascaded decoding units of a decoder network of the image de-filter network, to output the second image.
  • the performing the feature extraction on the first image may include: performing, using a portion of the cascaded feature extraction units other than a last feature extraction unit, at least one convolution process and at least one down-sampling process on an input feature map; and performing, using the last feature extraction unit, at least two convolution processes on the processed input feature map, residual-connecting of the input feature map and the convolution-processed input feature map, and outputting the extracted feature map to the decoder network, and the decoding the extracted feature map may include: performing at least one up-sampling process and at least one convolution process on the extracted feature map.
  • the removing the filter effect may further include: down-scaling, using a pre-processing layer, the first image according to a preset size; and outputting, to the encoder network, the down-scaled first image.
  • the removing the filter effect may further include: obtaining, from an image segmentation network, at least one type of segmentation information from among segmentation information for the first image; concatenating the first image with the at least one type of segmentation information to obtain at least one type of merged first image; and inputting, to the encoder network, the at least one type of merged first image.
  • the removing the filter effect may further include: concatenating, using at least one decoding unit of the decoder network, an up-sampled feature map with the convolution-processed input feature map corresponding to a same scale feature map; and performing at least one convolution process on the concatenated feature map.
  • the obtaining the image filter may include: obtaining a mapping relationship of transforming the second image to the first image; and obtaining the image filter according to the mapping relationship.
  • the obtaining the mapping relationship may include: acquiring a plurality of first sampling points in the first image and a plurality of second sampling points corresponding to respective first sampling points in the second image; and obtaining the mapping relationship of transforming the second image to the first image by determining mapping relationships between the respective first sampling points in the first image and the corresponding second sampling points in the second image.
  • the acquiring the plurality of first sampling points in the first image may include: detecting a salient target in the first image; acquiring a first portion of the plurality of first sampling points in a first area where the salient target is located in the first image; and acquiring a second portion of the plurality of first sampling points in a second area outside the salient target in the first image, and a first density of the first portion of the plurality of first sampling points may be greater than a second density of the second portion of the plurality of first sampling points.
  • the rendering the third image may include: determining a rendering degree of the third image, according to at least one of a scene similarity between the first image and the third image, and one or more human face areas in the third image; and rendering the third image using the image filter, according to the rendering degree of the third image.
  • the determining the rendering degree of the third image may include: determining rendering degrees of the one or more human face areas according to the one or more detected human face areas; detecting the scene similarity between the first image and the third image; and determining a rendering degree of another area, other than the one or more human face areas in the third image, according to the detected scene similarity.
  • the detecting the scene similarity between the first image and the third image may include: performing, using a scene classification network, scene classification on the first image and the third image; and determining the scene similarity of the first image and the third image, based on a scene classification result and features extracted by a set feature extraction layer in the scene classification network.
  • the determining the rendering degrees of the one or more human face areas may include: dividing each detected human face area of the one or more human face areas into a corresponding set of blocks; obtaining a human face color protection level of each block in the corresponding sets of blocks, according to a Gaussian distribution; and obtaining a rendering degree of each block in the corresponding sets of blocks, according to the human face color protection level of that block.
  • the rendering of the third image may include: fusing the third image and the fourth image based on the rendering degree of the third image to obtain a fifth image.
  • the obtaining of the image filter may include: segmenting the first image to obtain corresponding first segmented areas; acquiring an image filter corresponding to at least one first segmented area of the corresponding first segmented areas; and acquiring an image filter corresponding to the first image based on the image filter corresponding to the at least one first segmented area and the second image.
  • image filters may be pre-set in software or apparatus. That is, various styles of image filter parameters may be pre-stored in the software or apparatus, and when the user calls an image filter function, the software or apparatus directly calls the stored image filter parameters for calculation, to output the image filtered image. For example, the user can select a color filter function, a nostalgic filter function, a sketch filter function and the like.
  • a preset filter method has a good filter effect, a number of image filters is limited, and which is difficult to meet the user's growing demand for personalized image filters.
  • related art image filters may also provide for user-defined image filters.
  • some image editing functions support user-defined (e.g., personalized) image filters. That is, the user can adjust parameters of the image filters to generate custom image filters. This mode may require users to have some basic knowledge of image processing and experience in image filters to generate a good effect of custom image filter. However, it may be difficult for most users with zero foundation to bring up a better effect of the image filter. Thus, the user experience may not be satisfactory.
  • the related art methods for generating custom image filter based on modifying image filter parameters may require users to have certain image processing technology foundation and image filter technical knowledge.
  • the present disclosure proposes an image processing method for obtaining an image filter based on an image de-filter network.
  • the proposed image processing method may also adjust a rendering degree of the image filter in consideration of a scene similarity and human face areas, such that the effect of the image filter can be improved when compared to related art image filter technologies.
  • FIG. 1 is a flowchart illustrating a method 100 of image processing according to an embodiment of the present disclosure.
  • a first image is selected by a user, wherein the first image may be any image input by the user.
  • the first image may be acquired based on the selection from the user.
  • the first image refers to an image of which the user wishes to obtain a filter effect, and therefore, hereinafter, the first image may be referred to as a filter template image.
  • the filter template image may be any image, selected by the user, of which a style or an effect is desired to be obtained.
  • the filter template image is not necessarily an image with filter effect, but also an image without filter effect. As long as the user wishes to obtain a style or an effect of an image, the user may select that image as the filter template image.
  • the filter template image may be acquired from a local storage or a local database, or received from an external data source (e.g., the Internet, a server, a database, etc.) through an input device or a transmission medium, as needed.
  • an external data source e.g., the Internet, a server, a database, etc.
  • a second image in which a filter effect applied to the first image is removed is obtained, using an image de-filter network.
  • the second image may also be referred to as a de-filter template image.
  • a step of generating the second image with the de-filter effect corresponding to the first image using the image de-filter network is specifically described first.
  • the image de-filter network may be a deep convolutional neural network.
  • Convolutional Neural Networks are a type of Feedforward Neural Networks (FNNs) that includes convolution calculations and has a deep structure, which is one of representative algorithms of deep learning.
  • the CNNs may be constructed by imitating a biological visual perception mechanism, and can perform supervised learning and unsupervised learning.
  • convolution kernel parameters sharing in hidden layers and sparsity of connection between layers enable the CNNs learn grid-like topology features, such as pixels and audio, in a small amount of computation, which has a stable effect, and has no additional feature engineering requirements for data.
  • the CNNs generally include an input layer, a hidden layer, and an output layer, wherein the hidden layer of the CNNs include constructions of a convolutional layer, a normalization layer, an up-sampler layer and the like.
  • the Deep Convolutional Neural Networks refers to deeper convolutional neural networks with more than 3 layers.
  • the DCNNs proposed by various embodiments of the present disclosure may have more than 3 layers, and include an input layer, convolutional layers, normalization layers, a convolutional layer and a normalization layer with a skip connection structure, up-sampler layers, and an output layer. A function of each layer of the DCNNs is described in detail below.
  • the input layer of the DCNNs can process multi-dimensional data.
  • the input layer of one-dimensional (1D) convolutional neural network receives 1D or two-dimensional (2D) array, wherein the 1D array is usually time or spectrum sampling, the 2D array may contain multiple channels, the input layer of 2D convolutional neural network receives 2D or 3D array, the input layer of 3D convolutional neural network receives 4D arrays.
  • CNNs are widely used in a computer vision field, many studies presuppose 3D input data, that is, 2D pixel points on a plane and RGB channels, when introducing their structures.
  • the input layer of the DCNNs proposed by various embodiments of the present disclosure may be used to input a filter template image.
  • Convolutional layer the function of DCNNs is to perform feature extraction on the input data, which contains multiple convolution kernels therein.
  • Each element constituting the convolution kernels corresponds to a weight coefficient and a bias vector, which is similar to a neuron of one FNN.
  • Each neuron within the convolutional layer is connected to multiple neurons in an area which is close to the neuron in a previous layer, and a size of the area depends on a size of the convolution kernel, which can be referred to a "receptive field" and the meaning thereof can be compared to the receptive field of visual cortical cells.
  • the convolution kernel when working, will scan through the input features regularly, and within the receptive field, perform matrix elements multiplication and summation for the input features and superimpose deviation.
  • Convolutional layer parameters include a size of convolution kernel, a step size, and padding, which jointly determine a size of feature map output by the convolutional layer and are hyper-parameters of the convolutional neural network, wherein the size of the convolution kernel can be specified as any value smaller than a size of an input image, and the larger the convolution kernel is, the more complex the input features can be extracted.
  • the convolution step size defines a distance of the positions when the convolution kernel scans through the feature map twice adjacent to each other. When the convolution step is 1, the convolution kernel will scan the elements of the feature map one by one, and when the convolution step is n, the convolution kernel will skip n-1 pixels at the next scan.
  • Padding is a method of artificially increasing the size of the feature map before it passes through the convolution kernel to offset an effect of size shrinkage during calculation. Common padding methods are padding by 0 and padding by replicating boundary value (replication padding).
  • the convolutional layer contains excitation functions to assist in expressing complex features. Similar to other deep learning algorithms, convolution neural network often use a linear rectifier function (e.g., Rectified Linear Unit (ReLU)).
  • ReLU Rectified Linear Unit
  • Normalization layer in a deep network, as the input data is transferred step by step in the hidden layer, its mean value and standard deviation will change, resulting in covariate shift phenomenon. Covariant shift is considered as one of reasons of vanishing gradient in the deep networks.
  • the normalization layer partially solves such a problem at the cost of introducing additional learning parameters, in which the strategy is to standardize features in the hidden layer first, and then amplify the normalized features as new inputs, using two linear parameters.
  • the neural network will update normalization layer parameters of the normalization layer during the learning process.
  • the normalization layer parameters in the convolutional neural network have the same properties as the convolution kernel parameters, that is, pixels of a same channel in the feature map share a set of normalization layer parameters.
  • General normalization layer may use Batch Normalization.
  • Batch Normalization normalizes features on a training batch, and ensures consistent data distribution.
  • the present disclosure may use Instance Normalization in order to improve the effect of the proposed deep convolution neural network.
  • Instance Normalization normalizes features on image pixels, the effect of which may be obvious in the stylization migration.
  • Up-sampler layer up-sampler is also called as image enlarging or image interpolation.
  • An up-sampler interpolation method used in a deep neural network proposed in various embodiments of the present disclosure is the nearest neighbor method, which is a simplest kind of interpolation method.
  • the nearest neighbor method does not need to calculate an average value or generate an intermediate value based on complicated rules. Instead, for example, given four neighbor pixels of a pixel to be processed, a gray scale of a neighbor pixel closest to the pixel to be processed is assigned to the pixel to be processed.
  • Output layer the output layer is a last layer of the neural network, and generally has different output forms according to different tasks.
  • the output layer of a deep neural network proposed in various embodiments of the present disclosure is used to output a feature map with a same size as the input image.
  • Skip connection structure in order to improve the performance of the deep convolutional neural network, skip connection is generally added to the network.
  • Skip connection or shortcut connection is derived from skip connection and various gating algorithms in recurrent neural network (RNN), which is technique used to alleviate a problem of gradient disappearance in deep structures.
  • RNN recurrent neural network
  • the skip connection in the convolutional neural network can span any number of hidden layers, which is illustrated by skip between adjacent hidden layers as shown in Equation 1.
  • a l f Z l + u Z l ⁇ 1
  • Equation 1 Z l represents the l-th feature (a convoluted feature), Z l -1 represents (l-1)-th feature (a convoluted feature), f is the activation function and A is an activated feature, u is a conversion coefficient of a feature map.
  • the conversion coefficient converts the feature map with smaller size, generally converts Z l-1 to the size of Z l , to make sure that the operations of the matrix elements work.
  • the output of the convolutional layer l approximates an equivalent function and has no negative effect on a feature transfer of the layer.
  • a learning baseline of the l layer is set, so that the layer does not degenerate at least in iteration.
  • some errors can skip layer l to directly act on layer l-1 when back propagation compensates for a gradient loss caused by stepwise propagation in the deep structure. Therefore, it may be beneficial to an error propagation of the deep structure.
  • a combination of multiple convolutional layers including the skip connection may be referred to as a residual block.
  • the image de-filter network may be a U-NET type of deep convolutional neural network. It may be understood that the image de-filter network is not limited thereto, and may also comprise other artificial neural networks.
  • the U-NET is a fully convolutional network that includes multi-layer down-sampling and multi-layer up-sampling, and its characteristic is that the convolutional layers in the down-sampling part and the up-sampling part are completely symmetric.
  • the feature map at the down-sampling end can skip deep layer sampling and be concatenated to the corresponding up-sampling end.
  • the sizes of the input layer and the output layer of the network may be the same.
  • the U-NET type of CNNs are Fully Convolutional Networks (FCNs) organized under the encoder-decoder paradigm.
  • the U-NET type of deep convolutional neural network proposed by various embodiments of the present disclosure may also be referred to as an Encoder-Decoder Network.
  • the image de-filter network proposed by various embodiments of the present disclosure may be a U-NET type of deep convolutional neural network, and may be designed to include an input layer, convolutional layers and normalization layers, a down-sampling layers, a convolutional layer and a normalization layer with the skip connection structure, up-sampling layers and an output layer.
  • FIG. 2 illustrates a schematic diagram of a structure of an image de-filter network according to an embodiment of the present disclosure.
  • the present disclosure proposes an image de-filter network that improves the U-NET type of network.
  • the U-NET type of network proposed by the present disclosure can reduce an image size, a number of pooling operations, and a number of channels, increase skip connection structures (e.g., residual connection blocks), and remove a copy and crop structure.
  • the image de-filter network 200 may include an encoder network 210 and a decoder network 220.
  • the encoder network 210 performs feature extraction on the first image via several cascaded feature extraction units (e.g., 211a, 211b, and 211c, hereinafter "211" generally).
  • a last feature extraction unit 211 e.g., 211c
  • a last feature extraction unit 211 in the encoder network 210 performs convolution processing on the input feature map at least twice, and outputs after residual-connecting the input feature map and the convolved feature map for outputting
  • other feature extraction units 211 e.g., 211a, 211b
  • the encoder network 210 may include three feature extraction units 211a, 211b, and 211c.
  • the first feature extraction unit 211a may include an input layer for receiving the input first image, and includes two convolution layers for performing convolution process on the input first image twice, wherein a size of a convolution kernel for the first convolution process is 7*7, a size of a convolution kernel for the second convolution process is 3 *3, and a convolution step length of each convolution process is 1.
  • the first feature extraction unit 211a may further include two normalization layers for performing normalization process after each convolution process, and instance normalization process may be used.
  • the first feature extraction unit 211a may include a down-sampling layer for performing down-sampling (pooling) process on the feature map after convolution process or normalization process (e.g., maximum 2*2 pooling can be performed), and outputting a down-sampled feature map, wherein 211a contains three arrows, the first arrow represents that 7*7 convolution process is performed on the input 64*64 image to output the feature map, and the second arrow represents that 3*3 convolution process is performed on the above-described output feature map, and the third arrow represents that down-sampling process (also called as pooling process) is performed on the 3*3 convolution processed feature map and a 128*128 feature map is output.
  • a down-sampling layer for performing down-sampling (pooling) process on the feature map after convolution process or normalization process (e.g., maximum 2*2 pooling can be performed), and outputting a down-sampled feature map, wherein 211a contains three arrows,
  • the second feature extraction unit 211b may include two convolution layers for performing convolution process twice on the feature map output by the first feature extraction unit 211a, wherein a size of a convolution kernel for each convolution processing is 3*3, a convolution step length of each convolution process is 1.
  • the second feature extraction unit 211b may also include two normalization layers for performing normalization process after each convolution process, and instance normalization process may be used.
  • the second feature extraction unit 211b may include a down-sampling layer for performing down-sampling (pooling) process on the convolution processed or normalization processed feature map (e.g., maximum 2*2 pooling can be performed), and output a down-sampled feature map, wherein 211b contains three arrows, the first and second arrows represent that 3*3 convolution process is performed twice in sequence on the 128*128 feature map output by the first feature extraction unit 211a, and the third arrow represents that the down-sampling process (also called as pooling process) is performed on the 3*3 convolution processed feature map, and a 64*64 feature map is output.
  • a down-sampling layer for performing down-sampling (pooling) process on the convolution processed or normalization processed feature map (e.g., maximum 2*2 pooling can be performed), and output a down-sampled feature map, wherein 211b contains three arrows, the first and second arrows represent that 3*3 convolution process is
  • the third feature extraction unit 211c may include a convolutional layer with a skip connection structure. That is, in addition to performing convolution process three times on the feature map output by the second feature extraction unit 211b (wherein, a size of a convolution kernel of each convolution processing is 3*3, and a convolution step length of each convolution process is 1), the feature map output by the second feature extraction unit 211b (that is, the 64*64 feature map shown in the FIG. 2 ) is also output to the feature map after convolution processed three times in order to perform residual connection and then the residual connected feature map is output.
  • the feature map output by the second feature extraction unit 211b and the feature map after convolution processed three times are merged and output, then the feature map output by the third feature extraction unit 211c not only includes the feature map convolution processed by the third feature extraction unit 211c, but also includes the feature map of more upper layer before the convolution process.
  • the third feature extraction unit 211c may also include three normalization layers for further performing normalization processing after each convolution processing, and instance normalization process may be used.
  • the decoder network 220 decodes the extracted feature map via several cascaded decoding units 221 (e.g., 221a, 221b, hereinafter "221" generally) to obtain the second image.
  • the decoding unit 221 in the decoder network performs at least one up-sampling process and at least one convolution process on the input feature map.
  • the decoder network 220 may include two decoding units 221a and 221b.
  • the first decoding unit 221a may include an up-sampling layer for receiving the feature map output from the encoder network 210, and performing up-sampling on the feature map once(e.g., an up-sampling coefficient may be 2).
  • the first decoding unit 221a may include two convolution layers for performing convolution process twice on the up-sampled feature map, wherein a size of a convolution kernel for each convolution process is 3*3, and a convolution step length for each convolution process is 1. As shown in FIG.
  • the first decoding unit 221a contains three arrows, the first arrow represents the up-sampling process that is performed on the feature map output by the encoder network 210, the second and third arrows represent that 3*3 convolution process is performed twice in sequence on the feature map obtained by up-sampling, and a 256*256 feature map is output.
  • the first decoding unit 221a may further include two normalization layers for performing normalization processing after each convolution processing, and instance normalization process may be used.
  • the first decoding unit 221a may output the convolution processed or normalization processed feature map.
  • the second decoding unit 221b may include an up-sampling layer for performing up-sampling once on the feature map output by the first decoding unit 221a (e.g., an up-sampling coefficient may be 2).
  • the second decoding unit 221b may include three convolutional layers for performing convolution process three times on up-sampled feature map, wherein a size of a convolution kernel for each convolution processing is 3*3, and a convolution step length of each convolution processing is 1. As shown in FIG.
  • the second decoding unit 221b contains four arrows, the first arrow represents the up-sampling process that is performed on the 256*256 feature map output by the first decoding unit 221a, the second, third and fourth arrows represent 3*3 convolution process is performed three times in sequence on the feature map obtained by up-sampling, and a 256*256 image is output.
  • the first decoding unit 221a may also include three normalization layers for performing normalization processing after each convolution processing, and instance normalization process may be used.
  • the second decoding unit 221b may include an output layer for outputting the convolution processed or normalization processed feature map as the second image.
  • the image de-filter network structure shown in FIG. 2 is only an embodiment of an improved U-NET type of the network proposed by various embodiments of the present disclosure.
  • the improved U-NET type of the network proposed by the various embodiments of the present disclosure is not limited thereto, and may also include an adjustable number of convolution process, down-sampling process and up-sampling process, etc.
  • the image de-filter network 200 proposed by various embodiments of the present disclosure may also include a pre-processing layer (not shown).
  • the pre-processing layer may down-scale the first image according to a preset size (such as 256*256), and output the down-scaled image to the encoder network 210.
  • the image de-filter network 200 proposed by the present disclosure can achieve the effect of reducing the size of the image to increase a computing speed.
  • the image de-filter network 200 proposed by various embodiments of the present disclosure can use a smaller storage space on a mobile device (e.g., the de-filter network model can be 6.5 MB) and run faster (e.g., only need 60ms to generate a de-filter image), and can use any image as the first image to generate the second image with de-filter effect.
  • the image de-filter network proposed in the present disclosure the copy and crop structure is deleted. After the structure is deleted, the image de-filter network can still maintain good performance and increase the processing speed.
  • the number of channels in each convolutional layer can be correspondingly reduced, or the number of pooling layer operations can be reduced (e.g., reduced from 4 to 2) to improve processing speed of the network while ensuring network performance.
  • FIG. 3 illustrates a schematic diagram of a structure of another image de-filter network according to an embodiment of the present disclosure.
  • a person in the image is suitable for a person filter
  • a sky in the image is suitable for a sky filter. Therefore, another improved image de-filter network proposed by the present disclosure can generate different suitable de-filter effects for different segmentation categories (e.g., character parts, sky parts). For this, a channel of segmentation information (e.g., the image segmentation result map) (twill block) is added to the input layer of the image filter network in FIG.
  • the image segmentation may be semantic segmentation, that is, the image is segmented according to the categories of the image content (e.g., the person part, the sky part). It may be understood that the image segmentation may also be other possible segmentation, and the present disclosure does not limit the manner of image segmentation.
  • Copy and crop operations or copy operation may also added to the image de-filter network in FIG. 3 , to allow global information of upper layers in the encoder network to be transferred to the decoder network, and to make the semantic information constraint the de-filter process better.
  • the image de-filter network 300 may include an encoder network 310, a decoder network 320, and an image segmentation network 330, wherein the functions of the encoder network 310 and the decoder network 320 are similar to the functions of the encoder network 210 and the decoder network 220 described with reference to FIG. 2 .
  • the decoder unit 321 in decoder networks 320 may differ from decoder unit 211 in that the decoder unit 321 may combine the up-sampled feature maps with the convolution processed feature maps by the feature extraction units 310 of corresponding same scale feature maps in encoder network 310, and perform convolution process at least once on the combined feature map.
  • the image segmentation network 330 may obtain at least one type of segmentation information in the segmentation information for the first image, merge the first image with the at least one segmentation information, obtain at least one type of merged first image, and input the at least one type of merged first image to the encoder network.
  • the image segmentation network 330 may obtain at least one type of segmentation information in the segmentation information for the first image, connect (or merge) the first image with the at least one type of segmentation information, and input at least one of the connected (or merged) first images to the encoder network.
  • the image segmentation network 330 may be a semantic segmentation network.
  • the segmentation information refers to information related to a segmentation result obtained by segmenting the image
  • the information related to the segmentation result may be the predicted category value of each pixel, for example, information about which category each pixel in the image belongs to, and may also be information about a number of categories, information about areas belonging to each category, and the like.
  • merging the first image with the at least one type of segmentation information may be adding respective channels in the first image to the channel in the segmentation information, for example, the size of the first image is 256*256*3 (3 is the number of channels), the size of segmentation information is 256*256*1 (1 is the number of channels), and after merging, 256*256*4 (4 is the number of channels) input image may be generated.
  • the image segmentation network 330 may segment the image according to different rules, or the segmentation information can have different representation methods, thus variety types of segmentation information may be obtained for a same image. For example, when there are two people in the image, one type of segmentation information can indicate that an area where the two people are located belongs to the category "person”, and the other type of segmentation information may indicate that an area where the first person is located belongs to the category "man”, and an area where the second person is located belongs to the category " woman”. That is, the segmentation category here may be defined differently according to user requirement.
  • the image segmentation network 330 may concatenate the first image with the at least one type of segmentation information, and input at least one of the concatenated first images to the encoder network.
  • the input of the encoder network 310 may be a first image (3+1) in which the first image is concatenated with the segmentation information.
  • the operations of the encoder network 310 and the decoder network 320 are described below with reference to FIG. 3 .
  • the encoder network 310 may include three feature extraction units 311a, 311b, and 311c.
  • the first feature extraction unit 311a may include an input layer for receiving a first image in which the input first image and the segmentation information is merged.
  • the first feature extraction unit 311a may include two convolutional layers for performing convolution process twice on the input merged first image, wherein a size of a convolution kernel of a first convolution process is 7*7, and a size of a convolution kernel of a second convolution process is 3*3, and a convolution step length of each convolution processing is 1.
  • the first feature extraction unit 311a may also include two normalization layers for performing normalization process after each convolution process, and instance normalization process may be used.
  • the first feature extraction unit 311a may include a down-sampling layer for performing down-sampling (pooling) process on the convolution processed or normalization processed feature map (e.g., maximum 2*2 pooling can be performed), and outputting the down-sampled feature map.
  • the second feature extraction unit 311b may include one convolution layer for performing convolution process once on the feature map output by the first feature extraction unit 311a, wherein a size of a convolution kernel is 3*3, and a convolution step size is 1. Alternatively or additionally, the second feature extraction unit 311b may further include a normalization layer for performing normalization process after convolution process, and instance normalization process may be used. The second feature extraction unit 311b may include a down-sampling layer for performing down-sampling (pooling) process on the convolution processed or normalization processed feature map (e.g., maximum 2*2 pooling can be performed), and outputting the down-sampled feature map.
  • a down-sampling layer for performing down-sampling (pooling) process on the convolution processed or normalization processed feature map (e.g., maximum 2*2 pooling can be performed), and outputting the down-sampled feature map.
  • the third feature extraction unit 311c includes a convolutional layer having a skip connection structure, that is, in addition to performing convolution process three times on the feature map output by the second feature extraction unit 311b (wherein, a size of a convolution kernel for each convolution process is 3*3, and a convolution step length of each convolution process is 1), the feature map output by the second feature extraction unit 211b is output to the three times of convolution processed feature map, in order to perform residual connection, and the residual connection processed feature map is output.
  • the third feature extraction unit 311c may also include three normalization layers for performing normalization processing after each convolution process, and instance normalization process may be used.
  • the decoder network 320 may include two decoding units 321a and 321b.
  • the first decoding unit 321a may include an up-sampling layer for receiving the feature map output from the encoder network 310, and performing up-sampling once on the feature map (e.g., an up-sampling coefficient may be 2), and includes a convolution layer for merging the up-sampled feature map with the feature map convolution processed or normalization processed by the second feature extraction unit 311b (i.e., copy and cut process, or copy process), and performing convolution process once on the merged feature map, wherein a size of a convolution kernel is 3*3, and a convolution step size is 1.
  • the first decoding unit 321a may also include a normalization layer for performing normalization process after the convolution process, and instance normalization process may be used.
  • the first decoding unit 321a may output the convolution processed or normalization processed feature map.
  • the second decoding unit 321b may include an up-sampling layer for performing up-sampling once on the feature map output by the first decoding unit 221a (e.g., an up-sampling coefficient may be 2), and include convolution layers for merging the up-sampled feature map with the feature map that is convolution processed or normalization processed by the first feature extraction unit 311a (i.e., copy and cut process, or copy process), and performing convolution process twice on the merged feature map, wherein a size of a convolution kernel for each convolution process is 3*3, and a convolution step size for each convolution process is 1.
  • the first decoding unit 321a may further include normalization layers for performing normalization process after each convolution process, and instance normalization processing may be used.
  • the second decoding unit 321b may include an output layer for outputting the convolution processed or normalization processed feature map as the second image.
  • the image de-filter network proposed by the present disclosure may be trained, using a training image set including a plurality of images with filter effects (filter images) and a plurality of corresponding images in which filter effects are removed (that is de-filter images) (i.e., multiple image pairs, each image pair includes a filter image and a de-filter image corresponding to the filter image), and the de-filter template image corresponding to the filter template image may be generated using the trained image de-filter network.
  • the training image set may be appropriately modified.
  • the de-filter images in the training image set are obtained using different appropriate de-filter process on the corresponding filter images according to different semantic segmentation categories.
  • an image filter corresponding to the filter effect is obtained based on the first image and the second image.
  • a mapping relationship of transforming the second image (de-filter template image) to the first image (filter template image) may be calculated, and the calculated mapping relationship may be used as the image filter.
  • the mapping relationship of transforming the de-filter template image to the filter template image may be calculated based on image attributes of the filter template image and the de-filter template image.
  • the mapping relationship of transforming the de-filter template image to the filter template image may be calculated based on color attributes of the filter template image and the de-filter template image.
  • FIG. 4 is a schematic diagram illustrating acquiring the mapping relationship of transforming the de-filter template image to the filter template image according to an embodiment of the present disclosure.
  • the filter template image and the de-filter template image is equally sampled to obtain multiple sampling point pairs. Specifically, multiple sampling points in the filter template image and multiple sampling points in the de-filter template image corresponding to the multiple sampling points in the filter template image may be acquired.
  • the filter template image and the de-filter template image may be uniform random sampled to obtain multiple first sampling points in the filter template image and multiple second sampling points in the de-filter template image corresponding to respective first sampling point in the filter template image respectively.
  • uniform sampling points of the filter template image and the de-filter template image may be obtained by performing uniform random sampling on the filter template image and the de-filter template image using a uniform random sampling method.
  • the uniform sampling points of the filter template image and the uniform sampling points of the de-filter template image correspond one-to-one in position.
  • N uniform sampling points of the filter template image and N uniform sampling points of the de-filter template image may be obtained, wherein N may be equal to or greater than 4000. It may be understood that the number of N is not limited to these values.
  • a density of the first sampling points in the area where the salient target is located may be greater than the density of the first sampling points in the area outside the salient target. Therefore, larger sampling weight may be applied to salient target areas in the image, and smaller sampling weight can be applied to other areas. For example, it is assumed that N pairs of sampling points are collected in the filter template image and the de-filter template image.
  • Salient target in the filter template image is extracted, for example, salient target may be detected and extracted using a salience detection network.
  • Salient detection network refers to neural network model which is trained to detect salient object and give more sampling rate to the salient object.
  • N*T points (T>0.5) are sampled in the area where the salient target of the filter template image is located and the corresponding position of the de-filter template image, wherein T may be an extraction ratio (i.e., the sampling weight), and N*(1-T) points are sampled in the area outside of the salient target of the filter template image and the corresponding position of the de-filter template image. Therefore, the sampling weight for the area where the salient object is located in the image is greater than the sampling weight for the area outside of the salient target. That is, the more sampling points are extracted in the area where the salient target is located in the image, wherein N may be equal to or greater than 4000. It may be understood that the number of N is not limited to this.
  • T may be set according to factors of user requirement, image features and the like. For example, a ratio t of the area where the salient target is located in the image may be calculated, and T may be set to 0.5+0.5t. Alternatively or additionally, the uniform random sampling method may be applied to the sampling of both the area where the salient target is located and the area outside of the salient target in the image.
  • FIG. 5 is a schematic diagram illustrating acquiring the mapping relationship of transforming the de-filter template image to the filter template image, in consideration of the salient targets in the filter template image and the de-filter template image according to an embodiment of the present disclosure.
  • salient targets for example, starfish (white area) in the image, in the filter template image and the de-filter template image are detected.
  • the filter template image and the de-filter template image are sampled in such a way that the sampling density of the salient target area (starfish area) is higher than that of the other areas (sea water area and beach area).
  • a mapping equation set is built based on the sample point pairs of the filter template image and the de-filter template image.
  • the built mapping equation set is solved to obtain the mapping relationship of transforming the de-filter template image to the filter template image. This is described in detail below.
  • different parts in the image may be considered to be sampled differently, and a mapping relationship may be generated for each part and/or an entire mapping relationship may be generated for all parts, thereby generating multiple image filters, so that different appropriate image filters are used for different parts of the image to enhance the performance of the image filters.
  • N pairs of sampling points are collected in the filter template image and the de-filter template image.
  • the filter template image is segmented into multiple different parts, for example, the filter template image is segmented into multiple different parts through semantic segmentation.
  • FIG. 6 is a schematic diagram illustrating acquiring a mapping relationship of transforming the de-filter template image to the filter template image in consideration of different categories of areas in the filter template image and the de-filter template image according to an embodiment of the present disclosure.
  • step 601 image segmentation is performed on the filter template image to obtain different categories of areas.
  • different categories of areas of the filter template image and the de-filter template image are differently sampled. For example, sampling may be performed according to ratios of the different categories of areas to the total image area, respectively.
  • mapping equation sets are built based on the sample point pairs of different categories of areas of the filter template image and the de-filter template image, respectively, and an entire mapping equation set is built based on the sample points pairs of all areas of the filter template image and the de-filter template image.
  • the respective built mapping equation sets are solved to obtain mapping relationships of the de-filter template image to the filter template image, of respective different categories of areas, and the built total mapping equation set is solved to obtain the entire mapping relationship of transforming the de-filter template image to the filter template image. This is described in detail below.
  • the mapping relationship of transforming the de-filter template image to the filter template image may be calculated by calculating the color mapping relationship between all the multiple sampling points in the filter template image and all the corresponding multiple sampling points in the de-filter template image, and the image filter is obtained according to the mapping relationship.
  • the first image (e.g., filter template image) may be segmented to obtain respective first segmented areas. For at least one first segmented area, an image filter corresponding to the at least one first segmented area is obtained. An image filter corresponding to the first image is obtained based on the image filter corresponding to the at least one first segmented area and the second image (e.g., de-filter template image).
  • the second image e.g., de-filter template image
  • a plurality of first sampling points in the at least one first segmented area and a plurality of second sampling points corresponding to the plurality of first sampling points respectively in the second image are acquired, and the image filter corresponding to the at least one first segmented area is acquired by determining the mapping relationship between the respective first sampling points in the at least one first segmented area and the corresponding second sampling points in the second image.
  • the mapping relationship of transforming the de-filter template image to the filter template image of each segmented part may be calculated by the color mapping relationship between the multiple sampling points of the filter template image and the corresponding multiple sampling points in the de-filter template image of each segmented part, thereby obtaining multiple mapping relationships, and/or an entire mapping relationship of transforming the de-filter template image to the filter template image of all segmented parts may be calculated by the color mapping relationship between all the sampling points of the filter template image and all the sampling points in the de-filter template image of all the segmented parts.
  • the mapping relationships between respective color channels of the corresponding second sampling point of the second image (e.g., de-filter template image) and the any color channel of the any first sampling point may be determined respectively, to obtain the mapping relationships between the any first sampling point and the corresponding second sampling point.
  • the color channel may be an RGB (red, green, blue) channel, or other possible color channels, for example, YCbCr, YUV channels, etc.
  • RGB red, green, blue
  • a mapping equation set may be built based on multiple sampling point pairs of the filter template image and the de-filter template image.
  • the first equation set may include mapping equation of the pixel value of the red channel, the green channel, and the blue channel of the sampling point of the de-filter template image to the pixel value of the red channel of the corresponding sampling point of the filter template image, built for each sampling point, that is, the first equation set may include N (or N*r) such mapping equations.
  • the second equation set may include mapping equation of the pixel value of the red channel, the green channel, and the blue channel of the sampling point of the de-filter template image to the pixel value of the green channel of the corresponding sampling point of the filter template image, built for each sampling point, that is, the second equation set may include N (or N*r) such mapping equations.
  • the third equation set may include mapping equation of the pixel value of the red channel, the green channel, and the blue channel of the sampling point of the de-filter template image to the pixel value of the blue channel of the corresponding sampling point of the filter template image, built for each sampling point, that is, the third equation set can include N (or N*r) such mapping equations.
  • multiple linear function may be used to build the model.
  • the present disclosure preferably uses a quartic linear function to establish the mapping equations in the first equation set, the second equation set and the third equation set. It may be understood that the mapping equation is not limited to the quartic linear function, and other preferred solutions may also be used to establish the mapping equations.
  • mapping equations in the first equation set may be expressed as Equation 2.
  • Ri kR1*ri ⁇ 4+kR2*ri ⁇ 3+kR3*ri ⁇ 2+kR4*ri+kR5*(gi+bi)/2+ kR6*(gi ⁇ 2+bi ⁇ 2)/2+kR7*(gi ⁇ 3+bi ⁇ 3)/2+kR8*(gi ⁇ 4+bi ⁇ 4)/2+kR9
  • the mapping equations in the second equation set may be expressed as Equation 3.
  • ri, gi, and bi respectively represent the pixel value of the red channel, the pixel value of the green channel, and the pixel value of the blue channel of the i-th sampling point of the de-filter template image.
  • kR1, kR2, kR3, kR4, kR5, kR6, kR7, kR8, and kR9 are image filter parameters to be solved for the first equation set.
  • kG1, kG2, kG3, kG4, kG5, kG6, kG7, kG8, and kG9 are image filter parameters to be solved for the second equation set.
  • kB1, kB2, kB3, kB4, kB5, kB6, kB7, kB8, and kB9 are image filter parameters to be solved for the third equation set.
  • the built mapping equations are solved to obtain the mapping relationship, wherein an estimated fitting curve is shown in FIG. 4 .
  • the first equation set, the second equation set, and the third equation set may be solved to obtain the mapping relationship of transforming the de-filter template image to the filter template image.
  • the mapping relationship of transforming the de-filter template image is converted to the filter template image may be obtained.
  • a number of unknowns of each set is 9, and a number of mapping equations is the number of sampling points N (or N*r), for example, since N is equal to or greater than 4000 (N is not limited to this) and N (or N*r) is much greater than 9, the first equation set, the second equation set, and the third equation set are all overdetermined equation sets.
  • a numerical optimization algorithm (such as L-BFGS algorithm) may be used to solve the above equation sets (i.e., the first equation set, the second equation set, and the third equation set).
  • the L-BFGS algorithm refers to the limited-memory BFGS algorithm (Limited-memory BFGS), the basic idea thereof is to save only the most recent m times of iteration information, thereby greatly reducing the storage space of data.
  • the two-loop-recursion algorithm used in the steps of L-BFGS algorithm is as follows:
  • the above-mentioned equation sets (i.e., the first equation set, the second equation set, and the third equation set) may be solved using least square method.
  • the least square method finds best function match of data by minimizing sum of squares of error.
  • the least square method may be used to easily obtain unknown data, and minimize the sum of squares of error between the obtained data and the actual data.
  • m means there are m equations
  • n means there are n unknowns p , m>n.
  • the unique solution is the solution of the unknown in the formula.
  • the image filter proposed by the present disclosure contains rich and diverse color information, the filter effect is softer, more natural, contains more color features, and the effect is more beautiful.
  • an image to be processed (e.g., a third image) may be rendered using the obtained image filter to output the fourth image.
  • a direct calculation method may be used to obtain pixel values of the image to be processed directly rendered by the image filter.
  • 27 image filter parameters may be obtained, that is, kR1, kR2, kR3, kR4, kR5, kR6, kR7, kR8, kR9, kG1, kG2, kG3, kG4, kG5, kG6, kG7, kG8, kG9, kB1, kB2, kB3, kB4, kB5, kB6, kB7, kB8, kB9.
  • These 27 image filter parameters may be stored in a memory or a server and the like.
  • these 27 image filter parameters are read, and the pixel value of red channel, the pixel value of green channel, and the pixel value of blue channel of each pixel point of the image to be processed are used as the pixel value of red channel, the pixel value of green channel, and the pixel value of blue channel of each sampling point of the de-filter template image, and substituted into the mapping equations in the above equation sets (i.e., the first equation set, the second equation set, and the third equation set), to calculate the pixel value of each pixel point of the image to be processed directly rendered by the image filter.
  • the mapping equations in the above equation sets i.e., the first equation set, the second equation set, and the third equation set
  • FIG. 7 is a schematic diagram illustrating a filter template image (a first image), a de-filter template (a second image), an image to be processed (a third image), and a fourth image according to an embodiment of the present disclosure.
  • the above three mapping equations may be established.
  • N*3 mapping equations may be established.
  • the L-BFGS algorithm or the least square method may be used to solve these mapping equations.
  • the above 27 image filter parameters may be solved as values shown in Table 1 below: [Table 1] k R1 k R2 k R3 k R4 k R5 k R6 k R7 k R8 k R9 - 0.03 0 .62 0 .86 - 0.46 - 0.08 0 .46 - 2.33 4 .90 - 3.21 k G1 k G2 k G3 k G4 k G5 k G6 k G7 k G8 k G9 - 0.02 - 0.33 5 .50 - 8.19 3 .84 0 .79 - 2.83 3 .86 - 1.73 k B1 k B2 k B3 k B4 k B5 k B6 k B7 k B8 k B9 - 0.02 0 .12 3
  • Ri, Gi and, Bi respectively represent the pixel value of red channel, the pixel value of green channel and the pixel value of blue channel of the i-th sampling point of the rendered image.
  • ri, gi, and bi respectively represent the pixel value of red channel, the pixel value of green channel and the pixel value of blue channel of the i-th sampling point of the third image.
  • the pixel value of the Pi sampling point of the third image (e.g., a preview image) is (187, 193, 207) in the RGB format
  • a color lookup table method may be used to directly render the image to be processed.
  • a 3D LUT (that is, a color look-up table method of three color channels (red channel, green channel, and blue channel)) may be used to obtain pixel values of the image to be processed after the image filter processing.
  • the 3D LUT may first use the direct calculation method, calculate corresponding output red channel pixel value, green channel pixel value and blue channel pixel value for each input red channel pixel value, green channel pixel value and blue channel pixel value, using the mapping equations in the above equation sets (i.e., the first equation set, the second equation set, and the third equation set), and generate and store the color lookup table including the corresponding input and output red channel pixel values, green channel pixel values and blue channel pixel values.
  • the color lookup table is called, and the red channel pixel value, green channel pixel value and blue channel pixel value of each pixel point of the image to be processed are used as the input red channel pixel value and green channel pixel value and blue channel pixel value respectively, to obtain output red channel pixel value, green channel pixel value, and blue channel pixel value, as pixel value of each pixel point of the image to be processed directly rendered by the image filter.
  • FIG. 8 is a schematic diagram illustrating the 3D LUT according to an embodiment of the present disclosure. Input and output relationship of the 3D LUT is as follows:
  • the following processes may be performed on the third image (image to be processed): segmenting the third image to obtain respective second segmented areas. For any second segmented area, obtaining an image filter corresponding to the second segmented area.
  • the first image (filter template image) is segmented to generate 3 categories of segmented parts (category A/ category B/ category C)
  • these 3 segmented parts may generate 3 different image filters, all the sampling points of the 3 segmented parts may generate an entire image filter.
  • the category A segmented part may be directly rendered using the image filter generated for the category A segmented part of the filter template image
  • the category B segmented part of the image to be processed may be directly rendered using the image filter generated for the category B segmented part of the filter template image
  • the category C segmented part of the image to be processed may be directly rendered using the image filter generated for the category C segmented part of the filter template image
  • the category D segmented part of the image to be processed may be directly rendered using the entire image filter generated for all segmented parts of the filter template image.
  • FIG. 9 is a flowchart of a method for rendering the image to be processed using multiple image filters according to an embodiment of the present disclosure.
  • a de-filter template image with a de-filter effect corresponding to the filter template image is obtained using an image de-filter network with image segmentation function, wherein, respective segmented areas in the filter template image and the de-filter template image have mapping relationships with different de-filter effects according to different categories.
  • multiple image filters are obtained, wherein, the multiple image filters include multiple image filters generated for respective different mapping relationships of respective segmented areas of different categories, and/or an entire image filter generated for an entire mapping relationship of all parts.
  • the image to be processed is rendered to output the rendered image, wherein, after the image to be processed is segmented, for a segmented area having a same category as any segmented area of the filter template image in the image to be processed, the image filter corresponding to the any segmented area of the filter template image is used as the image filter of the segmented area in the image to be processed for rendering, and for a segmented area having no same category as any segmented area of the filter template image in the image to be processed, the entire image filter is used as the image filter of the segmented area in the image to be processed for rendering.
  • FIG. 10 is a flowchart illustrating a method of rendering the image to be processed to output the rendered image based on the multiple image filters according to an embodiment of the present disclosure.
  • step 1001 image segmentation is performed on the image to be processed to obtain segmented regions of different categories.
  • step 1002 it is determined whether all the segmented areas of the image to be processed have corresponding image filters.
  • whether the segmented area of the image to be processed has corresponding image filter is determined by determining whether any of categories of all segmented areas of the filter template image is same as category of the segmented area of the image to be processed.
  • any of the categories of all segmented areas of the filter template image is same as the category of the segmented area of the image to be processed, it is determined that the segmented area of the image to be processed has a corresponding image filter, if any of the categories of all segmented areas of the filter template image is not same as the category of the segmented area of the image to be processed, it is determined that the segmented area of the image to be processed has no corresponding image filter.
  • each of all segmented areas of the image to be processed is rendered using an image filter corresponding to each segmented area respectively, to output the rendered image.
  • step 1004 it is determined whether a part of the segmented areas of the image to be processed have corresponding image filters.
  • step 1005 each of the part of the segmented areas of the image to be processed is rendered using an image filter corresponding to each segmented area respectively, and other segmented areas (the areas having no corresponding image filter) of the image to be processed are rendered using the entire image filter, to output the rendered image.
  • the image to be processed is rendered using the entire image filter, to output the rendered image.
  • a rendering degree may be introduced when the image to be processed is rendered. That is, the image to be processed may be rendered using the image filter according to the rendering degree of the image to be processed.
  • the Alpha blending algorithm may be used to control the rendering effect.
  • the method of rendering may be: directly rendering the image to be processed (e.g., the third image) first using the image filter to obtain a directly rendered image (e.g., fourth image), then fusing the image to be processed (e.g., the third image) and the directly rendered image (e.g., the fourth image) based on the rendering degree of the image to be processed (e.g., the third image), to obtain the final rendered image (e.g., the fifth image).
  • the final rendered result (the fifth image) the third image*(1-the rendering degree) + fourth image*the rendering degree.
  • the rendering degree used in the method for performing rendering according to the rendering degree of the image to be processed according to the embodiment of the present disclosure may include, but is not limited to, the rendering degree determined based on human face areas in the image to be processed and the rendering degree determined according to the scene similarity between the filter template image and the image to be processed, which is introduced below.
  • the rendering is performed further in consideration of the scene similarity between the filter template image (e.g., the first image) and the image to be processed and/or whether there are human face areas in the image to be processed, so as to improve the rendering fidelity and avoid unnatural human face skin color.
  • the filter template image e.g., the first image
  • the rendering degree of the third image may be determined based on the scene similarity between the first image (e.g., filter template image) and the third image (e.g., the image to be processed) and/or the human face areas in the third image (e.g., the image to be processed), and the third image (e.g., the image to be processed) is rendered using the calculated image filter, according to the determined rendering degree.
  • the first image e.g., filter template image
  • the third image e.g., the image to be processed
  • human face areas in the third image e.g., the image to be processed
  • the third image e.g., the image to be processed
  • Each pixel in the third image has its own rendering degree.
  • the rendering degree of each pixel is equal, which is determined by scene similarity.
  • the rendering degree of each pixel in the human face areas is calculated, and the rendering degree of other areas is determined by the scene similarity.
  • the human face areas of the third image may be detected, and the rendering degree of the human face areas may be determined based on the detected human face areas.
  • the scene similarity between the first image and the third image may be detected, and the rendering degree of other areas except for the human face areas in the third image is determined according to the detected scene similarity.
  • the rendering degree of the human face areas in the image to be processed is calculated separately, and for the area outside the human face areas in the image to be processed, the rendering degree of the area outside the human face areas in the image to be processed is determined using the scene similarity between filter template image and the image to be processed.
  • FIG. 11A is a flow chart illustrating a method of performing image processing in consideration of the scene similarity between the filter template image and the image to be processed and/or whether there are human face areas in the image to be processed according to an embodiment of the present disclosure.
  • the de-filter template image with the de-filter effect corresponding to the filter template image is obtained using the image de-filter network.
  • an image filter is obtained based on the filter template image and the de-filter template image.
  • the scene similarity between the filter template image and the image to be processed is detected.
  • step 1104 whether there are human faces in the image to be processed is detected.
  • rendering degrees of the human face areas in the image to be processed is separately determined, and at step 1106, a rendering degree of other areas except for the human face areas in the image to be processed is determined according to the scene similarity.
  • the rendering degree of the image to be processed is determined according to the scene similarity.
  • the image to be processed is rendered based on the obtained image filter to output the rendered image.
  • step 1102 to step 1104 is not limited, and for another example, step 1103 to step 1107 may also be performed before step 1101.
  • FIG. 11B is schematic diagram illustrating a method of performing image processing in consideration of the scene similarity between the filter template image and the image to be processed and/or whether there are human faces in the image to be processed according to an embodiment of the present disclosure.
  • the main bodies performing the method may include an image de-filter network 1110, a filter generation module 1120, a scene similarity detection module 1130, a human face protection module 1140, and a filter rendering module 1150.
  • the image de-filter network 1110 may be used to obtain a de-filter template image with a de-filter effect corresponding to the filter template image based on the filter template image input by the user.
  • the image de-filter network 1110 may be an image de-filter network without an image segmentation function or an image de-filter network with an image segmentation function.
  • the filter generation module 1120 may be used to obtain an image filter based on the filter template image and the de-filter template image.
  • the filter generation module 1120 may obtain a single image filter.
  • the filter generation module 1120 may obtain multiple image filters.
  • the scene similarity detection module 1130 may be used to detect the scene similarity between the filter template image and the image to be processed.
  • the human face protection module 1140 may be used to determine the rendering degree of the human face areas in the image to be processed. For example, human face detection can be performed on the image to be processed. The main body performing human face detection may be included in the human face protection module 1140 or may be a separate main body from the human face protection module 1140. When one or many human faces included in the image to be processed according to the result of the human face detection, the human face protection module 1140 may determine the rendering degree of the human face areas in the image to be processed. When no human face is included in the image to be processed according to the result of the human face detection, the human face protection module 1140 does not perform the operation.
  • the filter rendering module 1150 may be used to render the image to be processed based on the obtained image filter according to the determined rendering degree, to output the rendered image. Specifically, when the image to be processed includes human faces, the filter rendering module 1150 may render the human face areas in the image to be processed according to the determined rendering degree of the human face areas, and render other areas in the image to be processed according to the rendering degree determined by the scene similarity. When the image to be processed includes no human face, the filter rendering module 1150 may render the image to be processed according to the rendering degree determined by the scene similarity. In the following, a method for determining the rendering degree of the human face areas according to the detected human face areas according to an embodiment of the present disclosure is specifically introduced.
  • the detected human face areas may be divided into several blocks.
  • the detected human face areas may be detected actual human face areas, or may be human face areas after the detected actual human faces area are extended.
  • the human face areas may be extended by 20% in the image (extending the human face areas may effectively prevent the detected human face areas from being too small, and may protect the color rendering of the human face areas better).
  • a human face color protection level of each block is calculated according to Gaussian distribution.
  • 2D Gaussian function may be used to build a human face color protection level table for each block.
  • a rendering degree of each block is obtained according to the human face color protection level of each block.
  • the rendering degree of each block may be calculated as (1-human face color protection level). Calculating the human face color protection level through Gaussian distribution may make the filtering effect of the human face areas smoother and the color of the human face areas more natural.
  • FIG. 12 illustrates a schematic diagram of calculating the rendering degree of a human face area according to an embodiment of the present disclosure.
  • the detected human face area (dotted box) may be extended in the image to be processed 1201 to obtain the extended human face area (solid box).
  • the extended human face area may be divided into several blocks, for example, divided into k*k (e.g., 5*5) blocks.
  • the human face color protection level of each area e.g., a human face color protection level table 1202 of each area is established using 2D Gaussian function
  • the rendering degree of each area 1203 may be obtained according to the human face color protection level of each block.
  • the first image (e.g., filter template image) and the third image (e.g., image to be processed) may be scene classified using a scene classification network.
  • the scene similarity of the first image and the third image is determined based on scene classification result and features extracted by a set feature extraction layer in the scene classification network.
  • FIG. 13 is a schematic diagram illustrating a structure of the scene classification network according to an embodiment of the present disclosure.
  • the scene classification network may include 5 convolutional layers (e.g., conv1, conv2, conv3, conv4, conv5) and 2 fully connection layers (e.g., fc6 and fc7), a total of 7 layers.
  • the input of the scene classification network is the image, and the output is the classification result, middle features may also be output in respective middle layers of the scene classification network.
  • the rendering degree of the image to be processed may also be the maximum value (e.g., the rendering degree is 1).
  • the scene similarity of the filter template image and the image to be processed is calculated using the features extracted by the set feature extraction layer in the scene classification network (which may be the middle output value of the scene classification network), and the calculated scene similarity is used as the rendering degree of the image to be processed, wherein the scene similarity may also be referred to as scene consistency.
  • whether the two categories are consistent may be determined using the output of the scene classification network (i.e., the classification result). For example, if the classification result of the filter template image by the scene classification network is class c1, and the classification result of the image to be processed by the scene classification network is class c2, it is determined that the scene categories of the filter template image and the image to be processed are inconsistent, and if the classification results of the filter template image and the image to be processed by the scene classification network are both class c3, it is determined that the scene categories of the filter template image and the image to be processed are consistent.
  • the features extracted at the set feature extraction layer during the classification process of the scene classification network may be used to calculate the scene similarity of the filter template image and the image to be processed.
  • the Markov distance dis of the normalized conv5 layer feature of the filter template image and the image to be processed may be calculated, and the scene similarity of the filter template image and the image to be processed is calculated as (1-dis), wherein values of the scene similarity may range from 0 to 1. The higher the value of the scene similarity, the higher the degree of scene similarity.
  • the scene similarity may be considered as the maximum value (e.g., 1), which is used as the rendering degree.
  • the scene similarity is calculated using the features extracted in the set feature extraction layer during the classification process of the scene classification network, which is used as the rendering degree.
  • FIG. 14 is a flowchart illustrating a method for detecting the scene similarity between the filter template image and the image to be processed and determining the rendering degree according to the detected scene similarity, according to an embodiment of the present disclosure.
  • the filter template image and the image to be processed is classified using a scene classification network.
  • the features of the filter template image and the image to be processed extracted at the set feature extraction layer e.g., conv5 layer
  • the classification results of the filter template image and the image to be processed may be output.
  • the classification results of the filter template image and the image to be processed are compared to determine whether the categories of the filter template image and the image to be processed are consistent.
  • the rendering degree is set to 1.
  • the scene similarity of the filter template image and the image to be processed is calculated using the features of the filter template image and the image to be processed extracted in the set feature extraction layer during the classification process, as the rendering degree.
  • FIG. 15 is a block diagram illustrating an electronic apparatus according to an example of the present disclosure.
  • an electronic apparatus 1500 includes an image acquisition device 1501, a de-filter image acquisition device 1502, an image filter acquisition device 1503, and an image rendering device 1504.
  • the image acquisition device 1501 may acquire a first image input by a user, wherein the first image may be any image input by the user.
  • the first image refers to an image of which the user wishes to obtain a filter effect. Therefore, hereinafter, the first image may be referred to as a filter template image.
  • the filter template image may be any image, selected by the user, if which a style or an effect is desired to be obtained.
  • the filter template image is not necessarily an image with filter effect, but also an image without filter effect. As long as the user wishes to obtain a style or an effect of an image, the user may select the image as the filter template image.
  • the filter template image may be acquired from a local storage or a local database or received from an external data source (e.g., the Internet, a server, a database, etc.) through an input device or a transmission medium, as needed.
  • an external data source e.g., the Internet, a server, a database, etc.
  • the de-filter image acquisition device 1502 may obtain a second image with a de-filter effect corresponding to the first image based on the first image, using an image de-filter network.
  • the second image may also be referred to as a de-filter template image.
  • the image de-filter network may be a deep convolutional neural network.
  • the deep convolutional neural network proposed by various embodiments of the present disclosure has more than 3 layers, including an input layer, convolution layers, normalization layers, a convolution layer and a normalization layer with skip connection structure, up-sampling layers and an output layer.
  • the input layer of the deep convolutional neural network proposed by the present disclosure is used to input the filter template image.
  • the convolutional layer of the deep convolutional neural network proposed by the present disclosure is used for feature extraction of the input data.
  • the present disclosure uses instance normalization.
  • the instance normalization normalizes features on the image pixels, and the effect is obvious in stylization migration.
  • the up-sampling interpolation method used by up-sampling layer of the deep neural network proposed in the present disclosure is the nearest neighbor method, which is the simplest interpolation method and does not need calculating, and in the four neighboring pixels of the desired pixel, the gray scale of the neighbor pixel closest to the desired pixel is assigned to the desired pixel.
  • the output layer of the deep neural network proposed by the present disclosure is used to output a feature map with the same size as the input image.
  • the deep neural network proposed by the present disclosure also adds a convolutional layer and a normalization layer with skip connection structure, to improve the performance of the deep convolutional neural network.
  • the image de-filter network may be a U-NET type of deep convolutional neural network. It may be understood that the image de-filter network is not limited to this, and may also be other artificial neural networks.
  • U-NET is a fully convolutional network that includes multi-layer down-sampling, multi-layer up-sampling, and skip connection structure, and its characteristic is that the convolutional layer is completely symmetric in the down-sampling and up-sampling parts, and the feature map at the down-sampling end may skip the deep sampling and be concatenated to the corresponding up-sampling end.
  • the sizes of the input layer and output layer of the network are completely the same.
  • the U-NET type of convolutional neural network is a Fully Convolutional Networks (FCNs) organized under the encoder-decoder paradigm.
  • FCNs Fully Convolutional Networks
  • the U-NET deep convolutional neural network proposed by the present disclosure is also called as Encoder-Decoder Network.
  • the U-NET deep convolutional neural network proposed by the present disclosure may be designed to include an input layer, convolutional layers and normalization layers, down-sampling layers, and a convolutional layer and a normalization layer with skip connection structure, up-sampling layers and a output layer.
  • the image de-filter network proposed by the present disclosure which is a U-NET type of deep convolutional neural network, may be designed as an image de-filter network as shown in FIG. 2 or FIG. 3 .
  • the system 1500 may further include a training device (not shown).
  • the training device may use a training image set including a plurality of images with filter effects (filter images) and a plurality of corresponding images in which filter effects are removed (de-filter images) (i.e., multiple image pairs, each image pair includes filter image and de-filter image corresponding to the filter image) to train the image de-filter network proposed by the present disclosure.
  • the de-filter image acquisition device 1502 may use the trained image de-filter network to generate the de-filter template image corresponding to the filter template image.
  • the training device may appropriately modify the training image set. For example, the de-filter images in the training image set are obtained using different appropriate de-filter process on the corresponding filter images according to different image segmentation categories.
  • the image filter acquisition device 1503 may obtain an image filter based on the first image and the second image. For example, the image filter acquisition device 1503 may calculate a mapping relationship of transforming the second image (de-filter template image) to the first image (filter template image), and use the calculated mapping relationship as an image filter.
  • the image filter acquisition device 1503 may calculate the mapping relationship of transforming the de-filter template image to the filter template image based on image attributes of the filter template image and the de-filter template image. For example, the image filter acquisition device 1503 may calculate the mapping relationship of transforming the de-filter template image to the filter template image based on color attributes of the filter template image and the de-filter template image. According to an embodiment of the present disclosure, first, the image filter acquiring device 1503 may acquire multiple first sampling points in the filter template image and multiple second sampling points in the de-filter template image corresponding to the respective first sampling points in the filter template image may be acquired.
  • the image filter acquiring device 1503 may perform uniform random sampling on the filter template image and the de-filter template image to obtain multiple sampling points in the filter template image and multiple sampling points in the de-filter template image corresponding to the multiple sampling points in the filter template image.
  • the image filter acquiring device 1503 may obtain corresponding sampling points of the filter template image and the de-filter template image by performing sampling on the filter template image and the de-filter template image.
  • the sampling points of the filter template image and the sampling points of the de-filter template image correspond one-to-one in position.
  • the filter template image and the de-filter template image may be uniform random sampled using a uniform random sampling method.
  • N sampling points may be collected in the filter template image and N sampling points of corresponding positions may be collected in the de-filter template image, wherein N may be equal to or greater than 4000. It may be understood that the number of N is not limited to this.
  • a density of the first sampling points in the area where the salient target is located may be greater than the density of the first sampling points in the area outside the salient target. Therefore, larger sampling weight may be applied to salient target regions in the image, and smaller sampling weight may be applied to other regions. For example, it is assumed that N pairs of sampling points are collected in the filter template image and the de-filter template image.
  • Salient target in the filter template image is extracted, for example, salient target may be detected and extracted using a salience detection network.
  • N*T points (T>0.5) are sampled in the area where the salient target of the filter template image is located and the corresponding position of the de-filter template image, wherein T may be an extraction ratio (i.e., the sampling weight), and N*(1-T) points are sampled in the area outside of the salient target of the filter template image and the corresponding position of the de-filter template image, therefore, the sampling weight for the area where the salient object is located in the image is greater than the sampling weight for the area outside of the salient target, that is, the more sampling points are extracted in the area where the salient target is located in the image, wherein N may be equal to or greater than 4000.
  • T may be set according to factors of user requirement, image features and the like. For example, a ratio t of the area where the salient target is located in the image may be calculated, and T may be set to 0.5+0.5t. Alternatively or additionally, the uniform random sampling method may be applied to the sampling of both the area where the salient target is located and the area outside of the salient target in the image.
  • different parts in the image may be considered to be sampled differently, and a mapping relationship may be generated for each part and/or an entire mapping relationship may be generated for all parts, thereby generating multiple image filters, so that different appropriate image filters are used for different parts of the image to enhance the performance of the image filters.
  • N pairs of sampling points are collected in the filter template image and the de-filter template image.
  • the filter template image is segmented into multiple different parts, for example, the filter template image is segmented into multiple different parts through semantic segmentation.
  • the image filter acquiring device 1503 may calculate the mapping relationship of transforming the de-filter template image to the filter template image by calculating the color mapping relationship between all the multiple sampling points in the filter template image and all the corresponding multiple sampling points in the de-filter template image, and obtain the image filter according to the mapping relationship.
  • the image filter acquiring device 1503 may segment the first image (e.g., filter template image) to obtain respective first segmented areas. For at least one first segmented area, obtain an image filter corresponding to the at least one first segmented area. Obtain an image filter corresponding to the first image based on the first image and the second image (e.g., de-filter template image).
  • first image e.g., filter template image
  • second image e.g., de-filter template image
  • the image filter acquiring device 1503 may acquire a plurality of first sampling points in the first segmented area and a plurality of second sampling points corresponding to the plurality of first sampling points respectively in the second image, and acquire the image filter corresponding to the first segmented area by determining the mapping relationship between the respective first sampling points in the first segmented area and the corresponding second sampling points in the second image.
  • the mapping relationship of transforming the de-filter template image to the filter template image of each segmented part may be calculated by the color mapping relationship between the multiple sampling points of the filter template image and the corresponding multiple sampling points in the de-filter template image of each segmented part, thereby obtaining multiple mapping relationships, and/or an entire mapping relationship of transforming the de-filter template image to the filter template image for all segmented parts may be calculated by the color mapping relationship between all the sampling points of the filter template image and all the sampling points in the de-filter template image of all the segmented parts.
  • the mapping relationships between respective color channels of the corresponding second sampling point of the second image (e.g., de-filter template image) and the any color channel of the any first sampling point may be determined respectively, to obtain the mapping relationships between the any first sampling point and the corresponding second sampling point.
  • the color channel may be an RGB (red, green, blue) channel, or other possible color channels, for example, YCbCr, YUV channels, etc.
  • RGB red, green, blue
  • the image filter acquiring device 1503 may build a first equation set, a second equation set, and a third equation set.
  • the first equation set may include mapping equations of the pixel value of the red channel, the green channel, and the blue channel of the sampling point of the de-filter template image to the pixel values of the red channel of the corresponding sampling point of the filter template image, built for each sampling point, that is, the first equation set may include N (or N*r) such mapping equations.
  • the second equation set may include mapping equations of the pixel value of the red channel, the green channel, and the blue channel of the sampling point of the de-filter template image to the pixel values of the green channel of the corresponding sampling point of the filter template image, built for each sampling point, that is, the second equation set may include N (or N*r) such mapping equations.
  • the third equation set may include mapping equations of the pixel value of the red channel, the green channel, and the blue channel of the sampling point of the de-filter template image to the pixel values of the blue channel of the corresponding sampling points of the filter template image, built for each sampling point, that is, the third equation set can include N (or N*r) such mapping equations.
  • the present disclosure in order to improve generalization ability of the model, multiple linear functions may be used to build the model.
  • the present disclosure preferably uses a quartic linear function to establish the mapping equations in the first equation set, the second equation set and the third equation set. It may be understood that the mapping equation is not limited to the quartic linear function, and other preferred solutions may also be used to establish the mapping equations.
  • the image filter acquiring device 1503 may solve the first equation set, the second equation set, and the third equation set to obtain the mapping relationship of transforming the de-filter template image to the filter template image, as image filter.
  • the mapping relationship of transforming the de-filter template image to the filter template image may be obtained.
  • the image filter acquiring device 1503 may use numerical optimization algorithm (such as L-BFGS algorithm) to solve the first equation set, the second equation set, and the third equation set.
  • numerical optimization algorithm such as L-BFGS algorithm
  • the image filter acquiring device 1503 may use least square method to solve the first equation set, the second equation set, and the third equation set.
  • the image filter proposed by the present disclosure contains rich and diverse color information, the filter effect is softer, more natural, contains more color features, and the effect is more beautiful.
  • the image rendering device 1504 may render an image to be processed (e.g., a third image) using the obtained image filter to output the fourth image.
  • an image to be processed e.g., a third image
  • the image rendering device 1504 may use a direct calculation method to obtain pixel values of the image to be processed directly rendered by the image filter.
  • 27 image filter parameters may be obtained, that is, kR1, kR2, kR3, kR4, kR5, kR6, kR7, kR8, kR9, kG1, kG2, kG3, kG4, kG5, kG6, kG7, kG8, kG9, kB 1, kB2, kB3, kB4, kB5, kB6, kB7, kB8, kB9.
  • These 27 image filter parameters may be stored in a memory or a server and the like.
  • the image rendering device 1504 reads these 27 image filter parameters, and use the pixel value of red channel, the pixel value of green channel, and the pixel value of blue channel of each pixel point of the image to be processed as the pixel value of red channel, the pixel value of green channel, and the pixel value of blue channel of each sampling point of the de-filter template image, and substitute them into the mapping equations in the above equation sets (i.e., the first equation set, the second equation set, and the third equation set), to calculate the pixel value of each pixel point of the image to be processed directly rendered by the image filter.
  • the mapping equations in the above equation sets i.e., the first equation set, the second equation set, and the third equation set
  • the image rendering device 1504 may use a color lookup table method (LUT) to obtain the pixel value of the image to be processed directly rendered by the image filter.
  • LUT color lookup table method
  • a 3D LUT (that is, a color look-up table method of three color channels (red channel, green channel, and blue channel)) may be used to obtain pixel values of the image to be processed after the image filter processing.
  • the 3D LUT may first use the direct calculation method, calculate corresponding output red channel pixel value, green channel pixel value and blue channel pixel value for each input red channel pixel value, green channel pixel value and blue channel pixel value, using the mapping equations in the above equation sets (i.e., the first equation set, the second equation set, and the third equation set), and generate and store the color lookup table including the corresponding input and output red channel pixel values, green channel pixel values and blue channel pixel values.
  • the image rendering device 1504 may call the color lookup table, and use the red channel pixel value, green channel pixel value and blue channel pixel value of each pixel point of the image to be processed as the input red channel pixel value and green channel pixel value and blue channel pixel value respectively, to obtain output red channel pixel value, green channel pixel value, and blue channel pixel value, as pixel value of each pixel point of the image to be processed directly rendered by the image filter.
  • the image rendering device 1504 may perform the following processes: segmenting the third image to obtain respective second segmented areas. For any second segmented area, obtaining an image filter corresponding to the second segmented area.
  • the filter template image is segmented to generate 3 categories of segmented parts (category A/category B/category C)
  • these 3 segmented parts may generate 3 different image filters, all the sampling points of the 3 segmented parts may generate an entire image filter.
  • the category A segmented part may be directly rendered using the image filter generated for the category A segmented part of the filter template image
  • the category B segmented part of the image to be processed may be directly rendered using the image filter generated for the category B segmented part of the filter template image
  • the category C segmented part of the image to be processed may be directly rendered using the image filter generated for the category C segmented part of the filter template image
  • the category D segmented part of the image to be processed may be directly rendered using the entire image filter generated for all segmented parts of the filter template image.
  • the rendering degree used in the method for performing rendering according to the rendering degree of the image to be processed according to the embodiment of the present disclosure may include, but is not limited to, the rendering degree determined based on human face areas in the image to be processed and the rendering degree determined according to the scene similarity between the filter template image and the image to be processed, which is introduced below.
  • the image rendering device 1504 when the image rendering device 1504 renders the image to be processed (e.g., the third image), the image rendering device 1504 may perform rendering further in consideration of the scene similarity of the filter template image (e.g., first image) and the image to be processed and/or whether there is human face areas in the image to be processed, so as to improve the rendering fidelity and avoid unnatural human face skin color.
  • the image rendering device 1504 may determine the rendering degree of the image to be processed according to the scene similarity between the filter template image and the image to be processed and/or the human face areas in the image to be processed, and render the image to be processed using the calculated image filter, according to the determined rendering degree.
  • Each pixel in the image to be processed has its own rendering degree.
  • the rendering degree of each pixel is equal, which is determined by scene similarity.
  • the rendering degree of each pixel in the human face areas are calculated, and the rendering degree of other areas are determined by the scene similarity.
  • the image rendering device 1504 may detect the human face areas of the third image, and determine the rendering degree of the human face areas based on the detected human face areas. Detect the scene similarity between the first image and the third image, and determine the rendering degree of other areas except for the human face areas in the third image according to the detected scene similarity. In other words, for the human face areas in the image to be processed, the image rendering device 1504 may separately calculate the rendering degree of the human face areas in the image to be processed, and for the area outside the human face areas in the image to be processed, the image rendering device 1504 may determine the rendering degree of the area outside the human face areas in the image to be processed using the scene similarity between filter template image and the image to be processed.
  • the image rendering device 1504 may divide every detected human face area into several blocks.
  • the detected human face areas may be a detected actual human face areas, or may be a human face areas after the detected actual human face areas are extended.
  • the human face areas may be extended by 20% in the image (extending the human face areas may effectively prevent the detected human face areas from being too small, and may protect the color rendering of the human face areas better).
  • the image rendering device 1504 may calculate a human face color protection level of each block according to Gaussian distribution. For example, 2D Gaussian function may be used to build a human face color protection level table for each block.
  • the image rendering device 1504 may obtain a rendering degree of each block according to the human face color protection level of each block.
  • the rendering degree of each block may be calculated as (1minus human face color protection level). Calculating the human face color protection level through Gaussian distribution may make the filtering effect of the human face areas smoother and the color of the human face areas more natural.
  • the image rendering device 1504 may use a scene classification network to scene classify the first image (filter template image) and the third image (image to be processed). Determine the scene similarity of the first image and the third image based on scene classification result and features extracted by a set feature extraction layer in the scene classification network.
  • the image rendering device 1504 may determine that the scene similarity of the first image and the third image is a maximum value (e.g., the similarity is 1), therefore, the rendering degree of the image to be processed may also be the maximum value (e.g., the rendering degree is 1). If the two categories are inconsistent according to the scene classification result, the image rendering device 1504 may calculate the scene similarity of the filter template image and the image to be processed using the features extracted by the set feature extraction layer in the scene classification network (which may be the middle output value of the scene classification network), and use the calculated scene similarity as the rendering degree of the image to be processed, wherein the scene similarity may also be referred to as scene consistency.
  • the scene similarity may also be referred to as scene consistency.
  • the image rendering device 1504 may determine whether the two categories are consistent using the output of the scene classification network (i.e., the classification result). For example, if the classification result of the filter template image by the scene classification network is class c1, and the classification result of the image to be processed by the scene classification network is class c2, the image rendering device 1504 may determine that the scene categories of the filter template image and the image to be processed are inconsistent, and if the classification results of the filter template image and the image to be processed by the scene classification network are both class c3, the image rendering device 1504 may determine that the scene categories of the filter template image and the image to be processed are consistent.
  • the classification result of the filter template image by the scene classification network is class c1
  • the classification result of the image to be processed by the scene classification network is class c2
  • the image rendering device 1504 may determine that the scene categories of the filter template image and the image to be processed are inconsistent
  • the classification results of the filter template image and the image to be processed by the scene classification network are both class c3
  • the image rendering device 1504 may use the features extracted at the set feature extraction layer during the classification process of the scene classification network (e.g., the features extracted through the conv5 layer of the scene classification network) to calculate the scene similarity of the filter template image and the image to be processed.
  • the image rendering device 1504 may calculate the Markov distance dis of the normalized conv5layer feature of the filter template image and the image to be processed, and calculate the scene similarity of the filter template image and the image to be processed as (1-dis), wherein values of the scene similarity may range from 0 to 1. The higher the value of the scene similarity is, the higher the degree of scene similarity is.
  • the image rendering device 1504 may consider the scene similarity to be the maximum value (1), which is used as the rendering degree.
  • the image rendering device 1504 may calculate the scene similarity using the features extracted in the set feature extraction layer during the classification process of the scene classification network, which is used as the rendering degree.
  • an image de-filter network is used to generate a template image in which the filter effect is removed, without pre-establishing an image library that stores a large number of sample images without the filter effect, which greatly reduces the required storage space and saves storage resources.
  • a network for implementing image de-filter is proposed, thereby improving the performance of the image filter.
  • a template image in which the filter effect is removed with the same structure with thereof is generated, without finding an image with the similar structure without the filter effect, which improves the speed and efficiency of the image filter, and the filter method is more robust.
  • the image filter is generated by sampling the filter template image and the corresponding de-filter template image and using the color information mapping relationship of the sampling points. This method takes into account the global color information and partial color information of the image, and thus the effect of image filter is improved.
  • the scene similarity between the image to be processed e.g., the camera preview image
  • the filter template image is considered to generate the rendering degree for controlling the filter effect, thereby the filter effect that is not ideal, unreal, and not in line with the actual scene may be avoided.
  • the human face skin color protection in the image to be processed (e.g., the camera preview image) is considered to generate the rendering degree for the human face areas, so that unnatural skin color effect may be avoided.
  • image segmentation category e.g., semantic segmentation category
  • different image filters are generated using different categories of areas of the filter template image, and the different image filters are applied to different categories of areas in the image to be processed, thereby improving the fidelity of the image to enhance the effect of the image filter.
  • the systems, devices, and units shown in FIG. 15 may be respectively configured as software, hardware, firmware, or any combination thereof to perform specific functions.
  • these systems, devices, or units may correspond to dedicated integrated circuits, may also correspond to pure software codes, or may further correspond to a combined module of software and hardware.
  • one or more functions implemented by these systems, devices, or units may also be uniformly executed by components in physical entity devices (e.g., processors, clients, or servers, etc.).
  • the method described with reference to FIG. 1 may be implemented by programs (or instructions) recorded on a computer-readable storage medium.
  • a computer-readable storage medium may be provided, on which computer programs (or instructions) for executing the steps of the method of image processing described with reference to FIG. 1 is recorded.
  • the computer programs (or instructions) may be used to perform the following steps: selecting a first image by a user. Obtaining a second image in which a filter effect applied to the first image is removed, using an image de-filter network. Obtaining an image filter corresponding to the filter effect based on the first image and the second image. Rendering a third image based on the obtained image filter to output a fourth image.
  • the computer program in the above-mentioned computer-readable storage medium may run in an environment deployed in computer equipment such as a client, a host, an agent device, a server, etc. It should be noted that the computer program may also be used to perform additional steps in addition to the above-mentioned steps or perform more specific process when performing the above steps. These additional steps and further processed content have been mentioned in the description of the related art method with reference to FIG. 1 , so in order to avoid repetition, the description will not be repeated here.
  • the electronic apparatus of image processing may completely rely on the operation of the computer program to realize the corresponding function, that is, respective units correspond to respective steps in the functional structure of the computer program, so that the entire electronic apparatus is called by special software packages (e.g., lib library) to achieve the corresponding functions.
  • special software packages e.g., lib library
  • each device shown in FIG. 15 may also be implemented by hardware, software, firmware, middleware, microcode, or any combination thereof.
  • the program code or code segment used to perform the corresponding operation may be stored in a computer-readable medium such as a storage medium, so that the processor may perform the corresponding operation by reading and running the corresponding program code or code segment.
  • an embodiment of the present disclosure may also be implemented as a computing device including a storage component and a processor.
  • the storage component stores a set of computer executable instructions. When the set of computer executable instructions is executed by the processor, the method of image processing according to an embodiment of the present disclosure is performed.
  • the computing device may be deployed in a server or a client, and may also be deployed on a node device in a distributed network environment.
  • the computing device may be a PC computer, a tablet device, a personal digital assistant, a smart phone, a web application, or other devices capable of executing the above set of instructions.
  • the computing device does not have to be a single computing device, and may also be any device or collection of circuits that may execute the above-mentioned instructions (or instruction set) individually or jointly.
  • the computing device may also be a part of an integrated control system or a system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) in interface.
  • the processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor.
  • the processor may further include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, and the like.
  • Some operations described in the method of image processing according to the embodiment of the present disclosure may be implemented by software, and some operations may be implemented by hardware. In addition, these operations may also be implemented by a combination of software and hardware.
  • the processor may run instructions or codes stored in one of the storage components, wherein the storage component may also store data. Instructions and data may also be transmitted and received through a network via a network interface device, wherein the network interface device may use any known transmission protocol.
  • the storage component may be integrated with the processor, for example, RAM or flash memory is arranged in an integrated circuit microprocessor or the like.
  • the storage component may include an independent device, such as an external disk drive, a storage array, or other storage device that can be used by any database system.
  • the storage component and the processor may be operatively coupled, or may communicate with each other, for example, through an I/O port, a network connection, or the like, so that the processor can read files stored in the storage component.
  • the computing device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the computing device may be connected to each other via a bus and/or network.
  • a video display such as a liquid crystal display
  • a user interaction interface such as a keyboard, mouse, touch input device, etc.
  • the method described with reference to FIG. 1 may be implemented by a system including at least one computing device and at least one storage device storing instructions.
  • the at least one computing device is a computing device of image processing according to an embodiment of the present disclosure
  • a computer-executable instruction set is stored in the storage device.
  • the steps of the method of image processing described with reference to FIG. 1 are executed.
  • the computer-executable instruction set is executed by the at least one computing device, the following steps may be executed: selecting the first image by a user. Obtaining a second image in which a filter effect applied to the first image is removed, using an image de-filter network. Obtaining an image filter corresponding to the filter effect based on the first image and the second image. Rendering a third image using the obtained image filter to output a fourth image.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Multimedia (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Human Computer Interaction (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Claims (15)

  1. Bildverarbeitungsverfahren, wobei das Verfahren umfasst:
    Erhalten (101) eines ersten Bildes;
    Entfernen eines auf das erste Bild angewendeten Filtereffekts unter Verwendung eines Bildentfilterungsnetzwerks (200), um ein zweites Bild zu erhalten (102);
    Erhalten (103), basierend auf dem ersten Bild und dem zweiten Bild, eines Bildfilters, der dem Filtereffekt entspricht; und
    Rendern (104) eines dritten Bildes unter Verwendung des erhaltenen Bildfilters zum Ausgeben eines vierten Bildes.
  2. Verfahren nach Anspruch 1, wobei das Entfernen des Filtereffekts umfasst:
    Durchführen einer Merkmalsextraktion an dem ersten Bild über kaskadierte Merkmalsextraktionseinheiten (211) eines Codierernetzwerks (210) des Bildentfilterungsnetzwerks (200), um eine extrahierte Merkmalstafel zu erzeugen; und
    Decodieren der extrahierten Merkmalskarte über kaskadierte Decodereinheiten (221) eines Decodernetzwerks (220) des Bildentfilterungsnetzwerks (200), um das zweite Bild auszugeben.
  3. Verfahren nach Anspruch 2, wobei die Durchführung der Merkmalsextraktion an dem ersten Bild umfasst:
    Durchführen, unter Verwendung eines Abschnitts der kaskadierten Merkmalsextraktionseinheiten (211), der sich von einer letzten Merkmalsextraktionseinheit unterscheidet, mindestens eines Faltungsprozesses und mindestens eines Abwärtsabtastprozesses an einer Eingangsmerkmalskarte, und
    Durchführen, unter Verwendung der letzten Merkmalsextraktionseinheit, von mindestens zwei Faltungsprozessen an der verarbeiteten Eingangsmerkmalskarte, einer Restverknüpfung der Eingangsmerkmalskarte und der faltungsverarbeiteten Eingangsmerkmalskarte, und Ausgeben der extrahierten Merkmalskarte an das Decodernetzwerk (220), und
    das Decodieren der extrahierten Merkmalskarte umfasst:
    Durchführen mindestens eines Aufwärtsabtastprozesses und mindestens eines Faltungsprozesses an der extrahierten Merkmalskarte.
  4. Verfahren nach Anspruch 2, wobei das Entfernen des Filtereffekts weiterhin umfasst:
    Herunterskalieren des ersten Bildes unter Verwendung einer Vorverarbeitungsschicht entsprechend einer vorgegebenen Größe; und
    Ausgeben des herunterskalierten ersten Bildes an das Codierernetzwerk (210).
  5. Verfahren nach Anspruch 3, wobei das Entfernen des Filtereffekts weiterhin umfasst:
    Erhalten, von einem Bildsegmentierungsnetzwerk (330), mindestens eines Typs von Segmentierungsinformationen aus den Segmentierungsinformationen für das erste Bild;
    Verketten des ersten Bildes mit dem mindestens einen Typ von Segmentierungsinformation, um mindestens einen Typ von einem zusammengeführten ersten Bild zu erhalten; und
    Eingeben des mindestens einen Typs des zusammengeführten ersten Bildes in das Codierernetzwerk (210).
  6. Verfahren nach Anspruch 5, wobei das Entfernen des Filtereffekts weiterhin umfasst:
    Verketten, unter Verwendung mindestens einer Decodierungseinheit des Decodernetzwerks (220), einer aufwärtsabgetasteten Merkmalskarte mit der faltungsverarbeiteten Eingangsmerkmalskarte, die einer Merkmalskarte gleichen Maßstabs entspricht; und
    Durchführen mindestens eines Faltungsprozesses an der verketteten Merkmalskarte.
  7. Verfahren nach Anspruch 1, wobei das Erhalten des Bildfilters umfasst:
    Erhalten einer Abbildungsbeziehung zur Transformation des zweiten Bildes in das erste Bild; und
    Erhalten des Bildfilters entsprechend der Abbildungsbeziehung.
  8. Verfahren nach Anspruch 7, wobei das Erhalten der Abbildungsbeziehung umfasst:
    Erfassen einer Vielzahl von ersten Abtastpunkten in dem ersten Bild und einer Vielzahl von zweiten Abtastpunkten, die den jeweiligen ersten Abtastpunkten entsprechen, in dem zweiten Bild; und
    Erhalten der Abbildungsbeziehung des Transformierens des zweiten Bildes in das erste Bild durch Bestimmen von Abbildungsbeziehungen zwischen den jeweiligen ersten Abtastpunkten in dem ersten Bild und den entsprechenden zweiten Abtastpunkten in dem zweiten Bild.
  9. Verfahren nach Anspruch 8, wobei das Erfassen der Vielzahl von ersten Abtastpunkten in dem ersten Bild umfasst:
    Erfassen eines hervorstechenden Ziels im ersten Bild;
    Erfassen eines ersten Abschnitts der Vielzahl von ersten Abtastpunkten in einem ersten Bereich, in dem sich das hervorstechende Ziel in dem ersten Bild befindet; und
    Erfassen eines zweiten Abschnitts der Vielzahl von ersten Abtastpunkten in einem zweiten Bereich außerhalb des hervorstechenden Ziels im ersten Bild, und
    wobei eine erste Dichte des ersten Abschnitts der Vielzahl von ersten Abtastpunkten größer ist als eine zweite Dichte des zweiten Abschnitts der Vielzahl von ersten Abtastpunkten.
  10. Verfahren nach Anspruch 1, wobei das Rendern des dritten Bildes umfasst:
    Bestimmen eines Rendering-Grades des dritten Bildes entsprechend mindestens einer einer Szenenähnlichkeit zwischen dem ersten Bild und dem dritten Bild und eines oder mehrerer menschlicher Gesichtsbereiche in dem dritten Bild; und
    Rendern des dritten Bildes unter Verwendung des Bildfilters entsprechend dem Rendering-Grad des dritten Bildes.
  11. Verfahren nach Anspruch 10, wobei das Bestimmen des Rendering-Grades des dritten Bildes umfasst:
    Bestimmen von Rendering-Graden des einen oder der mehreren menschlichen Gesichtsbereiche entsprechend dem einen oder den mehreren erfassten menschlichen Gesichtsbereichen;
    Erfassen der Szenenähnlichkeit zwischen dem ersten Bild und dem dritten Bild; und
    Bestimmen eines Rendering-Grades eines anderen Bereichs, der nicht der eine oder die mehreren menschlichen Gesichtsbereiche im dritten Bild ist, in Abhängigkeit von der erfassten Szenenähnlichkeit.
  12. Verfahren nach Anspruch 11, wobei das Erfassen der Szenenähnlichkeit zwischen dem ersten Bild und dem dritten Bild umfasst:
    Durchführen einer Szenenklassifizierung am ersten Bild und am dritten Bild unter Verwendung eines Szenenklassifizierungsnetzwerks; und
    Bestimmen der Szenenähnlichkeit des ersten Bildes und des dritten Bildes auf der Grundlage eines Szenenklassifizierungsergebnisses und von Merkmalen, die von einer eingestellten Merkmalsextraktionsschicht in dem Szenenklassifizierungsnetzwerk extrahiert wurden.
  13. Verfahren nach Anspruch 11, wobei das Bestimmen der Rendering-Grade des einen oder der mehreren menschlichen Gesichtsbereiche umfasst:
    Unterteilen jedes erfassten menschlichen Gesichtsbereichs des einen oder der mehreren menschlichen Gesichtsbereiche in einen entsprechenden Satz von Blöcken;
    Erhalten eines Farbschutzniveaus des menschlichen Gesichts für jeden Block in den entsprechenden Blocksätzen gemäß einer Gaußschen Verteilung; und
    Erhalten eines Rendering-Grades für jeden Block in den entsprechenden Blocksätzen entsprechend dem Farbschutzniveau des menschlichen Gesichts für diesen Block.
  14. Verfahren nach Anspruch 11, wobei das Rendern des dritten Bildes umfasst:
    Zusammenführen des dritten Bildes und des vierten Bildes auf der Grundlage des Rendering-Grades des dritten Bildes, um ein fünftes Bild zu erhalten.
  15. Verfahren nach Anspruch 1, wobei das Erhalten des Bildfilters umfasst:
    Segmentieren des ersten Bildes, um entsprechende erste segmentierte Bereiche zu erhalten;
    Erfassen eines Bildfilters, der mindestens einem ersten segmentierten Bereich der entsprechenden ersten segmentierten Bereiche entspricht; und
    Erfassen eines dem ersten Bild entsprechenden Bildfilters auf der Grundlage des dem mindestens einen ersten segmentierten Bereich und dem zweiten Bild entsprechenden Bildfilters.
EP21753231.6A 2020-02-11 2021-02-05 Bildverarbeitungsverfahren Active EP4100914B1 (de)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
CN202010087287 2020-02-11
CN202010600882.0A CN113256504A (zh) 2020-02-11 2020-06-28 图像处理的方法和电子设备
PCT/KR2021/001557 WO2021162359A1 (en) 2020-02-11 2021-02-05 Image processing method and electronic apparatus

Publications (4)

Publication Number Publication Date
EP4100914A1 EP4100914A1 (de) 2022-12-14
EP4100914A4 EP4100914A4 (de) 2023-08-02
EP4100914B1 true EP4100914B1 (de) 2024-08-14
EP4100914C0 EP4100914C0 (de) 2024-08-14

Family

ID=77220120

Family Applications (1)

Application Number Title Priority Date Filing Date
EP21753231.6A Active EP4100914B1 (de) 2020-02-11 2021-02-05 Bildverarbeitungsverfahren

Country Status (4)

Country Link
US (2) US12288315B2 (de)
EP (1) EP4100914B1 (de)
CN (1) CN113256504A (de)
WO (1) WO2021162359A1 (de)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115908596B (zh) 2021-08-20 2023-11-24 荣耀终端有限公司 一种图像处理方法及电子设备
CN113781292B (zh) * 2021-08-23 2024-09-24 北京达佳互联信息技术有限公司 图像处理方法和装置、电子设备及存储介质
CN114092815B (zh) * 2021-11-29 2022-04-15 自然资源部国土卫星遥感应用中心 一种大范围光伏发电设施遥感智能提取方法
CN116366841B (zh) * 2021-12-28 2025-11-11 维沃移动通信有限公司 环路滤波方法及终端
CN114972762B (zh) * 2022-06-20 2025-04-18 京东科技控股股份有限公司 图像语义分割方法、装置、计算机设备及存储介质
CN115345801B (zh) * 2022-10-18 2023-03-24 中科方寸知微(南京)科技有限公司 一种基于图像去噪思想的图像压缩及滤镜去除方法及系统
CN117173056B (zh) * 2023-11-01 2024-04-09 欣瑞华微电子(上海)有限公司 用于解决信息丢失的图像处理方法、设备及可读存储介质

Family Cites Families (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2004004361A1 (ja) 2002-06-28 2004-01-08 Nokia Corporation 情報端末
KR100606076B1 (ko) 2004-07-02 2006-07-28 삼성전자주식회사 휴대단말기의 이미지 처리방법
JP4799101B2 (ja) * 2005-09-26 2011-10-26 富士フイルム株式会社 画像処理方法および装置ならびにプログラム
US8509569B2 (en) 2008-02-11 2013-08-13 Apple Inc. Optimization of image processing using multiple processing units
CN102360506B (zh) * 2011-09-30 2013-07-31 北京航空航天大学 一种基于局部线性保持的场景颜色风格一致化处理方法
US9154709B2 (en) * 2012-12-21 2015-10-06 Google Inc. Recommending transformations for photography
CN104700353B (zh) * 2015-02-11 2017-12-05 小米科技有限责任公司 图像滤镜生成方法及装置
US10319085B2 (en) * 2015-02-16 2019-06-11 Samsung Electronics Co., Ltd. Metadata-based image processing method and apparatus
CN105825494B (zh) * 2015-08-31 2019-01-29 维沃移动通信有限公司 一种图像处理方法及移动终端
KR102552747B1 (ko) * 2016-06-28 2023-07-11 주식회사 엘엑스세미콘 역 톤 매핑 방법
CN108205813B (zh) * 2016-12-16 2022-06-03 微软技术许可有限责任公司 基于学习网络的图像风格化
US10152768B2 (en) * 2017-04-14 2018-12-11 Facebook, Inc. Artifact reduction for image style transfer
KR102421376B1 (ko) * 2017-06-09 2022-07-15 에스케이텔레콤 주식회사 재설정 가능한 이미지 변환을 통한 비지도된 시각적 속성 전달
CN111066060B (zh) * 2017-07-13 2024-08-02 资生堂株式会社 虚拟面部化妆去除和模拟、快速面部检测和地标跟踪
US10891723B1 (en) * 2017-09-29 2021-01-12 Snap Inc. Realistic neural network based image style transfer
CN108171776B (zh) 2017-12-26 2021-06-08 浙江工业大学 一种基于改进的卷积神经网络实现图像编辑传播的方法
US10650495B2 (en) * 2018-06-04 2020-05-12 Adobe Inc. High resolution style transfer
CN110349102B (zh) * 2019-06-27 2025-10-10 腾讯科技（深圳）有限公司 图像美化的处理方法、图像美化的处理装置以及电子设备
US11722796B2 (en) * 2021-02-26 2023-08-08 Samsung Electronics Co., Ltd. Self-regularizing inverse filter for image deblurring
EP4325427A4 (de) * 2021-10-18 2024-09-25 Samsung Electronics Co., Ltd. Elektronische vorrichtung zur verarbeitung eines bildes und verfahren zur steuerung davon
US12505660B2 (en) * 2021-12-29 2025-12-23 Samsung Electronics Co., Ltd. Image processing method and apparatus using convolutional neural network

Also Published As

Publication number Publication date
EP4100914A4 (de) 2023-08-02
WO2021162359A1 (en) 2021-08-19
US12288315B2 (en) 2025-04-29
US20250252542A1 (en) 2025-08-07
EP4100914A1 (de) 2022-12-14
CN113256504A (zh) 2021-08-13
EP4100914C0 (de) 2024-08-14
US20220392038A1 (en) 2022-12-08

Similar Documents

Publication Publication Date Title
EP4100914B1 (de) Bildverarbeitungsverfahren
KR102640237B1 (ko) 이미지 프로세싱 방법, 장치, 전자 디바이스 및 컴퓨터 판독가능 저장 매체
CN110717851B (zh) 图像处理方法及装置、神经网络的训练方法、存储介质
CN111798400B (zh) 基于生成对抗网络的无参考低光照图像增强方法及系统
CN105069825B (zh) 基于深度置信网络的图像超分辨率重建方法
US10186023B2 (en) Unified multi-image fusion approach
CN106022237B (zh) 一种端到端的卷积神经网络的行人检测方法
CN109712165B (zh) 一种基于卷积神经网络的同类前景图像集分割方法
CN116681636A (zh) 基于卷积神经网络的轻量化红外与可见光图像融合方法
CN108470320A (zh) 一种基于cnn的图像风格化方法及系统
CN112750201B (zh) 三维重建方法及相关装置、设备
CN112288788B (zh) 单目图像深度估计方法
CN117576402B (zh) 一种基于深度学习的多尺度聚合Transformer遥感图像语义分割方法
CN111899169B (zh) 一种基于语义分割的人脸图像的分割网络的方法
US11615515B2 (en) Superpixel merging
CN115984583B (zh) 数据处理方法、装置、计算机设备、存储介质和程序产品
CN111127309A (zh) 肖像风格迁移模型训练方法、肖像风格迁移方法以及装置
CN110796716A (zh) 一种基于多重残差网络和正则化迁移学习的图像着色方法
WO2024099466A1 (zh) 基于3DLut的图像去雾处理方法、系统及可存储介质
CN114155368A (zh) 一种基于注意力迁移网络的抠图算法
US12118647B2 (en) Generating colorized digital images utilizing a re-colorization neural network with local hints
CN116091823A (zh) 一种基于快速分组残差模块的单特征无锚框目标检测方法
CN115004220A (zh) 用于原始低光图像增强的神经网络
CN115471896B (zh) 证件照制作方法及装置
CN117292015A (zh) 图像编辑方法及相关设备

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20220907

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20230629

RIC1 Information provided on ipc code assigned before grant

Ipc: G06T 5/50 20060101ALI20230623BHEP

Ipc: G06T 3/40 20060101ALI20230623BHEP

Ipc: G06T 7/11 20170101ALI20230623BHEP

Ipc: G06T 5/00 20060101AFI20230623BHEP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Free format text: PREVIOUS MAIN CLASS: G06T0005000000

Ipc: G06T0005600000

Ref country code: DE

Ref legal event code: R079

Ref document number: 602021017268

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: G06T0005000000

Ipc: G06T0005600000

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

RIC1 Information provided on ipc code assigned before grant

Ipc: G06T 5/90 20240101ALI20240215BHEP

Ipc: G06V 10/82 20220101ALI20240215BHEP

Ipc: G06V 10/44 20220101ALI20240215BHEP

Ipc: G06N 3/084 20230101ALI20240215BHEP

Ipc: G06N 3/045 20230101ALI20240215BHEP

Ipc: G06F 18/2413 20230101ALI20240215BHEP

Ipc: G06T 5/70 20240101ALI20240215BHEP

Ipc: G06T 5/60 20240101AFI20240215BHEP

INTG Intention to grant announced

Effective date: 20240314

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602021017268

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

U01 Request for unitary effect filed

Effective date: 20240911

U07 Unitary effect registered

Designated state(s): AT BE BG DE DK EE FI FR IT LT LU LV MT NL PT RO SE SI

Effective date: 20240926

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20241114

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240814

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20241115

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20241214

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240814

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20241114

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240814

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20241114

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240814

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20241114

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20241214

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240814

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20241115

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240814

U20 Renewal fee for the european patent with unitary effect paid

Year of fee payment: 5

Effective date: 20250221

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240814

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240814

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240814

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20250515

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240814

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20250228

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20250205

U20 Renewal fee for the european patent with unitary effect paid

Year of fee payment: 6

Effective date: 20260223

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20260122

Year of fee payment: 6