WO2024099004A1 - 一种图像处理模型训练方法、装置、电子设备、计算机可读存储介质及计算机程序产品 - Google Patents

一种图像处理模型训练方法、装置、电子设备、计算机可读存储介质及计算机程序产品 Download PDF

Info

Publication number
WO2024099004A1
WO2024099004A1 PCT/CN2023/123450 CN2023123450W WO2024099004A1 WO 2024099004 A1 WO2024099004 A1 WO 2024099004A1 CN 2023123450 W CN2023123450 W CN 2023123450W WO 2024099004 A1 WO2024099004 A1 WO 2024099004A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
processing model
image processing
face
swapped
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/123450
Other languages
English (en)
French (fr)
Inventor
贺珂珂
朱俊伟
邰颖
汪铖杰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to EP23887696.5A priority Critical patent/EP4560588A4/en
Publication of WO2024099004A1 publication Critical patent/WO2024099004A1/zh
Priority to US18/813,622 priority patent/US20240420288A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20112—Image segmentation details
    • G06T2207/20132—Image cropping
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Definitions

  • the embodiments of the present application relate to machine learning technology, and in particular, to an image processing model training method, device, electronic device, computer-readable storage medium, and computer program product.
  • the embodiments of the present application provide an image processing model training method, device, electronic device, computer-readable storage medium and computer program product, which can assist in training a first image processing model with a heavy parameterized structure through a pre-trained second image processing model, taking into account both the model performance and the model's computational complexity.
  • the present application embodiment provides an image processing model training method, which is performed by an electronic device and includes:
  • the first training sample set includes at least one triplet training sample
  • the triplet training sample includes: a source image, a template image, and a true value image
  • the first image processing model is trained according to the fusion loss function, and when the training convergence condition of the first image processing model is reached, the model parameters of the first image processing model are determined.
  • the present application also provides an image processing model training device, the training device comprising:
  • a data transmission module is configured to obtain a first training sample set, wherein the first training sample set includes at least one triplet training sample, and the triplet training sample includes: a source image, a template image, and a true value image;
  • an image processing model training module configured to perform face swapping on the source image and the template image through a first image processing model to obtain a first face swapped image, wherein the first image processing model is a re-parameterized structure;
  • the image processing model training module is configured to obtain a second image processing model corresponding to the first image processing model, wherein the second image processing model is a pre-trained image processing model;
  • the image processing model training module is configured to calculate a fusion loss function of the first image processing model according to the second image processing model, the first face-swapped image and the true value image;
  • the image processing model training module is configured to train the first image processing model according to the fusion loss function, and determine the model parameters of the first image processing model when the training convergence condition of the first image processing model is reached.
  • the present application also provides an electronic device, the electronic device comprising:
  • a memory for storing computer executable instructions
  • the processor is used to implement the image processing model training method provided in the embodiment of the present application when running the computer executable instructions stored in the memory.
  • An embodiment of the present application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the image processing model training method provided in the embodiment of the present application.
  • An embodiment of the present application also provides a computer program product, including computer executable instructions, characterized in that when the computer executable instructions are executed by a processor, the image processing model training method provided in the embodiment of the present application is implemented.
  • the embodiment of the present application obtains a triplet training sample including a source image, a template image and a true image, and performs face-swapping on the source image and the template image through a first image processing model to obtain a first face-swapping image.
  • the first image processing model is a re-parameterized structure. Due to the characteristics of the re-parameterized structure, the first image processing model will be more lightweight in the application stage, thereby reducing resource consumption when the model is applied; a pre-trained second image processing model corresponding to the first image processing model is obtained, and a fusion loss function of the first image processing model is calculated according to the second image processing model, the first face-swapping image and the true image.
  • the first image processing model is trained.
  • the model parameters of the first image processing model are determined.
  • the first image processing model finally obtained can achieve lightweight application and can have high accuracy.
  • FIG1 is a schematic diagram of a use environment of an image processing model training method provided in an embodiment of the present application.
  • FIG2 is a schematic diagram of the composition structure of an image processing model training device provided in an embodiment of the present application.
  • FIG3 is a schematic diagram of generating image processing results in a related solution
  • FIG4 is a flow chart of an image processing model training method provided in an embodiment of the present application.
  • FIG5 is a schematic diagram of a facial image acquisition process in an embodiment of the present application.
  • FIG6 is a schematic diagram of a facial image acquisition process in an embodiment of the present application.
  • FIG7 is a schematic diagram of a model structure of a first image processing model in an embodiment of the present application.
  • FIG8 is a schematic diagram of a test process of a first image processing model in an embodiment of the present application.
  • FIG9 is a schematic diagram of the working process of a trained image processing model in an embodiment of the present application.
  • FIG. 10 is a schematic diagram of the face-changing effect in an embodiment of the present application.
  • first ⁇ second ⁇ third involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that “first ⁇ second ⁇ third” can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.
  • Video face swapping The input source image is swapped onto the template face, and the face in the output image retains the expression, angle, background and other information of the template face. The information in the output image other than the face is the same as the source image.
  • Neural Network In the field of machine learning and cognitive science, it is a mathematical model or computational model that imitates the structure and function of biological neural networks and is used to estimate or approximate functions.
  • Model parameters It is a quantity that uses universal variables to establish the relationship between functions and variables. In artificial neural networks, model parameters are usually real number matrices.
  • knowledge transfer refers to using the output data of the training sample data in the intermediate network layer or the final network layer of the teacher network to assist in the training of a student network with faster but poorer performance, thereby migrating the teacher network with good performance to the student network.
  • knowledge distillation refers to the technique of using the smoothed category posterior probabilities output by the teacher network to train the student network in classification problems.
  • Teacher Network A high-performance neural network that provides more accurate supervision information to the student network during the knowledge transfer process.
  • Student Network A single neural network with fast computing speed but poor performance, suitable for deployment in practical application scenarios with high real-time requirements. Compared with the teacher network, the student network has a larger computing throughput and fewer model parameters.
  • Downsampling sampling is performed at intervals of several sample values in the sample sequence, so that the new sequence is the downsampling of the original sequence. For example, for an image I with a size of M*N, it is downsampled s times to obtain a low-resolution image of size (M/s)*(N/s), where s is the common divisor of M and N.
  • Generative Adversarial Networks is a deep learning model that produces better outputs through the mutual game learning of at least two modules in the framework: the generative model G (Generative Model) and the discriminative model D (Discriminative Model).
  • G is a model for making high-resolution images (also called reconstructed images in this article)
  • D is a model for detecting whether it is an original natural image.
  • the goal of G is to make D unable to judge whether the high-resolution image generated by G is an unnatural image.
  • D should try its best to distinguish whether the input image is an original natural image or an unnatural image generated by G.
  • the parameters of G and D are continuously iterated and updated until the generative adversarial network meets the convergence conditions.
  • a generator network used to generate high-resolution images from low-resolution images.
  • the generator can be a convolutional neural network based on deep learning.
  • Discriminator network used to determine whether the input image x is an unnatural image generated by the generator or a natural image.
  • the discriminator outputs a probability value D1(x) in the range of 0-1.
  • D1(x) When D1(x) is 0, it means that the image x input to the discriminator is a natural image.
  • D1(x) When D1(x) is 1, it means that the image x input to the discriminator is an unnatural image.
  • the three-primary color encoding method also known as the RGB color mode, is the color standard in the industry. It obtains various colors by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other. RGB represents the colors of the three channels of red, green, and blue. This standard covers almost all colors that can be perceived by human vision and is one of the most widely used color systems at present.
  • Face swapping using the target part of the object in the image to be processed to replace the part of the object in other images that corresponds to the target part.
  • FIG1 is a schematic diagram of a related art super-resolution processing of an image based on a super-resolution generative adversarial network.
  • the structure of the super-resolution generative adversarial network is shown in FIG1, and includes a generator network 301 and a discriminator network 302.
  • the generator network 301 and the discriminator network 302 are deep neural network models.
  • a high-definition image is used as a training sample image and down-sampled to form a low-resolution (relative to the high-definition image) training sample image.
  • the low-resolution training sample image is reconstructed by the generator network 301 in the super-resolution generative adversarial network model to form a reconstructed image; the discriminator network 302 in the super-resolution generative adversarial network model identifies the reconstructed image, and adjusts the parameters of the generator network 301 and/or the discriminator network 302 according to the corresponding identification results, until the generator network 301 and the discriminator network 302 can reach Nash equilibrium, and the training of the super-resolution generative adversarial network model is completed, so that the super-resolution generative adversarial network model can reconstruct the input image with a lower resolution to form an image with a higher resolution.
  • the above-mentioned solutions of the related art have the following problems: generating high-resolution images requires the model to have a very large number of parameters, such as the Pix2PixHD model has about 100 million parameters.
  • the disadvantage of this large-scale model is that the test speed is slow and it is difficult to deploy on mobile devices, so the image processing model needs to be compressed.
  • the time consumption of the face-changing model is often not considered in the related art.
  • the supervised face-changing model is trained through a complex network structure, resulting in the model's computational complexity being too high and unable to run on mobile devices.
  • the embodiments of the present application are aimed at the computational complexity of the model in the related art and cannot be run on mobile devices.
  • the embodiments of the present application provide an image processing model training method, device, electronic device, computer-readable storage medium and computer program product, which can optimize the model's floating-point transport times per second to 544 megabytes by cutting the model structure, introducing the ideas of structural reparameterization and knowledge distillation, and reducing the computational workload by 94% compared with 9373 megabytes in the related art.
  • the number of frames per second of screen transmission can be between 17 and 20, and the time consumption basically meets the real-time requirements of the mobile terminal.
  • the image processing model training method provided in the embodiment of the present application can be implemented by the terminal/server alone; or it can be implemented by the terminal and the server in collaboration, for example, the terminal alone undertakes the following image processing model training method, or the terminal sends a training request to the server, and the server The image processing model training method is executed according to the received training request.
  • the terminal sends an image processing request to the server.
  • the server generates an image processing result for the target image to be processed by calling the generator network in the set image processing model, and returns the image processing result to the terminal.
  • the electronic device for executing the image processing model training method provided in the embodiment of the present application can be various types of terminal devices or servers, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services; the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto.
  • the terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.
  • the server can be a server cluster deployed in the cloud, opening artificial intelligence cloud services (AI as a Service, AIaaS) to users.
  • AIaaS artificial intelligence cloud services
  • the AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service model is similar to an AI theme mall. All users can access one or more artificial intelligence services provided by the AIaaS platform through an application programming interface.
  • FIG. 2 is a schematic diagram of the use scenario of the image processing model training method provided in an embodiment of the present application.
  • a client of an image processing software is set on the terminal (including terminal 10-1 and terminal 10-2), and the user can input the corresponding image to be processed through the set image processing software client.
  • the image processing client can also receive the corresponding image processing result and display the received image processing result to the user;
  • the terminal is connected to the server 200 through the network 300, and the network 300 can be a wide area network or a local area network, or a combination of the two, and a wireless link is used to realize data transmission.
  • the server 200 is used to set up an image processing model and train the image processing model to iteratively update the generator parameters and discriminator parameters of the image processing model, so as to generate an image processing result for the target image to be processed through the generator network in the image processing model, and display the image processing result corresponding to the image to be processed generated by the image processing model through the terminal (terminal 10-1 and/or terminal 10-2).
  • the image processing model needs to be trained. After the parameters of the image processing model are determined, it is deployed in the mobile terminal for the user to use, and can also be saved in the cloud server network waiting for the user to download and use.
  • the image processing model training method provided in the embodiment of the present application can be implemented based on artificial intelligence.
  • Artificial Intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
  • artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence.
  • Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, testing and decision-making.
  • Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies.
  • Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating/interactive systems, mechatronics, and other technologies.
  • Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, and machine learning/deep learning.
  • FIG3 is a schematic diagram of the composition structure of the image processing model training device provided in the embodiment of the present application, which can be It should be understood that FIG3 only shows an exemplary structure of the image processing model training device rather than the entire structure, and part or all of the structure shown in FIG3 may be implemented as needed.
  • the image processing model training device includes: at least one processor 201, a memory 202, a user interface 203 and at least one network interface 204.
  • the various components in the image processing model training device 20 are coupled together through a bus system 205.
  • the bus system 205 is used to realize the connection and communication between these components.
  • the bus system 205 also includes a power bus, a control bus and a status signal bus.
  • various buses are marked as bus systems 205 in Figure 2.
  • the user interface 203 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad or a touch screen.
  • the memory 202 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories.
  • the memory 202 in the embodiment of the present application can store data to support the operation of the terminal (such as 10-1). Examples of these data include: any computer program for operating on the terminal (such as 10-1), such as an operating system and an application.
  • the operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks.
  • the application can include various applications.
  • the image processing model training device provided in the embodiment of the present application can be implemented in a combination of software and hardware.
  • the image processing model training device provided in the embodiment of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image processing model training method provided in the embodiment of the present application.
  • the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.
  • ASICs application specific integrated circuits
  • DSPs digital signal processor
  • PLDs programmable logic devices
  • CPLDs complex programmable logic devices
  • FPGAs field programmable gate arrays
  • the image processing model training device provided in an embodiment of the present application can be directly embodied as a combination of software modules executed by a processor 201, and the software module can be located in a storage medium, and the storage medium is located in a memory 202.
  • the processor 201 reads the executable instructions included in the software module in the memory 202, and combines with necessary hardware (for example, including a processor 201 and other components connected to a bus 205) to complete the image processing model training method provided in an embodiment of the present application.
  • processor 201 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.
  • DSP digital signal processor
  • the device provided in the embodiment of the present application can be directly executed by a processor 201 in the form of a hardware decoding processor.
  • the image processing model training method provided in the embodiment of the present application can be implemented by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic components.
  • ASICs application specific integrated circuits
  • DSPs digital signal processor
  • PLDs programmable logic devices
  • CPLDs complex programmable logic devices
  • FPGAs field programmable gate arrays
  • the memory 202 in the embodiment of the present application is used to store various types of data to support the operation of the image processing model training device 20.
  • Examples of such data include: any executable instructions for operating on the image processing model training device 20, such as executable instructions, and the program for implementing the image processing model training method of the embodiment of the present application can be included in the executable instructions.
  • the image processing model training device provided in the embodiment of the present application can be implemented in a software manner.
  • FIG. 3 shows a storage
  • the image processing model training device 255 stored in the memory 250 can be software in the form of a program or a plug-in, and includes the following software modules: a data transmission module 2551, an image processing model training module 2552. These modules are logical, and thus can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.
  • FIG. 4 is a flow chart of the image processing model training method provided in an embodiment of the present application. It can be understood that the steps shown in FIG. 4 can be performed by various electronic devices running an image processing model training device, such as a small program running terminal with a face image detection and adjustment function, or a terminal with an image processing model training function. The steps shown in FIG. 4 are described below.
  • Step 401 Obtain the first training sample set
  • the first training sample set includes at least one triplet training sample
  • the triplet training sample includes: a source image, a template image and a true image.
  • a face image in the environment where the mobile terminal is located can be collected as the source image, where the source image can be image A including object A, the template image can be image B including object B, and the true image can be an image in which the face of object B in image B is replaced with the face of object A.
  • FIG. 5 is a schematic diagram of the facial image acquisition process in an embodiment of the present application.
  • dark channel dehazing processing can be performed on the facial image to form an enhanced image.
  • the enhanced image formed may include facial features and/or limb features.
  • the dark channel defogging process here is as follows: determine the dark channel value of the facial image, the grayscale value of the facial image, and the defogging adjustment value; determine the atmospheric light value of the facial image based on the dark channel value, the defogging adjustment value, and the grayscale value of the facial image; process the facial image according to the atmospheric light value and the light adjustment value of the facial image to form an enhanced image.
  • Dark channel refers to the grayscale image formed by taking the minimum value from the three RGB channels of the acquired facial image when the facial image is acquired, and performing minimum value filtering on the grayscale image formed by the minimum value.
  • the defogging adjustment value can be obtained; after converting the acquired facial image into a grayscale image, the grayscale value of the facial image and the dark channel value can be obtained.
  • the dark channel value be Dark_channel
  • the grayscale value of the facial image be Mean_H and Mean_V
  • the atmospheric light value of the facial image be AirLight
  • the dehazing adjustment value be P
  • the light adjustment value be A
  • the facial image to be enhanced be Input
  • a face image acquired by a terminal in the use environment of the first image processing model can be acquired through a mobile terminal; image augmentation processing is performed on the face image; based on the image augmentation processing result, a corresponding face position is determined through a face detection algorithm, and a face image including a background image is intercepted based on the face position; the face image including the background image is processed Cropping is performed to obtain the source image.
  • the face detection algorithm may be an algorithm for detecting the position of a face in an image, such as a face detection and face alignment method based on deep learning.
  • FIG. 6 is a schematic diagram of the facial image acquisition process in an embodiment of the present application. Since the position of the image acquisition device is fixed and the height of the target object is different, the comprehensiveness of the acquired facial image is also different (it may be that the target object is too short or too tall and an accurate facial image cannot be acquired).
  • the acquired facial image can be augmented; based on the processing result of the image augmentation, the corresponding face position is determined by a face detection algorithm, and the facial image including the background image is captured; the facial image including the background image is cleared to form the corresponding facial image of the target user.
  • the face detection technology can be used to frame the area where the user's face is located, and then expand it by 2 times with this area as the center, as shown in Figure 6, the detection area of the detection frame 601 is adjusted to the detection area of the detection frame 602, so as to obtain more background content and crop the facial image including the background content; for example, the following methods can be used: using a face detection algorithm to frame the face position of the target object; using a facial feature positioning algorithm to mark the feature points of the eyes, mouth, nose, etc.; and capturing the facial image including the background content according to the detected face position.
  • the deep processing network may include but is not limited to: LeNet, AlexNet, VGG, Inception series networks, ResNet networks; by extracting features of the face image (for example, extracting grayscale-based features such as mean and variance and features based on distribution histograms, features based on correlation matrices such as GLCM and GLRLM, or signal features after Fourier transform of the image), and performing background removal processing based on the extracted features, a real face cropped by the deep processing network is obtained, and a depth map corresponding to the face is calculated.
  • features of the face image for example, extracting grayscale-based features such as mean and variance and features based on distribution histograms, features based on correlation matrices such as GLCM and GLRLM, or signal features after Fourier transform of the image
  • a real face cropped by the deep processing network is obtained, and a depth map corresponding to the face is calculated.
  • the facial image calculated by using a real human face includes a depth map, while the depth map corresponding to the attack image (such as a face photo) is a black background map.
  • the facial image of the target object can be obtained by restoring the depth map.
  • the obtained facial image of the target object does not include the background image, which can make the face-changing function processing result of the image processing model more accurate.
  • Step 402 Process the first training sample set through a first image processing model to obtain a first face-swapped image, wherein the first image processing model is a re-parameterized structure.
  • the reparameterized structure means that the first image processing model is obtained based on the structural reparameterization technology.
  • Structural reparameterization means first constructing a series of structures (generally used for training) and converting their parameters into another set of parameters (generally used for reasoning), thereby converting this series of structures into another series of structures.
  • the structure during training is larger and has a good property (higher accuracy or other useful properties, such as sparsity), and the converted structure during reasoning is smaller and retains this property (same accuracy or other useful properties).
  • the original meaning of the term "structural reparameterization" is: converting a set of parameters of a structure into another set of parameters, and using the converted parameters to parameterize another structure. As long as the parameter conversion is equivalent, the replacement of the two structures is equivalent.
  • the first training sample set can be denoised to save the training time of the first image processing model and improve the training accuracy of the first image processing model.
  • the usage environment of the trained first image processing model includes: film and television production scenes, game image production scenes, live virtual image production scenes, and ID photo production scenes.
  • an image containing a character object can be used as a source image, and an image containing a game character can be used as a template image.
  • the source image and the template image are input into the trained first image processing model, and a corresponding face-changing image is output.
  • the output face-changing image replaces the identity of the game character in the template image with the identity of the character object in the source image. It can be seen that a unique game character can be designed for a character through a face-changing image.
  • the image containing the virtual image can be used as the source image, and each image frame containing the human object in the live video can be used as the template image and input into the trained first image processing model respectively with the source image, and the corresponding face-changing image is output.
  • the output face-changing image replaces the identity of the human object in the template image with the virtual image. It can be seen that the virtual image can be used to replace the identity in the live broadcast scene to enhance the fun of the live broadcast scene.
  • the image of the object for which the ID photo needs to be made can be used as the source image, the source image and the ID photo template image are input into the trained first image processing model, and the corresponding face-changing image is output.
  • the output face-changing image replaces the identity of the template object in the ID photo template image with the object for which the ID photo needs to be made. It can be seen that through the face-changing image, the object for which the ID photo needs to be made can directly make the ID photo by providing an image without shooting, which greatly reduces the production cost of the ID photo.
  • a dynamic noise threshold that matches the usage environment of the first image processing model can be determined; the first training sample set is denoised according to the dynamic noise threshold to form a second training sample set that matches the dynamic noise threshold, thereby ensuring the training accuracy of the image processing model.
  • a fixed noise threshold corresponding to the second image processing model is determined, and the first training sample set is denoised according to the fixed noise threshold to form a second training sample set that matches the fixed noise threshold, which can further compress the training time of the image processing model.
  • Step 403 Obtain a second image processing model corresponding to the first image processing model, wherein the second image processing model is a pre-trained image processing model, and the model parameters of the second image processing model remain unchanged during subsequent training.
  • the second image processing model may be a trained neural network, and the second image processing model may be a large-scale neural network, for example, the number of network parameters of the second image processing model is greater than a certain value, but the embodiment of the present application does not limit this.
  • the second image processing model may be a convolutional neural network (CNN Convolutional Neural Network), a deep neural network (DNN Deep Neural Network), a recurrent neural network (RNN Recurrent Neural Network), etc., and the embodiment of the present application does not limit the type of the second image processing model.
  • the second image processing model may be a neural network applicable to different computer vision tasks, such as: target recognition tasks, target classification tasks, target detection tasks or posture estimation tasks, etc.
  • the second image processing model may also be a neural network applicable to different application scenarios, such as: security detection scenarios, face unlocking scenarios, intelligent driving or remote sensing scenarios, etc., and the embodiment of the present application does not limit the scope of application of the first image processing model.
  • the network structure of the second image processing model may be designed according to computer vision tasks, or the network structure of the second image processing model may adopt at least a part of the existing network structure, such as: deep residual network, visual geometry group network (VGGNet Visual Geometry Group Network), etc.
  • the first image processing model may be a neural network to be trained, and the first image processing model may be a neural network with a smaller scale. It is convenient to deploy in a mobile terminal by taking advantage of the low floating-point operation amount.
  • the number of network parameters of the first image processing model is less than a certain value (at least the number of network parameters of the first image processing model is less than the number of network parameters of the second image processing model), but the embodiment of the present application is not limited to this.
  • the network scale of the second image processing model is larger than the network scale of the first image processing model.
  • the second image processing model can be a teacher network, and the first image processing model can be a student network. Using the teacher network to train the student network can improve the performance of the trained student network.
  • the first image processing model can be trained using knowledge distillation methods or other methods, which is not limited to this in the embodiment of the present application.
  • FIG7 is a schematic diagram of a model structure of the first image processing model in an embodiment of the present application, wherein the structure of the encoder and decoder in the model of the first image processing model is a re-parameterized structure RepVGG (Visual Geometry Group), as shown in FIG7 ,
  • a in FIG7 represents the original ResNet network, which contains the residual structure of Conv1*1 and the residual structure of Identity. The existence of these residual structures solves the gradient vanishing problem in the deep network and makes the network easier to converge.
  • FIG7 B represents the RepVGG network architecture in the training phase, in which the main body of the entire network contains the residual structure. At the same time, the residual blocks in the RepVGG network do not cross layers, and the entire network contains two residual structures.
  • the first image processing model uses the structure shown in FIG7 , which is actually similar to training multiple networks and integrating multiple networks into one network, which has higher training efficiency.
  • C in Figure 7 represents the RepVGG network in the test phase.
  • the structure of the network is very simple. The entire network is formed by the following network Conv3*3+Relu connection, which is easy to test and accelerate the model.
  • Step 404 Calculate the fusion loss function of the first image processing model based on the second image processing model and the first face-swapped image.
  • the fusion loss function of the first image processing model is composed of a combination of different loss functions, and a second face-changing image output by the second image processing model is obtained, and the reconstruction loss function of the first image processing model is calculated using the first face-changing image and the second face-changing image;
  • the feature loss function of the first image processing model is calculated based on the first face-changing image and the second face-changing image;
  • the estimation loss function of the first image processing model is calculated based on the first face-changing image and the source image;
  • the adversarial loss function of the first image processing model is calculated based on the first face-changing image and the true image;
  • the sum of the reconstruction loss function, the feature loss function, the estimation loss function and the adversarial loss function is calculated to obtain the fusion loss function of the first image processing model;
  • the training effect of the first image processing model can be improved from multiple dimensions, thereby improving the face-changing accuracy of the first image processing model.
  • Reconstruction_loss is the reconstruction loss function
  • LPIPS_loss is the feature loss function
  • ID_loss is the estimation loss function
  • D_loss is the discriminator loss
  • G_loss is the generator loss
  • (D_loss+G_loss) constitutes the adversarial loss function.
  • the second face-changing image calculated by the second image processing model can be represented as BigModel_fake, and the first face-changing image calculated by the first image processing model can be represented as fake; through the embodiments of the present application, the training effect of the second image processing model can be transferred to the first image processing model, thereby playing a teaching role.
  • BigModel_fake is the second face-swapped image
  • BigModel_swap represents the forward processing process of the second face-swapped model
  • source is the source image
  • template is the template image
  • Reconstruction_loss is the reconstruction loss function
  • fake is the first face-swapped image.
  • LPIPS_loss
  • alexnet_feature(fake) means inputting the first face image (fake) into the alexnet network model and outputting the features output by the four feature extraction layers (corresponding to different levels) of the alexnet network model
  • result_fea1, result_fea2, result_fea3 and result_fea4 are the decoded face features of the first face-changing image output by each of the four feature extraction layers.
  • alexnet_feature(gt_img) means inputting the second face-swapped image gt_img into the alexnet network model and outputting the features of gt_img output by the four feature extraction layers (corresponding to different levels) of the alexnet network model.
  • gt_img_fea1, gt_img_fea2, gt_img_fea3 and gt_img_fea4 are the standard facial features of the second face-swapped image gt_img output by each of the four feature extraction layers.
  • ID_loss 1 - cosine_similarity (fake_id_features, true_id_features) (4);
  • ID_loss is the estimated loss function
  • fake_id_features is the feature vector of the first face-swapped image
  • socre_id_features is the feature vector of the source image
  • cosine_similarity is the cosine similarity
  • the generator network which can be called G
  • the discriminator network which can be called D
  • D the discriminator network
  • a high-resolution image x is input and a number D(x) in the range of 0-1 is output.
  • D(x) is used to determine whether the input image is generated by the generator, where 0 means no and 1 means yes.
  • D_loss is the discriminator loss
  • G_loss is the generator loss
  • (D_loss+G_loss) constitutes the adversarial loss function loss.
  • the calculation of the adversarial loss function refers to formula (5):
  • D_loss is the discriminator loss
  • G_loss is the generator loss
  • D(gt_img) is the discriminant result of the discriminator output for the true value image
  • D(fake) is the discriminant result of the discriminator output for the first face-swapped image
  • loss is the adversarial loss function.
  • the discriminant result here can be a probability, that is, the probability of belonging to the real image.
  • Step 405 Train the first image processing model according to the fusion loss function, and when the training convergence condition of the first image processing model is reached, determine the model parameters of the first image processing model.
  • the training convergence condition here may be that a set number of training times is reached, or that the fusion loss function converges to a minimum value.
  • step 401-step 405 the training of the first image processing model is completed, and the parameters of the first image processing model are determined.
  • the trained first image processing model can be deployed in the mobile terminal to perform the face-changing function.
  • the floating-point operation FLOPs of the first image processing model is optimized to 544 megabytes, which is 94% less than the 9373 megabytes of FLOPs of the second image processing model.
  • the number of frames per second can be transmitted between 17 and 20, so that the time consumption of the face-changing function meets the real-time requirements of the mobile terminal.
  • FIG. 9 is a schematic diagram of the working process of the trained image processing model in an embodiment of the present application, specifically including the following steps:
  • Step 901 When the image processing model training is completed and deployed in the mobile terminal, the target face image (corresponding to the source image) and the face image to be replaced (corresponding to the template image) are obtained.
  • Step 902 Encode the target face image and the face image to be replaced through the encoder network of the image processing model to obtain a face image vector.
  • Step 903 Decode the face image vector through the decoder network of the image processing model to obtain the face-swapped image.
  • the generator uses asymmetric input and output. Since the screen of the mobile terminal is small, the decoder downgrades the network output resolution from 512 pixels to 256 pixels, and designs the input resolution to 128 pixels to meet the use of mobile terminals.
  • the encoder network uses convolution to halve the input and gradually increase the number of channels. Specifically, the input is encoded from 128*128*6 (the target face image and the face image to be replaced, each with 3 RGB channels) to 64*64*32, 32*32*64, 16*16*128, and so on.
  • the decoder network gradually doubles the resolution through deconvolution operations and decodes it to 32*32*64, 64*64*32, 128*128*16, 256*256*3, and finally obtains the face-changing result.
  • FIG10 is a schematic diagram of the face-changing effect in an embodiment of the present application.
  • the target source facial image may be, for example, the facial image (a) in FIG10
  • the target template facial image may be, for example, the facial image (b) in FIG10
  • the facial replacement image may be, for example, the facial image (c) in FIG4 .
  • the facial image (c) is obtained by replacing the face in the facial image (a) with the face in the facial image (b). It can be seen from the facial image (c) that the identity and additional image of the facial image (c) are consistent with those of the facial image (b), that is, the facial image (c) and the facial image (b) are of the same person. and face image (c) includes the same glasses as face image (b).
  • the attributes of face image (c) are consistent with those of face image (a). For example, it can be seen from face image (c) that the hairstyle of face image (c) is consistent with that of face image (a), and the angle of the mouth opening of face image (c) is larger than that of face image (b), thereby meeting the angle of the mouth opening of face image (a), thereby achieving the face-changing processing effect required by the user.
  • the software module stored in the image processing model training device in the memory may include: a data transmission module 2081, configured to obtain a first training sample set, wherein the first training sample set includes at least one triple training sample, and the triple training sample includes: a source image, a template image, and a true value image; an image processing model training module 2082, configured to perform face-swapping on the source image and the template image through a first image processing model to obtain a first face-swapping image, wherein the first image processing model is a re-parameterized structure; the image processing model training module 2082 is configured to obtain a second image processing model corresponding to the first image processing model, wherein the second image processing model is a pre-trained image processing model; the image processing model training module 2082 is configured to calculate the fusion loss function of the first image processing model according to the second image processing
  • the image processing model training module 2082 is further configured to determine a dynamic noise threshold that matches the usage environment of the first image processing model; denoise the first training sample set according to the dynamic noise threshold to form a second training sample set that matches the dynamic noise threshold; or, determine a fixed noise threshold corresponding to the second image processing model, and denoise the first training sample set according to the fixed noise threshold to form a second training sample set that matches the fixed noise threshold.
  • the image processing model training module 2082 is also configured to obtain a facial image collected by a terminal in the use environment of the first image processing model; perform image augmentation processing on the facial image; determine the corresponding facial position based on the image augmentation processing result, and capture the facial image including the background image based on the facial position; and crop the facial image including the background image to obtain the source image.
  • the image processing model training module 2082 is further configured to obtain a second face-swapped image output by the second image processing model, and use the first face-swapped image and the second face-swapped image to calculate the reconstruction loss function of the first image processing model; calculate the feature loss function of the first image processing model; calculate the estimation loss function of the first image processing model; calculate the adversarial loss function of the first image processing model; and fuse the reconstruction loss function, the feature loss function, the estimation loss function and the adversarial loss function to obtain the fusion loss function of the first image processing model.
  • the image processing model training module 2082 is further configured to calculate the pixel level difference between the first face-swapped image and the second face-swapped image; and determine the reconstruction loss function of the first image processing model based on the pixel level difference.
  • the image processing model training module 2082 is further configured to extract features from the first face-swapped image through a pre-trained feature extraction network to obtain features of multiple levels of the first face-swapped image;
  • the second face-swapped image is subjected to feature extraction to obtain features of multiple levels of the second face-swapped image; and a feature loss function of the first image processing model is determined based on the difference between the features of multiple levels of the first face-swapped image and the features of multiple levels of the second face-swapped image.
  • the image processing model training module 2082 is further configured to extract a first face-swapped image feature vector of the first face-swapped image; extract a source image feature vector of the source image; and calculate an estimated loss function of the first image processing model using the similarity between the first face-swapped image feature vector and the source image feature vector.
  • the image processing model training module 2082 is also configured to obtain a target face image and a face image to be replaced when the first image processing model is trained and deployed in a mobile terminal; encode the target face image and the face image to be replaced through the encoder network of the first image processing model to obtain a face image vector; and decode the face image vector through the decoder network of the first image processing model to obtain a third face-changing image.
  • the embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer executable instruction is stored in a computer readable storage medium.
  • the processor of the electronic device reads the computer executable instruction from the computer readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the image processing model training method described in the embodiment of the present application.
  • An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein computer-executable instructions are stored.
  • the processor will execute the image processing model training method provided by the embodiment of the present application.
  • the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.
  • computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.
  • computer-executable instructions may, but do not necessarily, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).
  • HTML HyperText Markup Language
  • computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.
  • the embodiment of the present application obtains a first training sample set, wherein the first training sample set includes at least one triplet training sample, and the triplet training sample includes: a source image, a template image, and a true value; the first training sample set is processed by a first image processing model to obtain a first face-changing image, wherein the first image processing model is a re-parameterized structure, and a second image processing model corresponding to the first image processing model is obtained, wherein the second image processing model is a pre-trained image processing model, and the model parameters of the second image processing model are fixed; the fusion loss function of the first image processing model is calculated according to the second image processing model and the first face-changing image; the first image processing model is trained according to the fusion loss function, and when the convergence condition of the first image processing model is reached, the model parameters of the first image processing model are determined.
  • the first image processing model is a re-parameterized structure
  • the structure of the first image processing model is complex when it is trained.
  • the first image processing model has strong processing capabilities and can learn complex data, while the structure is simple during testing, which can reduce the time consumption during testing and reduce the amount of floating-point operations, making it easier to deploy on mobile terminals.
  • using the second image processing model for training guidance can steadily improve the accuracy of smaller-scale image processing models without increasing the total amount of training samples and without the need for retraining. It is also generally applicable to most neural network models and data.
  • the training of smaller-scale image processing models takes into account the training accuracy while reducing the overfitting of the neural network model and enhancing the generalization ability of the neural network model, making it easier to deploy the image processing model in mobile terminals and realize large-scale application of image processing models.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Computing Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

本申请实施例提供了一种图像处理模型训练方法，所述方法由电子设备执行，包括：获取第一训练样本集合，其中，第一训练样本集合包括至少一个三元组训练样本，三元组训练样本包括：源图像、模板图像和真值图像；通过第一图像处理模型对源图像以及模板图像进行换脸，得到第一换脸图像，其中，第一图像处理模型为重参数化结构；获取与第一图像处理模型相对应的第二图像处理模型，其中，第二图像处理模型为经过预训练的图像处理模型；根据第二图像处理模型、第一换脸图像以及真值图像，计算第一图像处理模型的融合损失函数；根据融合损失函数，对第一图像处理模型进行训练，当达到第一图像处理模型的训练收敛条件时，确定第一图像处理模型的模型参数。

Description

一种图像处理模型训练方法、装置、电子设备、计算机可读存储介质及计算机程序产品
相关申请的交叉引用
本申请基于申请号为202211397807.4、申请日为2022年11月09日的中国专利申请提出，并要求中国专利申请的优先权，中国专利申请的全部内容在此引入本申请作为参考。
技术领域
本申请实施例涉及机器学习技术，尤其涉及一种图像处理模型训练方法、装置、电子设备、计算机可读存储介质及计算机程序产品。
背景技术
相关技术中，深度神经网络在许多计算机视觉任务中获得了非常好的性能。一般而言，网络的参数量和计算量越大，网络的性能越好。然而在资源受限的嵌入式系统上部署这类规模较大的网络却非常困难，而直接训练规模较小的网络得到的网络的性能要远低于规模较大的网络的性能。因此相关技术中难以在计算量较小的情况下实现性能最优。
发明内容
本申请实施例提供一种图像处理模型训练方法、装置、电子设备、计算机可读存储介质及计算机程序产品，能够通过预训练的第二图像处理模型对具有重参数化结构的第一图像处理模型进行辅助训练，兼顾模型性能以及模型的计算量。
本申请实施例的技术方案是这样实现的：
本申请实施例提供了一种图像处理模型训练方法，所述方法由电子设备执行，所述方法包括：
获取第一训练样本集合，其中，所述第一训练样本集合包括至少一个三元组训练样本，所述三元组训练样本包括：源图像、模板图像和真值图像；
通过第一图像处理模型对所述源图像以及所述模板图像进行换脸，得到第一换脸图像，其中，所述第一图像处理模型为重参数化结构；
获取与所述第一图像处理模型相对应的第二图像处理模型，其中，所述第二图像处理模型为经过预训练的图像处理模型；
根据所述第二图像处理模型、所述第一换脸图像以及所述真值图像，计算所述第一图像处理模型的融合损失函数；
根据所述融合损失函数，对所述第一图像处理模型进行训练，当达到所述第一图像处理模型的训练收敛条件时，确定所述第一图像处理模型的模型参数。
本申请实施例还提供了一种图像处理模型训练装置，所述训练装置包括：
数据传输模块，配置为获取第一训练样本集合，其中所述第一训练样本集合包括至少一个三元组训练样本，所述三元组训练样本包括：源图像、模板图像和真值图像；
图像处理模型训练模块，配置为通过第一图像处理模型对所述源图像以及所述模板图像进行换脸，得到第一换脸图像，其中，所述第一图像处理模型为重参数化结构；
所述图像处理模型训练模块，配置为获取与所述第一图像处理模型相对应的第二图像处理模型，其中，所述第二图像处理模型为经过预训练的图像处理模型；
所述图像处理模型训练模块，配置为根据所述第二图像处理模型、所述第一换脸图像以及所述真值图像，计算所述第一图像处理模型的融合损失函数；
所述图像处理模型训练模块，配置为根据所述融合损失函数，对所述第一图像处理模型进行训练，当达到所述第一图像处理模型的训练收敛条件时，确定所述第一图像处理模型的模型参数。
本申请实施例还提供了一种电子设备，所述电子设备包括：
存储器，用于存储计算机可执行指令；
处理器，用于运行所述存储器存储的计算机可执行指令时，实现本申请实施例提供的图像处理模型训练方法。
本申请实施例还提供了一种计算机可读存储介质，存储有计算机可执行指令，所述计算机可执行指令被处理器执行时实现本申请实施例提供的图像处理模型训练方法。
本申请实施例还提供了一种计算机程序产品，包括计算机可执行指令，其特征在于，所述计算机可执行指令被处理器执行时实现本申请实施例提供的图像处理模型训练方法。
本申请实施例具有以下有益效果：
本申请实施例获取包括源图像、模板图像和真值图像的三元组训练样本，并通过第一图像处理模型对源图像以及模板图像进行换脸，得到第一换脸图像，第一图像处理模型为重参数化结构，由于结构重参数化的特性会导致第一图像处理模型应用阶段更加轻量化，从而降低模型应用时的资源消耗；获取与第一图像处理模型相对应的经过预训练的第二图像处理模型，并根据第二图像处理模型、第一换脸图像以及真值图像，计算第一图像处理模型的融合损失函数，这里相当于利用预训练得到的第二图像处理模型进行协助训练，从而有助于第一图像处理模型学习到第二图像处理模型的能力，最后根据融合损失函数，对第一图像处理模型进行训练，当达到第一图像处理模型的训练收敛条件时，确定第一图像处理模型的模型参数，最终得到的第一图像处理模型可以实现轻量化的应用，并且能够具有较高的准确度。
附图说明
图1是本申请实施例提供的一种图像处理模型训练方法的使用环境示意图；
图2为本申请实施例提供的图像处理模型训练装置的组成结构示意图；
图3为相关方案中生成图像处理结果的示意图；
图4为本申请实施例提供的图像处理模型训练方法一个流程示意图；
图5为本申请实施例中脸部图像的采集过程的示意图；
图6为本申请实施例中脸部图像的采集过程的示意图；
图7为本申请实施例中第一图像处理模型的一个模型结构示意图；
图8为本申请实施例中第一图像处理模型的测试过程示意图；
图9为本申请实施例中经过训练的图像处理模型的工作过程示意图；
图10为本申请实施例中换脸效果示意图。
具体实施方式
为了使本申请实施例的目的、技术方案和优点更加清楚，下面将结合附图对本申请实施例作进一步地详细描述，所描述的实施例不应视为对本申请实施例的限制，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其它实施例，都属于本申请实施例保护的范围。
在以下的描述中，涉及到“一些实施例”，其描述了所有可能实施例的子集，但是可以理解，“一些实施例”可以是所有可能实施例的相同子集或不同子集，并且可以在不冲突的情况下相互结合。
在以下的描述中，所涉及的术语“第一\第二\第三”仅仅是是区别类似的对象，不代表针对对象的特定排序，可以理解地，“第一\第二\第三”在允许的情况下可以互换特定的顺序或先后次序，以使这里描述的本申请实施例能够以除了在这里图示或描述的以外的顺序实施。
除非另有定义，本文所使用的所有的技术和科学术语与属于本申请的技术领域的技术人员通常理解的含义相同。本文中所使用的术语只是为了描述本申请实施例的目的，不是旨在限制本申请。
对本申请实施例进行进一步详细说明之前，对本申请实施例中涉及的名词和术语进行说明，本申请实施例中涉及的名词和术语适用于如下的解释。
1)视频换脸：将输入的源图换到模板人脸中上，并使输出图像中的人脸保持模板人脸的表情、角度、背景等信息，输出图像中除人脸之外的信息与源图相同。
2)神经网络：在机器学习和认知科学领域，是一种模仿生物神经网络结构和功能的数学模型或计算模型，用于对函数进行估计或近似。
3)模型参数：是使用通用变量来建立函数和变量之间关系的一个数量。在人工神经网络中，模型参数通常是实数矩阵。
4)知识迁移(Knowledge Transfer)：在深度神经网络中，知识迁移是指利用训练样本数据在教师网络的中间网络层或最终网络层的输出数据，辅助训练速度较快但性能较差的学生网络，从而将性能优良的教师网络迁移到学生网络上。
5)知识提取(Knowledge Distill)：在深度神经网络中，知识提取是指在分类问题中利用教师网络输出的平滑类别后验概率训练学生网络的技术。
6)教师网络(Teacher Network)：在知识迁移过程中为学生网络提供更加准确的监督信息的高性能神经网络。
7)学生网络-(Student Network)：计算速度快但性能较差的适合部署到对实时性要求较高的实际应用场景中的单个神经网络，学生网络相比于教师网，具有更大的运算吞吐量和更少的模型参数。
8)下采样处理，在样值序列中间隔几个样值进行采样，这样得到新序列就是原序列的下采样，例如：对于一幅图像I尺寸为M*N，对其进行s倍下采样，即得到(M/s)*(N/s)尺寸的低分辨率图像，这里s 是M和N的公约数。
9)生成对抗网络(Generative Adversarial Networks，GAN)，是一种深度学习模型，模型通过框架中至少两个模块：生成模型G(Generative Model)和判别模型D(Discriminative Model)的互相博弈学习产生更好的输出。比如：G是制作高分图像(本文中也称为重构图像)的模型，D是检测是否为原始自然图像的模型。G的目标是使D判断不出G生成的高分图像是否为非自然图像，D则要尽可能的分辨出输入的图像是原始自然图像还是G生成的非自然图像，通过G和D的参数不断的迭代更新，直到生成对抗网络满足收敛条件。
10)生成器网络，用于通过低分辨率的图像生成高分辨率图像，生成器可为基于深度学习的卷积神经网络。
11)判别器网络，用于判断输入的图像x是生成器生成的非自然图像，还是自然图像。判别器输出一个0-1区间的概率值D1(x)，当D1(x)为0时，表示判别器输入的图像x是自然图像，当D1(x)为1时，表示判别器输入的图像x是非自然图像。
12)三原色编码方法，又可以称作RGB色彩模式，是工业界的颜色标准，是通过对红(R)、绿(G)、蓝(B)三个颜色通道的变化以及它们相互之间的叠加来得到各式各样的颜色，RGB即是代表红、绿、蓝三个通道的颜色，这个标准几乎包括了人类视力所能感知的所有颜色，是目前运用最广的颜色系统之一。
13)换脸，利用待处理图像中对象的目标部位替换其他图像的对象中与该目标部位相对应的部位。
图1是相关技术基于超分生成对抗网络对图像进行超分处理的示意图，超分生成对抗网络的结构如图1所示，包括有生成器网络301和判别器网络302。生成器网络301和判别器网络302为深度神经网络模型。高清图片被作为训练样本图像并进行下采样处理，形成低分辨率(相对于高清图片而言)的训练样本图像，通过超分生成对抗网络模型中的生成器网络301对低分辨率的训练样本图像进行图像重构，形成重构图像；超分生成对抗网络模型中的判别器网络302对重构图像进行鉴别，根据相应的鉴别结果调整生成器网络301和/或判别器网络302的参数，直至生成器网络301和判别器网络302能够达到纳什平衡，完成对超分生成对抗网络模型的训练，从而超分生成对抗网络模型能够对所输入的分辨率较低的图像进行重构，形成分辨率较高的图像。
相关技术的上述方案存在的问题包括：生成高分辨率的图像需要模型拥有非常大参数量，比如Pix2PixHD模型大约有100百万的参数。但是这种规模较大的模型缺点在于测试速度慢，难以部署在移动设备，因此需要对图像处理模型进行压缩。相关技术中往往不考虑换脸模型的耗时，通过复杂的网络结构，进行有监督的换脸模型的训练，导致模型的计算复杂度太高，无法在移动端设备上运行。本申请实施例针对相关技术中模型的计算度复杂无法在移动端设备上运行，本申请实施例提供一种图像处理模型训练方法、装置、电子设备、计算机可读存储介质及计算机程序产品，能够通过在模型结构上剪裁，引入结构重参数化和知识蒸馏的思想，将模型的每秒浮点运输次数优化至544兆，同比相关技术中9373兆，减少了94％的计算量。最终部署到手机后可以实现每秒画面传输帧数在17-20之间，耗时基本满足移动端实时性要求。
本申请实施例所提供的图像处理模型训练方法，可以由终端/服务器独自实现；也可以由终端和服务器协同实现，例如终端独自承担下文的图像处理模型训练方法，或者，终端向服务器发送训练请求，服务器 根据接收的训练请求执行图像处理模型训练方法，终端向服务器发送图像处理请求，服务器通过调用设置的图像处理模型中的生成器网络生成针对目标待处理图像的图像处理结果，并将图像处理结果返回至终端。
本申请实施例提供的用于执行图像处理模型训练方法的电子设备可以是各种类型的终端设备或服务器，其中，服务器可以是独立的物理服务器，也可以是多个物理服务器构成的服务器集群或者分布式系统，还可以是提供云计算服务的云服务器；终端可以是智能手机、平板电脑、笔记本电脑、台式计算机、智能音箱、智能手表等，但并不局限于此。终端以及服务器可以通过有线或无线通信方式进行直接或间接地连接，本申请在此不做限制。
以服务器为例，例如可以是部署在云端的服务器集群，向用户开放人工智能云服务(AI as a Service，AIaaS)，AIaaS平台会把几类常见的AI服务进行拆分，并在云端提供独立或者打包的服务，这种服务模式类似于一个AI主题商城，所有的用户都可以通过应用程序编程接口的方式来接入使用AIaaS平台提供的一种或者多种人工智能服务。
图2为本申请实施例提供的图像处理模型训练方法的使用场景示意图，参考图2，终端(包括终端10-1和终端10-2)上设置有图像处理软件的客户端，用户通过所设置的图像处理软件客户端可以输入相应的待处理图像，图像处理客户端也可以接收相应的图像处理结果，并将所接收的图像处理结果向用户进行展示；终端通过网络300连接服务器200，网络300可以是广域网或者局域网，又或者是二者的组合，使用无线链路实现数据传输。
在一些实施例中，服务器200用于设置图像处理模型并对图像处理模型进行训练，以对图像处理模型的生成器参数和判别器参数进行迭代更新，以通过图像处理模型中的生成器网络生成针对目标待处理图像的图像处理结果，并通过终端(终端10-1和/或终端10-2)展示图像处理模型所生成的与待处理图像相对应的图像处理结果。当然在通过图像处理模型对目标待处理图像进行处理以生成相应的图像处理结果之前，还需要对图像处理模型进行训练，当确定图像处理模型的参数之后，部署在移动终端中供用户使用，也可以保存在云服务器网络中等待用户的下载使用。
本申请实施例所提供的图像处理模型训练方法可以基于人工智能实现，人工智能(Artificial Intelligence，AI)是利用数字计算机或者数字计算机控制的机器模拟、延伸和扩展人的智能，感知环境、获取知识并使用知识获得最佳结果的理论、方法、技术及应用系统。换句话说，人工智能是计算机科学的一个综合技术，它企图了解智能的实质，并生产出一种新的能以人类智能相似的方式做出反应的智能机器。人工智能也就是研究各种智能机器的设计原理与实现方法，使机器具有感知、测试与决策的功能。
人工智能技术是一门综合学科，涉及领域广泛，既有硬件层面的技术也有软件层面的技术。人工智能基础技术一般包括如传感器、专用人工智能芯片、云计算、分布式存储、大数据处理技术、操作/交互系统、机电一体化等技术。人工智能软件技术主要包括计算机视觉技术、语音处理技术、自然语言处理技术以及机器学习/深度学习等几大方向。
下面对本申请实施例的图像处理模型训练装置的结构做详细说明，图像处理模型训练装置可以各种形式来实施，如带有图像处理模型训练功能的专用终端，也可以为设置有图像处理模型训练功能的服务器，例如前序图2中的服务器200。图3为本申请实施例提供的图像处理模型训练装置的组成结构示意图，可 以理解，图3仅仅示出了图像处理模型训练装置的示例性结构而非全部结构，根据需要可以实施图3示出的部分结构或全部结构。
本申请实施例提供的图像处理模型训练装置包括：至少一个处理器201、存储器202、用户接口203和至少一个网络接口204。图像处理模型训练装置20中的各个组件通过总线系统205耦合在一起。可以理解，总线系统205用于实现这些组件之间的连接通信。总线系统205除包括数据总线之外，还包括电源总线、控制总线和状态信号总线。但是为了清楚说明起见，在图2中将各种总线都标为总线系统205。
其中，用户接口203可以包括显示器、键盘、鼠标、轨迹球、点击轮、按键、按钮、触感板或者触摸屏等。
可以理解，存储器202可以是易失性存储器或非易失性存储器，也可包括易失性和非易失性存储器两者。本申请实施例中的存储器202能够存储数据以支持终端(如10-1)的操作。这些数据的示例包括：用于在终端(如10-1)上操作的任何计算机程序，如操作系统和应用程序。其中，操作系统包含各种系统程序，例如框架层、核心库层、驱动层等，用于实现各种基础业务以及处理基于硬件的任务。应用程序可以包含各种应用程序。
在一些实施例中，本申请实施例提供的图像处理模型训练装置可以采用软硬件结合的方式实现，作为示例，本申请实施例提供的图像处理模型训练装置可以是采用硬件译码处理器形式的处理器，其被编程以执行本申请实施例提供的图像处理模型训练方法。例如，硬件译码处理器形式的处理器可以采用一个或多个应用专用集成电路(ASIC，Application Specific Integrated Circuit)、DSP、可编程逻辑器件(PLD，Programmable Logic Device)、复杂可编程逻辑器件(CPLD，Complex Programmable Logic Device)、现场可编程门阵列(FPGA，Field-Programmable Gate Array)或其他电子元件。
作为本申请实施例提供的图像处理模型训练装置采用软硬件结合实施的示例，本申请实施例所提供的图像处理模型训练装置可以直接体现为由处理器201执行的软件模块组合，软件模块可以位于存储介质中，存储介质位于存储器202，处理器201读取存储器202中软件模块包括的可执行指令，结合必要的硬件(例如，包括处理器201以及连接到总线205的其他组件)完成本申请实施例提供的图像处理模型训练方法。
作为示例，处理器201可以是一种集成电路芯片，具有信号的处理能力，例如通用处理器、数字信号处理器(DSP，Digital Signal Processor)，或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等，其中，通用处理器可以是微处理器或者任何常规的处理器等。
作为本申请实施例提供的图像处理模型训练装置采用硬件实施的示例，本申请实施例所提供的装置可以直接采用硬件译码处理器形式的处理器201来执行完成，例如，被一个或多个应用专用集成电路(ASIC，Application Specific Integrated Circuit)、DSP、可编程逻辑器件(PLD，Programmable Logic Device)、复杂可编程逻辑器件(CPLD，Complex Programmable Logic Device)、现场可编程门阵列(FPGA，Field-Programmable Gate Array)或其他电子元件执行实现本申请实施例提供的图像处理模型训练方法。
本申请实施例中的存储器202用于存储各种类型的数据以支持图像处理模型训练装置20的操作。这些数据的示例包括：用于在图像处理模型训练装置20上操作的任何可执行指令，如可执行指令，实现本申请实施例的从图像处理模型训练方法的程序可以包含在可执行指令中。
在一些实施例中，本申请实施例提供的图像处理模型训练装置可以采用软件方式实现，图3示出了存 储在存储器250中的图像处理模型训练装置255，其可以是程序和插件等形式的软件，包括以下软件模块：数据传输模块2551、图像处理模型训练模块2552，这些模块是逻辑上的，因此根据所实现的功能可以进行任意的组合或进一步拆分。将在下文中说明各个模块的功能。
参见图4，图4为本申请实施例提供的图像处理模型训练方法一个流程示意图，可以理解地，图4所示的步骤可以由运行图像处理模型训练装置的各种电子设备执行，例如可以是如带有人脸图像检测与调整功能的小程序运行终端、或者是带有图像处理模型训练功能的终端。下面针对图4示出的步骤进行说明。
步骤401：获取第一训练样本集合
作为示例，第一训练样本集合包括至少一个三元组训练样本，三元组训练样本包括：源图像、模板图像和真值图像，可以采集移动终端所处环境中的人脸图像作为源图像，这里的源图像可以是包括对象A的图像A，模板图像可以是包括对象B的图像B，真值图像可以是将图像B中对象B的脸部替换为对象A的脸部的图像。
参考图5，图5为本申请实施例中脸部图像的采集过程的示意图，当图像采集设备的拍摄环境光线较暗时，可以对脸部图像中进行暗通道去雾处理，形成增强图像，所形成的增强图像可以包括人脸特征和/或肢体特征。
这里的暗通道去雾处理的过程如下：确定脸部图像的暗通道值，脸部图像的灰度值，去雾调节值；基于脸部图像的暗通道值、去雾调节值和灰度值确定脸部图像的大气光值；根据脸部图像的大气光值和光线调节值对脸部图像进行处理，以形成增强图像。
暗通道是指在对脸部图像进行采集时，所获取的脸部图像的RGB三个通道中取最小值组成灰度图，并对最小值组成灰度图进行最小值滤波处理所得到的。通过对移动终端所采集的脸部图像的图像参数进行分析，可以获得去雾调节值；将所采集的脸部图像转换为灰度图后，可以得到脸部图像的灰度值以及暗通道值。
记暗通道值为Dark_channel，脸部图像的灰度值为Mean_H和Mean_V，脸部图像的大气光值为AirLight；去雾调节值为P，光线调节值为A，待增强的脸部图像为Input，其取相反数的结果为IR，对于任意一幅输入图像，取输入图像的暗通道图像灰度值最大的M％个像素点，确定M％个像素点对应于每个通道的灰度值的平均值，其中M的取值范围为0.1-0.3，从而计算出每个通道的大气光值，即大气光值AirLight是一个三元素向量，每一个元素对应于每一个颜色通道。
在一些实施例中，对脸部图像进行采集时，可以确定脸部图像的每个像素点三个通道中的最小值；将去雾图像的每个像素点三个通道中的最小值赋值给暗通道的图像中对应的像素点，其中，通过公式：Dark_channel＝min(Input_R，Input_G，Input_B)可以确定脸部图像的暗通道值，这里的Input_R待增强的脸部图像的R通道的数值，这里的Input_G待增强的脸部图像的G通道的数值，这里的Input_B待增强的脸部图像的B通道的数值，并通过大气光值和光线调节值对所采集的脸部图像进行调整，以此获得更加清晰的脸部图像采集结果。
在一些实施例中，当进行人脸图像采集时，可以通过移动终端获取第一图像处理模型的使用环境中的终端所采集的人脸图像；对人脸图像进行图像增广处理；基于图像增广的处理结果，通过人脸检测算法确定相应的人脸位置，并基于所述人脸位置截取包括背景图像的人脸图像；对包括背景图像的人脸图像进行 裁剪处理，得到源图像。
作为示例，人脸检测算法可以是基于深度学习的人脸检测和人脸对齐方法等用于检测图像中人脸位置的算法。
参考图6，图6为本申请实施例中脸部图像的采集过程的示意图，由于图像采集设备的位置固定，目标对象的高矮不同，所采集的人脸图像的全面性也不同(可能出现目标对象身高过低或者过高均不能采集准确人脸图像的缺陷)，为了实现获得更加全面的人脸图像，可以对所采集的人脸图像进行图像增广处理；基于图像增广的处理结果，通过人脸检测算法确定相应的人脸位置，并截取包括背景图像的人脸图像；对包括背景图像的人脸图像进行背景清除处理，形成相应的目标用户的人脸图像。得到通过支付电子设备所采集的用户图像后，可以首先通过脸部检测的技术来框定用户脸部所在区域，并以此区域为中心扩大2倍，如图6所示由检测框601的检测区域调整至检测框602的检测区域，得到更多的背景内容，并对包括背景内容的脸部图像进行裁剪；例如：可以采用如下方式：采用人脸检测算法，框选目标对象的人脸位置；使用五官定位算法、标出人眼、嘴、鼻、等脸部的特征点；根据检测到的人脸位置截取包括背景内容的脸部图像。
在获取包括背景图像的人脸图像后，可以触发经过预训练的深度处理网络，深度处理网络可以包括但不限于：LeNet，AlexNet，VGG，Inception系列网络，ResNet网络；通过提取人脸图像的特征(例如提取均值，方差等基于灰度的特征和基于分布直方图的特征，GLCM和GLRLM等基于相关矩阵的特征或基于图像傅里叶变换之后的信号特征)，并基于所提取的特征进行背景清除处理，得到通过深度处理网络裁剪出来的真人人脸计算得到脸部对应的深度图。
在一些实施例中，通过真人人脸计算得到脸部图像包括深度图，而采用攻击图片(例如人脸照片)对应的深度图则是黑色的底图，通过对深度图的还原可以得到目标对象的人脸图像，所得到的目标对象的人脸图像并不包括背景图像，能够使得图像处理模型的换脸功能处理结果更加准确。
步骤402：通过第一图像处理模型对第一训练样本集合进行处理，得到第一换脸图像，其中，第一图像处理模型为重参数化结构。
作为示例，重参数化结构指的是第一图像处理模型是基于结构重参数化技术得到的，结构重参数化(structural re-parameterization)指的是首先构造一系列结构(一般用于训练)，并将其参数等价转换为另一组参数(一般用于推理)，从而将这一系列结构等价转换为另一系列结构。训练时的结构较大，具备好的某种性质(更高的精度或其他有用的性质，如稀疏性)，转换得到的推理时结构较小且保留这种性质(相同的精度或其他有用的性质)。“结构重参数化”这个词的本意就是：用一个结构的一组参数转换为另一组参数，并用转换得到的参数来参数化(parameterize)另一个结构。只要参数的转换是等价的，这两个结构的替换就是等价的。
在一些实施例中，由于从移动终端的环境中所采集的人脸图像可能会出现噪声，因此，在利用第一训练样本集合进行图像处理模型的训练之前，可以对第一训练样本集合进行除噪，以节省第一图像处理模型的训练时间，并提高第一图像处理模型的训练精度，具体来说，经过训练的第一图像处理模型使用环境包括：影视制作场景、游戏形象制作场景、直播虚拟形象制作场景以及证件照制作场景。
在影视制作中，一些专业的动作镜头由专业人员完成，后期可以通过换脸图像自动将演员替换进去； 具体来说，可以获取动作镜头视频片段中包含专业人员的图像帧，将包含替换演员的图像作为源图像，将每张包含专业人员的图像帧作为模板图像分别与源图像输入训练好的第一图像处理模型中，输出对应的换脸图像，输出的换脸图像将模板图像中专业人员的身份置换为替换演员的身份。可见，通过换脸图像，使得影视制作更加便利，避免重复拍摄，节约影视制作的成本。
在游戏形象制作中，可以将包含人物对象的图像作为源图像，将包含游戏形象的图像作为模板图像，将源图像与模板图像输入训练好的第一图像处理模型中，输出对应的换脸图像，输出的换脸图像将模板图像中游戏形象的身份置换为源图像中人物对象的身份。可见，通过换脸图像，可以为人物设计专属的游戏形象。
在直播场景中，可以将包含虚拟形象的图像作为源图像，将直播视频中每张包含人物对象的图像帧作为模板图像分别与源图像输入训练好的第一图像处理模型中，输出对应的换脸图像，输出的换脸图像将模板图像中人物对象的身份置换为虚拟形象。可见，可以在直播场景中利用虚拟形象进行身份置换，提升直播场景的趣味性。
在证件照的制作过程中，可以将需要制作证件照的对象的图像作为源图像，将源图像与证件照模板图像输入训练好的第一图像处理模型中，输出对应的换脸图像，输出的换脸图像将证件照模板图像中模板对象的身份置换为需要制作证件照的对象。可见，通过换脸图像，需要制作证件照的对象提供一张图像便可以直接制作证件照，不需要拍摄，大大降低了证件照的制作成本。
上述四种场景中，影视制作和游戏形象制作由于噪声不固定，因此可以确定与第一图像处理模型的使用环境相匹配的动态噪声阈值；根据动态噪声阈值对第一训练样本集合进行去噪处理，以形成与动态噪声阈值相匹配的第二训练样本集合，保证图像处理模型的训练精度。
对于直播虚拟形象制作和证件照制作来说，由于噪声不固定，确定与第二图像处理模型相对应的固定噪声阈值，并根据固定噪声阈值对第一训练样本集合进行去噪处理，以形成与固定噪声阈值相匹配的第二训练样本集合，可以进一步地压缩图像处理模型的训练时间。
步骤403：获取与第一图像处理模型相对应的第二图像处理模型，其中，第二图像处理模型为经过预训练的图像处理模型，在后续训练过程中第二图像处理模型的模型参数保持不变。
在一些实施例中，第二图像处理模型可以为训练好的神经网络，并且第二图像处理模型可以为规模较大的神经网络，例如：第二图像处理模型的网络参数的数量大于某一数值，但本申请实施例对此不作限定。第二图像处理模型可以为卷积神经网络(CNN ConvolutionalNeural Network)，深度神经网络(DNN Deep Neural Network)，循环神经网络(RNN Recurrent Neural Network)等，本申请实施例对第二图像处理模型的类型不作限定。第二图像处理模型可以为适用于不同的计算机视觉任务的神经网络，例如：目标识别任务、目标分类任务、目标检测任务或姿态估计任务等。第二图像处理模型也可以为适用于不同的应用场景的神经网络，例如：安防检测场景、人脸解锁场景、智能驾驶或遥感场景等，本申请实施例对第一图像处理模型的适用范围不作限定。第二图像处理模型的网络结构可以根据计算机视觉任务设计，或者，第二图像处理模型的网络结构可以采用现有的网络结构的至少一部分，例如：深度残差网络，视觉几何组网络(VGGNet Visual Geometry Group Network)等。
第一图像处理模型可以为待训练的神经网络，并且第一图像处理模型可以为规模较小的神经网络，以 便于通过低浮点运算量的优势部署在移动终端中，例如：第一图像处理模型的网络参数的数量小于某一数值(至少第一图像处理模型的网络参数的数量小于第二图像处理模型的网络参数的数量)，但本申请实施例对此不作限定。第二图像处理模型的网络规模大于第一图像处理模型的网络规模，第二图像处理模型可以为教师网络(teacher network)，第一图像处理模型可以为学生网络(student network)，利用教师网络来训练学生网络，能够提高训练得到的学生网络的性能。在一些实施例中，可以利用知识蒸馏方法或其他方法来进行对第一图像处理模型的训练，本申请实施例对此不作限定。
图7为本申请实施例中第一图像处理模型的一个模型结构示意图，其中，第一图像处理模型的模型中编码器和解码器的结构为重参数化结构RepVGG(Visual Geometry Group)，如图7所示，图7中的A表示的是原始的ResNet网络，该网络中包含着Conv1*1的残差结构和Identity的残差结构，这些残差结构的存在解决了深层网路中的梯度消失问题，使得网络更加易于收敛。图7中的B表示的是训练阶段的RepVGG网络架构，整个网络的主体中包含残差结构，同时，RepVGG网络中的残差块并没有跨层，并且整个网络包含2种残差结构。由于残差结构具有多个分支，就相当于给网络增加了多条梯度流动的路径，第一图像处理模型使用图7所示的结构，其实类似于训练了多个网络，并将多个网络融合在一个网络中，训练效率更高。图7中的C表示的是测试阶段的RepVGG网络，该网络的结构非常简单，整个网络均是由以下网络Conv3*3+Relu连接形成，，易于模型的测试和加速。
图8为本申请实施例中第一图像处理模型的测试过程示意图，其中，当前大多数移动设备中的测试引擎都对Conv3*3做了特定的加速，假如整个网络中的每一个Conv3*3都能节省3ms，如果一个网络中包含30个卷积层，那么整个网络就可以节省3*30＝90ms的时间。对于残差节点而言，需要当所有的残差分支都计算出对应的结果之后，才能获得最终的结果，这些残差分支的中间结果都会保存在设备的内存中，这样会对移动终端的内存具有较大的要求，来回的内存操作会降低整个网络的测试速度。而测试阶段首先在线下将模型转换为单分支结构，在移动终端测试阶段就能更好的提升设备的内存利用率，从而提升模型的测试速度
步骤404：根据第二图像处理模型和第一换脸图像，计算第一图像处理模型的融合损失函数。
在一些实施例中，第一图像处理模型的融合损失函数有不同的损失函数组合而成，获取第二图像处理模型输出的第二换脸图像，利用第一换脸图像和第二换脸图像计算第一图像处理模型的重构损失函数；基于所述第一换脸图像以及所述第二换脸图像计算第一图像处理模型的特征损失函数；基于所述第一换脸图像以及所述源图像计算第一图像处理模型的估计损失函数；基于所述第一换脸图像以及所述真值图像计算第一图像处理模型的对抗损失函数；计算重构损失函数、特征损失函数、估计损失函数以及对抗损失函数的加和，得到第一图像处理模型的融合损失函数；通过本申请实施例可以从多个维度提高第一图像处理模型的训练效果，从而提高第一图像处理模型的换脸准确度。
具体来说，融合损失函数loss的计算参考公式(1)：
loss＝Reconstruction_loss+LPIPS_loss+ID_loss+D_loss+G_loss(1)
其中，Reconstruction_loss为重构损失函数；LPIPS_loss为特征损失函数；ID_loss为估计损失函数；D_loss为判别器损失；G_loss为生成器损失，(D_loss+G_loss)构成了对抗损失函数。
下面对公式(1)中的每一个维度的损失函数分别进行介绍：
1)对于作为教师网络的第二图像处理模型(BigModel)来说，第二图像处理模型所计算得到的第二换脸图像可以表示为BigModel_fake，第一图像处理模型所计算得到的第一换脸图像可以表示为fake；通过本申请实施例可以使得第二图像处理模型的训练效果被迁移至第一图像处理模型，从而起到教导作用。
那么重构损失函数Reconstruction_loss的计算参考公式(2)：
BigModel_fake＝BigModel_swap(source，template)
Reconstruction_loss＝|BigModel_fake-fake|(2)；
其中，BigModel_fake是第二换脸图像，BigModel_swap表示第二换脸模型的前向处理过程，source是源图像，template是模板图像，Reconstruction_loss是重构损失函数，fake是第一换脸图像。
需要说明的的是，fake图和BigModel_fake的像素级差异越小，说明第二图像处理模型对第一图像处理模型训练指导的效果越好，使得第一图像处理模型继承了第二图像处理模型的精确度。
2)特征损失函数的计算：通过预训练的特征提取网络，对第一换脸图像进行特征提取，得到第一换脸图像的多个层级的特征；通过预训练的特征提取网络，对第二换脸图像进行特征提取，得到第二换脸图像的多个层级的特征；基于第一换脸图像的多个层级的特征与第二换脸图像的多个层级的特征之间的差值，确定第一图像处理模型的特征损失函数。通过本申请实施例可以确保第一图像处理模型针对中间各个特征的提取能力，从而提高第一图像处理模型的换脸准确度。
特征损失函数的计算参考公式(3)：
LPIPS_loss＝|result_fea1-gt_img_fea1|+|result_fea2-gt_img_fea2|+
|result_fea3-gt_img_fea3|+|result_fea4-gt_img_fea4|(3)；
上述公式(3)中，(result_fea1，result_fea2，result_fea3，result_fea4)＝alexnet_feature(fake)；(gt_img_fea1，gt_img_fea2，gt_img_fea3，gt_img_fea4)＝alexnet_feature(BigModel_fake)，其中，特征提取网络Alexnet由5个卷积层、3个池化层，以及3个全连接层构成。特征提取网络Alexnet的输出结果中，alexnet_feature(fake)表示将第一人脸图像(fake)输入到alexnet网络模型中并输出fake在alexnet网络模型的四个特征提取层(分别对应不同层级)所输出的特征，result_fea1、result_fea2、result_fea3以及result_fea4分别为四个特征提取层中的各个特征提取层所输出的第一换脸图像的解码人脸特征。alexnet_feature(gt_img)表示将第二换脸图像gt_img输入到alexnet网络模型中并输出gt_img在alexnet网络模型的四个特征提取层(分别对应不同层级)所输出的特征，gt_img_fea1、gt_img_fea2、gt_img_fea3以及gt_img_fea4分别为四个特征提取层中的各个特征提取层所输出的第二换脸图像gt_img的标准人脸特征。
3)估计损失函数的计算：提取第一换脸图像的第一换脸图像特征向量fake_id_features；提取源图像的源图像特征向量socre_id_features；利用第一换脸图像特征向量和源图像特征向量的相似度，计算第一图像处理模型的估计损失函数，通过本申请实施例可以确保第一图像处理模型在进行换脸的过程中不会出现失真的现象，从而提高第一图像处理模型的换脸准确度。
参考公式(4)：
ID_loss＝1-cosine_similarity(fake_id_features，socre_id_features)(4)；
其中，ID_loss是估计损失函数，fake_id_features是第一换脸图像特征向量，socre_id_features是源图像特征向量，cosine_similarity是余弦相似度。
4)对抗损失函数的计算：生成器网络，可称为G，判别器网络，可称为D，用于区分输入的图片是否为生成器网络所生成，输入一张高分辨率图像x，输出一个0-1区间的数D(x)，D(x)用于判断输入图是否为生成器生成的，其中，0表示不是，1表示是。其中，D_loss为判别器损失，G_loss为生成器损失，(D_loss+G_loss)构成了对抗损失函数loss，对抗损失函数的计算参考公式(5)：
其中，D_loss为判别器损失，G_loss为生成器损失，D(gt_img)是判别器针对真值图像输出的判别结果，D(fake)是判别器针对第一换脸图像输出的判别结果，loss是对抗损失函数。这里的判别结果可以是概率，即属于真实图像的概率。
步骤405：根据融合损失函数，对第一图像处理模型进行训练，当达到第一图像处理模型的训练收敛条件时，确定第一图像处理模型的模型参数。
作为示例，这里的训练收敛条件可以是达到设定的训练次数，可以是融合损失函数收敛于最小值。
至此，通过步骤401-步骤405完成了对第一图像处理模型的训练，确定了第一图像处理模型的参数，经过训练的第一图像处理模型可以部署在移动终端中，执行换脸功能，此时第一图像处理模型的浮点运算量FLOPs优化至544兆，同比第二图像处理模型的9373兆的FLOPs，减少了94％的计算量。最终部署到手机后可以实现每秒画面传输帧数在17-20之间，使得换脸功能的耗时满足移动终端实时性要求。
为了更好的说明本申请提供的图像处理模型的工作过程，参考图9，图9为本申请实施例中经过训练的图像处理模型的工作过程示意图，具体包括以下步骤：
步骤901：当图像处理模型训练完成，并部署在移动终端中时，获取目标人脸图像(对应源图像)和待替换人脸图像(对应模板图像)。
步骤902：通过图像处理模型的编码器网络，对目标人脸图像和待替换人脸图像进行编码，得到人脸图像向量。
步骤903：通过图像处理模型的解码器网络，对人脸图像向量进行解码，得到换脸图像。
其中，如图9所示，生成器为了采用非对称输入输出。由于移动终端的屏幕较小，因此解码器为网络输出分辨率从512像素降级到256像素，并且将输入的分辨率设计为128像素，以满足移动终端的使用。
编码器网络通过卷积计算将输入不断减半，通道逐渐增加。具体地，输入从128*128*6(目标人脸图像和待替换人脸图像，每张图的RGB通道数为3)，逐步编码为64*64*32，32*32*64，16*16*128，以此类推。
解码器网络通过反卷积运算，将分辨率逐渐增倍，解码为32*32*64，64*64*32，128*128*16，256*256*3，最终得到换脸结果。
图10为本申请实施例中换脸效果示意图，目标源面部图像例如可以是图10中的人脸图像(a)，目标模板面部图像例如可以是图10中的人脸图像(b)，面部更换图像例如可以是图4中的人脸图像(c)，人脸图像(c)是通过将人脸图像(a)中的人脸替换为人脸图像(b)中得到的，从人脸图像(c)可以看出，人脸图像(c)与人脸图像(b)的身份以及附加形象一致，即人脸图像(c)与人脸图像(b)是同一个人 的人脸，并且人脸图像(c)中包括了与人脸图像(b)相同的眼镜，人脸图像(c)与人脸图像(a)的属性一致，例如从人脸图像(c)可以看出，人脸图像(c)与人脸图像(a)的发型一致，并且人脸图像(c)嘴巴张开的角度比人脸图像(b)大，从而符合了人脸图像(a)的嘴巴张开的角度，达到用户所需要的换脸处理效果。
可以理解的是，在本申请实施例中，涉及到用户信息、人物图像等相关的数据，当本申请实施例运用到具体产品或技术中时，需要获得用户许可或者同意，且相关数据的收集、使用和处理需要遵守相关国家和地区的相关法律法规和标准。
下面继续说明本申请实施例提供的图像处理模型训练装置的实施为软件模块的示例性结构，在一些实施例中，如图2所示，存储在存储器的图像处理模型训练装置中的软件模块可以包括：数据传输模块2081，配置为获取第一训练样本集合，其中所述第一训练样本集合包括至少一个三元组训练样本，所述三元组训练样本包括：源图像、模板图像和真值图像；图像处理模型训练模块2082，配置为通过第一图像处理模型对所述源图像以及所述模板图像进行换脸，得到第一换脸图像，其中，所述第一图像处理模型为重参数化结构；所述图像处理模型训练模块2082，配置为获取与所述第一图像处理模型相对应的第二图像处理模型，其中，所述第二图像处理模型为经过预训练的图像处理模型；所述图像处理模型训练模块2082，配置为根据所述第二图像处理模型、所述第一换脸图像以及所述真值图像，计算所述第一图像处理模型的融合损失函数；所述图像处理模型训练模块2082，配置为根据所述融合损失函数，对所述第一图像处理模型进行训练，当达到所述第一图像处理模型的训练收敛条件时，确定所述第一图像处理模型的模型参数。
在一些实施例中，所述图像处理模型训练模块2082，还配置为确定与所述第一图像处理模型的使用环境相匹配的动态噪声阈值；根据所述动态噪声阈值对所述第一训练样本集合进行去噪处理，以形成与所述动态噪声阈值相匹配的第二训练样本集合；或者，确定与所述第二图像处理模型相对应的固定噪声阈值，并根据所述固定噪声阈值对所述第一训练样本集合进行去噪处理，以形成与所述固定噪声阈值相匹配的第二训练样本集合。
在一些实施例中，所述图像处理模型训练模块2082，还配置为获取所述第一图像处理模型的使用环境中的终端所采集的人脸图像；对所述人脸图像进行图像增广处理；基于图像增广的处理结果，确定相应的人脸位置，并基于所述人脸位置截取包括背景图像的人脸图像；对所述包括背景图像的人脸图像进行裁剪处理，得到所述源图像。
在一些实施例中，所述图像处理模型训练模块2082，还配置为获取所述第二图像处理模型输出的第二换脸图像，利用所述第一换脸图像和所述第二换脸图像计算所述第一图像处理模型的重构损失函数；计算所述第一图像处理模型的特征损失函数；计算所述第一图像处理模型的估计损失函数；计算所述第一图像处理模型的对抗损失函数；对所述重构损失函数、所述特征损失函数、所述估计损失函数以及所述对抗损失函数进行融合处理，得到所述第一图像处理模型的融合损失函数。
在一些实施例中，所述图像处理模型训练模块2082，还配置为计算所述第一换脸图像和所述第二换脸图像的像素级差值；根据所述像素级差值，确定所述第一图像处理模型的重构损失函数。
在一些实施例中，所述图像处理模型训练模块2082，还配置为通过预训练的特征提取网络，对所述第一换脸图像进行特征提取，得到所述第一换脸图像的多个层级的特征；通过预训练的特征提取网络，对所 述第二换脸图像进行特征提取，得到所述第二换脸图像的多个层级的特征；基于所述第一换脸图像的多个层级的特征与所述第二换脸图像的多个层级的特征之间的差值，确定所述第一图像处理模型的特征损失函数。
在一些实施例中，所述图像处理模型训练模块2082，还配置为提取所述第一换脸图像的第一换脸图像特征向量；提取所述源图像的源图像特征向量；利用所述第一换脸图像特征向量和所述源图像特征向量的相似度，计算所述第一图像处理模型的估计损失函数。
在一些实施例中，所述图像处理模型训练模块2082，还配置为当所述第一图像处理模型训练完成，并部署在移动终端中时，获取目标人脸图像和待替换人脸图像；通过所述第一图像处理模型的编码器网络，对所述目标人脸图像和所述待替换人脸图像进行编码，得到人脸图像向量；通过所述第一图像处理模型的解码器网络，对所述人脸图像向量进行解码，得到第三换脸图像。
本申请实施例提供了一种计算机程序产品，该计算机程序产品包括计算机程序或者计算机可执行指令，该计算机可执行指令存储在计算机可读存储介质中。电子设备的处理器从计算机可读存储介质读取该计算机可执行指令，处理器执行该计算机可执行指令，使得该电子设备执行本申请实施例上述的图像处理模型训练方法。
本申请实施例提供一种存储有计算机可执行指令的计算机可读存储介质，其中存储有计算机可执行指令，当计算机可执行指令被处理器执行时，将被处理器执行本申请实施例提供的图像处理模型训练方法。
在一些实施例中，计算机可读存储介质可以是FRAM、ROM、PROM、EPROM、EEPROM、闪存、磁表面存储器、光盘、或CD-ROM等存储器；也可以是包括上述存储器之一或任意组合的各种设备。
在一些实施例中，计算机可执行指令可以采用程序、软件、软件模块、脚本或代码的形式，按任意形式的编程语言(包括编译或解释语言，或者声明性或过程性语言)来编写，并且其可按任意形式部署，包括被部署为独立的程序或者被部署为模块、组件、子例程或者适合在计算环境中使用的其它单元。
作为示例，计算机可执行指令可以但不一定对应于文件系统中的文件，可以可被存储在保存其它程序或数据的文件的一部分，例如，存储在超文本标记语言(HTML，Hyper Text Markup Language)文档中的一个或多个脚本中，存储在专用于所讨论的程序的单个文件中，或者，存储在多个协同文件(例如，存储一个或多个模块、子程序或代码部分的文件)中。
作为示例，计算机可执行指令可被部署为在一个电子设备上执行，或者在位于一个地点的多个电子设备上执行，又或者，在分布在多个地点且通过通信网络互连的多个电子设备上执行。
本申请实施例具有以下有益技术效果：
1)本申请实施例通过获取第一训练样本集合，其中第一训练样本集合包括至少一个三元组训练样本，三元组训练样本包括：源图像、模板图像和真值；通过第一图像处理模型对第一训练样本集合进行处理，得到第一换脸图像，其中，第一图像处理模型为重参数化结构，获取与第一图像处理模型相对应的第二图像处理模型，其中，第二图像处理模型为经过预训练的图像处理模型，第二图像处理模型的模型参数固定不变；根据第二图像处理模型和第一换脸图像，计算第一图像处理模型的融合损失函数；根据融合损失函数，对第一图像处理模型进行训练，当达到第一图像处理模型的收敛条件时，确定第一图像处理模型的模型参数。因此，由于第一图像处理模型为重参数化结构，使得第一图像处理模型训练的时候结构是复杂的， 这样第一图像处理模型的处理能力强，能够学习复杂的数据，而测试的时候结构是简单的，能够减低测试时的耗时，并且减少浮点运算量，方便移动端的部署。此外，利用第二图像处理模型进行训练指导，能够在不增加训练样本总量和无需重新训练的前提下，稳定提高规模较小的图像处理模型的准确率，且对于大部分神经网络模型和数据普遍适用，通过知识蒸馏的方式实现对规模较小的图像处理模型训练兼顾了训练精确性的同时，减轻神经网络模型的过拟合，增强神经网络模型的泛化能力，便于将图像处理模型部署于移动终端中，实现图像处理模型的大规模应用。
2)通过对第一训练样本集合除噪，可以实现针对影视人像制作、游戏人物设计、虚拟形象、隐私保护等不同的换脸场景进行设计，使得对不同换脸场景的处理更加灵活，使得用户在实现换脸功能时灵活地选择训练样本。
以上，仅为本申请实施例而已，并非用于限定本申请的保护范围，凡在本申请实施例的精神和原则之内所作的任何修改、等同替换和改进等，均应包含在本申请实施例的保护范围之内。

Claims (12)

  1. 一种图像处理模型训练方法，所述方法由电子设备执行，所述方法包括：
    获取第一训练样本集合，其中，所述第一训练样本集合包括至少一个三元组训练样本，所述三元组训练样本包括：源图像、模板图像和真值图像；
    通过第一图像处理模型对所述源图像以及所述模板图像进行换脸，得到第一换脸图像，其中，所述第一图像处理模型为重参数化结构；
    获取与所述第一图像处理模型相对应的第二图像处理模型，其中，所述第二图像处理模型为经过预训练的图像处理模型；
    根据所述第二图像处理模型、所述第一换脸图像以及所述真值图像，计算所述第一图像处理模型的融合损失函数；
    根据所述融合损失函数，对所述第一图像处理模型进行训练，当达到所述第一图像处理模型的训练收敛条件时，确定所述第一图像处理模型的模型参数。
  2. 根据权利要求1所述的方法，其中，所述方法还包括：
    确定与所述第一图像处理模型的使用环境相匹配的动态噪声阈值；
    根据所述动态噪声阈值对所述第一训练样本集合进行去噪处理，以形成与所述动态噪声阈值相匹配的第二训练样本集合；或者，
    确定与所述第二图像处理模型相对应的固定噪声阈值，并根据所述固定噪声阈值对所述第一训练样本集合进行去噪处理，以形成与所述固定噪声阈值相匹配的第二训练样本集合。
  3. 根据权利要求1所述的方法，其中，所述方法还包括：
    获取所述第一图像处理模型的使用环境中的终端所采集的人脸图像；
    对所述人脸图像进行图像增广处理；
    基于图像增广的处理结果，确定相应的人脸位置，并基于所述人脸位置截取包括背景图像的人脸图像；
    对所述包括背景图像的人脸图像进行裁剪处理，得到所述源图像。
  4. 根据权利要求1所述的方法，其中，所述根据所述第二图像处理模型、所述第一换脸图像以及所述真值图像，计算所述第一图像处理模型的融合损失函数，包括：
    获取所述第二图像处理模型输出的第二换脸图像，利用所述第一换脸图像和所述第二换脸图像计算所述第一图像处理模型的重构损失函数；
    基于所述第一换脸图像以及所述第二换脸图像计算所述第一图像处理模型的特征损失函数；
    基于所述第一换脸图像以及所述源图像计算所述第一图像处理模型的估计损失函数；
    基于所述第一换脸图像以及所述真值图像计算所述第一图像处理模型的对抗损失函数；
    对所述重构损失函数、所述特征损失函数、所述估计损失函数以及所述对抗损失函数进行融合处理，得到所述第一图像处理模型的融合损失函数。
  5. 根据权利要求4所述的方法，其中，所述利用所述第一换脸图像和所述第二换脸图像计算所述第一图像处理模型的重构损失函数，包括：
    计算所述第一换脸图像和所述第二换脸图像的像素级差值；
    根据所述像素级差值，确定所述第一图像处理模型的重构损失函数。
  6. 根据权利要求4所述的方法，其中，所述基于所述第一换脸图像以及所述第二换脸图像计算所述第一图像处理模型的特征损失函数，包括：
    通过预训练的特征提取网络，对所述第一换脸图像进行特征提取，得到所述第一换脸图像的多个层级的特征；
    通过预训练的特征提取网络，对所述第二换脸图像进行特征提取，得到所述第二换脸图像的多个层级的特征；
    基于所述第一换脸图像的多个层级的特征与所述第二换脸图像的多个层级的特征之间的差值，确定所述第一图像处理模型的特征损失函数。
  7. 根据权利要求4所述的方法，其中，所述基于所述第一换脸图像以及所述源图像计算所述第一图像处理模型的估计损失函数，包括：
    提取所述第一换脸图像的第一换脸图像特征向量；
    提取所述源图像的源图像特征向量；
    利用所述第一换脸图像特征向量和所述源图像特征向量的相似度，计算所述第一图像处理模型的估计损失函数。
  8. 根据权利要求1所述的方法，其中，所述方法还包括：
    当所述第一图像处理模型训练完成，并部署在移动终端中时，获取目标人脸图像和待替换人脸图像；
    通过所述第一图像处理模型的编码器网络，对所述目标人脸图像和所述待替换人脸图像进行编码，得到人脸图像向量；
    通过所述第一图像处理模型的解码器网络，对所述人脸图像向量进行解码，得到第三换脸图像。
  9. 一种图像处理模型训练装置，所述训练装置包括：
    数据传输模块，配置为获取第一训练样本集合，其中所述第一训练样本集合包括至少一个三元组训练样本，所述三元组训练样本包括：源图像、模板图像和真值图像；
    图像处理模型训练模块，配置为通过第一图像处理模型对所述源图像以及所述模板图像进行换脸，得到第一换脸图像，其中，所述第一图像处理模型为重参数化结构；
    所述图像处理模型训练模块，配置为获取与所述第一图像处理模型相对应的第二图像处理模型，其中，所述第二图像处理模型为经过预训练的图像处理模型；
    所述图像处理模型训练模块，配置为根据所述第二图像处理模型、所述第一换脸图像以及所述真值图像，计算所述第一图像处理模型的融合损失函数；
    所述图像处理模型训练模块，配置为根据所述融合损失函数，对所述第一图像处理模型进行训练，当达到所述第一图像处理模型的训练收敛条件时，确定所述第一图像处理模型的模型参数。
  10. 一种电子设备，所述电子设备包括：
    存储器，用于存储计算机可执行指令；
    处理器，用于运行所述存储器存储的计算机可执行指令时，实现权利要求1至8任一项所述的图像处理模型训练方法。
  11. 一种计算机程序产品，包括计算机可执行指令，所述计算机可执行指令被处理器执行时，实现权利要求1至8任一项所述的图像处理模型训练方法。
  12. 一种计算机可读存储介质，存储有计算机可执行指令，所述计算机可执行指令被处理器执行时实现权利要求1至8任一项所述的图像处理模型训练方法。
PCT/CN2023/123450 2022-11-09 2023-10-08 一种图像处理模型训练方法、装置、电子设备、计算机可读存储介质及计算机程序产品 Ceased WO2024099004A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP23887696.5A EP4560588A4 (en) 2022-11-09 2023-10-08 IMAGE PROCESSING MODEL TRAINING METHOD AND APPARATUS, AND ELECTRONIC DEVICE, COMPUTER-READABLE STORAGE MEDIUM, AND COMPUTER PROGRAM PRODUCT
US18/813,622 US20240420288A1 (en) 2022-11-09 2024-08-23 Method and apparatus for training image processing model, electronic device, computer-readable storage medium, and computer program product

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211397807.4A CN117011665A (zh) 2022-11-09 2022-11-09 一种图像处理模型训练方法、装置、电子设备及存储介质
CN202211397807.4 2022-11-09

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/813,622 Continuation US20240420288A1 (en) 2022-11-09 2024-08-23 Method and apparatus for training image processing model, electronic device, computer-readable storage medium, and computer program product

Publications (1)

Publication Number Publication Date
WO2024099004A1 true WO2024099004A1 (zh) 2024-05-16

Family

ID=88569795

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/123450 Ceased WO2024099004A1 (zh) 2022-11-09 2023-10-08 一种图像处理模型训练方法、装置、电子设备、计算机可读存储介质及计算机程序产品

Country Status (4)

Country Link
US (1) US20240420288A1 (zh)
EP (1) EP4560588A4 (zh)
CN (1) CN117011665A (zh)
WO (1) WO2024099004A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119600104A (zh) * 2024-11-20 2025-03-11 深圳市大成机电技术有限公司 基于机器学习的工业机器人视觉定位方法及系统
CN120807296A (zh) * 2025-09-16 2025-10-17 济南大学 基于知识蒸馏的医学图像超分辨率重建方法及系统

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119443195A (zh) * 2023-07-30 2025-02-14 鸿海精密工业股份有限公司 机器学习方法
US20250245886A1 (en) * 2024-01-31 2025-07-31 Google Llc Optimization of overall editing vector to achieve target expression photo editing effect

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110826593A (zh) * 2019-09-29 2020-02-21 腾讯科技（深圳）有限公司 融合图像处理模型的训练方法、图像处理方法、装置及存储介质
CN111783603A (zh) * 2020-06-24 2020-10-16 有半岛(北京)信息科技有限公司 生成对抗网络训练方法、图像换脸、视频换脸方法及装置
CN111860043A (zh) * 2019-04-26 2020-10-30 北京陌陌信息技术有限公司 人脸换像模型训练方法、装置、设备和介质
CN113411425A (zh) * 2021-06-21 2021-09-17 深圳思谋信息科技有限公司 视频超分模型构建处理方法、装置、计算机设备和介质
US20220004803A1 (en) * 2020-06-29 2022-01-06 L'oreal Semantic relation preserving knowledge distillation for image-to-image translation
CN114004772A (zh) * 2021-09-30 2022-02-01 阿里巴巴（中国）有限公司 图像处理方法、图像合成模型的确定方法、系统及设备
CN114387656A (zh) * 2022-01-14 2022-04-22 平安科技（深圳）有限公司 基于人工智能的换脸方法、装置、设备及存储介质
CN114611700A (zh) * 2022-01-23 2022-06-10 杭州领见数字农业科技有限公司 一种基于结构重参数化的模型推理速度提升方法及装置
WO2022179401A1 (zh) * 2021-02-26 2022-09-01 腾讯科技（深圳）有限公司 图像处理方法、装置、计算机设备、存储介质和程序产品
CN115294423A (zh) * 2022-08-15 2022-11-04 网易（杭州）网络有限公司 模型确定方法、图像处理方法、装置、设备及存储介质

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111881838B (zh) * 2020-07-29 2023-09-26 清华大学 具有隐私保护功能的运动障碍评估录像分析方法及设备
US12452385B2 (en) * 2022-03-29 2025-10-21 Disney Enterprises, Inc. Method and system for deep learning based face swapping with multiple encoders
US12211178B2 (en) * 2022-04-21 2025-01-28 Adobe Inc. Transferring faces between digital images by combining latent codes utilizing a blending network

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111860043A (zh) * 2019-04-26 2020-10-30 北京陌陌信息技术有限公司 人脸换像模型训练方法、装置、设备和介质
CN110826593A (zh) * 2019-09-29 2020-02-21 腾讯科技（深圳）有限公司 融合图像处理模型的训练方法、图像处理方法、装置及存储介质
CN111783603A (zh) * 2020-06-24 2020-10-16 有半岛(北京)信息科技有限公司 生成对抗网络训练方法、图像换脸、视频换脸方法及装置
US20220004803A1 (en) * 2020-06-29 2022-01-06 L'oreal Semantic relation preserving knowledge distillation for image-to-image translation
WO2022179401A1 (zh) * 2021-02-26 2022-09-01 腾讯科技（深圳）有限公司 图像处理方法、装置、计算机设备、存储介质和程序产品
CN113411425A (zh) * 2021-06-21 2021-09-17 深圳思谋信息科技有限公司 视频超分模型构建处理方法、装置、计算机设备和介质
CN114004772A (zh) * 2021-09-30 2022-02-01 阿里巴巴（中国）有限公司 图像处理方法、图像合成模型的确定方法、系统及设备
CN114387656A (zh) * 2022-01-14 2022-04-22 平安科技（深圳）有限公司 基于人工智能的换脸方法、装置、设备及存储介质
CN114611700A (zh) * 2022-01-23 2022-06-10 杭州领见数字农业科技有限公司 一种基于结构重参数化的模型推理速度提升方法及装置
CN115294423A (zh) * 2022-08-15 2022-11-04 网易（杭州）网络有限公司 模型确定方法、图像处理方法、装置、设备及存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4560588A4 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119600104A (zh) * 2024-11-20 2025-03-11 深圳市大成机电技术有限公司 基于机器学习的工业机器人视觉定位方法及系统
CN120807296A (zh) * 2025-09-16 2025-10-17 济南大学 基于知识蒸馏的医学图像超分辨率重建方法及系统

Also Published As

Publication number Publication date
EP4560588A4 (en) 2025-10-22
CN117011665A (zh) 2023-11-07
EP4560588A1 (en) 2025-05-28
US20240420288A1 (en) 2024-12-19

Similar Documents

Publication Publication Date Title
Ren et al. Low-light image enhancement via a deep hybrid network
Wan et al. CoRRN: Cooperative reflection removal network
US10614574B2 (en) Generating image segmentation data using a multi-branch neural network
WO2024099004A1 (zh) 一种图像处理模型训练方法、装置、电子设备、计算机可读存储介质及计算机程序产品
CN112528764B (zh) 人脸表情识别方法、系统、装置及可读存储介质
CN111079764B (zh) 一种基于深度学习的低照度车牌图像识别方法及装置
CN111833360B (zh) 一种图像处理方法、装置、设备以及计算机可读存储介质
EP4668206A1 (en) Image enhancement method and apparatus, electronic device, computer-readable storage medium, and computer program product
CN112750176B (zh) 一种图像处理方法、装置、电子设备及存储介质
CN112329752B (zh) 人眼图像处理模型的训练方法、图像处理方法及装置
CN113762032B (zh) 图像处理方法、装置、电子设备及存储介质
CN113658065A (zh) 图像降噪方法及装置、计算机可读介质和电子设备
EP4617999A1 (en) Image enhancement method and apparatus, electronic device, computer-readable storage medium and computer program product
CN119693632B (zh) 对象的识别方法和装置、存储介质及电子设备
CN111382654A (zh) 图像处理方法和装置以及存储介质
CN114694065B (zh) 视频处理方法、装置、计算机设备及存储介质
CN116958306A (zh) 图像合成方法和装置、存储介质及电子设备
CN115861637A (zh) 一种基于双分支网络的显著目标检测方法
CN113822117B (zh) 一种数据处理方法、设备以及计算机可读存储介质
CN115965839A (zh) 图像识别方法、存储介质及设备
CN118411320A (zh) 图像重建方法、电子设备及存储介质
CN116797466B (zh) 一种图像处理方法、装置、设备及可读存储介质
CN116958758A (zh) 数据处理方法及装置、设备、存储介质、程序产品
CN116777766A (zh) 图像增强方法、电子设备及存储介质
Li et al. RSID: A remote sensing image dehazing network

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23887696

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023887696

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2023887696

Country of ref document: EP

Effective date: 20250219

WWE Wipo information: entry into national phase

Ref document number: 11202500820S

Country of ref document: SG

WWP Wipo information: published in national office

Ref document number: 11202500820S

Country of ref document: SG

WWP Wipo information: published in national office

Ref document number: 2023887696

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE