WO2023093244A1 - 一种模型训练方法、装置、设备、介质和程序产品 - Google Patents

一种模型训练方法、装置、设备、介质和程序产品 Download PDF

Info

Publication number
WO2023093244A1
WO2023093244A1 PCT/CN2022/120169 CN2022120169W WO2023093244A1 WO 2023093244 A1 WO2023093244 A1 WO 2023093244A1 CN 2022120169 W CN2022120169 W CN 2022120169W WO 2023093244 A1 WO2023093244 A1 WO 2023093244A1
Authority
WO
WIPO (PCT)
Prior art keywords
key point
key
target
point detection
training
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2022/120169
Other languages
English (en)
French (fr)
Inventor
温少国
王君乐
许子潇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology (Shenzhen) Co Ltd
Original Assignee
Tencent Technology (Shenzhen) Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology (Shenzhen) Co Ltd filed Critical Tencent Technology (Shenzhen) Co Ltd
Priority to EP22897330.1A priority Critical patent/EP4390728A4/en
Priority to JP2024520050A priority patent/JP7635466B2/ja
Priority to US18/199,235 priority patent/US20230290003A1/en
Publication of WO2023093244A1 publication Critical patent/WO2023093244A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/088—Non-supervised learning, e.g. competitive learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/776—Validation; Performance evaluation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/23—Recognition of whole body movements, e.g. for sport training

Definitions

  • This application relates to the technical field of artificial intelligence, in particular to model training.
  • Human body key point detection technology is also known as human body pose estimation technology. This technology is used to identify important joint points on human bones in images (such as nose, left and right eyes, left and right ears, left and right knee, left and right ankle, etc.). Human body key point detection technology is now widely used in many scenarios; for example, in somatosensory game scenes, human body key point detection technology is used to detect changes in the player's body movements, so that the corresponding game logic can be executed based on the detection results.
  • the realization of the human body key point detection technology mainly relies on the human body key point detection model, that is, through the human body key point detection model, the position of the important joint points on the human skeleton is determined according to the input image.
  • the human body key point detection model should not be too complicated; therefore, when training the human body key point detection model that is actually put into use in related technologies, the human body key point detection model with complex structure and better performance is usually used to detect human body key point detection for unlabeled images processing, determine the position of the key points of the human body in the image as the pseudo-label of the image, and then use the image and its pseudo-label as a training sample, and use the training sample to detect the key points of the human body that need to be put into practical use and have a relatively simple structure
  • the model is trained.
  • Embodiments of the present application provide a model training method and a related device, which can improve the accuracy of the determined pseudo-label, and then improve the performance of the human body key point detection model trained based on the training samples including the pseudo-label.
  • the first aspect of the present application provides a model training method, the method comprising:
  • the key point detection processing is performed on the target training image respectively, and key point detection results corresponding to the m reference object key point detection models are obtained;
  • the key point detection results include the The predicted positions of the key points corresponding to the key parts of multiple objects in the target training image;
  • the m is an integer greater than 1;
  • For each key part of the object determine whether the key part of the object is a key part of the target object according to the predicted position of the key point corresponding to the key point of the object in the m key point detection results; The predicted position of the key point corresponding to the key part of the target object in the m key point detection results, and determine the label position corresponding to the key part of the target object;
  • the target training image and its corresponding key point labeling results are used as training samples;
  • the key point labeling results include the corresponding labeling positions of each of the target object key parts;
  • a target object key point detection model is trained.
  • the second aspect of the present application provides a model training device, the device comprising:
  • the key point detection module is used to respectively perform object key point detection processing on the target training image through m reference object key point detection models, and obtain key point detection results corresponding to the m reference object key point detection models;
  • the key point detection results include the predicted positions of key points corresponding to the key parts of multiple objects in the target training image; the m is an integer greater than 1;
  • a part labeling module for each key part of the object, according to the predicted positions of the key points corresponding to the key parts of the object in the m key point detection results, determine whether the key part of the object is a key part of the target object If so, then according to the predicted position of the key point corresponding to the key point of the target object in the m key point detection results, determine the label position corresponding to the key part of the target object;
  • the sample generation module is used to use the target training image and its corresponding key point labeling results as training samples; the key point labeling results include the corresponding labeling positions of the key parts of the target objects;
  • a model training module configured to train a target object key point detection model based on the training samples.
  • the third aspect of the present application provides a computer device, the device includes a processor and a memory:
  • the memory is used to store computer programs
  • the processor is configured to execute the steps of the model training method described in the first aspect above according to the computer program.
  • a fourth aspect of the present application provides a computer-readable storage medium, where the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the steps of the model training method described in the first aspect above.
  • a fifth aspect of the present application provides a computer program product or computer program, where the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium.
  • the processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps of the model training method described in the first aspect above.
  • the embodiment of the present application provides a model training method.
  • the method When the method generates training samples for training the key point detection model of the target object that needs to be put into practical application, it will pass m (m is an integer greater than 1) key points of the reference object
  • the detection model is to perform object key point detection processing on the target training image respectively, and obtain key point detection results corresponding to m reference object key point detection models, where each key point detection result includes multiple object key parts in the target training image The predicted positions of the respective keypoints.
  • FIG. 1 is a schematic diagram of an application scenario of a model training method provided in an embodiment of the present application
  • Fig. 2 is a schematic flow chart of the model training method provided by the embodiment of the present application.
  • FIG. 3 is a schematic diagram of the working principle of the reference object key point detection model provided by the embodiment of the present application.
  • FIG. 4 is a schematic diagram of the training process of the target object key point detection model provided by the embodiment of the present application.
  • FIG. 5 is an overall implementation architecture diagram of the somatosensory game provided by the embodiment of the present application.
  • FIG. 6 is a schematic diagram of the training framework of the human body key point detection model applied in the somatosensory game provided by the embodiment of the present application;
  • FIG. 7 is a schematic diagram of the test results of the human body key point detection model provided in the embodiment of the present application.
  • FIG. 8 is a schematic structural diagram of a model training device provided in an embodiment of the present application.
  • FIG. 9 is a schematic structural diagram of another model training device provided in the embodiment of the present application.
  • FIG. 10 is a schematic structural diagram of another model training device provided in the embodiment of the present application.
  • FIG. 11 is a schematic structural diagram of a terminal device provided in an embodiment of the present application.
  • FIG. 12 is a schematic structural diagram of a server provided by an embodiment of the present application.
  • artificial intelligence technology has been researched and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones , robots, intelligent medical care, intelligent customer service, Internet of Vehicles, autonomous driving, intelligent transportation, etc., I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important value.
  • the key point detection result of the object key point detection model with complex structure for the unlabeled image is directly used as the pseudo label corresponding to the unlabeled image;
  • the training samples are used to train the object key point detection model with a simple structure that needs to be put into practical use.
  • the pseudo-labels generated in this way usually have more noise and lower accuracy, which will have a negative impact on the model performance of the trained object keypoint detection model.
  • the embodiment of the present application provides a model training method.
  • m (m is an integer greater than 1) key point detection models of reference objects are used to detect the target points respectively.
  • the training image is subjected to object key point detection processing, and the key point detection results corresponding to the m reference object key point detection models are obtained.
  • Each key point detection result here includes the key points corresponding to the key parts of multiple objects in the target training image. predicted location.
  • the target training image and the marked positions corresponding to the key parts of each target object are used as training samples; and based on the training samples, a target object key point detection model with a simpler structure that needs to be put into practical application is trained.
  • the target key point detection process is performed on the target training image through multiple reference object key point detection models with complex structures, and the obtained Multiple keypoint detection results. Then, based on the principle that the predicted positions of the key parts of the same object in each key point detection result should be basically the same, for each key part of the object, according to the predicted position of the key point corresponding to the key part of the object in each key point detection result, measure Whether the key point detection model of each reference object is reliable for the position prediction result of the key part of the object is to determine whether the key part of the object is the key part of the target object.
  • the training samples After confirming that the position prediction results of the key point detection models of each reference object are reliable for the key parts of the object, further determine the labeling position corresponding to the key part of the target object, that is, the pseudo-label; then, use the target training image and each target in it
  • the corresponding labeled positions of the key parts of the object constitute the training samples.
  • the training samples including such pseudo-labels the training needs to be put into practical application and the target object key point detection model with a relatively simple structure can guarantee the performance of the trained target object key point detection model to a certain extent.
  • the model training method provided in the embodiment of the present application may be executed by a computer device capable of image processing, and the computer device may be a terminal device or a server.
  • the terminal device can specifically be a computer, a smart phone, a tablet computer, a personal digital assistant (Personal Digital Assistant, PDA), etc.
  • the server can specifically be an application server or a Web server, and in actual deployment, it can be an independent server or a A cluster server or cloud server composed of multiple physical servers.
  • the processed data involved in the embodiment of the present application (such as training images, key point detection results, key point labeling results, etc.) can be saved on the blockchain.
  • the application scenario of the model training method is exemplarily introduced below by taking the execution subject of the model training method as a server as an example.
  • FIG. 1 is a schematic diagram of an application scenario of a model training method provided in an embodiment of the present application.
  • the application scenario includes a server 110 and a database 120 , and the server 110 may retrieve data from the database 120 through a network, or the database 120 may also be integrated in the server 110 .
  • the server 110 is used to execute the model training method provided by the embodiment of the present application, so as to train a target object key point detection model with a relatively simple structure that needs to be put into practical application; the database 120 is used to store unlabeled training images.
  • the server 110 can retrieve the target training image from the database 120, the target training image includes the object to be detected, and the object to be detected includes several key parts of the object; for example, the target training image can include clear and complete of the human body to be tested.
  • the server 110 can use m (m is an integer greater than 1) reference object key point detection models to respectively perform object key point detection processing on the retrieved target training images, and obtain the m reference object key point detection models corresponding to The key point detection results; each key point detection result here may include the predicted positions of the key points corresponding to the key parts of multiple objects in the target training image.
  • m is an integer greater than 1
  • reference object key point detection model is a pre-trained object key point detection model, which can usually more accurately detect the positions of key parts of each object on the object to be detected in the image, and its model structure is usually relatively complicated .
  • each of the above reference object key point detection models may be a reference human body key point detection model, which is used to detect the position of each important joint on the human skeleton.
  • each reference human body key point detection model is used to perform human body key point detection processing on the target training image, and the corresponding key point detection results of each reference human body key point detection model will be obtained.
  • the key point detection results include the target training image. The predicted positions of key points corresponding to each important joint on the human body to be detected.
  • the server 110 can judge the position of the m reference object key point detection models for the key part of the object according to the predicted position of the key point corresponding to the key point of the object in the m key point detection results Whether the prediction result is reliable means to determine whether the key part of the object is the key part of the target object. If it is determined that the key part of the object is the key part of the target object, then the marked position corresponding to the key part of the target object can be determined according to the predicted position of the key point corresponding to the key part of the target object in the m key point detection results.
  • the prediction results of different reference object key point detection models for the key parts of the same object in the target training image should be basically the same, that is, the distances between the predicted positions of the same object key parts by different reference object key point detection models should be smaller. If the distance between the predicted positions of the key parts of the same object in the m key point detection results is relatively large, it means that the predicted positions corresponding to the key parts of the object in the m key point detection results are unreliable, and there may be a large gap between them.
  • the key parts of the object can be ignored, and the corresponding label position of the key parts of the object can be determined not based on the predicted positions of the key parts of the object in the m key point detection results.
  • the m key point detection results have a small distance between the predicted positions of the key parts of the same object, it means that the predicted positions corresponding to the key parts of the object in the m key point detection results are more reliable.
  • the key part of the object is regarded as the key part of the target object, and the marked position corresponding to the key part of the target object is determined according to the predicted position corresponding to the key part of the target object in the m key point detection results.
  • the corresponding label positions of the key parts of each target object in the target training image will be obtained, and then the corresponding label positions of the key parts of each target object in the target training image can be used.
  • the key point labeling result corresponding to the target training image is composed, that is, the pseudo-label corresponding to the target training image. Furthermore, the target training image and its corresponding key point labeling results can be used as training samples.
  • the target object key point detection model can be trained based on the training samples constructed in the above manner.
  • the key point detection model of the target object is an object key point detection model that needs to be put into practical application.
  • the detection model needs to have high detection efficiency. Therefore, an object key point detection model with a relatively simple model structure is usually set as the target object key point detection model.
  • FIG. 2 is a schematic flowchart of a model training method provided in an embodiment of the present application.
  • the model training method includes the following steps:
  • Step 201 Through m reference object key point detection models, respectively perform object key point detection processing on the target training image, and obtain key point detection results corresponding to each of the m reference object key point detection models; the key point detection results It includes predicted positions of key points corresponding to each of the key parts of the object in the target training image; the m is an integer greater than 1.
  • the server when it generates its corresponding pseudo-label for the target training image, it can first use m reference object key point detection models to perform object key point detection processing on the target training image respectively, and obtain the m reference object key point detection models.
  • the key point detection results of the object key point detection models for the target training image that is, the key point detection results corresponding to the m reference object key point detection models.
  • each key point detection result includes the predicted positions of the key points corresponding to the key parts of the object in the target training image.
  • pseudo-labels are commonly used training data in semi-supervised learning. Usually, unlabeled data can be processed through complex models with better performance to obtain pseudo-labels corresponding to the unlabeled data. The pseudo-labels may is not accurate.
  • the pseudo-label corresponding to the target training image can be determined according to the key point detection results of the target training image by m reference object key point detection models;
  • the reference object key point detection model processes the key point detection results of the target training image, and obtains a pseudo-label that can more accurately reflect the position of the key part of the object in the target training image, thereby facilitating the improvement of training samples based on the pseudo-label.
  • the performance of the target object keypoint detection model is commonly used training data in semi-supervised learning. Usually, unlabeled data can be processed through complex models with better performance to obtain pseudo-labels corresponding to the unlabeled data.
  • the pseudo-labels may is not accurate.
  • the pseudo-label corresponding to the target training image can be
  • the reference object key point detection model is a pre-trained model used to detect the position of the key parts of the object on the object to be detected in the image, and it can usually detect the position of the key parts of the object more accurately;
  • the key point detection model of the reference object is set to have a relatively complex model structure.
  • each of the above-mentioned key point detection models of reference objects may be a reference human body key point detection model, which is used to detect the position of each important joint on the human body in the image, and the key point detection results of the reference human body key point detection model may be Used to determine the pose of the human body.
  • the m reference object key point detection models in the embodiment of the present application may be trained based on the same training sample set, or may be trained based on different training sample sets, and this application does not make any limited.
  • the target training image is an image including the object to be detected, and the object to be detected includes several key parts of the object; the key parts of the object here include the parts on the object to be detected that can reflect the posture of the object to be detected.
  • the target training image may be an image including a clear and complete human body to be detected.
  • the human body to be detected includes several important joints, such as the nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right Hips, left and right knees, left and right ankles, etc.
  • the key point detection result corresponding to the reference object key point detection model is obtained by performing object key point detection processing on the target training image through the reference object key point detection model, and the key point detection result includes the target training image The predicted positions of the key points corresponding to the key parts of each object on the object to be detected.
  • the target training image is an image with a size of H ⁇ W
  • a thermal feature map with a size of K ⁇ H ⁇ W will be output, Among them, K is the number of key parts of the object to be detected, and the k-th thermal feature map in the K ⁇ H ⁇ W thermal feature map is used to represent the predicted position of the key point corresponding to the k-th key part of the object in the target training image .
  • the server can obtain the key point detection result corresponding to each reference object key point detection model in the following manner: for each reference object key point detection model, the target The training image is subjected to object key point detection processing to determine the confidence that each pixel in the target training image belongs to the key part of the object. Then, the key point detection result corresponding to the reference object key point detection model is determined according to the pixels in the target training image that belong to the key parts of the object with a confidence degree higher than a preset reliability threshold.
  • FIG. 3 is a schematic diagram of a working principle of a reference object key point detection model provided in the embodiment of the present application.
  • the reference object key point detection model will analyze and process the target training image and output the corresponding size is a thermal feature map of K ⁇ H ⁇ W, which is essentially composed of K thermal feature maps with a size of H ⁇ W, where each thermal feature map with a size of H ⁇ W includes the target training image Confidence of each pixel in the key part of the object corresponding to the thermal feature map, for example, in the k-th thermal feature map of size H ⁇ W, each pixel has its corresponding confidence, the confidence It is used to represent the probability that the corresponding pixel belongs to the key part of the kth object.
  • the server can ignore the pixels whose corresponding confidence is less than or equal to the preset confidence (such as 0.5), for example, directly set the confidence corresponding to such pixels Adjusted to 0, by default it does not belong to the key part of the object corresponding to the thermal feature map.
  • the preset confidence such as 0.5
  • the position of the pixel corresponding to the thermal feature map with a confidence equal to 1 can reflect the predicted position of the key part of the object corresponding to the thermal feature map .
  • the thermal feature maps with a size of K ⁇ H ⁇ W obtained after the primary screening can be used as the reference object detection model The corresponding keypoint detection results.
  • the key point detection results corresponding to the key point detection model of the reference object can be obtained, which can further reduce the noise in the key point detection results, and at the same time avoid the server’s subsequent Unnecessary processing is performed on such noise points, which reduces required processing resources and can also improve subsequent processing efficiency.
  • Step 202 For each key part of the object, according to the predicted position of the key point corresponding to the key part of the object in the m key point detection results, determine whether the key part of the object is a key part of the target object; if so, Then, according to the predicted positions of the key points corresponding to the key parts of the target object in the m key point detection results, the marked positions corresponding to the key parts of the target object are determined.
  • the server can judge whether the position prediction results of the m reference object key point detection models for the key parts of each object are reliable according to the m key point detection results. That is, for each key part of an object, the server can determine whether the key part of the object is a key part of the target object according to the predicted positions of the key points corresponding to the key parts of the object in the m key point detection results; In the case of the key parts of the target object, the server can further determine the label position corresponding to the key parts of the target object according to the predicted positions of the key points corresponding to the key parts of the target object in the m key point detection results; furthermore, the server can use the target training The corresponding labeling positions of the key parts of each target object in the image constitute the key point labeling result corresponding to the target training image.
  • different reference object key point detection models should basically have the same prediction results for the key parts of the same object in the target training image, that is, the prediction results of different reference object key point detection models for the key parts of the same object The locations are relatively close to each other.
  • the embodiment of the present application chooses to ignore the key parts of the object, and does not determine the corresponding label position of the key parts of the object based on the predicted positions corresponding to the key parts of the object in the m key point detection results, That is, the pseudo-label corresponding to the target training image is not generated based on the key parts of the object.
  • the predicted positions corresponding to the key parts of the same object in the m key point detection results are relatively close to each other, it means that the predicted positions corresponding to the key parts of the object in the m key point detection results are more reliable.
  • the key part of the object is regarded as the key part of the target object, and according to the predicted position corresponding to the key part of the target object in the m key point detection results, the label position corresponding to the key part of the target object is determined, and the follow-up can also be based on the key point of the target object
  • the annotated position corresponding to the part determines the pseudo-label corresponding to the target training image.
  • the server can judge whether the position prediction results of the key points of each object by the m reference object key point detection models are reliable, that is, determine whether the key points of each object are is the key part of the target object: according to the predicted position of the key point corresponding to the key part of the object in the m key point detection results, determine the reference position corresponding to the key part of the object; then, for each key point detection result, determine the object The distance between the predicted position of the key point corresponding to the key part and the reference position corresponding to the key point of the object, and according to the distance, determine whether the key point detection result belongs to the target key point detection result corresponding to the key part of the object; and then, statistics The target number of target key point detection results corresponding to the key parts of the object in the m key point detection results; if the target number exceeds the preset number, it can be determined that the key part of the object is the key part of the target object; if the target number does not exceed preset quantity, then it can be determined that the key part of
  • the predicted position of the key point corresponding to the key part of the object is obtained by performing object key point detection processing on the target training image with reference to the key point detection model of the object, that is, the position prediction of the key point of the object output by the reference object key point detection model result.
  • the reference position corresponding to the key part of the object is determined according to the position prediction results of the key part of the object in the key point detection results of multiple reference object key point detection models, for example, the key part of the object in the multiple key point detection results
  • the corresponding predicted positions are obtained by averaging; it is used to reflect the distribution of the predicted positions corresponding to the key parts of the object in the multiple key point detection results.
  • the server can calculate the average value of the predicted positions of the key points corresponding to the head in the m key point detection results, as the reference position corresponding to the head; then, use the head The corresponding reference position is the center, and for each key point detection result, determine the distance between the predicted position of the key point corresponding to the head and the reference position, and determine whether the distance is less than the first preset distance (for example, in the Under the situation that each position carries out normalization processing, this first preset distance can be 0.1), if less than, then can determine that this key point detection result belongs to the target key point detection result corresponding to head, otherwise, if not less than, Then it can be determined that the key point detection result does not belong to the target key point detection result corresponding to the
  • the above parameters such as the first preset distance and the preset number can be set according to actual needs, and the present application does not make any limitation thereto.
  • the implementation of the above-mentioned method of determining the reference position corresponding to the key part of the object and the method of judging whether the key point detection result belongs to the target key point detection result corresponding to the key part of the object can also be set according to actual needs. There is no limit to it either.
  • the reliability of the judgment results can be ensured, and the position prediction results of the key parts of the object with low reliability can be effectively filtered out.
  • the pseudo-label corresponding to the subsequently determined target training image has high reliability, avoiding a large amount of noise.
  • the server may further determine the label position corresponding to the key parts of the target object in the following manner: for each target key point detection result corresponding to the key parts of the target object, Determine the predicted position of the key point corresponding to the key part of the target object as the predicted target position corresponding to the key part of the target object; then, according to the predicted positions of each target corresponding to the key part of the target object, determine the corresponding target position of the key part of the target object Label the location.
  • the server may acquire the predicted position of the key point corresponding to the key part of the target object in the detection results of key points of the target corresponding to the key part of the target object, as the target corresponding to the key target of the target object predicted location. Furthermore, the server may calculate an average value of the predicted positions of the targets corresponding to the key parts of the target object as the labeled position corresponding to the key parts of the target object.
  • the labeling position corresponding to the key part of the target object is determined, which can ensure the accuracy of the determined labeling position.
  • the server may also use other methods to determine the label position corresponding to the key part of the target object, for example, directly refer to the reference position corresponding to the key part of the target object (that is, the key part of the target object in the m key point detection results The mean value of the predicted position of the corresponding key point) is used as the label position corresponding to the key part of the target object.
  • the present application does not make any limitation on the manner in which the server determines the marked position corresponding to the key part of the target object.
  • the server may further evaluate the key parts of the above target object from the perspective of time series continuity to determine whether to retain the key parts of the above target object .
  • the server can determine the video frame adjacent to the target training image in the target video as the reference training image, and obtain the key point labeling result corresponding to the reference training image, and the key point labeling result includes each target object in the reference training image The corresponding label positions of the key parts. Then, for each key part of the target object, determine its labeling position in the key point labeling result corresponding to the target training image as the first position, and determine its labeling position in the key point labeling result corresponding to the reference training image as the second position; according to the distance between the first position and the second position, determine whether to keep the first position in the key point labeling result corresponding to the target training image, and whether to keep the key point labeling result corresponding to the reference training image the second location.
  • the server may determine the target video to which the target training image belongs, and determine a video frame adjacent to the target training image in the target video as a reference training image, the reference training image may be a previous video frame of the target training image, It can also be the next video frame of the target training image. And obtain the key point labeling result corresponding to the reference training image.
  • the key point labeling result corresponding to the reference training image is generated in the same way as the key point labeling result corresponding to the target training image introduced above.
  • the reference training image corresponds to
  • the key point labeling results of the reference training image include the corresponding labeling positions of the key parts of each target object in the reference training image.
  • the server can determine the labeling position corresponding to the key point of the target object in the key point labeling result corresponding to the target training image as the first position, and determine the corresponding position of the target object in the reference training image.
  • the marked position corresponding to the key part of the target object in the key point labeling result is used as the second position; then, calculate the distance between the first position and the second position, and judge whether the distance is less than the second preset distance (for example, in the Under the situation that each position carries out normalization processing, this second preset distance can be 0.2), if less than, then can keep target training image and reference training image respectively in the corresponding key point labeling result of this target object key position corresponding Marking positions (i.e., the first position and the second position), on the contrary, if not less than, the marking position corresponding to the key part of the target object in the key point marking results corresponding to the target training image and the reference training image can be eliminated, that is, the target Object key parts are regarded as invalid object key parts.
  • the second preset distance for example, in the Under the situation that each position carries out normalization processing, this second preset distance can be 0.2
  • the further screening process of the key parts of the target object based on the time series continuity is helpful to improve the accuracy of the determined key point labeling results, and can further filter out the possible noise in the key point labeling results.
  • Step 203 The target training image and its corresponding key point labeling results are used as training samples; the key point labeling results include the labeling positions corresponding to the key parts of the target object.
  • the server After the server obtains the key point labeling results corresponding to the target training image (including the labeling positions corresponding to the key parts of each target object in the target training image) through the above method, the server can use the target training image and its corresponding key point labeling results , to form the training samples used to train the target object key point detection model that needs to be put into practical application.
  • the key point labeling result corresponding to the target training image here is essentially the pseudo-label corresponding to the target training image.
  • Step 204 Based on the training samples, train a target object key point detection model.
  • the server can generate a large number of training samples through the above steps 201 to 203, and then use the generated training samples to train the key point detection model of the target object that needs to be put into practical application.
  • the key point detection model of the target object is a model to be trained to detect the position of the key parts of the object on the object to be detected in the image. It is a model that needs to be put into practical application.
  • the model structure of the target object key point detection model is usually relatively simple, that is, the complexity of the target object key point detection model is usually lower than the complexity of the above reference object key point detection model.
  • the key point detection model of the target object may be a model applied in a somatosensory game to detect changes in player's body movements, which is used to detect the position of important joint points of the human body in the image captured by the camera, so as to determine the position of the human body action posture.
  • the server may implement the training of the target object key point detection model based on the process shown in FIG. 4 . As shown in Figure 4, the server specifically needs to perform the following steps when training the target key point detection model:
  • Step 2041 Through the target object key point detection model, perform object key point detection processing on the target training image in the training sample, and obtain the training key point detection result corresponding to the target training image; the training key The point detection result includes the training predicted positions of the key points corresponding to the key parts of the object.
  • the server When the server specifically uses the training samples to train the target object key point detection model, it can input the target object key point detection model in the training sample into the target object key point detection model, and the target object key point detection model analyzes and processes the input target training image, A training key point detection result corresponding to the target training image may be correspondingly output, and the training key point detection result includes training prediction positions of key points corresponding to each of the key parts of the object in the target training image.
  • the working principle of the key point detection model of the target object is basically the same as that of the key point detection model of the reference object above, but there are differences in the model structure of the two.
  • the structure of the key point detection model of the target object is higher than that of the key point detection model of the reference object.
  • the structure of the detection model is simpler.
  • the working principle of the target object key point detection model can also be shown in Figure 3, that is, the target object key point detection model with a size of H ⁇ W is input into the target object key point detection model, and the target object key point detection model is trained by processing the target object.
  • the output training key point detection result should be a thermal feature map with a size of K ⁇ H ⁇ W, where the kth thermal feature map yk is used to represent the predicted position of the key point corresponding to the key part of the kth object.
  • Step 2042 Construct a target loss function according to the training key point detection results corresponding to the target training image and the key point labeling results in the training samples.
  • the server uses the key point detection model of the target object to perform key point detection processing on the target training image, and after obtaining the training key point detection result corresponding to the target training image, it may The difference, construct the target loss function.
  • the server can construct the target loss function in the following way: For each key point in the training key point detection result, determine whether the key part of the object corresponding to the key point is the target in the key point labeling result The key part of the object; if so, according to the training prediction position corresponding to the key point and the label position of the key part of the target object corresponding to the key point, the loss item corresponding to the key point is constructed, and the loss item corresponding to the key point is configured correspondingly The first loss weight of the key point; if not, according to the training prediction position corresponding to the key point, construct the loss item corresponding to the key point, and configure the corresponding second loss weight for the loss item corresponding to the key point; here the first The first loss weight is greater than 0, and the second loss weight is equal to 0. Furthermore, according to the loss items corresponding to each key point in the training key point detection results and their corresponding loss weights, the target loss function is constructed.
  • step 202 when determining the key point labeling result corresponding to the target training image, it will judge whether it is a key part of the target object for each key part of the object, and only determine the key part corresponding to the target object. Only use the labeling position corresponding to the key part of the target object to determine the key point labeling result. During this process, the key points corresponding to some key parts of the object will be ignored. Although the key points corresponding to these key parts of the object may actually exist, the embodiment of the present application considers the key points corresponding to the key parts of the object through a series of evaluation processes. is not reliable, therefore, it is not added to the key point labeling results corresponding to the target training image.
  • MSE weighted mean squared error
  • the server can first judge whether the key part of the object corresponding to the key point is a key part of a target object in the key point labeling result, that is, judge the key point labeling result Whether there is a label position of the key part of the object corresponding to the key point in . If so, the server can construct the loss item corresponding to the key point according to the difference between the training prediction position corresponding to the key point and the labeling position of the key part of the object corresponding to the key point in the key point labeling result, and at the same time, the loss item Configure the first loss weight greater than 0 (for example, 1).
  • the server can construct the loss item corresponding to the key point according to the training prediction position corresponding to the key point and the preset reference label position (can be set randomly), and configure the second loss weight equal to 0 for the loss item . Furthermore, the server may construct a target loss function according to the loss items corresponding to each key point in the training key point detection result and their corresponding loss weights.
  • L mse is the target loss function constructed;
  • K is the total number of key parts of the object to be detected by the target object key point detection model.
  • the key point labeling result includes the kth object
  • the label position corresponding to the key part is the labeling position corresponding to the key part of the k-th object in the key point labeling result; in the case where the key point labeling result does not include the labeling position corresponding to the key part of the k-th object, Annotate locations for randomly set references.
  • the server may also construct a first reference loss function from the perspective of temporal continuity, so that based on the above target loss function and the first reference loss function, Co-training object keypoint detection models.
  • the server can determine the video frame adjacent to the target training image in the target video as the reference training image, and obtain the training key point detection result corresponding to the reference training image, where the training key point detection result corresponding to the reference training image, It is obtained by performing object key point detection processing on the reference training image through the target object key point detection model, including the training predicted positions of the key points corresponding to the key parts of the object in the reference training image. Then, the server may construct a first reference loss function according to the corresponding training key point detection results of the target training image and the reference training image.
  • the embodiment of the present application innovatively designs the first reference loss function for constraining the continuity between frames, that is, according to the key point detection results of the target object key point detection model for the training key point detection results of two adjacent video frames, construct the The first reference loss function of the key point detection model of the target object is assisted in training, so that the position of the key parts of the object between frames will not change too much through the first reference loss function.
  • the server can determine the training prediction position of the key point corresponding to the key part of the object in the target training image as the third position, and determine the training prediction position of the key point corresponding to the key part of the object in the reference training image.
  • the predicted position is the fourth position; then, according to the distance between the third position and the fourth position, a loss item corresponding to the key part of the object is constructed, and a corresponding loss weight is configured for the loss item.
  • the server may construct the first reference loss function according to the loss items corresponding to each key part of the object and the corresponding loss weight.
  • the server may determine the predicted position of the key point corresponding to the key part of the object in the prediction result of the key point of training corresponding to the target training image as the third position, and determine the training key point corresponding to the reference training image The predicted position of the key point corresponding to the key part of the object in the prediction result is used as the fourth position.
  • the server can calculate the distance between the third position and the fourth position, and construct the loss item corresponding to the key part of the object according to the distance, and determine the loss weight corresponding to the loss item according to the distance; for example, if the If the distance is less than the third preset distance (for example, in the case of normalizing each position, the third preset distance can be 0.1), it can be determined that the loss weight corresponding to the loss item is equal to 0, if the distance is not is less than the third preset distance, it can be determined that the loss weight corresponding to the loss item is equal to 1; of course, for the two cases where the distance is less than the third preset distance and not less than the third preset distance, the server can also be configured accordingly For other loss weights, it is only necessary to ensure that the loss weight configured when the distance is less than the third preset distance is smaller than the loss weight configured when the distance is not less than the third preset distance. Furthermore, the server may construct the first reference loss function according to the loss items and loss weights corresponding to each key part of
  • L consistency is the first reference loss function constructed; K is the total number of key parts of the object to be detected by the key point detection model of the target object.
  • (y t, k -y t-1, k ) 2 is the loss item corresponding to the key part of the k-th object; among them, y t, k is the key part of the k-th object in the training key point detection result corresponding to the target training image
  • the training prediction position of the corresponding key point, y t-1,k is the training prediction position of the key point corresponding to the kth object key part in the training key point detection result corresponding to the reference training image.
  • w k is the loss weight corresponding to the above loss item; in the case where the distance between y t, k and y t-1, k is less than the third preset distance, w k can be equal to 0, for example; when y t, k and In the case where the distance between y t-1, k is not less than the third preset distance, w k may be equal to 1, for example.
  • the embodiment of the present application may also introduce the idea of knowledge distillation into the training process of the target object key point detection model, so as to further improve the model performance of the trained target object key point detection model.
  • Knowledge distillation is a model training method that uses the knowledge learned by a complex model (also known as a teacher model) to guide a simple model (also known as a student model), aiming to make the simple model have comparable performance to the complex model, and the simple Compared with complex models, the number of parameters of the model is greatly reduced, so that the compression and acceleration of the model can be achieved.
  • the server may obtain a first intermediate processing result generated when a certain reference object key point detection processes the target training image, and acquire a second intermediate processing result generated when the target object key point detection model processes the target training image. Furthermore, according to the first intermediate processing result and the second intermediate processing result, a second reference loss function is constructed.
  • the server may select a reference object key point detection model that is closer to the model structure of the trained target object key point detection model from the m reference object key point detection models as the teacher model; then, obtain the teacher
  • the intermediate processing result generated when the model processes the target training image is used as the first intermediate processing result.
  • the teacher model processes the target training image
  • the features extracted by a certain feature extraction layer from the target training image are obtained as the first intermediate process result.
  • the server uses the target object key point detection model to process the target training image, it can obtain the intermediate processing result generated when the target object key point detection model processes the target training image as the second intermediate processing result, for example, obtain the target object key
  • the point detection model processes the target training image, the features extracted by a certain feature extraction layer from the target training image are used as the second intermediate result.
  • first intermediate processing result and the second intermediate processing result obtained here should have the same dimension and should come from the same processing stage, for example, the first intermediate processing result and the second intermediate processing result are from the reference object
  • the encoding stage of the key point detection model and the key point detection model of the target object, and/or the first intermediate processing result and the second intermediate processing result are respectively from the decoding stage of the reference object key point detection model and the target object key point detection model; the present application
  • the embodiment does not make any limitation on the first intermediate processing result and the second intermediate processing result.
  • the server may construct a second reference loss function according to a difference between the first intermediate processing result and the second intermediate processing result.
  • the minimization of the second loss function can be set as the training goal, and the model parameters of the key point detection model of the target object to be trained can be adjusted.
  • the second reference loss function based on knowledge distillation is introduced, and the key point detection model of the reference object with better model performance is used to guide the training of the key point detection model of the target object, which helps to improve the key point detection of the target object.
  • the model performance of the model and helps to make the trained target object key point detection model converge quickly, and improve the efficiency of model training.
  • Step 2043 Adjust model parameters of the target object key point detection model based on the target loss function.
  • step 2042 after the server constructs the target loss function for training the target object key point detection model, it can set the minimum target loss function as the training goal, and adjust the model parameters of the target object key point detection model so that the target object key point detection model The point detection model tends to converge.
  • the server can adjust the model parameters of the target object key point detection model based on the target loss function and the first reference loss function .
  • the server can construct the total loss function in the model training phase according to the target loss function and the first reference loss function through the following formula (3):
  • L is the total loss function in the model training phase
  • L mse is the target loss function
  • L consistency is the first reference loss function
  • ⁇ is the weight configured in advance for the first reference loss function.
  • the model parameters of the target object key point detection model are adjusted so that the target object key point detection model tends to converge.
  • the server can adjust the model parameters of the target object key point detection model based on the target loss function and the second reference loss function.
  • the server can construct the total loss function in the model training phase according to the target loss function and the second reference loss function in a manner similar to the above-mentioned construction of the total loss function based on the target loss function and the first reference loss function. Furthermore, with the minimization of the total loss function as the training goal, the model parameters of the target object key point detection model are adjusted so that the target object key point detection model tends to converge.
  • the server can adjust the key point detection of the target object based on the target loss function, the first reference loss function and the second reference loss function.
  • the model parameters for the model That is, the server can configure corresponding weights for the first reference loss function and the second reference loss function, and then, according to the target loss function, the first reference loss function and its corresponding weight, and the second reference loss function and its corresponding weight Weights, constructing the total loss function in the model training phase; and taking the minimization of the total loss function as the training model, adjusting the model parameters of the target object key point detection model, so that the target object key point detection model tends to converge.
  • the server Based on different target training images, the server repeatedly executes the above step 2041 to step 2043 to realize the iterative training of the key point detection model of the target object until the key point detection model of the target object satisfies the end condition of model training.
  • the object key point detection model is the target object key point detection model that can be put into practical application.
  • the training end condition here, for example, can be that the model performance of the trained target object key point detection model meets the preset requirements (such as the detection accuracy of the target object key point detection model reaches the preset accuracy threshold, etc.), and for example can The model performance of the key point detection model of the target object is no longer significantly improved (such as using the test sample set to test the key point detection model of the target object obtained by several rounds of training, and determine the detection of the key point detection model of the target object obtained by several rounds of training. The difference between the accuracy rates is less than the preset gap threshold, etc.), and for example, the number of iterative training for the key point detection model of the target object reaches the preset number of times.
  • This application does not make any limitation on the training end condition.
  • the target key point detection process is performed on the target training image through multiple reference object key point detection models with complex structures, and the obtained Multiple keypoint detection results. Then, based on the principle that the predicted positions of the key parts of the same object in each key point detection result should be basically the same, for each key part of the object, according to the predicted position of the key point corresponding to the key part of the object in each key point detection result, measure Whether the key point detection model of each reference object is reliable for the position prediction result of the key part of the object is to determine whether the key part of the object is the key part of the target object.
  • the training samples After confirming that the position prediction results of the key point detection models of each reference object are reliable for the key parts of the object, further determine the labeling position corresponding to the key part of the target object, that is, the pseudo-label; then, use the target training image and each target in it
  • the corresponding labeled positions of the key parts of the object constitute the training samples.
  • the training samples including such pseudo-labels the training needs to be put into practical application and the target object key point detection model with a relatively simple structure can guarantee the performance of the trained target object key point detection model to a certain extent.
  • model training method is used as an example to train a human body key point detection model for detecting player gestures in a somatosensory game, and an overall exemplary introduction to the model training method is given below.
  • Somatosensory games refer to games that use the body to feel, breaking through the traditional operation method of simply inputting with handle buttons.
  • Somatosensory games are a new type of electronic games that are operated through changes in body movements.
  • the camera can capture images including the player's limbs, and transmit the captured images to the game server in the background.
  • the game server detects the key points of the human body through the human body key point detection model to detect the position of the important joint points of the player's bones in the image. And based on this, the body movements made by the player are recognized, and then the corresponding game logic is executed.
  • Figure 5 is the overall implementation architecture diagram of the above-mentioned somatosensory game.
  • FIG. 6 is a schematic diagram of a training framework for the human body key point detection model applied in the above-mentioned somatosensory game provided by the embodiment of the present application.
  • the server can select two adjacent video frames x t-1 and x t from it, and then, through pseudo The label generation network generates pseudo-labels for each of the two video frames and Furthermore, through the human body key point detection model that needs to be trained, the human body key point detection process is performed on the two video frames, and the corresponding thermal feature maps of the two video frames are obtained.
  • the training goal of the model training process is to make the two A thermal feature map is similar to the corresponding pseudo-label.
  • the embodiment of this application is also based on and A specific loss function is designed for the continuity between frames, enabling the model to achieve self-supervised learning. In the test or application stage, it is only necessary to input pictures or videos into the trained human body key point detection model, and the pseudo-label generation network only exists in the model training stage.
  • the core modules involved in the technical solution provided by the embodiment of the present application mainly include two: one is the pseudo-label generation network, which uses the existing multiple human body key point detection models with complex structure and better performance to detect unlabeled video
  • the video frame in is subjected to human body key point detection processing, and then based on the respective detection results of each human body key point detection model, a pseudo-label corresponding to the video frame is generated and used in the subsequent model training process.
  • the other is the inter-frame continuity constraint loss function, which is based on the a priori condition that the range of character movements between video frames will not change greatly, and the inter-frame continuity loss function is designed so that the model can achieve unsupervised learning.
  • the pseudo-label generation network includes a human body key point prediction module and a prediction result integration module.
  • the human body key point prediction module includes multiple At present, there are existing human body key point detection models with complex structure and better performance; in the case of m (m is an integer greater than 1) human body key point detection model included in the human body key point detection module, it is assumed that a picture is input x t , then through the m human body key point detection model to perform human body key point detection processing on the picture, the corresponding prediction results of the m human body key point detection models will be obtained
  • the prediction result integration module is used to complete the integration of m prediction results through the following steps.
  • the embodiment of the present application performs strict filtering processing on the prediction results in both the spatial dimension and the time dimension, so as to ensure The generated pseudo-labels have higher accuracy.
  • the specific integration method is as follows:
  • the prediction result includes the confidence that each pixel in the picture x t-1 belongs to a key point of the human body.
  • the embodiment of the present application may default that it does not belong to the key point of the human body, so It can be ignored.
  • the server can calculate the mean value of the predicted coordinates of the key point corresponding to the human body part in the m prediction results, as the reference coordinate corresponding to the human body part. Furthermore, it is possible to determine whether the predicted result belongs to the qualified predicted result corresponding to the human body part according to the distance between the predicted coordinate of the key point corresponding to the human body part in each predicted result and the reference coordinate; specifically, when the predicted result When the distance between the predicted coordinates of the key points corresponding to the human body part and the reference coordinates is less than 0.1 (after normalization), the predicted result can be considered as a qualified predicted result corresponding to the human body part; otherwise, the predicted result can be considered as The result does not belong to the qualified prediction result corresponding to the body part.
  • the server can determine that the human body part is a valid human body part, and the key points corresponding to the valid human body part in the qualified prediction results corresponding to the valid human body part The predicted coordinates are averaged to obtain the label coordinates corresponding to the valid body part; otherwise, the server can determine that the body part belongs to an invalid body part, ignore the invalid body part, and determine its corresponding label coordinates. Furthermore, the server can use the label coordinates corresponding to each effective human body part to form a pseudo-label corresponding to the picture x t
  • the server can also obtain the video frame x t-1 adjacent to the picture x t in the video; and then, according to this Pseudo-labels for each of the two video frames and For these two pseudo-labels and The label coordinates included in are further screened. Specifically, if the pseudo-label The label coordinates corresponding to the kth human body part in with pseudo tags The label coordinates corresponding to the kth human body part in The distance between exceeds 0.2 (after normalization), then the pseudo-label can be middle and pseudo tags middle are ignored.
  • L mse is the loss function constructed; K is the total number of human body parts to be detected by the human body key point detection model. is the loss item corresponding to the k-th human body part; among them, y t,k is the human body key point detection model to be trained for the predicted position of the k-th human body part; in the pseudo-label Including the marked position corresponding to the kth human body part, pseudo label The marked position corresponding to the k-th human body part in ; in the pseudo-label In the case of not including the marked position corresponding to the kth human body part, Annotate locations for randomly set references.
  • the embodiment of the present application finds that there will not be too much change difference between two adjacent video frames before and after a video, that is, the change range of human body parts in adjacent video frames will not be very large. Based on empirical knowledge, this application also designs a constraint function of inter-frame continuity, which is used to realize the unsupervised learning of the human body key point detection model, so as to ensure that the key point detection model of the human body does not predict the key points between frames. The point coordinates will not jump too much.
  • the loss function is as follows:
  • L consistency is the constraint function of inter-frame continuity
  • K is the total number of human body parts to be detected by the human body key point detection model.
  • (y t, k -y t-1, k ) 2 is the loss item corresponding to the k-th human body part; where, y t, k is the loss item corresponding to the k-th human body part in the training prediction result corresponding to the t-th video frame
  • the training prediction position of the key point, y t-1,k is the training prediction position of the key point corresponding to the kth human body part in the training prediction result corresponding to the t-1th video frame.
  • ⁇ is the pre-configured weight for L consistency .
  • the server can train the human key point detection model to be trained based on the loss function L.
  • the average accuracy rate (mean Average Precision, mAP) index of the whole class increased from 0.457 to 0.521.
  • the inventor of the present application also conducted a comparative test on the key point detection model of the human body currently used in somatosensory games and the key point detection model of the human body trained by the above method.
  • the results of the comparison test are shown in Figure 7, where (a) is passed The detection result determined by the human body key point detection model obtained by the program training provided by the embodiment of the present application, (b) is the detection result determined by the existing human body key point detection model, which can be found by comparison, (a) is compared with (b) ) is more accurate, the key points corresponding to the left and right feet in (a) are not transposed, and the key points corresponding to the left and right feet in (b) are transposed.
  • model training method provided in the embodiment of the present application can not only be used to train the human body key point detection model applied in the somatosensory game scene, but also can be used to train the object key point detection model in other scenes, such as video shooting
  • the function of adding special effects to human body or animals in the scene often requires the object key point detection model to detect the key points of the object in the image.
  • the solution provided by the embodiment of this application can also be used to train the object key point detection model in this kind of scene , there is no limitation on the applicable application scenarios of the solutions provided in the embodiments of the present application.
  • the present application also provides a corresponding model training device, so that the above model training method can be applied and realized in practice.
  • FIG. 8 is a schematic structural diagram of a model training device 800 corresponding to the model training method shown in FIG. 2 above.
  • the model training device 800 includes:
  • the key point detection module 801 is used to respectively perform object key point detection processing on the target training image through m reference object key point detection models, and obtain key point detection results corresponding to each of the m reference object key point detection models;
  • the key point detection results include the predicted positions of the key points corresponding to the key parts of multiple objects in the target training image; the m is an integer greater than 1;
  • Part labeling module 802 for each key part of the object, according to the predicted position of the key point corresponding to the key point of the object in the m key point detection results, determine whether the key part of the object is a key part of the target object position; if so, according to the predicted position of the key point corresponding to the key point of the target object in the m key point detection results, determine the label position corresponding to the key part of the target object;
  • the sample generation module 803 is used to use the target training image and its corresponding key point labeling results as training samples; the key point labeling results include the respective labeling positions corresponding to the key parts of the target object;
  • a model training module 804 configured to train a target object key point detection model based on the training samples.
  • the part labeling module 802 is specifically used for:
  • For each key point detection result determine the distance between the predicted position of the key point corresponding to the key point of the object and the reference position corresponding to the key point of the object, and determine the key point according to the distance Whether the detection result belongs to the target key point detection result corresponding to the key part of the object;
  • the target quantity exceeds the preset quantity, determine the key part of the object as the key part of the target object; if the target quantity does not exceed the preset quantity, determine the key part of the object as an invalid key part of the object.
  • the part labeling module 802 is specifically used for:
  • the marked position corresponding to the key part of the target object is determined.
  • FIG. 9 is a schematic structural diagram of another model training device 900 provided in an embodiment of the present application.
  • the device when the target training image is a video frame in the target video, the device also includes:
  • a reference data acquisition module 901 configured to determine a video frame adjacent to the target training image in the target video as a reference training image; and obtain a key point labeling result corresponding to the reference training image;
  • the labeling result screening module 902 is configured to, for each key part of the target object, determine that its labeling position in the key point labeling result corresponding to the target training image is the first position, and determine that its labeling position in the reference training image corresponds to The labeling position in the key point labeling result is the second position; according to the distance between the first position and the second position, determine whether to retain the key point labeling result corresponding to the target training image. A position, and whether to retain the second position in the key point labeling result corresponding to the reference training image.
  • the key point detection module 801 is specifically used for:
  • the object key point detection process is performed on the target training image through the reference object key point detection model, and it is determined that each pixel in the target training image belongs to the object key point. Confidence of the position;
  • a key point detection result corresponding to the reference object key point detection model is determined according to pixels in the target training image that belong to key parts of the object with a confidence level higher than a preset level.
  • FIG. 10 is a schematic structural diagram of another model training device 1000 provided in the embodiment of the present application.
  • the model training module 804 includes:
  • the key point detection sub-module 1001 is configured to use the target object key point detection model to perform object key point detection processing on the target training image in the training sample, and obtain the training key point detection corresponding to the target training image Result;
  • the training key point detection results include the training predicted positions of key points corresponding to each of the key parts of a plurality of objects;
  • a loss function construction sub-module 1002 configured to construct a target loss function according to the training key point detection results corresponding to the target training image and the key point labeling results in the training samples;
  • the parameter adjustment sub-module 1003 is configured to adjust the model parameters of the key point detection model of the target object based on the target loss function.
  • the loss function construction submodule 1002 is specifically used for:
  • the key part of the object corresponding to the key point is the key part of the target object in the key point labeling result; if so, then according to the key point corresponding to the key point Training the predicted position and the marked position of the key part of the target object corresponding to the key point, constructing the loss item corresponding to the key point, and configuring the corresponding first loss weight for the loss item corresponding to the key point; if not, According to the training prediction position corresponding to the key point, construct the loss item corresponding to the key point, and configure the corresponding second loss weight for the loss item corresponding to the key point; the first loss weight is greater than 0, the The second loss weight is equal to 0;
  • the target loss function is constructed according to the corresponding loss items and the corresponding loss weights of each key point in the training key point detection result.
  • the loss function construction submodule 1002 is also used for:
  • parameter adjustment submodule 1003 is specifically used for:
  • the loss function construction sub-module 1002 is specifically used for:
  • the training prediction position For each key part of the object, determine the training prediction position of the key point corresponding to the key part of the object in the target training image as the third position, and determine the key point corresponding to the key part of the object in the reference training image
  • the training prediction position is the fourth position; according to the distance between the third position and the fourth position, construct the loss item corresponding to the key part of the object, and configure the corresponding loss weight for the loss item;
  • the first reference loss function is constructed according to the loss items corresponding to the key parts of the object and the corresponding loss weights.
  • the loss function construction submodule 1002 is also used for:
  • parameter adjustment submodule 1003 is specifically used for:
  • the model training device When the above-mentioned model training device generates training samples for training the target object key point detection model that needs to be put into practical application, it will respectively perform object key point detection processing on the target training image through multiple reference object key point detection models with complex structures, and obtain Multiple keypoint detection results. Then, based on the principle that the predicted positions of the key parts of the same object in each key point detection result should be basically the same, for each key part of the object, according to the predicted position of the key point corresponding to the key part of the object in each key point detection result, measure Whether the key point detection model of each reference object is reliable for the position prediction result of the key part of the object is to determine whether the key part of the object is the key part of the target object.
  • the training samples After confirming that the position prediction results of the key point detection models of each reference object are reliable for the key parts of the object, further determine the labeling position corresponding to the key part of the target object, that is, the pseudo-label; then, use the target training image and each target in it
  • the corresponding labeled positions of the key parts of the object constitute the training samples.
  • the training samples including this kind of pseudo-label the training needs to be put into practical application and the target object key point detection model with a relatively simple structure can guarantee the performance of the trained target object key point detection model to a certain extent.
  • the embodiment of the present application also provides a computer device for training a model.
  • the computer device may specifically be a terminal device or a server.
  • the following will introduce the terminal device and the server provided in the embodiment of the present application from the perspective of hardware realization.
  • FIG. 11 is a schematic structural diagram of a terminal device provided by an embodiment of the present application. As shown in FIG. 11 , for ease of description, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application.
  • the terminal can be any terminal device including mobile phone, tablet computer, personal digital assistant, point of sales (POS), vehicle-mounted computer, etc. Taking the terminal as a computer as an example:
  • FIG. 11 is a block diagram showing a partial structure of a computer related to the terminal provided by the embodiment of the present application.
  • the computer includes: a radio frequency (Radio Frequency, RF) circuit 1110, a memory 1120, an input unit 1130 (including a touch panel 1131 and other input devices 1132), a display unit 1140 (including a display panel 1141), a sensor 1150 , an audio circuit 1160 (which can be connected to a speaker 1161 and a microphone 1162), a wireless fidelity (wireless fidelity, WiFi) module 1170, a processor 1180, and a power supply 1190 and other components.
  • RF Radio Frequency
  • FIG. 11 is not limited to the computer, and may include more or less components than those shown in the illustration, or combine some components, or arrange different components.
  • the memory 1120 can be used to store software programs and modules, and the processor 1180 executes various functional applications and data processing of the computer by running the software programs and modules stored in the memory 1120 .
  • the memory 1120 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playback function, an image playback function, etc.) and the like; Data created by the use of computers (such as audio data, phone books, etc.), etc.
  • the memory 1120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices.
  • the processor 1180 is the control center of the computer. It uses various interfaces and lines to connect various parts of the entire computer. By running or executing software programs and/or modules stored in the memory 1120, and calling data stored in the memory 1120, execution Various functions and processing data of the computer, so as to monitor the computer as a whole.
  • the processor 1180 may include one or more processing units; preferably, the processor 1180 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc. , the modem processor mainly handles wireless communications. It can be understood that the foregoing modem processor may not be integrated into the processor 1180 .
  • the processor 1180 included in the terminal also has the following functions:
  • the key point detection processing is performed on the target training image respectively, and key point detection results corresponding to the m reference object key point detection models are obtained;
  • the key point detection results include the The predicted positions of the key points corresponding to the key parts of multiple objects in the target training image;
  • the m is an integer greater than 1;
  • For each key part of the object determine whether the key part of the object is a key part of the target object according to the predicted position of the key point corresponding to the key point of the object in the m key point detection results; The predicted position of the key point corresponding to the key part of the target object in the m key point detection results, and determine the label position corresponding to the key part of the target object;
  • the target training image and its corresponding key point labeling results are used as training samples;
  • the key point labeling results include the corresponding labeling positions of the key parts of each target object;
  • a target object key point detection model is trained.
  • the processor 1180 is further configured to execute the steps of any implementation manner of the model training method provided in the embodiment of the present application.
  • FIG. 12 is a schematic structural diagram of a server 1200 provided in an embodiment of the present application.
  • the server 1200 can have relatively large differences due to different configurations or performances, and can include one or more central processing units (central processing units, CPU) 1222 (for example, one or more processors) and memory 1232, one or one
  • the storage medium 1230 (for example, one or more mass storage devices) for storing the application program 1242 or the data 1244.
  • the memory 1232 and the storage medium 1230 may be temporary storage or persistent storage.
  • the program stored in the storage medium 1230 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server.
  • the central processing unit 1222 may be configured to communicate with the storage medium 1230 , and execute a series of instruction operations in the storage medium 1230 on the server 1200 .
  • the server 1200 can also include one or more power supplies 1226, one or more wired or wireless network interfaces 1250, one or more input and output interfaces 1258, and/or, one or more operating systems, such as Windows Server TM , Mac OS XTM , UnixTM , LinuxTM , FreeBSDTM, etc.
  • one or more operating systems such as Windows Server TM , Mac OS XTM , UnixTM , LinuxTM , FreeBSDTM, etc.
  • the steps performed by the server in the foregoing embodiments may be based on the server structure shown in FIG. 12 .
  • CPU 1222 is used for carrying out following steps:
  • the key point detection processing is performed on the target training image respectively, and key point detection results corresponding to the m reference object key point detection models are obtained;
  • the key point detection results include the The predicted positions of the key points corresponding to the key parts of multiple objects in the target training image;
  • the m is an integer greater than 1;
  • For each key part of the object determine whether the key part of the object is a key part of the target object according to the predicted position of the key point corresponding to the key point of the object in the m key point detection results; The predicted position of the key point corresponding to the key part of the target object in the m key point detection results, and determine the label position corresponding to the key part of the target object;
  • the target training image and its corresponding key point labeling results are used as training samples;
  • the key point labeling results include the corresponding labeling positions of the key parts of each target object;
  • a target object key point detection model is trained.
  • the CPU 1222 can also be used to execute the steps of any implementation of the model training method provided in the embodiment of the present application.
  • An embodiment of the present application further provides a computer-readable storage medium for storing a computer program, and the computer program is used to execute any one of the implementation manners of a model training method described in the foregoing embodiments.
  • the embodiment of the present application also provides a computer program product or computer program, where the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium.
  • the processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes any one of the model training methods described in the foregoing embodiments.
  • the disclosed system, device and method can be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the units is only a logical function division. In actual implementation, there may be other division methods.
  • multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.
  • the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
  • the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disc, etc., which can store various media of computer programs. .
  • At least one (item) means one or more, and “multiple” means two or more.
  • “And/or” is used to describe the association relationship of associated objects, indicating that there can be three types of relationships, for example, “A and/or B” can mean: only A exists, only B exists, and A and B exist at the same time , where A and B can be singular or plural.
  • the character “/” generally indicates that the contextual objects are an “or” relationship.
  • At least one of the following” or similar expressions refer to any combination of these items, including any combination of single or plural items.
  • At least one item (piece) of a, b or c can mean: a, b, c, "a and b", “a and c", “b and c", or "a and b and c ", where a, b, c can be single or multiple.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Software Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Human Computer Interaction (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Mathematical Physics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Image Analysis (AREA)

Abstract

本申请实施例公开了一种人工智能领域的模型训练方法及相关装置，其中该方法包括：通过m个参考对象关键点检测模型分别对目标训练图像进行对象关键点检测处理，得到m个包括目标训练图像中多个对象关键部位各自对应的关键点的预测位置的关键点检测结果；针对每个对象关键部位，根据m个关键点检测结果中该对象关键部位对应的关键点的预测位置，确定该对象关键部位是否为目标对象关键部位；若是，则确定该目标对象关键部位对应的标注位置；将目标训练图像以及其中各目标对象关键部位各自对应的标注位置作为训练样本；基于训练样本，训练目标对象关键点检测模型。该方法能够提高所训练的目标对象关键点检测模型的性能。

Description

一种模型训练方法、装置、设备、介质和程序产品
本申请要求于2021年11月24日提交中国专利局、申请号为202111398623.5、申请名称为“一种模型训练方法及相关装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及人工智能技术领域，尤其涉及模型训练。
背景技术
人体关键点检测技术又被称为人体姿态估计技术，该技术用于识别图像中人体骨骼上重要关节点(例如鼻子、左右眼、左右耳、左右肩、左右肘、左右腕、左右臀、左右膝、左右脚踝等)的位置。人体关键点检测技术如今在很多场景中被广泛应用；例如，在体感游戏场景中，人体关键点检测技术用于检测玩家的肢体动作变化，以便后续根据检测结果执行相应的游戏逻辑。
人体关键点检测技术的实现主要依靠人体关键点检测模型，即通过人体关键点检测模型，根据输入的图像确定其中人体骨骼上重要关节点的位置。在实际应用中，由于标注人体关键点数据非常耗时费力，难以标注出充足的用于训练人体关键点检测模型的训练数据，并且为了保证人体关键点的检测效率，实际投入使用的人体关键点检测模型也不能太复杂；因此，相关技术训练实际投入使用的人体关键点检测模型时，通常先利用结构复杂、且性能较优的人体关键点检测模型，针对未标注的图像进行人体关键点检测处理，确定该图像中人体关键点的位置作为该图像的伪标签，进而，将该图像及其伪标签作为训练样本，利用该训练样本对需要实际投入使用的、结构较简单的人体关键点检测模型进行训练。
然而，即使是结构复杂的人体关键点检测模型，其模型性能也存在局限，难以保证所确定的伪标签的准确度，即通过上述方式生成的伪标签通常存在很多噪声。相应地，利用包括此种伪标签的训练样本训练需要投入实际使用的人体关键点检测模型，将会影响所训练的人体关键点检测模型的性能。
发明内容
本申请实施例提供了一种模型训练方法及相关装置，能够提高所确定的伪标签的准确度，进而提高基于包括该伪标签的训练样本训练的人体关键点检测模型的性能。
有鉴于此，本申请第一方面提供了一种模型训练方法，所述方法包括：
通过m个参考对象关键点检测模型，分别对目标训练图像进行对象关键点检测处理，得到所述m个参考对象关键点检测模型各自对应的关键点检测结果；所述关键点检测结果包括所述目标训练图像中多个对象关键部位各自对应的关键点的预测位置；所述m为大于1的整数；
针对每个所述对象关键部位，根据所述m个关键点检测结果中所述对象关键部位对应的关键点的预测位置，确定所述对象关键部位是否为目标对象关键部位；若是，则根据所述m个关键点检测结果中所述目标对象关键部位对应的关键点的预测位置，确定所述目标对象关键部位对应的标注位置；
将所述目标训练图像及其对应的关键点标注结果，作为训练样本；所述关键点标注结 果包括各所述目标对象关键部位各自对应的标注位置；
基于所述训练样本，训练目标对象关键点检测模型。
本申请第二方面提供了一种模型训练装置，所述装置包括：
关键点检测模块，用于通过m个参考对象关键点检测模型，分别对目标训练图像进行对象关键点检测处理，得到所述m个参考对象关键点检测模型各自对应的关键点检测结果；所述关键点检测结果包括所述目标训练图像中多个对象关键部位各自对应的关键点的预测位置；所述m为大于1的整数；
部位标注模块，用于针对每个所述对象关键部位，根据所述m个关键点检测结果中所述对象关键部位对应的关键点的预测位置，确定所述对象关键部位是否为目标对象关键部位；若是，则根据所述m个关键点检测结果中所述目标对象关键部位对应的关键点的预测位置，确定所述目标对象关键部位对应的标注位置；
样本生成模块，用于将所述目标训练图像及其对应的关键点标注结果，作为训练样本；所述关键点标注结果包括各所述目标对象关键部位各自对应的标注位置；
模型训练模块，用于基于所述训练样本，训练目标对象关键点检测模型。
本申请第三方面提供了一种计算机设备，所述设备包括处理器以及存储器：
所述存储器用于存储计算机程序；
所述处理器用于根据所述计算机程序，执行如上述第一方面所述的模型训练方法的步骤。
本申请第四方面提供了一种计算机可读存储介质，所述计算机可读存储介质用于存储计算机程序，所述计算机程序用于执行上述第一方面所述的模型训练方法的步骤。
本申请第五方面提供了一种计算机程序产品或计算机程序，该计算机程序产品或计算机程序包括计算机指令，该计算机指令存储在计算机可读存储介质中。计算机设备的处理器从计算机可读存储介质读取该计算机指令，处理器执行该计算机指令，使得该计算机设备执行上述第一方面所述的模型训练方法的步骤。
从以上技术方案可以看出，本申请实施例具有以下优点：
本申请实施例提供了一种模型训练方法，该方法生成用于训练需要投入实际应用的目标对象关键点检测模型的训练样本时，会通过m(m为大于1的整数)个参考对象关键点检测模型，分别对目标训练图像进行对象关键点检测处理，得到m个参考对象关键点检测模型各自对应的关键点检测结果，此处每个关键点检测结果包括目标训练图像中多个对象关键部位各自对应的关键点的预测位置。然后，基于各关键点检测结果中相同的对象关键部位的预测位置应基本一致的原理，针对每个对象关键部位，根据各个关键点检测结果中该对象关键部位对应的关键点的预测位置，衡量各个参考对象关键点检测模型对于该对象关键部位的位置预测结果是否可靠，即确定该对象关键部位是否为目标对象关键部位；在确定各个参考对象关键点检测模型对于该对象关键部位的位置预测结果可靠的情况下，再进一步确定该目标对象关键部位对应的标注位置即伪标签；进而，利用该目标训练图像以及其中各目标对象关键部位各自对应的标注位置组成训练样本；如此，忽略位置预测结果不可靠的对象关键部位，仅利用位置预测结果可靠的对象关键部位对应的标注位置作为伪标 签，能够有效地减少所确定的伪标签中存在的噪声，提高该伪标签的准确度。相应地，基于包括该种伪标签的训练样本，训练需要投入实际应用、结构比较简单的目标对象关键点检测模型，能够在一定程度上保证所训练的目标对象关键点检测模型的性能。
附图说明
图1为本申请实施例提供的模型训练方法的应用场景示意图；
图2为本申请实施例提供的模型训练方法的流程示意图；
图3为本申请实施例提供的参考对象关键点检测模型的工作原理示意图；
图4为本申请实施例提供的目标对象关键点检测模型的训练流程示意图；
图5为本申请实施例提供的体感游戏的整体实现架构图；
图6为本申请实施例提供的应用在体感游戏中的人体关键点检测模型的训练架构示意图；
图7为本申请实施例提供的人体关键点检测模型的测试结果示意图；
图8为本申请实施例提供的一种模型训练装置的结构示意图；
图9为本申请实施例提供的另一种模型训练装置的结构示意图；
图10为本申请实施例提供的又一种模型训练装置的结构示意图；
图11为本申请实施例提供的终端设备的结构示意图；
图12为本申请实施例提供的服务器的结构示意图。
具体实施方式
为了使本技术领域的人员更好地理解本申请方案，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”、“第三”、“第四”等(如果存在)是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便这里描述的本申请的实施例能够以除了在这里图示或描述的那些以外的顺序实施。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或单元的过程、方法、系统、产品或设备不必限于清楚地列出的那些步骤或单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。
随着人工智能技术研究和进步，人工智能技术在多个领域展开研究和应用，例如常见的智能家居、智能穿戴设备、虚拟助理、智能音箱、智能营销、无人驾驶、自动驾驶、无人机、机器人、智能医疗、智能客服、车联网、自动驾驶、智慧交通等，相信随着技术的发展，人工智能技术将在更多的领域得到应用，并发挥越来越重要的价值。
本申请实施例提供的方案涉及人工智能的计算机视觉技术，且可以应用于上述各类领域中，具体通过如下实施例进行说明：
相关技术中，直接将结构复杂的对象关键点检测模型对于未标注图像的关键点检测结果，作为该未标注图像对应的伪标签；进而，利用由该未标注图像及其对应的伪标签组成 的训练样本，对结构简单、需要投入实际使用的对象关键点检测模型进行训练。通过该种方式生成的伪标签通常存在较多的噪声，准确度较低，而这将对所训练的对象关键点检测模型的模型性能产生负面影响。
为了解决上述相关技术存在的问题，本申请实施例提供了一种模型训练方法，在该模型训练方法中，先通过m(m为大于1的整数)个参考对象关键点检测模型，分别对目标训练图像进行对象关键点检测处理，得到m个参考对象关键点检测模型各自对应的关键点检测结果，此处的每个关键点检测结果包括目标训练图像中多个对象关键部位各自对应的关键点的预测位置。然后，针对每个对象关键部位，根据m个关键点检测结果中该对象关键部位对应的关键点的预测位置，确定该对象关键部位是否为目标对象关键部位；若是，则根据m个关键点检测结果中该目标对象关键部位对应的关键点的预测位置，确定该目标对象关键部位对应的标注位置。进而，利用该目标训练图像以及其中各目标对象关键部位各自对应的标注位置作为训练样本；并且基于该训练样本，训练需要投入实际应用的、结构较简单的目标对象关键点检测模型。
上述模型训练方法生成用于训练需要投入实际应用的目标对象关键点检测模型的训练样本时，会通过多个结构复杂的参考对象关键点检测模型分别对目标训练图像进行对象关键点检测处理，得到多个关键点检测结果。然后，基于各关键点检测结果中相同的对象关键部位的预测位置应基本一致的原理，针对每个对象关键部位，根据各个关键点检测结果中该对象关键部位对应的关键点的预测位置，衡量各个参考对象关键点检测模型对于该对象关键部位的位置预测结果是否可靠，即确定该对象关键部位是否为目标对象关键部位。在确定各个参考对象关键点检测模型对于该对象关键部位的位置预测结果可靠的情况下，再进一步确定该目标对象关键部位对应的标注位置即伪标签；进而，利用该目标训练图像以及其中各目标对象关键部位各自对应的标注位置组成训练样本。如此，忽略位置预测结果不可靠的对象关键部位，仅利用位置预测结果可靠的对象关键部位对应的标注位置作为伪标签，能够有效地减少所确定的伪标签中存在的噪声，提高该伪标签的准确度。相应地，基于包括该种伪标签的训练样本，训练需要投入实际应用、结构比较简单的目标对象关键点检测模型，能够在一定程度上保证所训练的目标对象关键点检测模型的性能。
应理解，本申请实施例提供的模型训练方法可以由具备图像处理能力的计算机设备执行，该计算机设备可以是终端设备或服务器。其中，终端设备具体可以为计算机、智能手机、平板电脑、个人数字助理(Personal Digital Assistant，PDA)等；服务器具体可以为应用服务器或Web服务器，在实际部署时，可以为独立服务器，也可以为由多个物理服务器构成的集群服务器或云服务器。本申请实施例涉及的处理数据(如训练图像、关键点检测结果、关键点标注结果等)可以保存于区块链上。
为了便于理解本申请实施例提供的模型训练方法，下面以该模型训练方法的执行主体为服务器为例，对该模型训练方法的应用场景进行示例性介绍。
参见图1，图1为本申请实施例提供的模型训练方法的应用场景示意图。如图1所示，该应用场景中包括服务器110和数据库120，服务器110可以通过网络从数据库120中调取数据，或者数据库120也可以集成在服务器110中。其中，服务器110用于执行本申请实施例提供的 模型训练方法，以训练需要投入实际应用的、结构较简单的目标对象关键点检测模型；数据库120用于存储未标注的训练图像。
在实际应用中，服务器110可以从数据库120中调取目标训练图像，该目标训练图像中包括待检测对象，该待检测对象上包括若干对象关键部位；例如，目标训练图像中可以包括清晰且完整的待检测人体。
然后，服务器110可以通过m(m为大于1的整数)个参考对象关键点检测模型，分别对所调取的目标训练图像进行对象关键点检测处理，得到m个参考对象关键点检测模型各自对应的关键点检测结果；此处的每个关键点检测结果中可以包括目标训练图像中多个对象关键部位各自对应的关键点的预测位置。需要说明的是，上述参考对象关键点检测模型是预先训练好的对象关键点检测模型，其通常能够较准确地检测出图像中待检测对象上各对象关键部位的位置，其模型结构通常比较复杂。
作为一种示例，上述各参考对象关键点检测模型可以是参考人体关键点检测模型，其用于检测人体骨骼上各重要关节的位置。相应地，通过各参考人体关键点检测模型分别对目标训练图像进行人体关键点检测处理，将得到各参考人体关键点检测模型各自对应的关键点检测结果，该关键点检测结果包括目标训练图像中待检测人体上各重要关节各自对应的关键点的预测位置。
接着，服务器110可以针对每个对象关键部位，根据这m个关键点检测结果中该对象关键部位对应的关键点的预测位置，评判这m个参考对象关键点检测模型对于该对象关键部位的位置预测结果是否可靠，即确定该对象关键部位是否为目标对象关键部位。若确定该对象关键部位为目标对象关键部位，则可以根据m个关键点检测结果中该目标对象关键部位对应的关键点的预测位置，确定该目标对象关键部位对应的标注位置。
应理解，不同的参考对象关键点检测模型对于目标训练图像中同一对象关键部位的位置预测结果应基本一致，即不同的参考对象关键点检测模型对于同一对象关键部位的预测位置彼此之间的距离应较小。若m个关键点检测结果中对于同一对象关键部位的预测位置彼此之间的距离较大，则说明这m个关键点检测结果中该对象关键部位对应的预测位置不可靠，可能存在较大的噪声；为了避免将此噪声引入后续的模型训练，可以忽略该对象关键部位，不基于m个关键点检测结果中对于该对象关键部位的预测位置，确定该对象关键部位对应的标注位置。反之，若m个关键点检测结果对于同一对象关键部位的预测位置彼此之间的距离较小，则说明这m个关键点检测结果中该对象关键部位对应的预测位置较可靠，相应地，可以将该对象关键部位视为目标对象关键部位，并根据m个关键点检测结果中该目标对象关键部位对应的预测位置，确定该目标对象关键部位对应的标注位置。
针对目标训练图像中各对象关键部位进行上述处理后，将得到目标训练图像中各目标对象关键部位各自对应的标注位置，然后可以利用该目标训练图像中各目标对象关键部位各自对应的标注位置，组成该目标训练图像对应的关键点标注结果，也即该目标训练图像对应的伪标签。进而，可以将目标训练图像及其对应的关键点标注结果作为训练样本。
进而，可以基于通过上述方式构建出的训练样本，训练目标对象关键点检测模型。该目标对象关键点检测模型是需要投入实际应用的对象关键点检测模型，例如，可以是应用 在体感游戏中用于识别玩家肢体动作的人体关键点检测模型；考虑到实际应用中的对象关键点检测模型需要具有较高的检测效率，因此，通常设置模型结构较简单的对象关键点检测模型作为上述目标对象关键点检测模型。
应理解，图1所示的应用场景仅为示例，在实际应用中，本申请实施例提供的模型训练方法还可以应用于其它场景，在此不对本申请实施例提供的模型训练方法适用的应用场景做任何限定。
下面通过方法实施例对本申请提供的模型训练方法进行详细介绍。
参见图2，图2为本申请实施例提供的模型训练方法的流程示意图。为了便于描述，下述实施例仍以该模型训练方法的执行主体为服务器为例进行介绍。如图2所示，该模型训练方法包括以下步骤：
步骤201：通过m个参考对象关键点检测模型，分别对目标训练图像进行对象关键点检测处理，得到所述m个参考对象关键点检测模型各自对应的关键点检测结果；所述关键点检测结果包括所述目标训练图像中多个对象关键部位各自对应的关键点的预测位置；所述m为大于1的整数。
在本申请实施例中，服务器针对目标训练图像生成其对应的伪标签时，可以先通过m个参考对象关键点检测模型，分别对该目标训练图像进行对象关键点检测处理，得到这m个参考对象关键点检测模型各自对于该目标训练图像的关键点检测结果，即这m个参考对象关键点检测模型各自对应的关键点检测结果。此处，每个关键点检测结果中均包括目标训练图像中多个对象关键部位各自对应的关键点的预测位置。
需要说明的是，伪标签是半监督学习中常用的训练数据，通常情况下，可以通过性能更优的复杂模型对无标注数据进行处理，得到该无标注数据对应的伪标签，该伪标签可能是不准确的。在本申请实施例中，目标训练图像对应的伪标签，可以根据m个参考对象关键点检测模型对于该目标训练图像的关键点检测结果确定；本申请旨在通过一系列处理流程，对m个参考对象关键点检测模型对于该目标训练图像的关键点检测结果进行处理，得到能够较准确地反映该目标训练图像中对象关键部位位置的伪标签，从而便于提高基于包括该伪标签的训练样本训练的目标对象关键点检测模型的性能。
需要说明的是，参考对象关键点检测模型是预先训练好的用于检测图像中待检测对象上对象关键部位所处位置的模型，其通常能够较准确地检测出对象关键部位所处的位置；通常情况下，为了使得参考对象关键点检测模型能够较准确地检测对象关键部位的位置，会设置参考对象关键点检测模型具有比较复杂的模型结构。示例性的，上述各参考对象关键点检测模型可以为参考人体关键点检测模型，其用于检测图像中人体上各重要关节所处的位置，该参考人体关键点检测模型的关键点检测结果可以用于确定人体姿态。应理解，本申请实施例中的m个参考对象关键点检测模型，可以是基于相同的训练样本集训练得到的，也可以是基于不同的训练样本集训练得到的，本申请对此不做任何限定。
需要说明的是，目标训练图像是包括待检测对象的图像，其中的待检测对象上包括若干对象关键部位；此处的对象关键部位包括待检测对象上能够反映该待检测对象的姿态的部位。示例性的，该目标训练图像可以是包括清晰且完整的待检测人体的图像，该待检测 人体上包括若干重要关节，如鼻子、左右眼、左右耳、左右肩、左右肘、左右腕、左右臀、左右膝、左右脚踝等。
需要说明的是，参考对象关键点检测模型对应的关键点检测结果，是通过该参考对象关键点检测模型对目标训练图像进行对象关键点检测处理得到的，该关键点检测结果中包括目标训练图像中待检测对象上各对象关键部位各自对应的关键点的预测位置。示例性的，假设目标训练图像是尺寸为H×W的图像，参考对象关键点检测模型对该目标训练图像进行对象关键点检测处理后，将输出尺寸为K×H×W的热力特征图，其中，K为所需检测的对象关键部位的数目，K×H×W的热力特征图中第k张热力特征图用于表示目标训练图像中第k个对象关键部位对应的关键点的预测位置。
在一种可能的实现方式中，服务器可以通过以下方式获得每个参考对象关键点检测模型对应的关键点检测结果：针对每个参考对象关键点检测模型，通过该参考对象关键点检测模型对目标训练图像进行对象关键点检测处理，以确定该目标训练图像中各像素点属于对象关键部位的置信度。然后，根据该目标训练图像中属于对象关键部位的置信度高于预设置信度阈值的像素点，确定该参考对象关键点检测模型对应的关键点检测结果。
具体的，图3所示为本申请实施例提供的一种参考对象关键点检测模型的工作原理示意图。如图3所示，服务器将尺寸为H×W的目标训练图像输入某参考对象关键点检测模型中后，该参考对象关键点检测模型通过对该目标训练图像进行分析处理，将相应地输出尺寸为K×H×W的热力特征图，该热力特征图本质上是由K张尺寸为H×W的热力特征图组成的，其中，每张尺寸为H×W的热力特征图包括目标训练图像中各个像素点属于该张热力特征图对应的对象关键部位的置信度，例如，在第k张尺寸为H×W的热力特征图中，每个像素点具有其对应的置信度，该置信度用于表征对应的像素点属于第k个对象关键部位的概率。
针对每张尺寸为H×W的热力特征图，服务器可以将其中所对应的置信度小于或等于预设置信度(如0.5)的像素点忽略，例如，直接将此类像素点对应的置信度调整为0，默认其不属于该张热力特征图对应的对象关键部位。仅根据所对应的置信度高于该预设置信度的像素点，确定该张热力特征图对应的对象关键部位的预测位置，例如，将此类像素点对应的置信度调整为1，默认其属于该张热力特征图对应的对象关键部位；相应地，该张热力特征图中所对应的置信度等于1的像素点的位置，即可反映该张热力特征图对应的对象关键部位的预测位置。如此，通过上述方式完成对K张尺寸为H×W的热力特征图的初筛处理后，可以将初筛处理后得到的尺寸为K×H×W的热力特征图，作为该参考对象检测模型对应的关键点检测结果。
如此，通过对参考对象关键点检测模型的实际检测结果进行预处理，得到该参考对象关键点检测模型对应的关键点检测结果，可以进一步减少关键点检测结果中的噪声，同时可以避免服务器后续对于此类噪声点进行不必要的处理，减少所需耗费的处理资源，并且还能够提高后续的处理效率。
步骤202：针对每个所述对象关键部位，根据所述m个关键点检测结果中所述对象关键部位对应的关键点的预测位置，确定所述对象关键部位是否为目标对象关键部位；若是，则根据所述m个关键点检测结果中所述目标对象关键部位对应的关键点的预测位置，确定 所述目标对象关键部位对应的标注位置。
服务器获得m个参考对象关键点检测模型各自对应的关键点检测结果后，可以根据这m个关键点检测结果，评判m个参考对象关键点检测模型对各个对象关键部位的位置预测结果是否可靠。即，针对每个对象关键部位，服务器可以根据m个关键点检测结果中该对象关键部位对应的关键点的预测位置，确定该对象关键部位是否为目标对象关键部位；在确定该对象关键部位为目标对象关键部位的情况下，服务器可以进一步根据m个关键点检测结果中该目标对象关键部位对应的关键点的预测位置，确定该目标对象关键部位对应的标注位置；进而，服务器可以利用目标训练图像中各目标对象关键部位各自对应的标注位置，组成该目标训练图像对应的关键点标注结果。
从空间一致性的角度来看，不同的参考对象关键点检测模型对目标训练图像中同一对象关键部位的位置预测结果应基本一致，即不同的参考对象关键点检测模型对于同一对象关键部位的预测位置彼此之间距离较近。若m个关键点检测结果中同一对象关键部位对应的预测位置彼此之间距离较远，则说明这m个关键点检测结果中该对象关键部位对应的预测位置不可靠，可能存在较大的噪声；为了避免将此噪声引入后续的模型训练，本申请实施例选择忽略该对象关键部位，不基于m个关键点检测结果中该对象关键部位对应的预测位置确定该对象关键部位对应的标注位置，即不基于该对象关键部位生成目标训练图像对应的伪标签。反之，若m个关键点检测结果中同一对象关键部位对应的预测位置彼此之间距离较近，则说明这m个关键点检测结果中该对象关键部位对应的预测位置较可靠，相应地，可以将该对象关键部位视为目标对象关键部位，并根据m个关键点检测结果中该目标对象关键部位对应的预测位置，确定该目标对象关键部位对应的标注位置，后续也可基于该目标对象关键部位对应的标注位置确定目标训练图像对应的伪标签。
在一种可能的实现方式中，服务器可以通过以下方式评判m个参考对象关键点检测模型对每个对象关键部位的位置预测结果是否可靠，也即通过以下方式针对每个对象关键部位确定其是否为目标对象关键部位：根据m个关键点检测结果中该对象关键部位对应的关键点的预测位置，确定该对象关键部位对应的参考位置；然后，针对每个关键点检测结果，确定其中该对象关键部位对应的关键点的预测位置与该对象关键点对应的参考位置之间的距离，并根据该距离，确定该关键点检测结果是否属于对象关键部位对应的目标关键点检测结果；进而，统计m个关键点检测结果中该对象关键部位对应的目标关键点检测结果的目标数量；若该目标数量超过预设数量，则可以确定该对象关键部位为目标对象关键部位；若该目标数量未超过预设数量，则可以确定该对象关键部位为非目标对象关键部位。
应理解，对象关键部位对应的关键点的预测位置，是参考对象关键点检测模型对目标训练图像进行对象关键点检测处理得到的，即是参考对象关键点检测模型输出的对象关键部位的位置预测结果。对象关键部位对应的参考位置，是根据多个参考对象关键点检测模型的关键点检测结果中该对象关键部位的位置预测结果确定的，例如，是对多个关键点检测结果中该对象关键部位对应的预测位置进行求平均处理得到的；其用于反映多个关键点检测结果中该对象关键部位对应的预测位置的分布情况。
为了便于理解上述实现过程，下面以针对对象关键部位“头部”确定其是否为目标对 象关键部位为例，对上述实现过程进行示例性介绍。针对头部确定其是否属于目标对象关键部位时，服务器可以计算m个关键点检测结果中头部对应的关键点的预测位置的平均值，作为头部对应的参考位置；然后，以该头部对应的参考位置为中心，针对每个关键点检测结果，确定其中头部对应的关键点的预测位置与该参考位置之间的距离，并判断该距离是否小于第一预设距离(例如在对各位置进行归一化处理的情况下，该第一预设距离可以为0.1)，若小于，则可以确定该关键点检测结果属于头部对应的目标关键点检测结果，反之，若不小于，则可以确定该关键点检测结果不属于头部对应的目标关键点检测结果；针对m个关键点检测结果均完成上述处理后，可以统计这m个关键点检测结果中头部对应的目标关键点检测结果的目标数量；若该头部对应的目标数量超过预设数量(例如m/2)，则可以确定头部属于目标对象关键部位；反之，若该头部对应的目标数量没有超过预设数量，则可以确定头部属于非目标对象关键部位。
应理解，上述第一预设距离、预设数量等参数均可以根据实际需求设定，本申请在此不对其做任何限定。此外，上述确定对象关键部位对应的参考位置的实现方式、以及判断关键点检测结果是否属于对象关键部位对应的目标关键点检测结果的实现方式，也均可以根据实际需求设定，本申请在此也不对其做任何限定。
通过上述方式评判m个参考对象关键点检测模型对每个对象关键部位的位置预测结果是否可靠，可以保证评判结果的可靠性，有效地过滤掉可靠性较低的对象关键部位的位置预测结果。相应地，可以保证后续确定的目标训练图像对应的伪标签具有较高的可靠性，避免其中掺杂大量噪声。
在通过上述方式确定各对象关键部位是否为目标对象关键部位的情况下，服务器可以进一步通过以下方式确定目标对象关键部位对应的标注位置：针对目标对象关键部位对应的每个目标关键点检测结果，确定其中该目标对象关键部位对应的关键点的预测位置，作为该目标对象关键部位对应的目标预测位置；进而，根据该目标对象关键部位对应的各目标预测位置，确定该目标对象关键部位对应的标注位置。
示例性的，针对某目标对象关键部位，服务器可以获取该目标对象关键部位对应的各目标关键点检测结果中该目标对象关键部位对应的关键点的预测位置，作为该目标对象关键目标对应的目标预测位置。进而，服务器可以计算该目标对象关键部位对应的各目标预测位置的平均值，作为该目标对象关键部位对应的标注位置。
如此，仅根据目标对象关键部位对应的目标关键点检测结果中该目标对象关键部位的位置预测结果，确定该目标对象关键部位对应的标注位置，能够保证所确定的标注位置的准确性。
应理解，在实际应用中，服务器也可以采用其它方式确定目标对象关键部位对应的标注位置，例如，直接将目标对象关键部位对应的参考位置(即m个关键点检测结果中该目标对象关键部位对应的关键点的预测位置的均值)作为该目标对象关键部位对应的标注位置。本申请在此不对服务器确定目标对象关键部位对应的标注位置的方式做任何限定。
可选的，在目标训练图像为目标视频中的视频帧的情况下，服务器还可以从时序连贯性的角度出发，对上述目标对象关键部位进行进一步的评判，以确定是否保留上述目标对 象关键部位。
即，服务器可以确定目标视频中与目标训练图像相邻的视频帧作为参考训练图像，并且获取该参考训练图像对应的关键点标注结果，该关键点标注结果中包括该参考训练图像中各目标对象关键部位各自对应的标注位置。然后，针对每个目标对象关键部位，确定其在目标训练图像对应的关键点标注结果中的标注位置为第一位置，确定其在参考训练图像对应的关键点标注结果中的标注位置为第二位置；根据该第一位置与该第二位置之间的距离，确定是否保留目标训练图像对应的关键点标注结果中的该第一位置、以及是否保留参考训练图像对应的关键点标注结果中的该第二位置。
在实际应用中，视频中相邻的视频帧之间不会存在很大的差异，这意味着相邻的视频帧中待检测对象上对象关键部位的变化幅度也不会很大，基于此，本申请实施例可以对视频中相邻的两个视频帧各自对应的关键点标注结果所涉及的目标对象关键部位做进一步评判处理。
具体的，服务器可以确定目标训练图像所属的目标视频，并确定该目标视频中与该目标训练图像相邻的视频帧作为参考训练图像，该参考训练图像可以是目标训练图像的前一个视频帧，也可以是目标训练图像的后一个视频帧。并且获取该参考训练图像对应的关键点标注结果，该参考训练图像对应的关键点标注结果的产生方式与上文介绍的目标训练图像对应的关键点标注结果的产生方式相同，该参考训练图像对应的关键点标注结果中包括该参考训练图像中各目标对象关键部位各自对应的标注位置。
针对目标训练图像和参考训练图像中的每个目标对象关键部位，服务器可以确定目标训练图像对应的关键点标注结果中该目标对象关键部位对应的标注位置作为第一位置，确定参考训练图像对应的关键点标注结果中该目标对象关键部位对应的标注位置作为第二位置；然后，计算该第一位置与第二位置之间的距离，并判断该距离是否小于第二预设距离(例如在对各位置进行归一化处理的情况下，该第二预设距离可以为0.2)，若小于，则可以保留目标训练图像和参考训练图像各自对应的关键点标注结果中该目标对象关键部位对应的标注位置(即第一位置和第二位置)，反之，若不小于，则可以剔除目标训练图像和参考训练图像各自对应的关键点标注结果中该目标对象关键部位对应的标注位置，即将该目标对象关键部位视为无效对象关键部位。
应理解，上述第二预设距离可以根据实际需求设定，本申请对此不做任何限定。
如此，通过上述方式，基于时序连贯性对目标对象关键部位做进一步的筛选处理，有助于提高所确定的关键点标注结果的准确性，可以进一步滤除关键点标注结果中可能存在的噪声。
步骤203：将所述目标训练图像及其对应的关键点标注结果，作为训练样本；所述关键点标注结果包括各所述目标对象关键部位各自对应的标注位置。
服务器通过上述方式得到目标训练图像对应的关键点标注结果(其中包括该目标训练图像中各目标对象关键部位各自对应的标注位置)后，即可利用该目标训练图像及其对应的关键点标注结果，组成用于训练需要投入实际应用的目标对象关键点检测模型的训练样本。应理解，此处的目标训练图像对应的关键点标注结果本质上即是该目标训练图像对应 的伪标签。
步骤204：基于所述训练样本，训练目标对象关键点检测模型。
在实际应用中，服务器可以通过上述步骤201至步骤203生成大量的训练样本，进而，利用所生成的训练样本，训练需要投入实际应用的目标对象关键点检测模型。
需要说明的是，目标对象关键点检测模型是待训练的用于检测图像中待检测对象上对象关键部位所处位置的模型，其是需要投入实际应用的模型，通常情况下，为了保证实际应用中的检测效率，该目标对象关键点检测模型的模型结构通常比较简单，即该目标对象关键点检测模型的复杂度通常低于上述参考对象关键点检测模型的复杂度。示例性的，该目标对象关键点检测模型可以是应用在体感游戏中用于检测玩家肢体动作变化的模型，其用于对摄像头拍摄的图像中的人体进行重要关节点位置检测，以确定该人体的动作姿态。
在一种可能的实现方式中，服务器可以基于图4所示的流程，实现对于目标对象关键点检测模型的训练。如图4所示，服务器训练目标关键点检测模型时具体需要执行以下步骤：
步骤2041：通过所述目标对象关键点检测模型，对所述训练样本中的所述目标训练图像进行对象关键点检测处理，得到所述目标训练图像对应的训练关键点检测结果；所述训练关键点检测结果包括多个对象关键部位各自对应的关键点的训练预测位置。
服务器具体利用训练样本训练目标对象关键点检测模型时，可以将训练样本中的目标训练图像输入该目标对象关键点检测模型，该目标对象关键点检测模型通过对输入的目标训练图像进行分析处理，可以相应地输出该目标训练图像对应的训练关键点检测结果，该训练关键点检测结果中包括目标训练图像中多个对象关键部位各自对应的关键点的训练预测位置。
应理解，目标对象关键点检测模型与上文中参考对象关键点检测模型的工作原理基本相同，只是二者的模型结构存在区别，通常情况下，目标对象关键点检测模型的结构比参考对象关键点检测模型的结构更简单。基于此，目标对象关键点检测模型的工作原理也可以如图3所示，即将尺寸为H×W的目标训练图像输入目标对象关键点检测模型，该目标对象关键点检测模型通过处理该目标训练图像，输出的训练关键点检测结果应为尺寸为K×H×W的热力特征图，其中，第k张热力特征图yk用于表征第k个对象关键部位对应的关键点的预测位置。
步骤2042：根据所述目标训练图像对应的训练关键点检测结果、以及所述训练样本中的所述关键点标注结果，构建目标损失函数。
服务器利用目标对象关键点检测模型对目标训练图像进行关键点检测处理，得到该目标训练图像对应的训练关键点检测结果后，可以根据该训练关键点检测结果和训练样本中的关键点标注结果间的差异，构建目标损失函数。
在一种可能的实现方式中，服务器可以通过以下方式构建目标损失函数：针对训练关键点检测结果中的每个关键点，判断该关键点对应的对象关键部位是否为关键点标注结果中的目标对象关键部位；若是，则根据该关键点对应的训练预测位置以及该关键点对应的目标对象关键部位的标注位置，构建该关键点对应的损失项，并且为该关键点对应的损失项配置对应的第一损失权重；若否，根据该关键点对应的训练预测位置，构建该关键点对 应的损失项，并且为所述关键点对应的损失项配置对应的第二损失权重；此处的第一损失权重大于0，第二损失权重等于0。进而，根据训练关键点检测结果中各关键点各自对应的损失项及其对应的损失权重，构建目标损失函数。
根据对于步骤202的介绍内容可知，本申请实施例在确定目标训练图像对应的关键点标注结果时，会针对每个对象关键部位判断其是否为目标对象关键部位，并且仅确定目标对象关键部位对应的标注位置，仅利用目标对象关键部位对应的标注位置确定关键点标注结果。在此过程中，会忽略掉一些对象关键部位对应的关键点，虽然这些对象关键部位对应的关键点可能实际存在，但是本申请实施例通过一系列评判处理，认为这些对象关键部位对应的关键点并不可靠，因此，没有将其加入到目标训练图像对应的关键点标注结果中。为了避免构建目标损失函数时因误认为上述被忽略的对象关键部位对应的关键点实际是不存在的，而导致错误地梯度反向传播，影响所训练的目标对象关键点检测模型的性能，本申请实施例创新性地设计了带权重的平均平方误差(Mean squared Error，MSE)损失函数，作为目标损失函数。
具体的，针对训练关键点检测结果中的每个关键点，服务器可以先判断该关键点对应的对象关键部位是否为关键点标注结果中的某个目标对象关键部位，即判断该关键点标注结果中是否存在该关键点对应的对象关键部位的标注位置。若是，则服务器可以根据该关键点对应的训练预测位置与关键点标注结果中该关键点对应的对象关键部位的标注位置之间的差异，构建该关键点对应的损失项，同时为该损失项配置大于0的第一损失权重(例如1)。若否，则服务器可以根据该关键点对应的训练预测位置以及预设的参考标注位置(可以随机设置)，构建该关键点对应的损失项，同时为该损失项配置等于0的第二损失权重。进而，服务器可以根据训练关键点检测结果中各关键点各自对应的损失项及其对应的损失权重，构建目标损失函数。
目标损失函数的具体构建公式可以如式(1)所示：
Figure PCTCN2022120169-appb-000001
其中，L mse即为所构建的目标损失函数；K为目标对象关键点检测模型所要检测的对象关键部位的总数目。
Figure PCTCN2022120169-appb-000002
为第k个对象关键部位对应的损失项；其中，y t,k为训练关键点检测结果中第k个对象关键部位对应的关键点的训练预测位置；在关键点标注结果包括第k个对象关键部位对应的标注位置的情况下，
Figure PCTCN2022120169-appb-000003
为关键点标注结果中第k个对象关键部位对应的标注位置；在关键点标注结果不包括第k个对象关键部位对应的标注位置的情况下，
Figure PCTCN2022120169-appb-000004
为随机设置的参考标注位置。v t,k为上述损失项对应的损失权重；在关键点标注结果包括第k个对象关键部位对应的标注位置的情况下，v t,k等于第一损失权重，例如v t,k＝1；在关键点标注结果不包括第k个对象关键部位对应的标注位置的情况下，v t,k等于第二损失权重，即v t,k＝0。
可选的，在目标训练图像为目标视频中的视频帧的情况下，服务器还可以从时序连贯性的角度出发构建第一参考损失函数，以便基于上述目标损失函数和该第一参考损失函数， 对目标对象关键点检测模型进行协同训练。
即，服务器可以确定目标视频中与目标训练图像相邻的视频帧作为参考训练图像，并获取该参考训练图像对应的训练关键点检测结果，此处的参考训练图像对应的训练关键点检测结果，是通过目标对象关键点检测模型对该参考训练图像进行对象关键点检测处理得到的，其中包括该参考训练图像中多个对象关键部位各自对应的关键点的训练预测位置。然后，服务器可以根据目标训练图像和参考训练图像各自对应的训练关键点检测结果，构建第一参考损失函数。
在实际应用中，视频中相邻的视频帧之间不会存在很大的差异，这意味着相邻的视频帧中待检测对象上对象关键部位的变化幅度也不会很大；基于此，本申请实施例创新性地设计了用于约束帧间连续性的第一参考损失函数，即，根据目标对象关键点检测模型对于相邻的两个视频帧的训练关键点检测结果，构建用于辅助训练目标对象关键点检测模型的第一参考损失函数，以通过该第一参考损失函数约束帧间的对象关键部位的位置变化幅度不会太大。
具体实现时，服务器可以针对每个对象关键部位，确定目标训练图像中该对象关键部位对应的关键点的训练预测位置为第三位置，确定参考训练图像中该对象关键部位对应的关键点的训练预测位置为第四位置；然后，根据第三位置与第四位置之间的距离，构建该对象关键部位对应的损失项，并为该损失项配置对应的损失权重。进而，服务器可以根据各对象关键部位各自对应的损失项及其对应的损失权重，构建该第一参考损失函数。
示例性的，针对每个对象关键部位，服务器可以确定目标训练图像对应的训练关键点预测结果中该对象关键部位对应的关键点的预测位置作为第三位置，确定参考训练图像对应的训练关键点预测结果中该对象关键部位对应的关键点的预测位置作为第四位置。然后，服务器可以计算该第三位置与该第四位置之间的距离，并根据该距离构建该对象关键部位对应的损失项，以及根据该距离确定该损失项对应的损失权重；例如，若该距离小于第三预设距离(例如在对各位置进行归一化处理的情况下，该第三预设距离可以为0.1)，则可以确定该损失项对应的损失权重等于0，若该距离不小于第三预设距离，则可以确定该损失项对应的损失权重等于1；当然，针对该距离小于第三预设距离和不小于第三预设距离这两种情况，服务器也可以相应地配置其它损失权重，只需保证在小于第三预设距离的情况下配置的损失权重小于在不小于第三预设距离的情况下配置的损失权重即可。进而，服务器可以根据各对象关键部位各自对应的损失项以及损失权重，构建该第一参考损失函数。
第一参考损失函数的具体构建公式可以如式(2)所示：
Figure PCTCN2022120169-appb-000005
其中，L consistency即为所构建的第一参考损失函数；K为目标对象关键点检测模型所要检测的对象关键部位的总数目。(y t，k-y t-1，k) 2为第k个对象关键部位对应的损失项；其中，y t,k为目标训练图像对应的训练关键点检测结果中第k个对象关键部位对应的关键点的训练预测位置，y t-1,k为参考训练图像对应的训练关键点检测结果中第k个对象关键部位对应的关键点的训练预测位置。w k为上述损失项对应的损失权重；在y t,k与y t-1,k之间的距离小于第三预设距离的情况下，w k例如可以等于0；在y t,k与y t-1,k之间的距离不小于第三预设距离的情况 下，w k例如可以等于1。
如此，通过上述方式，引入用于约束帧间连续性的第一参考损失函数，有助于使所训练的目标对象关键点检测模型快速收敛，提高模型训练效率，并且提高所训练的目标对象关键点检测模型的模型性能。
可选的，本申请实施例还可以将知识蒸馏的思想引入对于目标对象关键点检测模型的训练过程中，以进一步提升所训练的目标对象关键点检测模型的模型性能。知识蒸馏是利用复杂模型(又被称为老师模型)学习到的知识指导简单模型(又被称为学生模型)的模型训练方式，旨在使得简单模型具有与复杂模型相当的性能，并且该简单模型的参数数量相比复杂模型的参数数量大幅降低，如此实现模型的压缩与加速。
即，服务器可以获取某个参考对象关键点检测处理目标训练图像时产生的第一中间处理结果，以及获取目标对象关键点检测模型处理目标训练图像时产生的第二中间处理结果。进而，根据该第一中间处理结果和该第二中间处理结果，构建第二参考损失函数。
示例性的，服务器可以从m个参考对象关键点检测模型中，选取与所训练的目标对象关键点检测模型的模型结构较接近的参考对象关键点检测模型，作为老师模型；然后，获取该老师模型处理目标训练图像时产生的中间处理结果，作为第一中间处理结果，例如，获取该老师模型处理目标训练图像时，其中某特征提取层从目标训练图像中提取出的特征，作为第一中间处理结果。服务器利用目标对象关键点检测模型处理该目标训练图像时，可以获取该目标对象关键点检测模型处理该目标训练图像时产生的中间处理结果，作为第二中间处理结果，例如，获取该目标对象关键点检测模型处理目标训练图像时，其中某特征提取层从目标训练图像中提取出的特征，作为第二中间结果。需要说明的是，此处获取的第一中间处理结果和第二中间处理结果应当具有相同的维度，并且应当来自同一处理阶段，例如，第一中间处理结果和第二中间处理结果分别来自参考对象关键点检测模型和目标对象关键点检测模型的编码阶段，和/或第一中间处理结果和第二中间处理结果分别来自参考对象关键点检测模型和目标对象关键点检测模型的解码阶段；本申请实施例在此不对第一中间处理结果和第二中间处理结果做任何限定。
服务器获取到第一中间处理结果和第二中间处理结果后，可以根据第一中间处理结果与第二中间处理结果之间的差异，构建第二参考损失函数。基于该第二参考损失函数训练目标对象关键点检测模型时，可以将最小化该第二损失函数为训练目标，调整所要训练的目标对象关键点检测模型的模型参数。
如此，通过上述方式，引入基于知识蒸馏思想构建的第二参考损失函数，利用模型性能较优的参考对象关键点检测模型指导目标对象关键点检测模型的训练，有助于提高目标对象关键点检测模型的模型性能，并且有助于使得所训练的目标对象关键点检测模型快速收敛，提高模型训练效率。
步骤2043：基于所述目标损失函数，调整所述目标对象关键点检测模型的模型参数。
服务器通过步骤2042，构建出用于训练目标对象关键点检测模型的目标损失函数后，可以将最小化该目标损失函数作为训练目标，调整目标对象关键点检测模型的模型参数，使得该目标对象关键点检测模型趋于收敛。
应理解，在服务器还引入了用于约束帧间连续性的第一参考损失函数的情况下，服务 器可以基于目标损失函数和该第一参考损失函数，调整该目标对象关键点检测模型的模型参数。
具体的，服务器可以通过如下式(3)根据目标损失函数和第一参考损失函数，构建模型训练阶段的总损失函数：
L＝L mse+λ·L consistency       (3)
其中，L为模型训练阶段的总损失函数，L mse为目标损失函数，L consistency为第一参考损失函数，λ为预先为第一参考损失函数配置的权重。
进而，以最小化该总损失函数为训练目标，调整目标对象关键点检测模型的模型参数，使得该目标对象关键点检测模型趋于收敛。
应理解，在服务器还引入了基于知识蒸馏思想构建的第二参考损失函数的情况下，服务器可以基于目标损失函数和第二参考损失函数，调整该目标对象关键点检测模型的模型参数。
具体的，服务器可以采用与上文中根据目标损失函数和第一参考损失函数构建总损失函数相类似的方式，根据目标损失函数和该第二参考损失函数，构建模型训练阶段的总损失函数。进而，以最小化该总损失函数为训练目标，调整目标对象关键点检测模型的模型参数，使得该目标对象关键点检测模型趋于收敛。
应理解，在服务器同时引入了上述第一参考损失函数和第二参考损失函数的情况下，服务器可以基于目标损失函数、第一参考损失函数和第二参考损失函数，调整该目标对象关键点检测模型的模型参数。即，服务器可以为第一参考损失函数和第二参考损失函数分别配置对应的权重，进而，根据目标损失函数、第一参考损失函数及其对应的权重、以及第二参考损失函数及其对应的权重，构建模型训练阶段的总损失函数；并以最小化该总损失函数为训练模型，调整目标对象关键点检测模型的模型参数，使得该目标对象关键点检测模型趋于收敛。
服务器基于不同的目标训练图像，反复执行上述步骤2041至步骤2043，实现对于目标对象关键点检测模型的迭代训练，直至该目标对象关键点检测模型满足模型训练结束条件为止，此时训练得到的目标对象关键点检测模型即为可以投入实际应用的目标对象关键点检测模型。此处的训练结束条件，例如可以是所训练的目标对象关键点检测模型的模型性能满足预设要求(如目标对象关键点检测模型的检测准确率达到预设准确率阈值等)，又例如可以是目标对象关键点检测模型的模型性能不再显著提高(如利用测试样本集对几轮训练得到的目标对象关键点检测模型分别进行测试，确定几轮训练得到的目标对象关键点检测模型的检测准确率之间的差距小于预设差距阈值等)，再例如可以是对于目标对象关键点检测模型的迭代训练次数达到预设次数，本申请在此不对该训练结束条件做任何限定。
上述模型训练方法生成用于训练需要投入实际应用的目标对象关键点检测模型的训练样本时，会通过多个结构复杂的参考对象关键点检测模型分别对目标训练图像进行对象关键点检测处理，得到多个关键点检测结果。然后，基于各关键点检测结果中相同的对象关键部位的预测位置应基本一致的原理，针对每个对象关键部位，根据各个关键点检测结果中该对象关键部位对应的关键点的预测位置，衡量各个参考对象关键点检测模型对于该对 象关键部位的位置预测结果是否可靠，即确定该对象关键部位是否为目标对象关键部位。在确定各个参考对象关键点检测模型对于该对象关键部位的位置预测结果可靠的情况下，再进一步确定该目标对象关键部位对应的标注位置即伪标签；进而，利用该目标训练图像以及其中各目标对象关键部位各自对应的标注位置组成训练样本。如此，忽略位置预测结果不可靠的对象关键部位，仅利用位置预测结果可靠的对象关键部位对应的标注位置作为伪标签，能够有效地减少所确定的伪标签中存在的噪声，提高该伪标签的准确度。相应地，基于包括该种伪标签的训练样本，训练需要投入实际应用、结构比较简单的目标对象关键点检测模型，能够在一定程度上保证所训练的目标对象关键点检测模型的性能。
为了便于进一步理解本申请实施例提供的模型训练方法，下面以通过该模型训练方法训练体感游戏中用于检测玩家姿态的人体关键点检测模型为例，对该模型训练方法进行整体示例性介绍。
体感游戏是指用身体去感受的游戏，突破传统的单纯以手柄按键输入的操作方式，体感游戏是一种通过肢体动作变化来进行操作的新型电子游戏。在进行体感游戏的过程中，摄像头可以拍摄包括玩家肢体的图像，并将拍摄的图像传输给后台的游戏服务器，由游戏服务器通过人体关键点检测模型，检测图像中玩家骨骼重要关节点的位置，并据此识别玩家所做的肢体动作，进而执行对应的游戏逻辑。图5所示即为上述体感游戏的整体实现架构图。
图6为本申请实施例提供的对于应用在上述体感游戏中的人体关键点检测模型的训练架构示意图。如图6所示，在模型训练过程中，对于任意一段没有标注过的且包括清晰完整人体的视频，服务器可以从中选取相邻的两个视频帧x t-1和x t，然后，通过伪标签生成网络生成这两个视频帧各自对应的伪标签
Figure PCTCN2022120169-appb-000006
和
Figure PCTCN2022120169-appb-000007
进而，通过所需训练的人体关键点检测模型，对这两个视频帧进行人体关键点检测处理，得到这两个视频帧各自对应的热力特征图，模型训练过程的训练目标即是使得这两个热力特征图与对应的伪标签相近。此外，本申请实施例还基于
Figure PCTCN2022120169-appb-000008
和
Figure PCTCN2022120169-appb-000009
之间具备的帧间连续性设计了特定的损失函数，使模型能够实现自我监督学习。在测试或者应用阶段，只需要将图片或视频输入到所训练的人体关键点检测模型即可，伪标签生成网络仅存在于模型训练阶段。
本申请实施例提供的技术方案涉及的核心模块主要包括两个：一个是伪标签生成网络，其借助现有的结构复杂、且性能较优的多个人体关键点检测模型，对无标注的视频中的视频帧进行人体关键点检测处理，然后基于各个人体关键点检测模型各自的检测结果，生成该视频帧对应的伪标签，并将其用于后续的模型训练过程中。另一个是帧间连续性约束损失函数，其基于视频帧间人物动作变化幅度不会很大的先验条件，设计了帧间连续性损失函数，使得模型可以实现无监督的学习。
下面对上述两个核心模块分别进行详细介绍。
如图6所示，伪标签生成网络包括人体关键点预测模块和预测结果集成模块。
其中，人体关键点预测模块中包括多个
Figure PCTCN2022120169-appb-000010
目前已有的结构复杂、且性能较优的人体关键点检测模型；在人体关键点检测模块中包括m(m为大于1的整数)个人体关键点检测模 型的情况下，假设输入一张图片x t，那么通过这m个人体关键点检测模型对该图片进行人体关键点检测处理，将得到这m个人体关键点检测模型各自对应的预测结果
Figure PCTCN2022120169-appb-000011
其中，预测结果集成模块用于通过以下步骤完成对m个预测结果的集成，在集成的过程中，本申请实施例对预测结果在空间维度和时间维度上均进行了严格的过滤处理，从而保证所生成的伪标签具有更高的准确度。具体集成方法如下：
(1)针对每个预测结果，对于其中所对应的置信度小于0.5的关键点均进行忽略处理。具体的，预测结果中包括图片x t-1中各个像素点属于人体关键点的置信度，对于所对应的置信度小于0.5的像素点，本申请实施例可以默认其不属于人体关键点，因此可以将其忽略掉。
(2)从空间上考虑，针对某个人体部位对应的关键点，服务器可以计算m个预测结果中该人体部位对应的关键点预测坐标的均值，作为该人体部位对应的参考坐标。进而，可以针对每个预测结果中该人体部位对应的关键点预测坐标与该参考坐标之间的距离，确定该预测结果是否属于该人体部位对应的合格预测结果；具体的，当预测结果中该人体部位对应的关键点预测坐标与该参考坐标之间的距离小于0.1(经归一化处理后)时，可以认为该预测结果属于该人体部位对应的合格预测结果；否则，则可以认为该预测结果不属于该人体部位对应的合格预测结果。当该人体部位对应的合格预测结果的数量大于或等于m/2时，服务器可以确定该人体部位属于有效人体部位，并对该有效人体部位对应的合格预测结果中该有效人体部位对应的关键点预测坐标取平均，得到该有效人体部位对应的标注坐标；否则，服务器可以确定该人体部位属于无效人体部位，忽略该无效人体部位，不确定其对应的标注坐标。进而，服务器可以利用各有效人体部位各自对应的标注坐标，组成图片x t对应的伪标签
Figure PCTCN2022120169-appb-000012
(3)从时序上考虑，在图片x t为某视频中的一个视频帧的情况下，服务器还可以获取该视频中与该图片x t相邻的视频帧x t-1；进而，根据这两个视频帧各自对应的伪标签
Figure PCTCN2022120169-appb-000013
和
Figure PCTCN2022120169-appb-000014
对这两个伪标签
Figure PCTCN2022120169-appb-000015
和
Figure PCTCN2022120169-appb-000016
中包括的标注坐标做进一步筛选处理。具体的，如果伪标签
Figure PCTCN2022120169-appb-000017
中第k个人体部位对应的标注坐标
Figure PCTCN2022120169-appb-000018
与伪标签
Figure PCTCN2022120169-appb-000019
中第k个人体部位对应的标注坐标
Figure PCTCN2022120169-appb-000020
之间的距离超过0.2(经归一化处理后)，则可以将伪标签
Figure PCTCN2022120169-appb-000021
中的
Figure PCTCN2022120169-appb-000022
和伪标签
Figure PCTCN2022120169-appb-000023
中的
Figure PCTCN2022120169-appb-000024
均忽略掉。
(4)通过上述空间维度和时间维度的过滤处理后，可以获取到准确度较高的伪标签，但是通过上述处理会存在很多被忽略掉的人体部位，这些人体部位实际是存在的，只不过本申请会认为人体关键点检测模型对于它们的检测结果不可靠，所以没有将其投入后续的模型训练过程。如果直接基于MSE损失函数进行模型训练，会错误地认为这些关键点实际就是不存在的，从而导致错误地梯度反向传播，从而影响所训练的人体关键点检测模型的性能。为了解决此问题，本申请实施例设计了带权重的MSE损失函数，当伪标签中该人体 部位被忽略时，设置该人体部位对应的损失项的损失权重为0，这样可以防止错误的反向传播。如下公式，即为本申请实施例提出的带权重的MSE损失函数：
Figure PCTCN2022120169-appb-000025
其中，L mse即为所构建的损失函数；K为人体关键点检测模型所要检测的人体部位的总数目。
Figure PCTCN2022120169-appb-000026
为第k个人体部位对应的损失项；其中，y t,k为所要训练的人体关键点检测模型对于第k个人体部位预测位置；在伪标签
Figure PCTCN2022120169-appb-000027
包括第k个人体部位对应的标注位置的情况下，
Figure PCTCN2022120169-appb-000028
为伪标签
Figure PCTCN2022120169-appb-000029
中第k个人体部位对应的标注位置；在伪标签
Figure PCTCN2022120169-appb-000030
不包括第k个人体部位对应的标注位置的情况下，
Figure PCTCN2022120169-appb-000031
为随机设置的参考标注位置。v t,k为上述损失项对应的损失权重；在伪标签
Figure PCTCN2022120169-appb-000032
包括第k个人体部位对应的标注位置的情况下，v t,k＝1；在伪标签
Figure PCTCN2022120169-appb-000033
不包括第k个人体部位对应的标注位置的情况下，v t,k＝0。
除此之外，本申请实施例发现一个视频前后相邻的两个视频帧之间不会存在太大的变化差异，即相邻视频帧中人体部位的变化幅度不会很大，基于该先验知识，本申请还设计了一个帧间连续性的约束函数，用来实现对于人体关键点检测模型的无监督学习，以保证该人体关键点检测模型在对视频做预测时，帧间的关键点坐标跳动幅度不会太大，该损失函数具体如下：
Figure PCTCN2022120169-appb-000034
其中，L consistency即为帧间连续性的约束函数；K为人体关键点检测模型所要检测的人体部位的总数目。(y t，k-y t-1，k) 2为第k个人体部位对应的损失项；其中，y t,k为第t个视频帧对应的训练预测结果中第k个人体部位对应的关键点的训练预测位置，y t-1,k为第t-1个视频帧对应的训练预测结果中第k个人体部位对应的关键点的训练预测位置。w k为上述损失项对应的损失权重；在y t,k与y t-1,k之间的距离小于0.1(经归一化处理后)的情况下，w k＝0；在y t,k与y t-1,k之间的距离不小于0.1(经归一化处理后)的情况下，w k＝1。
综上，本申请实施例提供的方案在训练阶段总的损失函数如下：
L＝L mse+λ·L consistency
其中，λ是预先为L consistency配置的权重。
进而，服务器可以基于该损失函数L，对所要训练的人体关键点检测模型进行训练。
利用COCO-wholebody人体关键点检测公开测试集，对通过上述方式训练得到的人体关键点检测模型进行测试，全类平均正确率(mean Average Precision，mAP)指标从0.457上升到了0.521。
本申请发明人还对体感游戏中目前应用的人体关键点检测模型和通过上述方式训练得到的人体关键点检测模型进行了对比测试，对比测试的结果如图7所示，其中(a)为通过本申请实施例提供的方案训练得到的人体关键点检测模型确定的检测结果，(b)为通过现 有的人体关键点检测模型确定的检测结果，通过对比可以发现，(a)相比(b)更加准确，(a)中的左右脚对应的关键点没有换位，而(b)中的左右脚对应的关键点出现了换位。
应理解，本申请实施例提供的模型训练方法，除了可以用于训练应用在体感游戏场景中的人体关键点检测模型外，还可以用于训练其它场景中的对象关键点检测模型，如视频拍摄场景中用于为人体或动物增加特效的功能，往往需要对象关键点检测模型检测图像中对象的关键点，本申请实施例提供的方案也可以用于训练此种场景中的对象关键点检测模型，在此不对本申请实施例提供的方案所适用的应用场景做任何限定。
可以理解的是，在本申请的具体实施方式中，涉及到用户信息(如摄像头拍摄的图像)等相关的数据，当本申请以上实施例运用到具体产品或技术中时，需要获得用户许可或者同意，且相关数据的收集、使用和处理需要遵守相关国家和地区的相关法律法规和标准。
针对上文描述的模型训练方法，本申请还提供了对应的模型训练装置，以使上述模型训练方法在实际中得以应用及实现。
参见图8，图8是与上文图2所示的模型训练方法对应的一种模型训练装置800的结构示意图。如图8所示，该模型训练装置800包括：
关键点检测模块801，用于通过m个参考对象关键点检测模型，分别对目标训练图像进行对象关键点检测处理，得到所述m个参考对象关键点检测模型各自对应的关键点检测结果；所述关键点检测结果包括所述目标训练图像中多个对象关键部位各自对应的关键点的预测位置；所述m为大于1的整数；
部位标注模块802，用于针对每个所述对象关键部位，根据所述m个关键点检测结果中所述对象关键部位对应的关键点的预测位置，确定所述对象关键部位是否为目标对象关键部位；若是，则根据所述m个关键点检测结果中所述目标对象关键部位对应的关键点的预测位置，确定所述目标对象关键部位对应的标注位置；
样本生成模块803，用于将所述目标训练图像及其对应的关键点标注结果，作为训练样本；所述关键点标注结果包括各所述目标对象关键部位各自对应的标注位置；
模型训练模块804，用于基于所述训练样本，训练目标对象关键点检测模型。
可选的，在图8所示的模型训练装置的基础上，所述部位标注模块802具体用于：
根据所述m个关键点检测结果中所述对象关键部位对应的关键点的预测位置，确定所述对象关键部位对应的参考位置；
针对每个所述关键点检测结果，确定其中所述对象关键部位对应的关键点的预测位置与所述对象关键部位对应的参考位置之间的距离，并根据所述距离，确定所述关键点检测结果是否属于所述对象关键部位对应的目标关键点检测结果；
统计所述m个关键点检测结果中所述对象关键部位对应的目标关键点检测结果的目标数量；
若所述目标数量超过预设数量，则确定所述对象关键部位为目标对象关键部位；若所述目标数量未超过所述预设数量，则确定所述对象关键部位为无效对象关键部位。
可选的，在图8所示的模型训练装置的基础上，所述部位标注模块802具体用于：
针对所述目标对象关键部位对应的每个目标关键点检测结果，确定其中所述目标对象 关键部位对应的关键点的预测位置，作为所述目标对象关键部位对应的目标预测位置；
根据所述目标对象关键部位对应的各目标预测位置，确定所述目标对象关键部位对应的标注位置。
可选的，在图8所示的模型训练装置的基础上，参见图9，图9为本申请实施例提供的另一种模型训练装置900的结构示意图。如图9所示，当所述目标训练图像为目标视频中的视频帧时，所述装置还包括：
参考数据获取模块901，用于确定所述目标视频中与所述目标训练图像相邻的视频帧，作为参考训练图像；并获取所述参考训练图像对应的关键点标注结果；
标注结果筛选模块902，用于针对每个所述目标对象关键部位，确定其在所述目标训练图像对应的关键点标注结果中的标注位置为第一位置，确定其在所述参考训练图像对应的关键点标注结果中的标注位置为第二位置；根据所述第一位置与所述第二位置之间的距离，确定是否保留所述目标训练图像对应的关键点标注结果中的所述第一位置、以及是否保留所述参考训练图像对应的关键点标注结果中的所述第二位置。
可选的，在图8所示的模型训练装置的基础上，所述关键点检测模块801具体用于：
针对每个所述参考对象关键点检测模型，通过所述参考对象关键点检测模型，对所述目标训练图像进行对象关键点检测处理，确定所述目标训练图像中各像素点属于所述对象关键部位的置信度；
根据所述目标训练图像中属于所述对象关键部位的置信度高于预设置信度的像素点，确定所述参考对象关键点检测模型对应的关键点检测结果。
可选的，在图8所示的模型训练装置的基础上，参见图10，图10为本申请实施例提供的另一种模型训练装置1000的结构示意图。如图10所示，所述模型训练模块804包括：
关键点检测子模块1001，用于通过所述目标对象关键点检测模型，对所述训练样本中的所述目标训练图像进行对象关键点检测处理，得到所述目标训练图像对应的训练关键点检测结果；所述训练关键点检测结果包括多个对象关键部位各自对应的关键点的训练预测位置；
损失函数构建子模块1002，用于根据所述目标训练图像对应的训练关键点检测结果、以及所述训练样本中的所述关键点标注结果，构建目标损失函数；
参数调整子模块1003，用于基于所述目标损失函数，调整所述目标对象关键点检测模型的模型参数。
可选的，在图10所示的模型训练装置的基础上，所述损失函数构建子模块1002具体用于：
针对所述训练关键点检测结果中的每个关键点，判断所述关键点对应的对象关键部位是否为所述关键点标注结果中的目标对象关键部位；若是，则根据所述关键点对应的训练预测位置、以及所述关键点对应的目标对象关键部位的标注位置，构建所述关键点对应的损失项，并为所述关键点对应的损失项配置对应的第一损失权重；若否，根据所述关键点对应的训练预测位置，构建所述关键点对应的损失项，并为所述关键点对应的损失项配置对应的第二损失权重；所述第一损失权重大于0，所述第二损失权重等于0；
根据所述训练关键点检测结果中各关键点各自对应的损失项及其对应的损失权重，构建所述目标损失函数。
可选的，在图10所示的模型训练装置的基础上，所述损失函数构建子模块1002还用于：
确定所述目标视频中与所述目标训练图像相邻的视频帧，作为参考训练图像；并获取所述参考训练图像对应的训练关键点检测结果；
根据所述目标训练图像和所述参考训练图像各自对应的训练关键点检测结果，构建第一参考损失函数；
则所述参数调整子模块1003具体用于：
基于所述目标损失函数和所述第一参考损失函数，调整所述目标对象关键点检测模型的模型参数。
可选的，所述损失函数构建子模块1002具体用于：
针对每个所述对象关键部位，确定所述目标训练图像中所述对象关键部位对应的关键点的训练预测位置为第三位置，确定所述参考训练图像中所述对象关键部位对应的关键点的训练预测位置为第四位置；根据所述第三位置与所述第四位置之间的距离，构建所述对象关键部位对应的损失项，并为所述损失项配置对应的损失权重；
根据各所述对象关键部位各自对应的损失项及其对应的损失权重，构建所述第一参考损失函数。
可选的，在图10所示的模型训练装置的基础上，所述损失函数构建子模块1002还用于：
获取所述参考对象关键点检测模型处理所述目标训练图像时产生的第一中间处理结果；以及，获取所述目标对象关键点检测模型处理所述目标训练图像时产生的第二中间处理结果；
根据所述第一中间处理结果和所述第二中间处理结果，构建第二参考损失函数；
则所述参数调整子模块1003具体用于：
基于所述目标损失函数和所述第二参考损失函数，调整所述目标对象关键点检测模型的模型参数。
上述模型训练装置生成用于训练需要投入实际应用的目标对象关键点检测模型的训练样本时，会通过多个结构复杂的参考对象关键点检测模型分别对目标训练图像进行对象关键点检测处理，得到多个关键点检测结果。然后，基于各关键点检测结果中相同的对象关键部位的预测位置应基本一致的原理，针对每个对象关键部位，根据各个关键点检测结果中该对象关键部位对应的关键点的预测位置，衡量各个参考对象关键点检测模型对于该对象关键部位的位置预测结果是否可靠，即确定该对象关键部位是否为目标对象关键部位。在确定各个参考对象关键点检测模型对于该对象关键部位的位置预测结果可靠的情况下，再进一步确定该目标对象关键部位对应的标注位置即伪标签；进而，利用该目标训练图像以及其中各目标对象关键部位各自对应的标注位置组成训练样本。如此，忽略位置预测结果不可靠的对象关键部位，仅利用位置预测结果可靠的对象关键部位对应的标注位置作为伪标签，能够有效地减少所确定的伪标签中存在的噪声，提高该伪标签的准确度。相应地，基于包括该种伪标签的训练样本，训练需要投入实际应用、结构比较简单的目标对象关键 点检测模型，能够在一定程度上保证所训练的目标对象关键点检测模型的性能。
本申请实施例还提供了一种用于训练模型的计算机设备，该计算机设备具体可以是终端设备或者服务器，下面将从硬件实体化的角度对本申请实施例提供的终端设备和服务器进行介绍。
参见图11，图11是本申请实施例提供的终端设备的结构示意图。如图11所示，为了便于说明，仅示出了与本申请实施例相关的部分，具体技术细节未揭示的，请参照本申请实施例方法部分。该终端可以为包括手机、平板电脑、个人数字助理、销售终端(Point of Sales，POS)、车载电脑等任意终端设备，以终端为计算机为例：
图11示出的是与本申请实施例提供的终端相关的计算机的部分结构的框图。参考图11，计算机包括：射频(Radio Frequency，RF)电路1110、存储器1120、输入单元1130(其中包括触控面板1131和其他输入设备1132)、显示单元1140(其中包括显示面板1141)、传感器1150、音频电路1160(其可以连接扬声器1161和传声器1162)、无线保真(wireless fidelity，WiFi)模块1170、处理器1180、以及电源1190等部件。本领域技术人员可以理解，图11中示出的计算机结构并不构成对计算机的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。
存储器1120可用于存储软件程序以及模块，处理器1180通过运行存储在存储器1120的软件程序以及模块，从而执行计算机的各种功能应用以及数据处理。存储器1120可主要包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需的应用程序(比如声音播放功能、图像播放功能等)等；存储数据区可存储根据计算机的使用所创建的数据(比如音频数据、电话本等)等。此外，存储器1120可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他易失性固态存储器件。
处理器1180是计算机的控制中心，利用各种接口和线路连接整个计算机的各个部分，通过运行或执行存储在存储器1120内的软件程序和/或模块，以及调用存储在存储器1120内的数据，执行计算机的各种功能和处理数据，从而对计算机进行整体监控。可选的，处理器1180可包括一个或多个处理单元；优选的，处理器1180可集成应用处理器和调制解调处理器，其中，应用处理器主要处理操作系统、用户界面和应用程序等，调制解调处理器主要处理无线通信。可以理解的是，上述调制解调处理器也可以不集成到处理器1180中。
在本申请实施例中，该终端所包括的处理器1180还具有以下功能：
通过m个参考对象关键点检测模型，分别对目标训练图像进行对象关键点检测处理，得到所述m个参考对象关键点检测模型各自对应的关键点检测结果；所述关键点检测结果包括所述目标训练图像中多个对象关键部位各自对应的关键点的预测位置；所述m为大于1的整数；
针对每个所述对象关键部位，根据所述m个关键点检测结果中所述对象关键部位对应的关键点的预测位置，确定所述对象关键部位是否为目标对象关键部位；若是，则根据所述m个关键点检测结果中所述目标对象关键部位对应的关键点的预测位置，确定所述目标对象关键部位对应的标注位置；
将所述目标训练图像及其对应的关键点标注结果，作为训练样本；所述关键点标注结果包括各所述目标对象关键部位各自对应的标注位置；
基于所述训练样本，训练目标对象关键点检测模型。
可选的，所述处理器1180还用于执行本申请实施例提供的模型训练方法的任意一种实现方式的步骤。
参见图12，图12为本申请实施例提供的一种服务器1200的结构示意图。该服务器1200可因配置或性能不同而产生比较大的差异，可以包括一个或一个以上中央处理器(central processing units，CPU)1222(例如，一个或一个以上处理器)和存储器1232，一个或一个以上存储应用程序1242或数据1244的存储介质1230(例如一个或一个以上海量存储设备)。其中，存储器1232和存储介质1230可以是短暂存储或持久存储。存储在存储介质1230的程序可以包括一个或一个以上模块(图示没标出)，每个模块可以包括对服务器中的一系列指令操作。更进一步地，中央处理器1222可以设置为与存储介质1230通信，在服务器1200上执行存储介质1230中的一系列指令操作。
服务器1200还可以包括一个或一个以上电源1226，一个或一个以上有线或无线网络接口1250，一个或一个以上输入输出接口1258，和/或，一个或一个以上操作系统，例如Windows Server TM，Mac OS X TM，Unix TM,Linux TM，FreeBSD TM等等。
上述实施例中由服务器所执行的步骤可以基于该图12所示的服务器结构。
其中，CPU 1222用于执行如下步骤：
通过m个参考对象关键点检测模型，分别对目标训练图像进行对象关键点检测处理，得到所述m个参考对象关键点检测模型各自对应的关键点检测结果；所述关键点检测结果包括所述目标训练图像中多个对象关键部位各自对应的关键点的预测位置；所述m为大于1的整数；
针对每个所述对象关键部位，根据所述m个关键点检测结果中所述对象关键部位对应的关键点的预测位置，确定所述对象关键部位是否为目标对象关键部位；若是，则根据所述m个关键点检测结果中所述目标对象关键部位对应的关键点的预测位置，确定所述目标对象关键部位对应的标注位置；
将所述目标训练图像及其对应的关键点标注结果，作为训练样本；所述关键点标注结果包括各所述目标对象关键部位各自对应的标注位置；
基于所述训练样本，训练目标对象关键点检测模型。
可选的，CPU 1222还可以用于执行本申请实施例提供的模型训练方法的任意一种实现方式的步骤。
本申请实施例还提供一种计算机可读存储介质，用于存储计算机程序，该计算机程序用于执行前述各个实施例所述的一种模型训练方法中的任意一种实施方式。
本申请实施例还提供了一种计算机程序产品或计算机程序，该计算机程序产品或计算机程序包括计算机指令，该计算机指令存储在计算机可读存储介质中。计算机设备的处理器从计算机可读存储介质读取该计算机指令，处理器执行该计算机指令，使得该计算机设备执行前述各个实施例所述的一种模型训练方法中的任意一种实施方式。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统，装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统，装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储计算机程序的介质。
应当理解，在本申请中，“至少一个(项)”是指一个或者多个，“多个”是指两个或两个以上。“和/或”，用于描述关联对象的关联关系，表示可以存在三种关系，例如，“A和/或B”可以表示：只存在A，只存在B以及同时存在A和B三种情况，其中A，B可以是单数或者复数。字符“/”一般表示前后关联对象是一种“或”的关系。“以下至少一项(个)”或其类似表达，是指这些项中的任意组合，包括单项(个)或复数项(个)的任意组合。例如，a，b或c中的至少一项(个)，可以表示：a，b，c，“a和b”，“a和c”，“b和c”，或“a和b和c”，其中a，b，c可以是单个，也可以是多个。
以上所述，以上实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围。

Claims (15)

  1. 一种模型训练方法，所述方法由计算机设备执行，所述方法包括：
    通过m个参考对象关键点检测模型，分别对目标训练图像进行对象关键点检测处理，得到所述m个参考对象关键点检测模型各自对应的关键点检测结果；所述关键点检测结果包括所述目标训练图像中多个对象关键部位各自对应的关键点的预测位置；所述m为大于1的整数；
    针对每个所述对象关键部位，根据m个关键点检测结果中所述对象关键部位对应的关键点的预测位置，确定所述对象关键部位是否为目标对象关键部位；若是，则根据所述m个关键点检测结果中所述目标对象关键部位对应的关键点的预测位置，确定所述目标对象关键部位对应的标注位置；
    将所述目标训练图像及其对应的关键点标注结果，作为训练样本；所述关键点标注结果包括各所述目标对象关键部位各自对应的标注位置；
    基于所述训练样本，训练目标对象关键点检测模型。
  2. 根据权利要求1所述的方法，所述根据m个关键点检测结果中所述对象关键部位对应的关键点的预测位置，确定所述对象关键部位是否为目标对象关键部位，包括：
    根据所述m个关键点检测结果中所述对象关键部位对应的关键点的预测位置，确定所述对象关键部位对应的参考位置；
    针对每个所述关键点检测结果，确定其中所述对象关键部位对应的关键点的预测位置与所述对象关键部位对应的参考位置之间的距离，并根据所述距离，确定所述关键点检测结果是否属于所述对象关键部位对应的目标关键点检测结果；
    统计所述m个关键点检测结果中所述对象关键部位对应的目标关键点检测结果的目标数量；
    若所述目标数量超过预设数量，则确定所述对象关键部位为目标对象关键部位；若所述目标数量未超过所述预设数量，则确定所述对象关键部位为非目标对象关键部位。
  3. 根据权利要求2所述的方法，所述根据所述m个关键点检测结果中所述目标对象关键部位对应的关键点的预测位置，确定所述目标对象关键部位对应的标注位置，包括：
    针对所述目标对象关键部位对应的每个目标关键点检测结果，确定其中所述目标对象关键部位对应的关键点的预测位置，作为所述目标对象关键部位对应的目标预测位置；
    根据所述目标对象关键部位对应的各目标预测位置，确定所述目标对象关键部位对应的标注位置。
  4. 根据权利要求1所述的方法，当所述目标训练图像为目标视频中的视频帧时，所述方法还包括：
    确定所述目标视频中与所述目标训练图像相邻的视频帧，作为参考训练图像；并获取所述参考训练图像对应的关键点标注结果；
    针对每个所述目标对象关键部位，确定其在所述目标训练图像对应的关键点标注结果中的标注位置为第一位置，确定其在所述参考训练图像对应的关键点标注结果中的标注位置为第二位置；根据所述第一位置与所述第二位置之间的距离，确定是否保留所述目标训 练图像对应的关键点标注结果中的所述第一位置、以及是否保留所述参考训练图像对应的关键点标注结果中的所述第二位置。
  5. 根据权利要求1所述的方法，所述通过m个参考对象关键点检测模型，分别对目标训练图像进行对象关键点检测处理，得到所述m个参考对象关键点检测模型各自对应的关键点检测结果，包括：
    针对每个所述参考对象关键点检测模型，通过所述参考对象关键点检测模型，对所述目标训练图像进行对象关键点检测处理，确定所述目标训练图像中各像素点属于所述对象关键部位的置信度；
    根据所述目标训练图像中属于所述对象关键部位的置信度高于预设置信度的像素点，确定所述参考对象关键点检测模型对应的关键点检测结果。
  6. 根据权利要求1所述的方法，所述基于所述训练样本，训练目标对象关键点检测模型，包括：
    通过所述目标对象关键点检测模型，对所述训练样本中的所述目标训练图像进行对象关键点检测处理，得到所述目标训练图像对应的训练关键点检测结果；所述训练关键点检测结果包括多个对象关键部位各自对应的关键点的训练预测位置；
    根据所述目标训练图像对应的训练关键点检测结果、以及所述训练样本中的所述关键点标注结果，构建目标损失函数；
    基于所述目标损失函数，调整所述目标对象关键点检测模型的模型参数。
  7. 根据权利要求6所述的方法，所述根据所述目标训练图像对应的训练关键点检测结果、以及所述训练样本中的所述关键点标注结果，构建目标损失函数，包括：
    针对所述训练关键点检测结果中的每个关键点，判断所述关键点对应的对象关键部位是否为所述关键点标注结果中的目标对象关键部位；若是，则根据所述关键点对应的训练预测位置、以及所述关键点对应的目标对象关键部位的标注位置，构建所述关键点对应的损失项，并为所述关键点对应的损失项配置对应的第一损失权重；若否，根据所述关键点对应的训练预测位置，构建所述关键点对应的损失项，并为所述关键点对应的损失项配置对应的第二损失权重；所述第一损失权重大于0，所述第二损失权重等于0；
    根据所述训练关键点检测结果中各关键点各自对应的损失项及其对应的损失权重，构建所述目标损失函数。
  8. 根据权利要求6所述的方法，当所述目标训练图像为目标视频中的视频帧时，所述方法还包括：
    确定所述目标视频中与所述目标训练图像相邻的视频帧，作为参考训练图像；并获取所述参考训练图像对应的训练关键点检测结果；
    根据所述目标训练图像和所述参考训练图像各自对应的训练关键点检测结果，构建第一参考损失函数；
    所述基于所述目标损失函数，调整所述目标对象关键点检测模型的模型参数，包括：
    基于所述目标损失函数和所述第一参考损失函数，调整所述目标对象关键点检测模型的模型参数。
  9. 根据权利要求8所述的方法，所述根据所述目标训练图像和所述参考训练图像各自对应的训练关键点检测结果，构建第一参考损失函数，包括：
    针对每个所述对象关键部位，确定所述目标训练图像中所述对象关键部位对应的关键点的训练预测位置为第三位置，确定所述参考训练图像中所述对象关键部位对应的关键点的训练预测位置为第四位置；根据所述第三位置与所述第四位置之间的距离，构建所述对象关键部位对应的损失项，并为所述损失项配置对应的损失权重；
    根据各所述对象关键部位各自对应的损失项及其对应的损失权重，构建所述第一参考损失函数。
  10. 根据权利要求6或8所述的方法，所述方法还包括：
    获取所述参考对象关键点检测模型处理所述目标训练图像时产生的第一中间处理结果；以及，获取所述目标对象关键点检测模型处理所述目标训练图像时产生的第二中间处理结果；
    根据所述第一中间处理结果和所述第二中间处理结果，构建第二参考损失函数；
    所述基于所述目标损失函数，调整所述目标对象关键点检测模型的模型参数，包括：
    基于所述目标损失函数和所述第二参考损失函数，调整所述目标对象关键点检测模型的模型参数。
  11. 一种模型训练装置，所述装置包括：
    关键点检测模块，用于通过m个参考对象关键点检测模型，分别对目标训练图像进行对象关键点检测处理，得到所述m个参考对象关键点检测模型各自对应的关键点检测结果；所述关键点检测结果包括所述目标训练图像中多个对象关键部位各自对应的关键点的预测位置；所述m为大于1的整数；
    部位标注模块，用于针对每个所述对象关键部位，根据m个关键点检测结果中所述对象关键部位对应的关键点的预测位置，确定所述对象关键部位是否为目标对象关键部位；若是，则根据所述m个关键点检测结果中所述目标对象关键部位对应的关键点的预测位置，确定所述目标对象关键部位对应的标注位置；
    样本生成模块，用于将所述目标训练图像及其对应的关键点标注结果，作为训练样本；所述关键点标注结果包括各所述目标对象关键部位各自对应的标注位置；
    模型训练模块，用于基于所述训练样本，训练目标对象关键点检测模型。
  12. 根据权利要求11所述的装置，所述部位标注模块具体用于：
    根据所述m个关键点检测结果中所述对象关键部位对应的关键点的预测位置，确定所述对象关键部位对应的参考位置；
    针对每个所述关键点检测结果，确定其中所述对象关键部位对应的关键点的预测位置与所述对象关键部位对应的参考位置之间的距离，并根据所述距离，确定所述关键点检测结果是否属于所述对象关键部位对应的目标关键点检测结果；
    统计所述m个关键点检测结果中所述对象关键部位对应的目标关键点检测结果的目标数量；
    若所述目标数量超过预设数量，则确定所述对象关键部位为目标对象关键部位；若所 述目标数量未超过所述预设数量，则确定所述对象关键部位为非目标对象关键部位。
  13. 一种计算机设备，所述设备包括处理器及存储器；
    所述存储器用于存储计算机程序；
    所述处理器用于根据所述计算机程序执行权利要求1至10中任一项所述的模型训练方法。
  14. 一种计算机可读存储介质，所述计算机可读存储介质用于存储计算机程序，所述计算机程序用于执行权利要求1至10中任一项所述的模型训练方法。
  15. 一种计算机程序产品，包括计算机程序或者指令，所述计算机程序或者所述指令被处理器执行时，实现权利要求1至10中任一项所述的模型训练方法。
PCT/CN2022/120169 2021-11-24 2022-09-21 一种模型训练方法、装置、设备、介质和程序产品 Ceased WO2023093244A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP22897330.1A EP4390728A4 (en) 2021-11-24 2022-09-21 MODEL TRAINING METHOD AND APPARATUS, DEVICE, MEDIUM AND PROGRAM PRODUCT
JP2024520050A JP7635466B2 (ja) 2021-11-24 2022-09-21 モデル訓練方法、装置、機器、媒体およびプログラム
US18/199,235 US20230290003A1 (en) 2021-11-24 2023-05-18 Model training method and apparatus, device, medium, and program product

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111398623.5 2021-11-24
CN202111398623.5A CN113822254B (zh) 2021-11-24 2021-11-24 一种模型训练方法及相关装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/199,235 Continuation US20230290003A1 (en) 2021-11-24 2023-05-18 Model training method and apparatus, device, medium, and program product

Publications (1)

Publication Number Publication Date
WO2023093244A1 true WO2023093244A1 (zh) 2023-06-01

Family

ID=78919790

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/120169 Ceased WO2023093244A1 (zh) 2021-11-24 2022-09-21 一种模型训练方法、装置、设备、介质和程序产品

Country Status (5)

Country Link
US (1) US20230290003A1 (zh)
EP (1) EP4390728A4 (zh)
JP (1) JP7635466B2 (zh)
CN (1) CN113822254B (zh)
WO (1) WO2023093244A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116863279A (zh) * 2023-09-01 2023-10-10 南京理工大学 用于移动端模型轻量化的基于可解释指导的模型蒸馏方法

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113822254B (zh) * 2021-11-24 2022-02-25 腾讯科技（深圳）有限公司 一种模型训练方法及相关装置
CN114492761B (zh) * 2022-02-09 2024-08-16 腾讯科技（深圳）有限公司 编码模型的获取方法、聚类方法、装置、设备及介质
CN114638999B (zh) * 2022-03-23 2025-06-03 支付宝(杭州)信息技术有限公司 图像质量评估模型的训练方法、资料上传处理方法及装置
CN115526842A (zh) * 2022-09-14 2022-12-27 浙江华诺康科技有限公司 鼻咽喉镜监测方法、装置、系统、计算机设备和存储介质
US12505654B2 (en) * 2023-01-17 2025-12-23 Adobe Inc. Material selection from images
CN116843051A (zh) * 2023-05-12 2023-10-03 中国人民解放军92578部队 一种基于数据裁剪的船舶设备剩余使用寿命预测方法
CN117095205A (zh) * 2023-08-10 2023-11-21 支付宝(杭州)信息技术有限公司 一种生物体的检测方法、装置及设备
CN117853462B (zh) * 2024-01-11 2024-06-18 连云港市第二人民医院(连云港市临床肿瘤研究所) 基于多模态成像的关节内压力检测及信息提取方法和系统
CN119130927A (zh) * 2024-08-19 2024-12-13 卓世未来(成都)科技有限公司 一种远程自动判断脸痣能否进行激光去除的方法和装置
CN119785007B (zh) * 2024-12-25 2025-09-26 温州电力设计有限公司普华招标咨询分公司 一种目标识别方法、系统、电子设备和存储介质
CN121616845B (zh) * 2026-02-03 2026-05-01 西安交通大学 面向医学图像关键点检测的单样本学习路径构建方法

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200394416A1 (en) * 2018-07-27 2020-12-17 Tencent Technology (Shenzhen) Company Limited Method and apparatus for training feature extraction model, computer device, and computer-readable storage medium
CN113554034A (zh) * 2020-04-24 2021-10-26 北京达佳互联信息技术有限公司 关键点检测模型构建方法、检测方法、装置、设备及介质
CN113822254A (zh) * 2021-11-24 2021-12-21 腾讯科技（深圳）有限公司 一种模型训练方法及相关装置

Family Cites Families (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108280388A (zh) * 2017-01-06 2018-07-13 富士通株式会社 训练面部检测模型的方法和装置以及面部检测方法和装置
CN108509894A (zh) * 2018-03-28 2018-09-07 北京市商汤科技开发有限公司 人脸检测方法及装置
CN110555339A (zh) * 2018-05-31 2019-12-10 北京嘀嘀无限科技发展有限公司 一种目标检测方法、系统、装置及存储介质
CN109657615B (zh) * 2018-12-19 2021-11-02 腾讯科技（深圳）有限公司 一种目标检测的训练方法、装置及终端设备
CN109359207B (zh) * 2018-12-24 2021-01-22 焦点科技股份有限公司 一种易于快速迭代更新的Logo检测方法
CN112183166B (zh) * 2019-07-04 2024-07-02 北京地平线机器人技术研发有限公司 确定训练样本的方法、装置和电子设备
CN111160269A (zh) * 2019-12-30 2020-05-15 广东工业大学 一种人脸关键点检测方法及装置
KR20210087680A (ko) * 2020-01-03 2021-07-13 네이버 주식회사 입력 영상에 포함된 객체의 3차원 포즈를 추정하기 위한 데이터를 생성하는 방법 및 장치와 3차원 포즈 추정을 위한 추론 모델
CN111444978B (zh) * 2020-04-03 2023-06-09 王银璇 椎体成形术骨水泥渗漏检测方法、系统及存储介质
CN111523422B (zh) * 2020-04-15 2023-10-10 北京华捷艾米科技有限公司 一种关键点检测模型训练方法、关键点检测方法和装置
CN112101105B (zh) * 2020-08-07 2024-04-09 深圳数联天下智能科技有限公司 人脸关键点检测模型的训练方法、装置以及存储介质
CN112115894B (zh) * 2020-09-24 2023-08-25 北京达佳互联信息技术有限公司 手部关键点检测模型的训练方法、装置及电子设备
SG10202009760XA (en) * 2020-10-01 2021-04-29 Yitu Pte Ltd Video-based pose estimation method and apparatus, medium, and electronic device
CN112257659B (zh) * 2020-11-11 2024-04-05 四川云从天府人工智能科技有限公司 检测跟踪方法、装置以及介质
CN112990298B (zh) * 2021-03-11 2023-11-24 北京中科虹霸科技有限公司 关键点检测模型训练方法、关键点检测方法及装置
CN113449718A (zh) * 2021-06-30 2021-09-28 平安科技（深圳）有限公司 关键点定位模型的训练方法、装置和计算机设备
CN113610126B (zh) * 2021-07-23 2023-12-05 武汉工程大学 基于多目标检测模型无标签的知识蒸馏方法及存储介质

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200394416A1 (en) * 2018-07-27 2020-12-17 Tencent Technology (Shenzhen) Company Limited Method and apparatus for training feature extraction model, computer device, and computer-readable storage medium
CN113554034A (zh) * 2020-04-24 2021-10-26 北京达佳互联信息技术有限公司 关键点检测模型构建方法、检测方法、装置、设备及介质
CN113822254A (zh) * 2021-11-24 2021-12-21 腾讯科技（深圳）有限公司 一种模型训练方法及相关装置

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
See also references of EP4390728A4 *
SUN XIAO-XUAN , ZHANG LEI , LI JIAN: "Research on Semi-Automatic Generation Technology of Object Detection Datasets", COMPUTER SYSTEMS & APPLICATIONS, vol. 28, no. 10, 15 October 2019 (2019-10-15), pages 8 - 14, XP093069588, ISSN: 1003-3254, DOI: 10.15888/j.cnki.csa.007101 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116863279A (zh) * 2023-09-01 2023-10-10 南京理工大学 用于移动端模型轻量化的基于可解释指导的模型蒸馏方法
CN116863279B (zh) * 2023-09-01 2023-11-21 南京理工大学 用于移动端模型轻量化的基于可解释指导的模型蒸馏方法

Also Published As

Publication number Publication date
JP2024534684A (ja) 2024-09-20
CN113822254B (zh) 2022-02-25
CN113822254A (zh) 2021-12-21
EP4390728A1 (en) 2024-06-26
US20230290003A1 (en) 2023-09-14
JP7635466B2 (ja) 2025-02-25
EP4390728A4 (en) 2024-11-27

Similar Documents

Publication Publication Date Title
JP7635466B2 (ja) モデル訓練方法、装置、機器、媒体およびプログラム
WO2021098616A1 (zh) 运动姿态识别方法、运动姿态识别装置、终端设备及介质
CN108140032B (zh) 用于自动视频概括的设备和方法
CN110866563B (zh) 相似视频检测、推荐方法、电子设备和存储介质
CN112307886A (zh) 行人重识别方法和装置
CN111898561A (zh) 一种人脸认证方法、装置、设备及介质
CN114842389A (zh) 一种实时的鲁棒的两阶段姿态预估方法
CN108961314A (zh) 运动图像生成方法、装置、电子设备及计算机可读存储介质
CN110795975B (zh) 人脸误检优化方法及装置
CN114401446B (zh) 人体姿态迁移方法、装置、系统、电子设备以及存储介质
Shrestha et al. Human pose estimation for yoga using VGG-19 and COCO dataset: Development and implementation of a mobile application
CN111753795A (zh) 一种动作识别方法、装置、电子设备及存储介质
Yang et al. A survey on soccer player detection and tracking with videos
CN108550132A (zh) 一种基于全局紧凑先验和全局相似度显著传播的协同显著目标检测方法
CN111291692A (zh) 视频场景识别方法、装置、电子设备及存储介质
CN115362481A (zh) 运动模糊鲁棒图像特征描述符
CN111598924A (zh) 目标跟踪方法、装置、计算机设备及存储介质
CN116403285A (zh) 动作识别方法、装置、电子设备以及存储介质
CN118334755B (zh) 一种半监督动物三维姿态估计方法、设备及存储介质
CN111598923A (zh) 目标跟踪方法、装置、计算机设备及存储介质
CN116580211B (zh) 关键点检测方法、装置、计算机设备及存储介质
Di Giambattista et al. On field gesture-based robot-to-robot communication with NAO soccer players
Sun et al. Intelligent optimization algorithm of 3d tracking technology in football player moving image analysis
Yang et al. [Retracted] Falling‐Point Recognition and Scoring Algorithm in Table Tennis Using Dual‐Channel Target Motion Detection
CN117455948A (zh) 基于深度学习算法的多视角行人轨迹提取分析方法

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 2022897330

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2024520050

Country of ref document: JP

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 2022897330

Country of ref document: EP

Effective date: 20240320

NENP Non-entry into the national phase

Ref country code: DE