WO2025200078A1 - Suivi de visage basé sur une agrégation spatio-temporelle et antérieur rigide - Google Patents

Suivi de visage basé sur une agrégation spatio-temporelle et antérieur rigide

Info

Publication number
WO2025200078A1
WO2025200078A1 PCT/CN2024/090761 CN2024090761W WO2025200078A1 WO 2025200078 A1 WO2025200078 A1 WO 2025200078A1 CN 2024090761 W CN2024090761 W CN 2024090761W WO 2025200078 A1 WO2025200078 A1 WO 2025200078A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature map
generating
neural network
blendshape
face
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2024/090761
Other languages
English (en)
Inventor
Ming Lu
Anbang YAO
Shandong WANG
Yangyuxuan KANG
Yurong Chen
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Intel Corp
Original Assignee
Intel Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Intel Corp filed Critical Intel Corp
Publication of WO2025200078A1 publication Critical patent/WO2025200078A1/fr
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20076—Probabilistic image processing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face

Definitions

  • This disclosure relates generally to computer vision, and more specifically, face tracking based on spatial-temporal aggregation and rigid prior.
  • DNNs deep neural networks
  • CNNs convolutional neural networks
  • Computer vision tasks include methods for acquiring, processing, analyzing, or understanding visual images or videos to produce information, such as face tracking (tracking face expression, tracking face motion, etc. ) , motion detection, and so on.
  • FIG. 1 illustrates an example computer vision system, in accordance with various embodiments.
  • FIG. 2 illustrates memory-based aggregation of spatial-temporal features, in accordance with various embodiments.
  • FIG. 3 illustrates an example read controller, in accordance with various embodiments.
  • FIG. 4 illustrates face tracking using a blendshape decoder and a pose decoder, in accordance with various embodiments.
  • FIG. 6 illustrates an example rigid prior module, in accordance with various embodiments.
  • FIG. 7 illustrates an example convolution, in accordance with various embodiments.
  • FIG. 8 illustrates an example DNN, in accordance with various embodiments.
  • FIG. 9 illustrates an AI-based visual recognition environment, in accordance with various embodiments.
  • FIG. 10 is a flowchart showing a method of face tracking, in accordance with various embodiments.
  • FIG. 11 is a block diagram of an example computing device, in accordance with various embodiments.
  • Some face tracking methods use DNNs that directly predict 3D face animation parameters from monocular videos. Such methods typically use a two-dimensional (2D) face detector to crop the face region in each video frame, then train a 3D face tracking network to regress the parameters in the frame-by-frame manner, and finally use Kalman filter to smooth the parameters over all the video frames.
  • 2D two-dimensional
  • Kalman filter uses Kalman filter to smooth the parameters over all the video frames.
  • These DNN-based method suffer from limitations. For example, they require usage of Kalman filter as the post processing step to achieve temporally-consistent 3D face tracking results, which results in additional computational cost during inference.
  • they usually lack rigid stabilization, which is important in monocular 3D face tracking due to the depth ambiguity of the monocular input.
  • Embodiments of the present disclosure may improve on at least some of the challenges and issues described above by providing a method of face tracking (e.g., facial expression tracking, head pose tracking, etc. ) based on spatial-temporal aggregation and semantic-aware rigid prior.
  • face tracking e.g., facial expression tracking, head pose tracking, etc.
  • spatial-temporal features extracted from multiple frames may be aggregated for predicting a facial expression (e.g., smiling, frowning, eye rolling, appearing bored, appearing interested, etc. ) .
  • a facial expression e.g., smiling, frowning, eye rolling, appearing bored, appearing interested, etc.
  • the aggregated spatial-temporal features can also be used in combination with a semantic-aware rigid prior to predict a head pose.
  • the semantic-aware rigid prior can enable rigid stabilization and reduce or even eliminate depth shifting during face tracking and therefore, improve the accuracy of head pose prediction.
  • a computer vision system may extract feature maps from various frames in a video, e.g., a monocular video.
  • the frames may include a present frame and one or more historical frames.
  • the present frame may be the frame that has the most recent time stamp, while each historical frame has an earlier time stamp and is temporally precedent to the present frame in the video.
  • the computer vision system may aggregate a feature map extracted from the present frame ( “present feature map” ) with a feature map extracted from each historical frame ( “historical feature map” ) .
  • the computer vision system may use one or more transformer blocks to generate an intermediate feature map from the present feature map and the historical feature map (s) .
  • the output of the transformer block (s) may be further concatenated with the feature map to generate an aggregated feature map.
  • a transformer block may include a transformer encoder or transformer decoder.
  • the computer vision system may use a decoder to determine blendshape coefficients from the aggregated feature map.
  • the blendshape coefficients may represent the predicted facial expression.
  • Blendshapes may be models (e.g., 3D models) of facial expressions. For instance, a blendshape may be approximate semantic parameterization of a facial expression.
  • the computer vision system may generate a rigid transformation from the aggregated feature map by imposing a semantic-based rigid prior module to the aggregated feature map.
  • the computer vision system may use the rigid transformation to predict a head pose, e.g., by using another decoder.
  • the other decoder may output pose coefficients, e.g., rotation parameters and translation parameters.
  • the face in the video may be reconstructed using the blendshape coefficients and pose coefficients. For instance, a 3D face animation may be generated.
  • the phrase “A or B” or the phrase “A and/or B” means (A) , (B) , or (A and B) .
  • the phrase “A, B, or C” or the phrase “A, B, and/or C” means (A) , (B) , (C) , (A and B) , (A and C) , (B and C) , or (A, B, and C) .
  • the terms “comprise, ” “comprising, ” “include, ” “including, ” “have, ” “having” or any other variation thereof, are intended to cover a non-exclusive inclusion.
  • a method, process, device, or DNN accelerator that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such method, process, device, or DNN accelerators.
  • the term “or” refers to an inclusive “or” and not to an exclusive “or. ”
  • the training module 130 trains the face tracking module 110, such as the DNNs in the face tracking module 110.
  • the DNN in the face tracking module 110 may include the DNN 160, transformer decoders in the feature aggregation module 170, transformer decoders in the blendshape module 180, transformer decoders in the pose module 190, CNN in the pose module 190, and so on.
  • the training module 130 may modify internal parameters of the face tracking module 110 based on the ground-truth labels of the training samples and the outputs of the face tracking module 110 that are generated by processing the training samples.
  • Training samples may include video frames.
  • the ground-truth labels of training sample may include verified or known face tracking parameters that encode face expressions and poses shown in the video frames.
  • a training sample may include a sequence of frames. For each training sample, the training module 130 may sample a batch of consecutive N F +1 frames from the training dataset. The training module 130 may also extract and store the feature of the first N F frames and compute the loss based on the prediction of the last frame
  • the training module 130 modifies the internal parameters of the face tracking module 110 to minimize the error between labels of the training samples that are generated by the face tracking module 110 and the ground-truth labels.
  • the training module 130 uses a cost function or loss function to minimize the error.
  • the training module 130 may use the Mean Squared Error (MSE) loss across blendshape parameters, rotation parameters, and translation parameters.
  • MSE loss may be denoted as:
  • ⁇ denotes the ground-truth blendshape parameters
  • ⁇ denotes the predicted rotation parameters
  • ⁇ denotes the ground-truth rotation parameters
  • t denotes the ground-truth translation parameters, and stands for loss.
  • the training module 130 may also form validation datasets for validating performance of the face tracking module 110 after training by the validating module 140.
  • a validation dataset may include validation samples and ground-truth labels of the validation samples.
  • the validation dataset may include different samples from the training dataset used for training the face tracking module 110.
  • a part of a training dataset may be used to initially train the face tracking module 110, and the rest of the training dataset may be held back as a validation subset used by the validating module 140 to validate performance of the face tracking module 110.
  • the portion of the training dataset not including the validation subset may be used to train the face tracking module 110.
  • the training module 130 may define the architecture of the face tracking module 110 (or part of the face tracking module 110, e.g., a DNN in the face tracking module 110) , e.g., based on some of the hyperparameters.
  • the architecture of the DNN may include an input layer, an output layer, and a plurality of hidden layers.
  • the input layer may include tensors (e.g., a multi-dimensional array) specifying attributes of the IFM, such as the height of the IFM, the width of the IFM, and the depth of the IFM (e.g., the number of channels in the IFM) .
  • the output layer includes labels of objects in the input layer.
  • the hidden layers are layers between the input layer and output layer.
  • the hidden layers include one or more convolutional layers and one or more other types of layers, such as pooling layers, fully-connected layers, layer normalization layers, self attention layers, cross attention layers, Softmax or logistic layers, and so on.
  • the training module 130 may train the face tracking module 110 for a predetermined number of epochs.
  • the number of epochs is a hyperparameter that defines the number of times that the deep learning algorithm will work through the entire training dataset.
  • One epoch means that each sample in the training dataset has had an opportunity to update the internal parameters of the face tracking module 110.
  • the training module 130 may stop updating the internal parameters of the face tracking module 110, and the face tracking module 110 is considered trained.
  • the validating module 140 verifies accuracy of the face tracking module 110 after the face tracking module 110 is trained.
  • the validating module 140 inputs samples in a validation dataset into the face tracking module 110 and uses the outputs of the face tracking module 110 to determine the model accuracy.
  • a validation dataset may be formed of some or all the samples in the training dataset. Additionally or alternatively, the validation dataset includes additional samples, other than those in the training sets.
  • the validating module 140 determines may determine an accuracy score measuring the precision, recall, or a combination of precision and recall of the DNN.
  • the validating module 140 may compare the accuracy score with a threshold score. In an example where the validating module 140 determines that the accuracy score is lower than the threshold score, the validating module 140 instructs the training module 130 to re-train the face tracking module 110. In one embodiment, the training module 130 may iteratively re-train the face tracking module 110 until the occurrence of a stopping condition, such as the accuracy measurement indication that the face tracking module 110 may be sufficiently accurate, or a number of training rounds having taken place.
  • a stopping condition such as the accuracy measurement indication that the face tracking module 110 may be sufficiently accurate, or a number of training rounds having taken place.
  • the datastore 150 stores data received, generated, used, or otherwise associated with the computer vision system 100.
  • the datastore 150 stores the datasets used by the training module 130 and validating module 140.
  • the datastore 150 may also store data generated by the training module 130 and validating module 140, such as the hyperparameters for training the face tracking module 110, internal parameters of the face tracking module 110, and so on.
  • the datastore 150 may store video frames to be processed by the face tracking module 110 for performing face tracking tasks.
  • the datastore 150 may also store outputs of the face tracking module 110 or components of the face tracking module 110, such as feature maps, blendshapes parameters, pose parameters, segmentation probability maps, and so on.
  • the datastore 150 is a component of the computer vision system 100.
  • the datastore 150 may be external to the computer vision system 100 and communicate with the computer vision system 100 through a network.
  • FIG. 2 illustrates memory-based aggregation of spatial-temporal features, in accordance with various embodiments.
  • the memory-based aggregation of spatial-temporal features in FIG. 2 is performed by a feature aggregation module 210, which receives outputs of a backbone network 220.
  • the backbone network 220 may be an example of the DNN 160 in FIG. 1.
  • the backbone network 220 receives a video 201 that includes a sequence of frames 202 (individually referred to as “frame 202” ) .
  • the video 201 captures one or more faces, e.g., a face of a person or other types of objects.
  • the video 201 may be a monocular video.
  • the frames 202 may be arranged in a temporal sequence.
  • a frame 202 that has an earlier time stamp may be arranged before a frame 202 that has a later time stamp.
  • the backbone network 220 and the feature aggregation module 210 may operate in an online mode. For instance, the backbone network 220 may receive each frame 202 as the frame 202 is being displayed. The latest frame 202 that is being received by the backbone network 220 may be the present frame, and the other frame (s) 202 arranged before the present frame are historical frame (s) .
  • Each frame 202 received by the backbone network 220 is an input to the backbone network 220.
  • the backbone network 220 may be a DNN that includes a plurality of layers, which may include one or more convolutional layers.
  • the backbone network 220 processes the frame 202 using the layers and generates a feature map.
  • a feature map may be a 2D or 3D tensor that includes data points (e.g., activations) representing features extracted by the backbone network 220 from the corresponding frame 202.
  • the feature map of each historical frame is referred to as a previous feature map 203H.
  • the feature map of the present frame is referred to as a current feature map 203P.
  • the historical feature map (s) 203H and the current feature map 203P are collectively referred to as “feature maps 203” or “feature map 203. ”
  • the feature aggregation module 210 receives feature maps 203 from the backbone network 220 and generate spatial-temporal aggregated features from the feature maps 203.
  • the feature aggregation module 210 may be an example of the feature aggregation module 170 in FIG. 1. As shown in FIG. 2, the feature aggregation module 210 includes a write controller 230, a buffer 240, and a read controller 250. In other embodiments, alternative configurations, different or additional components may be included in the feature aggregation module 210. Further, functionality attributed to a component of the feature aggregation module 210 may be accomplished by a different component included in the feature aggregation module 210 or by a different module or system.
  • the write controller 230 may write feature maps 203 generated by the backbone network 220 into the buffer 240.
  • the write controller 230 may be implemented by a queue with a maximum length of N F so that no more than N F feature maps 203 would be preserved in the buffer 240 at a time. Storing a certain number of historical frames can facilitate effective and efficient face tracking because changes in expression and pose can exhibit smooth transitions in 3D face tracking. Frames 202 closer to the current time can contain more useful information, while frames 202 beyond a certain time window can be unlikely to contribute to the prediction of the current frame and may even introduce noise to the prediction.
  • feature maps 203 beyond a specific time window can be removed from the buffer 240 while feature maps 203 within the time window are preserved and stored, which can ensure accuracy of the prediction and mitigate the accumulation of errors during the prediction process.
  • the read controller 250 reads feature maps 203 stored in the buffer 240.
  • the read controller 250 may aggregate the current feature map 203C with one or more previous feature maps 203P.
  • the aggregation may be at least part of a spatial-temporal aggregation.
  • the read controller 250 includes one or more transformer decoders that perform the aggregation.
  • the transformer decoder (s) may take the current feature map 203P, which may be denoted as as queries and take the previous feature map (s) 203P, which may be denoted as as keys and values.
  • the transformer decoder (s) may output an intermediate feature map.
  • the intermediate feature map is then added with the current feature map 203C by a concatenator 260 via a skip connection.
  • f d denote the transformer decoder (s) , and denotes a spatial-tempoeral aggregated feature map 204.
  • FIG. 3 illustrates an example read controller 300, in accordance with various embodiments.
  • the read controller 300 aggregates features extracted from video frames having different time stamps.
  • the read controller 300 may aggregate features extracted from a temporal sequence of frames from a monocular video for predicting face expression or head pose captured in one of the frames.
  • the read controller 300 may be an example of the read controller 250 in FIG. 2.
  • the read controller 300 includes transformer decoders 310, individually referred to as “transformer decoder 310. ”
  • Each transformer decoder 310 includes a plurality of layers, such as a self attention layer 320, a layer norm layer 330, a cross attention layer 340, another layer norm layer 350, a feed forward layer 360, and another layer norm layer 370.
  • a transformer decoder 310 may include different, fewer, or more layers.
  • the layer read controller 300 may include a different number of transformer decoders 310.
  • the self attention layer 320 or cross attention layer 340 may have an attention function.
  • an attention function may map one or more queries and one or more key-value pairs to an output, where the query, each key, each value, or the output may be a tensor, such as a vector.
  • the output may be computed as a weighted sum of the values.
  • the weight assigned to each value may be computed by a compatibility function of the query with the corresponding key.
  • the layer norm layer 330 may connect all positions with a constant number of sequentially executed operations.
  • Self attention sometimes called intra-attention, may be an attention mechanism relating different positions of a single sequence in order to compute a representation of the sequence.
  • the self attention layer 320 may allow each position in the corresponding transformer decoder 310 to attend to all positions in the transformer decoder 310 up to and including that position.
  • the self attention layer 320 receives a vector 302C.
  • the vector 302C is generated by flattening a current feature map 301C, which may be generated from a current frame in the video.
  • the current feature map 301C may be a 2D or 3D tensor, while the vector 302C may be a 1D tensor.
  • the data points in the current feature map 301C may be rearranged so that the data points are all in one dimension to generate the vector 302C.
  • the current feature map 301C may have three dimensions: X, Y, and Z, in which the Z dimension may be the channel dimension.
  • the data points of the current feature map 301C may be rearranged to be all in one of the X, Y, and Z dimensions to form the vector 302C.
  • the vector 302C is input into the self attention layer 320 and may be used as the query, keys, and values of the self attention function.
  • the self attention function in the self attention layer 320 may be denoted as:
  • Q, K, V denotes the query, keys, and values, respectively
  • f (Q, K, V) denotes the self attention function
  • ⁇ denotes a Softmax function
  • the output of the self attention layer 320 is further accumulated with the vector 302C by a concatenator 325.
  • the result of the concatenation is input into the layer norm layer 330.
  • the layer norm layer 330 applies a layer normalization operation on the output of the concatenator 325.
  • the layer norm layer 330 may normalize its input (i.e., the output of the concatenator 325) across the feature dimension (instead of the batch dimension) .
  • the feature dimension may be the dimension of the vector 302C or the vector 302P.
  • the layer normalization operation may include a sequence of computations.
  • the layer norm layer 330 may include a mean computation , which may be denoted as where A xyz denotes a data element in the input tensor, x may be the positional index of the data element in one of the spatial dimensions, y may be the positional index of the data element in the other one of the spatial dimensions, z may be the positional index of the data element in the channel dimension, and ⁇ xy denotes the output of the mean computation, which may be a 2D matrix.
  • the mean computation may be channel-wise reduction operation.
  • the layer norm layer 330 may perform data to convert ⁇ xy to a 3D tensor ⁇ xyz , e.g., by replicating every data element over z output points.
  • the layer norm layer 330 may also perform a variance computation, which may be denoted as
  • the layer norm layer 330 may further include a division computation denoted as M xy may be a 2D tensor.
  • the layer norm layer 330 may also convert M xy to a 3D tensor M xyz , e.g., by replicating every data element over z output points.
  • the layer norm layer 330 may have an element multiplication denoted as
  • the layer norm layer 330 may further compute LN xyz may be the output of the layer normalization operation.
  • the output of the layer norm layer 330 which encodes the flattened current feature, may be input into the cross attention layer 340.
  • the cross attention layer 340 also receives a vector 302P as another input.
  • the vector 302P is generated by flattening previous feature maps 301P, which may be generated from previous frames in the video.
  • a previous feature map 301P may be a 2D or 3D tensor, while the vector 302P may be a 1D tensor.
  • the data points in the previous feature maps 301P may be rearranged so that the data points are all in one dimension to generate the vector 302P.
  • each a previous feature map 301P may have three dimensions: X, Y, and Z, in which the Z dimension may be the channel dimension.
  • the data points of the previous feature maps 301P may be rearranged to be all in one of the X, Y, and Z dimensions to form the vector 302P.
  • the data points in the same previous feature map may be arranged together, e.g., in a manner that is not interrupted by any data points in a different previous feature map.
  • the vector 302P is in the same dimension as the vector 302C.
  • the vector 302C and the vector 302P are each represented by a sequence of boxes in FIG. 3 for the purpose of illustration.
  • the number of boxes in the vector 302C or the vector 302P does not represent the number of data points in the vector 302C or the vector 302P.
  • the number of data points in the vector 302C may equal the number of data points in the current feature map 301C, and the number of data points in the vector 302P may equal the total number of data points in all the previous feature maps 301P.
  • the cross attention layer 340 has a cross attention function that is applied on the vector 302P and the vector 302C.
  • the vector 302P may be used as the query, and the vector 302C may be used as the keys and values.
  • the cross attention function may be denoted as:
  • ⁇ is the Softmax function; denotes the vector 302C; denotes the vector 302P; and W Q , W K , are projection matrixes for query, keys, and values respectively.
  • the output of the cross attention layer 340 may be accumulated with the output of the layer norm layer 330 by another concatenator 345.
  • the output of the concatenator 345 is input into the layer norm layer 350 for performing another layer normalization operation in the layer norm layer 350.
  • the output of the layer norm layer 350 is input into the feed forward layer 360.
  • the feed forward layer 360 may include a linear transformation and a non-linear transformation.
  • the linear transformation may be the same or similar to the operation in a fully-connected layer, e.g., a fully-connected layer 830 in FIG. 8.
  • the non-linear transformation may be an activation function.
  • the feed forward layer 360 may multiple the input with keys to get a weight for each key.
  • the feed forward layer 360 may compute a weight sum of the values using the weights to produce the output.
  • the output of the feed forward layer 360 may be accumulated with the output of the layer norm layer 350 by a concatenator 365.
  • the output of the concatenator 365 may be input into the layer norm layer 370 in which another layer normalization operation is performed.
  • the output of the layer norm layer 370 may be input into the next transformer decoder 310 for further processing.
  • the output of the last transformer decoder 310 may constitute the output of the read controller 300, which may be an intermediate feature map that represents an aggregation of the current feature map 301C and the previous feature maps 301P.
  • FIG. 4 illustrates face tracking using a blendshape decoder 410 and a pose decoder 430 coupled with a rigid prior module 420, in accordance with various embodiments.
  • the blendshape decoder 410 receives a feature map 401.
  • the feature map 401 may be an aggregated feature map generated from a feature map extracted from a current frame and one or more feature maps extracted from one or more previous frames.
  • the feature map 401 is generated by the feature aggregation module 170 in FIG. 1.
  • the blendshape decoder 410 also receives a plurality of latent representations 402, individually referred to as latent representation 402.
  • Each latent representation 402 may be an implicit blendshape latent representation.
  • the latent representations 402 may represent various blendshapes corresponding to various face expressions.
  • Each latent representation 402 may be a learnable latent tensor.
  • a learnable latent tensor may be denoted as where N E is the number of blendshapes and C E is the channel number of the latent representations.
  • Each latent tensor may represent or encode a blendshape after training.
  • the blendshape decoder 410 uses the feature map 401 and the latent representations 402 to predict a face expression of the face shown in the current frame.
  • the blendshape decoder 410 may include one or more transformer decoders that processes the feature map 401 and the latent representations 402.
  • the blendshape decoder 410 generates blendshape parameters 403 (also referred to as “blendshape coefficients” ) , which may encode the predicted face expression.
  • blendshape parameters 403 may be denoted as:
  • the feature map 401 is also input into the rigid prior module 420.
  • the rigid prior module 420 may reduce or even eliminate depth shifting in the feature map 401 for achieving better rigid stabilization.
  • the rigid prior module 420 may use a segmentation map 404 of the face to generate a rigid transformation from the feature map 401.
  • the rigid transformation is input into the pose decoder 430.
  • the pose decoder 430 also receives a plurality of latent representations 405, individually referred to as latent representation 405.
  • Each latent representation 405 may be an implicit pose latent representation.
  • Each latent representation 405 may be a learnable latent tensor.
  • the latent representation 405 may represent various poses.
  • the pose decoder 430 may output pose parameters 406 (also referred to as “pose coefficients” ) that encode the predicted head pose.
  • the pose parameters 406 may be denoted as In some embodiments, the pose parameters may be decomposed into a six-degree-of-freedom vector, where rotation is parameterized by three Euler angles and translation is represented by a vector
  • FIG. 5 illustrates an example decoder module 500, in accordance with various embodiments.
  • the decoder module 500 may be an example of the blendshape decoder 410 or the pose decoder 430 in FIG. 4.
  • the decoder module 500 includes transformer decoders 510, individually referred to as “transformer decoder 510. ”
  • Each transformer decoder 510 includes a plurality of layers, such as a self attention layer 520, a layer norm layer 530, a cross attention layer 540, another layer norm layer 550, a feed forward layer 560, and another layer norm layer 570.
  • a transformer decoder 510 may include different, fewer, or more layers.
  • the layer decoder module 500 may include a different number of transformer decoders 510.
  • a transformer decoder 510 may be the same or similar as the transformer decoder 310 in FIG. 3.
  • the data points of the feature map 501 may be rearranged to be all in one of the X, Y, and Z dimensions to form the vector 502.
  • the vector 502 is input into the self attention layer 520 and may be used as the query, keys, and values of the self attention function.
  • the self attention function in the self attention layer 520 may be denoted as:
  • the output of the self attention layer 520 is further accumulated with the vector 502 by a concatenator 525.
  • the result of the concatenation is input into the layer norm layer 530.
  • the layer norm layer 530 applies a layer normalization operation on the output of the concatenator 525.
  • the layer norm layer 530 may normalize its input (i.e., the output of the concatenator 525) across the feature dimension (instead of the batch dimension) .
  • the feature dimension may be the dimension of the vector 502.
  • the output of the layer norm layer 530 may be input into the cross attention layer 540.
  • the cross attention layer 540 also receives latent representations 504 (individually referred to as “latent representation 504” ) as another input.
  • the latent representations 504 may be implicit blendshape latent representations, e.g., in embodiments where the decoder module 500 is used for predicting blendshapes. In other embodiments, (e.g., embodiments where the decoder module 500 is used for predicting poses) , the latent representations 504 may be implicit pose latent representations.
  • the cross attention function applied on the latent representations 503 and the output of the layer norm layer 530 may use the latent representations 503 as queries and use the output of the layer norm layer 530, which may represent the aggregated spatial-temporal features, as keys and values.
  • the cross attention function which may be the same or similar as the cross attention function described above in conjunction with FIG. 3 but with different query, keys, and values.
  • the output of the cross attention layer 540 may be accumulated with the output of the layer norm layer 530 by another concatenator 545.
  • the output of the concatenator 545 is input into the layer norm layer 550 for performing another layer normalization operation in the layer norm layer 550.
  • the output of the layer norm layer 550 is input into the feed forward layer 560.
  • the feed forward layer 560 may have the same or similar computations as the feed forward layer 360.
  • the output of the feed forward layer 560 may be accumulated with the output of the layer norm layer 550 by a concatenator 565.
  • the output of the concatenator 565 may be input into the layer norm layer 570 in which another layer normalization operation is performed.
  • the output of the layer norm layer 570 may be input into the next transformer decoder 510 for further processing.
  • the output of the last transformer decoder 510 may constitute the output of the decoder module 500, which may be blendshape coefficients or pose coefficients.
  • FIG. 6 illustrates an example rigid prior module 600, in accordance with various embodiments.
  • the rigid prior module 600 receives a segmentation map 602.
  • the segmentation map 602 is generated from an image 601 that captures a head.
  • the image 601 may be a frame in a video.
  • the segmentation map 602 includes information that indicates segmentation of the head in the image 601.
  • the segmentation map 602 may include a plurality of regions, each of which may correspond to a distinct portion of the head.
  • the segmentation map 602 includes a region corresponding to the hair, a region corresponding to the face, a region corresponding to the nose, a region corresponding to the glasses, and so on.
  • the segmentation map 602 may be a segmentation probability map of the current frame.
  • the segmentation map 602 may be generated by a face segmentation model, which segments a face into distinct regions.
  • the segmentation map 602 may be an example of the segmentation map 404 in FIG. 4.
  • one or more regions in the segmentation map 602 may be identified or labeled as rigid region (s) .
  • a rigid region may correspond to a portion of the head that has no or minimum distortion or change when the head pose changes. Examples of rigid regions include forehead, nose, and so on.
  • the segmentation map 602 may be used by the rigid prior module 600 to improve rigid stabilization for pose prediction.
  • a DNN 610 in the rigid prior module 600 receives the segmentation map 602 as an input and outputs a reweighting mask 603.
  • the DNN 610 may be a CNN, an example of which may be the DNN 800 in FIG. 8.
  • the reweighting mask 603 may be denoted as whose spatial dimension is the same as a feature map 604.
  • the feature map 604 may be an aggregation of a feature map extracted from the current frame and feature maps extracted from previous frames. In some embodiments, the feature map 604 is generated by the feature aggregation module 170 in FIG. 7.
  • a multiplier 620 in the rigid prior module 600 multiplies the reweighting mask 603 with the feature map 604 and computes a rigid transformation 605.
  • the rigid transformation 605 is a tensor that has the same spatial size or shape as the feature map 604. The values of the data points in the rigid transformation 605 may be different from those in the feature map 604. By multiplying the feature map 604 with the reweighting mask 603, the rigid regions can be enhanced.
  • the rigid transformation 605 may be used to predict the pose of the head.
  • the rigid transformation 605 may be processed by a pose decoder, such as the pose decoder 430 to generate pose parameters.
  • FIG. 7 illustrates an example convolution, in accordance with various embodiments.
  • the convolution may be a deep learning operation in a convolutional layer of a DNN, e.g., the DNN 160 in FIG. 1, the backbone network 220 in FIG. 2, the DNN 610 in FIG. 6, and so on.
  • the convolution may extract features from an input.
  • the convolution can be executed on an input tensor 710 and filters 720 (individually referred to as “filter 720” ) .
  • the result of the convolution is an output tensor 730.
  • the convolution is performed by a DNN accelerator.
  • the input tensor 710 includes activations (also referred to as “input activations, ” “elements, ” or “input elements” ) arranged in a 3D matrix.
  • An input element is a data point in the input tensor 710.
  • the input tensor 710 has a spatial size H in ⁇ W in ⁇ C in , where H in is the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of activations in a column in the 3D matrix of each input channel) , W in is the width of the 3D matrix (i.e., the length along the X axis, which indicates the number of activations in a row in the 3D matrix of each input channel) , and C in is the depth of the 3D matrix (i.e., the length along the Z axis, which indicates the number of input channels) .
  • the input tensor 710 has a spatial size of 7 ⁇ 7 ⁇ 3, i.e., the input tensor 710 includes three input channels and each input channel has a 7 ⁇ 7 2D matrix.
  • Each input element in the input tensor 710 may be represented by a (X, Y, Z) coordinate. In other embodiments, the height, width, or depth of the input tensor 710 may be different.
  • each filter 720 in FIG. 7 has a spatial size of 7 ⁇ 3 ⁇ 3, i.e., the filter 720 includes 7 convolutional kernels with a spatial size of 3 ⁇ 3.
  • the height, width, or depth of the filter 720 may be different.
  • the spatial size of the convolutional kernels is smaller than the spatial size of the 2D matrix of each input channel in the input tensor 710.
  • An activation or weight may take one or more bytes in a memory.
  • the number of bytes for an activation or weight may depend on the data format. For example, when the activation or weight has an INT8 format, the activation takes one byte. When the activation or weight has a FP16 format, the activation or weight takes two bytes. Other data formats may be used for activations or weights.
  • each filter 720 slides across the input tensor 710 and generates a 2D matrix for an output channel in the output tensor 730.
  • the 2D matrix has a spatial size of 5 ⁇ 5.
  • the output tensor 730 includes activations (also referred to as “output activations, ” “elements, ” or “output element” ) arranged in a 3D matrix.
  • An output activation is a data point in the output tensor 730.
  • the output tensor 730 has a spatial size H out ⁇ W out ⁇ C out , where H out is the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of output activations in a column in the 2D matrix of each output channel) , W out is the width of the 3D matrix (i.e., the length along the X axis, which indicates the number of output activations in a row in the 2D matrix of each output channel) , and C out is the depth of the 3D matrix (i.e., the length along the Z axis, which indicates the number of output channels) .
  • C out may equal the number of filters 720 in the convolution.
  • H out and W out may depend on the heights and weights of the input tensor 710 and each filter 720.
  • multiply-accumulate (MAC) operations can be performed on a 3 ⁇ 3 ⁇ 3 subtensor 715 (which is highlighted with a dotted pattern in FIG. 7) in the input tensor 710 and each filter 720.
  • the result of the MAC operations on the subtensor 715 and one filter 720 is an output activation.
  • an output activation may include 8 bits, e.g., one byte.
  • an output activation may include more than one byte. For instance, an output element may include two bytes.
  • a vector 735 is produced.
  • the vector 735 is highlighted with slashes in FIG. 7.
  • the vector 735 includes a sequence of output activations, which are arranged along the Z axis.
  • the output activations in the vector 735 have the same (X, Y) coordinate, but the output activations correspond to different output channels and have different Z coordinates.
  • the dimension of the vector 735 along the Z axis may equal the total number of output channels in the output tensor 730.
  • the output activations in the output tensor 730 may be further processed based on one or more activation functions before they are stored or inputted into the next layer of the CNN.
  • the processing based on the one or more activation functions may be at least part of the post processing of the convolution.
  • the post processing may include one or more other computations, such as offset computation, bias computation, and so on.
  • the results of the post processing may be stored in a local memory of the compute block and be used as input to the next layer.
  • the input activations in the input tensor 710 may be results of post processing of the previous layer. Even though the input tensor 710, filters 720, and output tensor 730 are 3D tensors in FIG. 7, the input tensor 710, a filter 720, or the output tensor 730 may be a 2D tensor in other embodiments.
  • the convolutional layers 810 summarize the presence of features in the input to the DNN 800.
  • the convolutional layers 810 function as feature extractors.
  • the first layer of the DNN 800 is a convolutional layer 810.
  • a convolutional layer 810 performs a convolution on an input tensor 840 (also referred to as IFM 840) and a filter 850.
  • the IFM 840 is represented by a 7 ⁇ 7 ⁇ 3 three-dimensional (3D) matrix.
  • the IFM 840 includes 3 input channels, each of which is represented by a 7 ⁇ 7 two-dimensional (2D) matrix.
  • the 7 ⁇ 7 2D matrix includes 7 input elements (also referred to as input points) in each row and seven input elements in each column.
  • the convolution includes MAC operations with the input elements in the IFM 840 and the weights in the filter 850.
  • the convolution may be a standard convolution 863 or a depthwise convolution 883. In the standard convolution 863, the whole filter 850 slides across the IFM 840. All the input channels are combined to produce an output tensor 860 (also referred to as OFM 860) .
  • the OFM 860 is represented by a 5 ⁇ 5 2D matrix.
  • the 5 ⁇ 5 2D matrix includes 5 output elements (also referred to as output points) in each row and five output elements in each column.
  • the standard convolution includes one filter in the embodiments of FIG. 8. In embodiments where there are multiple filters, the standard convolution may produce multiple output channels in the OFM 860.
  • the multiplication applied between a kernel-sized patch of the IFM 840 and a kernel may be a dot product.
  • a dot product is the elementwise multiplication between the kernel-sized patch of the IFM 840 and the corresponding kernel, which is then summed, always resulting in a single value. Because it results in a single value, the operation is often referred to as the “scalar product. ”
  • Using a kernel smaller than the IFM 840 is intentional as it allows the same kernel (set of weights) to be multiplied by the IFM 840 multiple times at different points on the IFM 840.
  • the kernel is applied systematically to each overlapping part or kernel-sized patch of the IFM 840, left to right, top to bottom.
  • the depthwise convolution 883 In the depthwise convolution 883, the input channels are not combined. Rather, MAC operations are performed on an individual input channel and an individual kernel and produce an output channel. As shown in FIG. 8, the depthwise convolution 883 produces a depthwise output tensor 880.
  • the depthwise output tensor 880 is represented by a 5 ⁇ 5 ⁇ 3 3D matrix.
  • the depthwise output tensor 880 includes 3 output channels, each of which is represented by a 5 ⁇ 5 2D matrix.
  • the 5 ⁇ 5 2D matrix includes 5 output elements in each row and five output elements in each column.
  • Each output channel is a result of MAC operations of an input channel of the IFM 840 and a kernel of the filter 850.
  • the first output channel (patterned with dots) is a result of MAC operations of the first input channel (patterned with dots) and the first kernel (patterned with dots)
  • the second output channel (patterned with horizontal strips) is a result of MAC operations of the second input channel (patterned with horizontal strips) and the second kernel (patterned with horizontal strips)
  • the third output channel (patterned with diagonal stripes) is a result of MAC operations of the third input channel (patterned with diagonal stripes) and the third kernel (patterned with diagonal stripes) .
  • the number of input channels equals the number of output channels, and each output channel corresponds to a different input channel.
  • the input channels and output channels are referred to collectively as depthwise channels.
  • a pointwise convolution 893 is then performed on the depthwise output tensor 880 and a 1 ⁇ 1 ⁇ 3 tensor 890 to produce the OFM 860.
  • the OFM 860 is then passed to the next layer in the sequence.
  • the OFM 860 is passed through an activation function.
  • An example activation function is rectified linear unit (ReLU) .
  • ReLU is a calculation that returns the value provided as input directly, or the value zero if the input is zero or less.
  • the convolutional layer 810 may receive several images as input and calculate the convolution of each of them with each of the kernels. This process can be repeated several times. For instance, the OFM 860 is passed to the subsequent convolutional layer 810 (i.e., the convolutional layer 810 following the convolutional layer 810 generating the OFM 860 in the sequence) .
  • a convolutional layer 810 has four hyperparameters: the number of kernels, the size F kernels (e.g., a kernel is of dimensions F ⁇ F ⁇ D pixels) , the S step with which the window corresponding to the kernel is dragged on the image (e.g., a step of one means moving the window one pixel at a time) , and the zero-padding P (e.g., adding a black contour of P pixels thickness to the input image of the convolutional layer 810) .
  • the convolutional layers 810 may perform various types of convolutions, such as 2-dimensional convolution, dilated or atrous convolution, spatial separable convolution, depthwise separable convolution, transposed convolution, and so on.
  • the DNN 800 includes 86 convolutional layers 810. In other embodiments, the DNN 800 may include a different number of convolutional layers.
  • a pooling layer 820 receives feature maps generated by the preceding convolution layer 810 and applies a pooling operation to the feature maps.
  • the pooling operation reduces the size of the feature maps while preserving their important characteristics. Accordingly, the pooling operation improves the efficiency of the CNN and avoids over-learning.
  • the pooling layers 820 may perform the pooling operation through average pooling (calculating the average value for each patch on the feature map) , max pooling (calculating the maximum value for each patch of the feature map) , or a combination of both.
  • the size of the pooling operation is smaller than the size of the feature maps.
  • the linear transformation may include a tensor multiplication between the input operand and the weight matrix.
  • the result of the linear transformation may be an output operand.
  • the fully-connected layer may further apply a non-linear transformation (e.g., by using a non-linear activation function) on the result of the linear transformation to generate an output operand.
  • the output operand may contain as many elements as there are classes: element i represents the probability that the image belongs to class i. Each element is therefore between 0 and 8, and the sum of all is worth one. These probabilities are calculated by the last fully-connected layer 830 by using a logistic function (binary classification) or a SoftMax function (multi-class classification) as an activation function.
  • FIG. 9 illustrates an AI-based face tracking environment 900, in accordance with various embodiments.
  • the AI-based face tracking environment 900 includes a computer vision system 910, client devices 920 (individually referred to as client device 920) , and a third-party system 930.
  • client devices 920 individually referred to as client device 920
  • third-party system 930 the AI-based face tracking environment 900 may include fewer, more, or different components.
  • the AI-based face tracking environment 900 may include a different number of client devices 920 or more than one third-party system 930.
  • the client devices 920 are in communication with the computer vision system 910.
  • the client device 920 may receive 3D face animations from the computer vision system 910 and display the 3D face animations to one or more users associated with the client device 920.
  • a client device 920 may execute one or more applications allowing one or more users of the client device 920 to interact with the computer vision system 910.
  • a client device 920 executes a browser application to enable interaction between the client device 920 and the computer vision system 910.
  • a client device 920 interacts with the computer vision system 910 through an application programming interface (API) running on a native operating system of the client device 920, such as or ANDROID TM .
  • API application programming interface
  • a client device 920 may be one or more computing devices capable of receiving user input as well as transmitting and/or receiving data via the network 940.
  • a client device 920 is a conventional computer system, such as a desktop or a laptop computer.
  • a client device 920 may be a device having computer functionality, such as a personal digital assistant (PDA) , a mobile telephone, a smartphone, an autonomous vehicle, or another suitable device.
  • PDA personal digital assistant
  • a client device 920 is configured to communicate via the network 940.
  • a client device 920 is an integrated computing device that operates as a standalone network-enabled device.
  • the client device 920 includes display, speakers, microphone, camera, and input device.
  • a client device 920 is a computing device for coupling to an external media device such as a television or other external display and/or audio output system.
  • the client device 920 may couple to the external media device via a wireless interface or wired interface and may utilize various functions of the external media device such as its display, speakers, microphone, camera, and input devices.
  • the client device 920 may be configured to be compatible with a generic external media device that does not have specialized software, firmware, or hardware specifically for interacting with the client device 920.
  • the third-party system 930 is an online system that may communicate with the computer vision system 910 or at least one of the client devices 920.
  • the third-party system 930 may provide data to the computer vision system 910 for 3D face tracking.
  • the data may include videos, data for training DNNs, data for validating DNNs, and so on.
  • the third-party system 930 may be a social media system, an online image gallery, an online searching system, and so on. Additionally or alternatively, the third-party system 930 may use results of 3D face tracking in various applications. For instance, the third-party system 930 may use face tracking results from the computer vision system 910 for virtual reality, augmented reality, film and game production, telepresence, and so on.
  • networking protocols used for communicating via the network 940 may include multiprotocol label switching (MPLS) , transmission control protocol/Internet protocol (TCP/IP) , hypertext transport protocol (HTTP) , simple mail transfer protocol (SMTP) , and file transfer protocol (FTP) .
  • MPLS multiprotocol label switching
  • TCP/IP transmission control protocol/Internet protocol
  • HTTP hypertext transport protocol
  • SMTP simple mail transfer protocol
  • FTP file transfer protocol
  • Data exchanged over the network 940 may be represented using any suitable format, such as hypertext markup language (HTML) or extensible markup language (XML) .
  • HTML hypertext markup language
  • XML extensible markup language
  • all or some of the communication links of the network 940 may be encrypted using any suitable technique or techniques.
  • FIG. 10 is a flowchart showing a method 1000 of face tracking, in accordance with various embodiments.
  • the method 1000 may be a method of 3D visual recognition.
  • the method 1000 may be performed by the face tracking module 110 in FIG. 1.
  • the method 1000 is described with reference to the flowchart illustrated in FIG. 10, many other methods for visual recognition may alternatively be used.
  • the order of execution of the steps in FIG. 10 may be changed.
  • some of the steps may be changed, eliminated, or combined.
  • the face tracking module 110 generates 1010 by a first neural network, a first feature map from a first frame in a video that captures a face.
  • the first neural network is the DNN 160 in FIG. 1.
  • the first frame is the current frame.
  • the first feature map is the current feature map.
  • the face tracking module 110 generates 1020, by the first neural network, a second feature map from a second frame in a video.
  • the second frame is temporally subsequent to the first frame in the video.
  • the first frame is a previous frame.
  • the first feature map is a previous feature map.
  • the face tracking module 110 generates 1030 an aggregated feature map by using the first feature map with the second feature map. In some embodiments, the face tracking module 110 writes the first feature map to a buffer after generating the first feature map and reads the first feature map from the buffer after generating the second feature map.
  • the face tracking module 110 generates 1150 an animation of the face based on the blendshape coefficients. In some embodiments, the face tracking module 110 generates the animation of the face based on the one or more blendshape coefficients and the one or more pose coefficients. In some embodiments, the face tracking module 110 generates the animation of the face further based on one or more blendshape templates.
  • FIG. 11 is a block diagram of an example computing device 1100, in accordance with various embodiments.
  • the computing device 1100 can be used as at least part of the computer vision system 100.
  • a number of components are illustrated in FIG. 11 as included in the computing device 1100, but any one or more of these components may be omitted or duplicated, as suitable for the application.
  • some or all of the components included in the computing device 1100 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated onto a single system on a chip (SoC) die. Additionally, in various embodiments, the computing device 1100 may not include one or more of the components illustrated in FIG.
  • SoC system on a chip
  • the computing device 1100 may include interface circuitry for coupling to the one or more components.
  • the computing device 1100 may not include a display device 1106, but may include display device interface circuitry (e.g., a connector and driver circuitry) to which a display device 1106 may be coupled.
  • the computing device 1100 may not include an audio input device 1118 or an audio output device 1108, but may include audio input or output device interface circuitry (e.g., connectors and supporting circuitry) to which an audio input device 1118 or audio output device 1108 may be coupled.
  • the computing device 1100 may include a processing device 1102 (e.g., one or more processing devices) .
  • the processing device 1102 processes electronic data from registers and/or memory to transform that electronic data into other electronic data that may be stored in registers and/or memory.
  • the computing device 1100 may include a memory 1104, which may itself include one or more memory devices such as volatile memory (e.g., DRAM) , nonvolatile memory (e.g., read-only memory (ROM) ) , high bandwidth memory (HBM) , flash memory, solid state memory, and/or a hard drive.
  • the memory 1104 may include memory that shares a die with the processing device 1102.
  • the memory 1104 includes one or more non-transitory computer-readable media storing instructions executable to perform operations for performing 3D face tracking, e.g., the method 1000 described above in conjunction with FIG. 10 or some operations performed by the computer vision system 100 or a component of the computer vision system 100 (e.g., the face tracking module 110) described above in conjunction with FIG. 1.
  • the instructions stored in the one or more non-transitory computer-readable media may be executed by the processing device 1102.
  • the computing device 1100 may include a communication chip 1112 (e.g., one or more communication chips) .
  • the communication chip 1112 may be configured for managing wireless communications for the transfer of data to and from the computing device 1100.
  • wireless and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate data through the use of modulated electromagnetic radiation through a nonsolid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not.
  • the communication chip 1112 may implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.10 family) , IEEE 802.16 standards (e.g., IEEE 802.16-2005 Amendment) , Long-Term Evolution (LTE) project along with any amendments, updates, and/or revisions (e.g., advanced LTE project, ultramobile broadband (UMB) project (also referred to as "3GPP2" ) , etc. ) .
  • IEEE Institute for Electrical and Electronic Engineers
  • Wi-Fi IEEE 802.10 family
  • IEEE 802.16 standards e.g., IEEE 802.16-2005 Amendment
  • LTE Long-Term Evolution
  • LTE Long-Term Evolution
  • UMB ultramobile broadband
  • WiMAX Broadband Wireless Access
  • the communication chip 1112 may operate in accordance with a Global System for Mobile Communication (GSM) , General Packet Radio Service (GPRS) , Universal Mobile Telecommunications System (UMTS) , High Speed Packet Access (HSPA) , Evolved HSPA (E-HSPA) , or LTE network.
  • GSM Global System for Mobile Communication
  • GPRS General Packet Radio Service
  • UMTS Universal Mobile Telecommunications System
  • HSPA High Speed Packet Access
  • E-HSPA Evolved HSPA
  • the communication chip 1112 may manage wired communications, such as electrical, optical, or any other suitable communication protocols (e.g., the Ethernet) .
  • the communication chip 1112 may include multiple communication chips. For instance, a first communication chip 1112 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, and a second communication chip 1112 may be dedicated to longer-range wireless communications such as global positioning system (GPS) , EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others.
  • GPS global positioning system
  • a first communication chip 1112 may be dedicated to wireless communications
  • a second communication chip 1112 may be dedicated to wired communications.
  • the computing device 1100 may include battery/power circuitry 1114.
  • the battery/power circuitry 1114 may include one or more energy storage devices (e.g., batteries or capacitors) and/or circuitry for coupling components of the computing device 1100 to an energy source separate from the computing device 1100 (e.g., AC line power) .
  • the computing device 1100 may include a display device 1106 (or corresponding interface circuitry, as discussed above) .
  • the display device 1106 may include any visual indicators, such as a heads-up display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD) , a light-emitting diode display, or a flat panel display, for example.
  • LCD liquid crystal display
  • the computing device 1100 may include an audio output device 1108 (or corresponding interface circuitry, as discussed above) .
  • the audio output device 1108 may include any device that generates an audible indicator, such as speakers, headsets, or earbuds, for example.
  • the computing device 1100 may include an audio input device 1118 (or corresponding interface circuitry, as discussed above) .
  • the audio input device 1118 may include any device that generates a signal representative of a sound, such as microphones, microphone arrays, or digital instruments (e.g., instruments having a musical instrument digital interface (MIDI) output) .
  • MIDI musical instrument digital interface
  • the computing device 1100 may include a GPS device 1116 (or corresponding interface circuitry, as discussed above) .
  • the GPS device 1116 may be in communication with a satellite-based system and may receive a location of the computing device 1100, as known in the art.
  • the computing device 1100 may include another input device 1120 (or corresponding interface circuitry, as discussed above) .
  • Examples of the other input device 1120 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a bar code reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.
  • the computing device 1100 may have any desired form factor, such as a handheld or mobile computer system (e.g., a cell phone, a smart phone, a mobile internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a PDA, an ultramobile personal computer, etc. ) , a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computer system.
  • the computing device 1100 may be any other electronic device that processes data.
  • Example 1 provides a method, including generating, by a first neural network, a first feature map from a first frame in a video that captures a face; generating, by the first neural network, a second feature map from a second frame in a video, in which the second frame is temporally subsequent to the first frame in the video; generating an aggregated feature map by using the first feature map with the second feature map; determining, by a second neural network, one or more blendshape coefficients by using the aggregated feature map and a trained blendshape representation, the one or more blendshape coefficients representing a predicted facial expression; and generating an animation of the face based on the one or more blendshape coefficients.
  • Example 2 provides the method of example 1, in which generating the aggregated feature map includes generating, by a third neural network, an intermediate feature map from the first feature map and the second feature map; and aggregating the intermediate feature map with the second feature map.
  • Example 3 provides the method of example 2, in which the third neural network includes a transformer decoder.
  • Example 4 provides the method of any one of examples 1-3, in which generating the aggregated feature map includes after generating the first feature map, writing the first feature map to a buffer; and after generating the second feature map, reading the first feature map from the buffer.
  • Example 5 provides the method of example any one of claims 1-4, in which the second neural network includes a transformer decoder that includes a cross attention module, and the cross attention module processes the aggregated feature map as a key or value and processes the trained blendshape representation as a query.
  • Example 6 provides the method of any one of examples 1-5, in which the trained blendshape representation includes a plurality of vectors, and each vector corresponds to a trained blendshape.
  • Example 7 provides the method of any one of examples 1-6, further including generating a segmentation probability map of the second frame, the segmentation probability map indicating segmentation of different portions of the face; generating a reweighting mask by applying a third neural network on the face segmentation probability map; and generating a reweighted feature map by aggregating the second feature map with the reweighting mask; and determining, by a fourth neural network, one or more pose coefficients by using the reweighted feature map.
  • Example 8 provides the method of example 7, in which the third neural network is a convolutional neural network, and the fourth neural network includes a transformer decoder.
  • Example 9 provides the method of example 7 or 8, in which generating the animation of the face includes generating the animation of the face based on the one or more blendshape coefficients and the one or more pose coefficients.
  • Example 10 provides the method of any one of examples 1-9, in which generating the animation of the face includes generating the animation of the face further based on one or more blendshape templates, a blendshape template corresponding to a reference facial expression.
  • Example 11 provides one or more non-transitory computer-readable media storing instructions executable to perform operations, the operations including generating, by a first neural network, a first feature map from a first frame in a video that captures a face; generating, by the first neural network, a second feature map from a second frame in a video, in which the second frame is temporally subsequent to the first frame in the video; generating an aggregated feature map by using the first feature map with the second feature map; determining, by a second neural network, one or more blendshape coefficients by using the aggregated feature map and a trained blendshape representation, the one or more blendshape coefficients representing a predicted facial expression; and generating an animation of the face based on the one or more blendshape coefficients.
  • Example 12 provides the one or more non-transitory computer-readable media of example 11, in which generating the aggregated feature map includes generating, by a third neural network, an intermediate feature map from the first feature map and the second feature map; and aggregating the intermediate feature map with the second feature map.
  • Example 14 provides the one or more non-transitory computer-readable media of any one of examples 11-13, in which generating the aggregated feature map includes after generating the first feature map, writing the first feature map to a buffer; and after generating the second feature map, reading the first feature map from the buffer.
  • Example 15 provides the one or more non-transitory computer-readable media of any one of examples 11-14, in which the second neural network includes a transformer decoder that includes a cross attention module, and the cross attention module processes the aggregated feature map as a key or value and processes the trained blendshape representation as a query.
  • the second neural network includes a transformer decoder that includes a cross attention module
  • the cross attention module processes the aggregated feature map as a key or value and processes the trained blendshape representation as a query.
  • Example 16 provides the one or more non-transitory computer-readable media of any one of examples 11-15, in which the trained blendshape representation includes a plurality of vectors, and each vector corresponds to a trained blendshape.
  • Example 17 provides the one or more non-transitory computer-readable media of any one of examples 11-16, in which the operations further include generating a segmentation probability map of the second frame, the segmentation probability map indicating segmentation of different portions of the face; generating a reweighting mask by applying a third neural network on the face segmentation probability map; and generating a reweighted feature map by aggregating the second feature map with the reweighting mask; and determining, by a fourth neural network, one or more pose coefficients by using the reweighted feature map.
  • Example 18 provides the one or more non-transitory computer-readable media of example 17, in which the third neural network is a convolutional neural network, and the fourth neural network includes a transformer decoder.
  • Example 20 provides the one or more non-transitory computer-readable media of any one of examples 11-19, in which generating the animation of the face includes generating the animation of the face further based on one or more blendshape templates, a blendshape template corresponding to a reference facial expression.
  • Example 22 provides the apparatus of example 21, in which generating the aggregated feature map includes generating, by a third neural network, an intermediate feature map from the first feature map and the second feature map; and aggregating the intermediate feature map with the second feature map.
  • Example 23 provides the apparatus of example 21 or 22, in which generating the aggregated feature map includes after generating the first feature map, writing the first feature map to a buffer; and after generating the second feature map, reading the first feature map from the buffer.
  • Example 24 provides the apparatus of any one of examples 21-23, in which the operations further include generating a segmentation probability map of the second frame, the segmentation probability map indicating segmentation of different portions of the face; generating a reweighting mask by applying a third neural network on the face segmentation probability map; and generating a reweighted feature map by aggregating the second feature map with the reweighting mask; and determining, by a fourth neural network, one or more pose coefficients by using the reweighted feature map.
  • Example 25 provides the apparatus of example 24, in which the third neural network is a convolutional neural network, and the fourth neural network includes a transformer decoder.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)

Abstract

L'invention concerne un système de vision artificielle qui peut extraire des cartes de caractéristiques de diverses trames dans une vidéo, par exemple, une vidéo monoculaire. Les trames peuvent comprendre une trame actuelle et une ou plusieurs trames historiques. Le système de vision artificielle peut agréger une carte de caractéristiques extraite de la trame actuelle (« carte de caractéristiques actuelle ») avec une carte de caractéristiques extraite de chaque trame historique (« carte de caractéristiques historiques »). Le système de vision artificielle peut utiliser un décodeur pour déterminer des coefficients de Blendshape à partir de la carte de caractéristiques agrégée. Les coefficients de Blendshape peuvent représenter l'expression faciale prédite. De plus, le système de vision artificielle peut utiliser un autre décodeur pour déterminer des coefficients de posture à partir de la carte de caractéristiques agrégée, par exemple, après avoir imposé un module antérieur rigide basé sur la sémantique à la carte de caractéristiques agrégée. Les coefficients de posture peuvent représenter la posture de tête prédite. Le système de vision artificielle peut générer une animation faciale à partir des coefficients Blendshape et des coefficients de posture.
PCT/CN2024/090761 2024-03-26 2024-04-30 Suivi de visage basé sur une agrégation spatio-temporelle et antérieur rigide Pending WO2025200078A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN2024083753 2024-03-26
CNPCT/CN2024/083753 2024-03-26

Publications (1)

Publication Number Publication Date
WO2025200078A1 true WO2025200078A1 (fr) 2025-10-02

Family

ID=97216295

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/090761 Pending WO2025200078A1 (fr) 2024-03-26 2024-04-30 Suivi de visage basé sur une agrégation spatio-temporelle et antérieur rigide

Country Status (1)

Country Link
WO (1) WO2025200078A1 (fr)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160275341A1 (en) * 2015-03-18 2016-09-22 Adobe Systems Incorporated Facial Expression Capture for Character Animation
CN112950640A (zh) * 2021-02-23 2021-06-11 Oppo广东移动通信有限公司 视频人像分割方法、装置、电子设备及存储介质
CN114092519A (zh) * 2021-11-23 2022-02-25 江西理工大学 一种使用卷积神经网络和双向匹配算法的视频多目标跟踪方法
CN114170558A (zh) * 2021-12-14 2022-03-11 北京有竹居网络技术有限公司 用于视频处理的方法、系统、设备、介质和产品
US20220237945A1 (en) * 2019-11-07 2022-07-28 Hyperconnect Inc. Method and Apparatus for Generating Reenacted Image

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160275341A1 (en) * 2015-03-18 2016-09-22 Adobe Systems Incorporated Facial Expression Capture for Character Animation
US20220237945A1 (en) * 2019-11-07 2022-07-28 Hyperconnect Inc. Method and Apparatus for Generating Reenacted Image
CN112950640A (zh) * 2021-02-23 2021-06-11 Oppo广东移动通信有限公司 视频人像分割方法、装置、电子设备及存储介质
CN114092519A (zh) * 2021-11-23 2022-02-25 江西理工大学 一种使用卷积神经网络和双向匹配算法的视频多目标跟踪方法
CN114170558A (zh) * 2021-12-14 2022-03-11 北京有竹居网络技术有限公司 用于视频处理的方法、系统、设备、介质和产品

Similar Documents

Publication Publication Date Title
US11704817B2 (en) Method, apparatus, terminal, and storage medium for training model
US12554988B2 (en) System and method for compressing convolutional neural networks
US12288389B2 (en) Learning orthogonal factorization in GAN latent space
US20230016455A1 (en) Decomposing a deconvolution into multiple convolutions
WO2023220859A1 (fr) Attention multidimensionnelle pour noyau convolutif dynamique
US20250232451A1 (en) Detecting moving objects
Sharjeel et al. Real time drone detection by moving camera using COROLA and CNN algorithm
US11790228B2 (en) Methods and systems for performing tasks on media using attribute specific joint learning
US20230298322A1 (en) Out-of-distribution detection using a neural network
US20230059976A1 (en) Deep neural network (dnn) accelerator facilitating quantized inference
EP4607477A1 (fr) Détermination de la position du regard sur de multiples écrans à l'aide d'une caméra monoculaire
WO2025200078A1 (fr) Suivi de visage basé sur une agrégation spatio-temporelle et antérieur rigide
US20240144447A1 (en) Saliency maps and concept formation intensity for diffusion models
WO2025123208A1 (fr) Réseau d'annotation pour estimation de pose tridimensionnelle
WO2025200079A1 (fr) Codeur assimilable convertissant un nuage de points en grille pour reconnaissance visuelle
WO2024072472A1 (fr) Génération de carte d'activation de classe efficace sans gradient
WO2025118238A1 (fr) Suivi de mouvement avec réseau neuronal à tâches multiples
WO2024077463A1 (fr) Modélisation séquentielle avec une mémoire contenant des réseaux à plages multiples
WO2025097349A1 (fr) Vision artificielle basée sur un graphe à l'aide d'un apprenant à grille progressive et d'un réseau neuronal convolutif
WO2025199844A1 (fr) Suivi de mouvement avec réseaux neuronaux et images de profondeur
US20240249392A1 (en) Minimal image signal processing pipeline for an early scene understanding
US20250363591A1 (en) High resolution patch management system in an early-stage image
WO2025102256A1 (fr) Entraînement de réseau neuronal avec distillation de connaissances contrastives
EP4738240A1 (fr) Super-échantillonnage efficace dans des vidéos à l'aide de caractéristiques intermédiaires historiques
CN118230426B (zh) 人体姿态识别方法、设备和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24932696

Country of ref document: EP

Kind code of ref document: A1