CN104123749A - Picture processing method and system - Google Patents

Abstract

The invention provides a picture processing method and system. The picture processing method and system includes that a model of a first person is simulated according to at least one picture which contains the first person; a target picture which contains a second person is determined; displaying of feature information of the second person is determined in the target picture; the displaying of the first person is adjusted according to the feature information in the model of the first person; the second person is replaced by the first person after displaying adjustment in the target picture. According to the technical scheme, the problem that replaced persons and backgrounds are inconsistent and conflicting due to absence of the relation between the persons before or after picture processing replacement is solved.

Description

Image processing method and system

Technical Field

The present invention relates to image processing technologies, and in particular, to an image processing method and system.

Background

When image processing is performed to replace one person in an image with another image, the head or the face of one person is simply cut along the outline and is superimposed on the corresponding position of the image of another person, which is similar to the effect of a sticker. On one hand, the replaced character is inconsistent with the background due to illumination, visual angle and the like, so that the situation that the color of the character is conflicted with the background and the color tone is not matched occurs; on the other hand, when the face of a person is replaced with the face of another person, only the expression of the original person can be kept, and the expression is usually inconsistent with the background, so that the incompatibility of the person and the background after the replacement according to the prior art obviously cannot meet the requirements of people.

The defects of the prior art are as follows:

the characters before and after replacement are not connected, so that the problems of incompatibility, conflict and the like of the replaced characters and the background occur.

Disclosure of Invention

The invention provides an image processing method and an image processing system aiming at the problems, which are used for solving the problem that the character image does not accord with the replaced image background when the image is replaced by simulation.

The embodiment of the invention provides an image processing method, which comprises the following steps:

simulating a model of a first person based on at least one image containing the first person;

determining a target image containing a second person;

determining feature information of a second person displayed in the target image;

adjusting the display of the first persona in the model of the first persona according to the characteristic information;

and replacing the second person with the first person after the display adjustment in the target image.

An embodiment of the present invention provides an image processing system, which may include:

the model simulation module is used for simulating a model of a first person according to at least one image containing the first person;

a target image determination module for determining a target image containing a second person;

the characteristic information determining module is used for determining characteristic information of a second person displayed in the target image;

an adjustment display module for adjusting the display of the first persona in the model of the first persona according to the characteristic information;

and the person replacing module is used for replacing the second person with the first person after the display adjustment in the target image.

The invention has the following beneficial effects:

in the technical scheme provided by the embodiment of the invention, the model of the first person is simulated firstly, and then the adjustment is carried out according to the characteristic information displayed in the target image by the second person as the replaced object, so that the first person and the second person have the same display characteristics in the target image, and the problem of incompatibility with the background of the target image and the like after replacement is solved.

Drawings

Specific embodiments of the present invention will now be described with reference to the accompanying drawings, in which:

FIG. 1 is a schematic flow chart illustrating an embodiment of an image processing method according to the present invention;

FIG. 2 is a schematic diagram of a face detection algorithm implementation flow in the embodiment of the present invention;

FIG. 3 is a schematic diagram of extracting Haar-like features according to an embodiment of the present invention;

FIG. 4 is a flow chart illustrating a method for integrating an integral graph according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of a cascade detector of the waterfall type according to an embodiment of the present invention;

FIG. 6 is a schematic diagram of a face calibrated according to an embodiment of the present invention;

FIG. 7 is a diagram illustrating a process of creating local features in an embodiment of the invention;

FIG. 8 is a flowchart illustrating an implementation of a method for calculating a new position of each feature point according to an embodiment of the present invention;

FIG. 9 is a schematic diagram of a face detection result according to an embodiment of the present invention;

FIG. 10 is a schematic diagram of a three-dimensional face reconstruction implementation process in the embodiment of the present invention;

FIG. 11 is a schematic diagram of an original image and a three-dimensional model in an embodiment of the invention;

FIG. 12 is a diagram illustrating an example of an expression of a model according to an embodiment of the present invention;

FIG. 13 is a schematic diagram of human expression feature points according to an embodiment of the present invention;

FIG. 14 is a diagram illustrating an exemplary image processing system according to the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.

Fig. 1 is a schematic flow chart of an implementation of an image processing method, as shown in fig. 1, the method may include the following steps:

step 101: simulating a model of a first person based on at least one image containing the first person;

step 102: determining a target image containing a second person;

step 103: determining feature information of a second person displayed in the target image;

step 104: adjusting the display of the first persona in the model of the first persona according to the characteristic information;

step 105: and replacing the second person with the first person after the display adjustment in the target image.

Specifically, when the image is replaced, the person can be automatically edited according to the picture or image sequence provided by the user. For example, the following may be used:

a. the user provides one or more pictures or image sequences as materials, and all the materials contain the same person, namely: a first character;

b. the system simulates a model of the first person based on material provided by the user. The model can be adjusted correspondingly according to different visual angles, illumination and the like, and can be deformed differently;

c. the user specifies another person in the picture or image sequence, namely: a second character;

d. the system detects the related feature information of the designated second person in each frame of image. The characteristic information refers to the characteristics of position, outline, relative visual angle, illumination, deformation and the like;

e. on each frame of image, the first person model is adjusted to the characteristics of the second person on the frame of image and replaces the second person.

In the implementation, the extraction, processing, adjustment, and conversion of a single image are described, and each frame of a plurality of image sequences and video images is composed of a single image, so based on the technical solution provided by the embodiment of the present invention, an image sequence composed of a plurality of or batch images or a processing of a video image can be easily obtained, for example, a simplest way is: and after each image of the image sequence or the video is subjected to replacement processing, the image sequence or the video after replacement is formed. It will be readily understood by those skilled in the art and modified accordingly, how to extend the processing to the entire sequence of images or video, based on the processing of a single image.

In practice, the characters in the embodiment of the present invention may be anthropomorphic characters, such as cartoon characters, 3D characters, etc., which are not limited to human characters, but are not necessarily naturally occurring characters, and are all referred to as "characters" in the embodiment. In the following embodiments, human image processing is also used as an example, because it is most representative and most complex. Therefore, the description is made by taking the portrait as an example; however, the technical solutions provided by the embodiments of the present invention may also be used for other image processing, because it discloses a solution related to replacement for image processing, that is, as long as all the objects in the field of image processing are replaced, the solution in the embodiments of the present invention may be adopted, theoretically, the solution is not limited to people, as long as the purpose of pattern replacement is achieved, and the figures are only used for teaching those skilled in the art how to implement the present invention specifically, but are not meant to be used only for figures, and the implementation process may be used in corresponding environments in combination with practical needs.

In an implementation, when simulating a model of a face of a first person based on at least one image including the first person, the simulating may include:

detecting a region of a face of a first person;

determining the areas of five sense organs and the contour of a cheek in the detected area of the face;

and fitting the detected five sense organ regions and the contour of the cheek to the existing three-dimensional human face 3D model to obtain the simulated model of the first person face.

Specifically, according to the simulated model of the first person, the whole person may be modeled, or the face of the person may be modeled, and the face of the person is used as an example in the embodiment, but it should be understood by those skilled in the art that a processing manner not limited to the face, such as a whole person model, may be obtained by performing processing with a corresponding image tool.

Taking the embodiment of the face of the person as an example, the following can be taken:

a. detecting the position and the area of the face of a first person from a picture or an image sequence provided by a user;

b. in the detected human face region, the facial five sense organs region and cheek contour are determined, such as: eyes, nose, eyebrows, mouth, and ears;

c. and (3) fitting the detected regions of five sense organs and the contour of the cheek to an existing human face 3-dimensional model, so that the human face 3-dimensional model can automatically present different visual angles, illumination and expression changes according to the setting of parameters.

In implementation, when determining that the feature information of the face of the second person is displayed in the target image, the method may include:

detecting a region of a face of a second person;

determining the areas of five sense organs and the contour of a cheek in the detected area of the face;

feature information of the face of the second person displayed in the target image is determined based on the detected regions of the five sense organs and the contour of the cheek.

Specifically, the information about the characteristics of the second person in each frame of image may be detected as follows:

a. detecting the position and the area of the face of a second person from the picture or the image sequence;

b. in the detected human face region, the facial five sense organs region and cheek contour are determined, such as: eyes, nose, eyebrows, mouth, and ears;

c. and deducing related characteristic information of the second person through the detected areas of the five sense organs and the contour of the cheek. The characteristic information includes a change in viewing angle, illumination, and expression, and the like.

Specifically, in the detected region of the face, the region of the five sense organs and the contour of the cheek are determined by using a face recognition algorithm, and the region of the five sense organs and the contour of the cheek may be determined by using an ASM (Active Shape Model) algorithm.

In the implementation, the ASM algorithm is used for explanation because the ASM algorithm is typical and common in the face recognition algorithm, and is easily understood and implemented by those skilled in the art, so the ASM algorithm is taken as an example here; however, theoretically, other algorithms are also possible as long as the purpose of determining the facial contour and the area of the five sense organs can be achieved, and for example, an algorithm such as AAM (active application Model) or SDM (supervisory gradient Descent Method) may be used. Therefore, the ASM algorithm is only used for teaching one skilled in the art how to implement the present invention specifically, but it is not meant to be limited to the ASM algorithm, and the corresponding algorithm can be determined according to practical needs during implementation.

In this embodiment, the second person is replaced with the first person who has been adjusted, and the face of the second person is replaced with the face of the first person who has been adjusted based on the region of the face of the first person and the region of the face of the second person.

Specifically, the first character model may further replace the second character as follows:

a. according to the related characteristic information of the second person, adjusting the model of the first person to enable the model to be similar to the related characteristic of the second person;

b. wiping off the face area of the second person in each frame of image through the detected five sense organs and cheek outlines;

c. in each frame of image, the adjusted model of the first person is placed in the face area of the second person.

In practice, in the detected face region, a face recognition algorithm may be used to determine the region of the five sense organs and the contour of the cheek.

In an implementation, after replacing the second person with the display-adjusted first person in the target image, the method may further include:

an image is added for a first person in the target image.

This is convenient for adding images of props and the like to the original person after replacement, and the props comprise glasses, hats, clothes, backpacks, shoes and the like.

In practice, there are many ways to detect the position and region of the face of the first person or the second person from the picture or image sequence provided by the user, as shown in fig. 3.

Among the listed methods, the statistical model-based method is a currently popular method, and specifically, see: the proposal has great superiority in the review of human face detection research (in the report of computer science Vol25No5May2002) by Lianglu hong et al. Its advantages are:

1. the method does not depend on the prior knowledge and the parameter model of the human face, and can avoid errors caused by inaccurate or incomplete knowledge;

2. the parameters of the model are obtained by adopting an example learning method, so that the statistical significance is more reliable;

3. the detection mode range can be expanded by increasing the learning examples, and the robustness is improved.

Method for statistical modeling

The face detection algorithm based on ensemble machine learning proposed by Viola and Jones in around 2001 has obvious advantages over other methods, which can be seen in particular in: face detection and retrieval, recorded in science foundation project 60273005, by ai hai boat et al; wubo et al, Multi-View face detection based on the continuous Adaboost Algorithm (in computer research and development, 2005). Recent literature also indicates that no other face detection methods have been found to date that are superior to the Viola and Jones methods, see in particular: n Degtyarev et al, comprehensive Testing of Face Detection Algorithms (Image and Signal processing, 2010). The method has high detection precision, and the most important is that the operation speed is greatly higher than that of other methods.

Several key steps in the Viola and Jones face detection methods can be found in detail in: paul Viola and Michael joints, "Rapid object detection using a bossed cassette of simplefeatures (loaded in Accepted Conference on Computer Vision and Pattern Recognition 2001):

1. extracting Haar-like features (Haar-like features)

A Haar-like type feature is a simple rectangular feature proposed by Viola et al, named for its Haar-like wavelet. The definition of a Haar-type feature is the difference between the sum of the weighted gray levels of the corresponding regions of the black and white rectangles in the image sub-window. As shown in fig. 4, the two simplest feature operators are shown. As can be seen in fig. 4, at a face-specific structure, the operator calculation yields a larger value.

2. Calculating an integrogram

When the number of operators is large, the calculation amount is too large, and Viola and the like invent an integral graph method, so that the calculation speed is greatly accelerated. As shown in fig. 5, the value at point 1 is the pixel integration of the a region, and the value at point 2 is the pixel integration of the AB region. The integral value of the pixels of any region D can be conveniently calculated to be 4+1-2-3 by carrying out integral operation on the whole picture once.

3. Training Adaboost model

In the discrete Adaboost algorithm, a certain threshold is subtracted from a calculation result of a Haar-like feature operator, and the result can be regarded as a face detector. Because of its low accuracy, it is called a weak classifier. In the cycle of the Adaboost algorithm, various weak classifiers are firstly used for classifying a training picture library, the weak classifier with the highest accuracy is reserved, meanwhile, the weight of the picture with the wrong judgment is increased, and the next cycle is entered. Finally, the weak classifiers reserved in each cycle are combined to form an accurate face detector, namely a strong classifier. The specific calculation process can be seen in the following concrete: wubo et al, "continuous adaboost algorithm based multi-view face detection" (in computer research and development, 2005); paul Viola and Michael joints, "Rapid object detection using a bossed cassette of simple defects" (described in Accepted Conference on Computer Vision and Pattern Recognition 2001).

4. Cascade detector of establishing waterfall type

The cascade detector is a detection structure provided for the problem of human face detection speed. As shown in fig. 6, each layer of the waterfall is a strong classifier trained by the adaboost algorithm. The threshold value of each layer is set so that most face images can pass through, and counterexamples are discarded as much as possible on the basis. The more the layers at the back of the position are more complex, and the stronger the classification capability is.

Such a detector configuration would have a series of screens with decreasing mesh size, each step being able to screen out some of the adverse instances of previous screen drops, and eventually the sample passing through all screens would be accepted as a human face. The waterfall type detector training algorithm can be specifically seen in: wubo et al, Multi-View face detection based on the continuous Adaboost Algorithm (in computer research and development 2005).

In the implementation of the above algorithm, an OpenCV (Open Source Computer Vision Library) face detection program flow is adopted, and specific program Source codes can be described in the following websites:

http：//www.opencv.org.cn/index.php/％E4％BA％BA％E8％84％B8％E6％A3％80％E6％B5％8B。

OpenCV is a (open source) based cross-platform computer vision library that can run on Linux, Windows, and Mac OS operating systems. The method is light and efficient, is composed of a series of C functions and a small number of C + + classes, provides interfaces of languages such as Python, Ruby, MATLAB and the like, and realizes a plurality of general algorithms in the aspects of image processing and computer vision.

The face detection program of OpenCV adopts a Viola and Jones face detection method, and mainly calls a trained cascade classifier cascade to perform pattern matching.

cvhaar detecteobjects grays the image, judges whether canny edge processing is performed according to the input parameters (default is not used), and then performs matching. And after matching, collecting the found matching blocks, filtering noise, and taking the output result if the number of the adjacent blocks exceeds a specified value (the incoming min _ neighbors), or deleting the output result.

Matching and circulating: and (3) amplifying the scale (incoming value) times of the matching classifier, simultaneously reducing the scale times of the original image, matching, and returning a matching result until the size of the matching classifier is larger than that of the original image. When matching, call cvRunHaarClassifierCascade to match, store all results in CvSeq _ Seq (dynamic growing element sequence), and transfer the results to cvHaarDetectObjects.

The cvRunHaarClassifierCascade function as a whole is matched based on the incoming image and the cascade. And different matching modes can be performed according to different types of the incoming cascade (tree type, stub, or other).

The function cvrunhaarclasssifierccascade is used for the detection of a single picture. The integrogram and the appropriate scaling factor (═ window size) were first set using cvsetimageforhaarclasssifierccascade before the function call. Positive values (which is a candidate target) are returned when the analyzed rectangular boxes all pass through each layer of the cascade classifier, otherwise 0 or negative values are returned.

The training of the classifier adopts a Haar classifier, and the training of the Haar classifier is independent of the human face detection process. The training of the classifier is divided into two stages:

A. creating a sample, and completing the sample by using the creatsampies.exe carried by OpenCV;

B. training a classifier, generating an xml file, and finishing by haartraining.exe carried by OpenCV.

The training process can be seen in detail in the following 1 and 2:

1、http：//034080116.blog.163.com/blog/static/334061912009641073715/；

2、\OpenCV\apps\HaarTraining\doc\haartraining.doc；

of the above addresses, address 1 can be seen in a blog, and the haar trained source file provided by address 2 can be found in an openCVS installation package directory after downloading and installation.

Meanwhile, the adaboost of the training algorithm adopted in the OpenCV is the generic adaboost, which is the most suitable scheme for face detection. See in particular:

1、http：//www.opencv.org.cn/forum/viewtopic.phpf＝1&t＝4264#p15258

2、http：//www.opencv.org.cn/forum/viewtopic.phpt＝3880

for example, within the detected face region, facial features, positional relationship and cheek contour information are determined, such as: eyes, nose, eyebrows, mouth, ears, etc., can be implemented by having many algorithms. The present invention preferably uses the ASM algorithm, which will be described below.

ASM is an algorithm based on a Distribution Model (PDM), in which the geometric shapes of objects to be similar in shape, such as human faces, human hands, hearts, lungs, etc., can be represented by sequentially connecting coordinates of several key feature points (landworks) in series to form a shape vector. This patent uses human face as an example to describe the basic principle and method of the algorithm. First, a picture of a face with 68 key feature points being scaled is given, as shown in fig. 6. In the practical application process, the ASM includes two parts of training and searching.

Training of ASM

ASM training consists of two parts.

1. Establishing a shape model: the part consists of the following steps

1.1 collecting n training samples

If ASM training needs to be carried out on the face key area of the face, n sample pictures containing the face area of the face need to be involved. It should be reminded that the collected picture only contains the face region, and the problem of normalization of image size and the like is not considered here.

1.2 manually record k key feature points in each training sample

As shown in fig. 7, for any one picture in the training set, it is necessary to record position coordinate information of several key feature points (68 in fig. 7) and store the coordinate information in a text file. This step can be done by a programmer in general. And the program loads a training sample each time, the user sequentially clicks key characteristic points in the picture, and the program automatically records the position coordinate of the current mouse click once clicking and stores the position coordinate for later use.

1.3 constructing the shape vector of the training set

And forming a shape vector by k key feature points calibrated in a graph.

<math> <mrow> <msub> <mi>a</mi> <mi>i</mi> </msub> <mo>=</mo> <mrow> <mo>(</mo> <msubsup> <mi>x</mi> <mn>1</mn> <mi>i</mi> </msubsup> <mo>,</mo> <msubsup> <mi>y</mi> <mn>1</mn> <mi>i</mi> </msubsup> <mo>,</mo> <msubsup> <mi>x</mi> <mn>2</mn> <mi>i</mi> </msubsup> <mo>,</mo> <msubsup> <mi>y</mi> <mn>2</mn> <mi>i</mi> </msubsup> <mo>,</mo> <mo>&CenterDot;</mo> <mo>&CenterDot;</mo> <mo>&CenterDot;</mo> <mo>&CenterDot;</mo> <mo>,</mo> <msubsup> <mi>x</mi> <mi>k</mi> <mi>i</mi> </msubsup> <mo>,</mo> <msubsup> <mi>y</mi> <mi>k</mi> <mi>i</mi> </msubsup> <mo>)</mo> </mrow> <mo>,</mo> <mi>i</mi> <mo>=</mo> <mn>1,2</mn> <mo>,</mo> <mo>.</mo> <mo>.</mo> <mo>.</mo> <mo>,</mo> <mi>n</mi> </mrow> </math> Formula (1)

Wherein,and the coordinates of the jth characteristic point on the ith training sample are shown, and n is the number of the training samples. Thus, n training samples constitute n shape vectors.

1.4 shape normalization

The step aims to perform normalization or alignment operation on the human face shape which is manually calibrated in the front, so that non-shape interference of the human face in the picture caused by external factors such as different angles, distance, posture change and the like is eliminated, and the point distribution model is more effective. Generally, this step is normalized using the Procrustes method. In short, the method aligns a series of point distribution models to the same point distribution model through proper translation, rotation and scaling transformation on the basis of not changing the point distribution models, thereby changing the disordered state of the acquired original data and reducing the interference of non-shape factors. Using Procrustes method to get pi ═ alpha₁，α₂，...，α_nThe alignment procedure for this training set needs to be performed for each α in the training set_iThere are 4 parameters calculated: rotation angle of theta_iScaling by a scale s_iAmount of translation in the horizontal directionTranslation in vertical directionLet M(s)_i，θ_i)[α_i]Is expressed as a pair_iMake a rotation angle theta_iScaling scale of s_iAnd (4) transforming. Alpha is alpha_iTo alpha_kThe alignment process is to find theta_i，s_i，So thatA process of minimization. Wherein <math> <mrow> <msub> <mi>Z</mi> <mi>i</mi> </msub> <mo>=</mo> <msub> <mi>&alpha;</mi> <mi>i</mi> </msub> <mo>-</mo> <mi>M</mi> <mrow> <mo>(</mo> <msub> <mi>s</mi> <mi>i</mi> </msub> <mo>,</mo> <msub> <mi>&theta;</mi> <mi>i</mi> </msub> <mo>)</mo> </mrow> <mo>[</mo> <msub> <mi>&alpha;</mi> <mi>i</mi> </msub> <mo>]</mo> <mo>-</mo> <mo>[</mo> <msub> <mi>X</mi> <msub> <mi>x</mi> <mi>i</mi> </msub> </msub> <mo>,</mo> <msub> <msub> <mi>X</mi> <mi>y</mi> </msub> <mi>i</mi> </msub> <mo>,</mo> <mo>.</mo> <mo>.</mo> <mo>.</mo> <mo>,</mo> <msub> <mi>X</mi> <msub> <mi>x</mi> <mi>i</mi> </msub> </msub> <mo>,</mo> <msub> <mi>X</mi> <msub> <mi>y</mi> <mi>i</mi> </msub> </msub> <mo>]</mo> <mo>.</mo> </mrow> </math> Where W is a diagonal matrix, which can be obtained by the following calculation: let R_kiRepresenting the distance between the kth point and the 1 st point in one image, andrepresenting R between different images in the entire training set_kiBy calculating the variance ofThereby obtaining:it is easy to find that the Procrustes method is only a method for solving the transformation matrix. In the ASM, the Procrustes is utilized to perform the alignment operation of the point distribution model, and the specific steps are as follows:

(1) aligning all face models in the training set to the 1 st face model;

(2) computing an average face model

(3) Aligning all face models to an average face model

(4) Repeating (2) and (3) until convergence.

1.5 PCA processing is carried out on the aligned shape vectors

(1) Calculate the average shape vector:

<math> <mrow> <mover> <mi>&alpha;</mi> <mo>&OverBar;</mo> </mover> <mo>=</mo> <mfrac> <mn>1</mn> <mi>n</mi> </mfrac> <munderover> <mi>&Sigma;</mi> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>n</mi> </munderover> <msub> <mi>&alpha;</mi> <mi>i</mi> </msub> </mrow> </math> formula (2)

(2) Calculating a covariance matrix:

<math> <mrow> <mi>S</mi> <mo>=</mo> <mfrac> <mn>1</mn> <mi>n</mi> </mfrac> <munderover> <mi>&Sigma;</mi> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>n</mi> </munderover> <msup> <mrow> <mo>(</mo> <msub> <mi>&alpha;</mi> <mi>i</mi> </msub> <mo>-</mo> <mover> <mi>&alpha;</mi> <mo>&OverBar;</mo> </mover> <mo>)</mo> </mrow> <mi>T</mi> </msup> <mo>&CenterDot;</mo> <msub> <mi>&alpha;</mi> <mi>i</mi> </msub> <mo>-</mo> <mover> <mi>&alpha;</mi> <mo>&OverBar;</mo> </mover> </mrow> </math> formula (3)

(3) Calculating the eigenvalues of the covariance matrix S and sequencing the eigenvalues in sequence from big to small:

thus, λ is obtained₁，λ₂，...，λ_qWherein λ is₁Is greater than 0. The first t eigenvectors P ═ P (P) are selected₁，p₂，...，p_t) So that the characteristic values corresponding thereto satisfy:

<math> <mrow> <mfrac> <mrow> <msubsup> <mi>&Sigma;</mi> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>t</mi> </msubsup> <msub> <mi>&lambda;</mi> <mi>i</mi> </msub> </mrow> <mrow> <msubsup> <mi>&Sigma;</mi> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>q</mi> </msubsup> <msub> <mi>&lambda;</mi> <mi>s</mi> </msub> </mrow> </mfrac> <mo>></mo> <msub> <mi>f</mi> <mi>v</mi> </msub> <msub> <mi>V</mi> <mi>T</mi> </msub> </mrow> </math> formula (4)

Where f is_vIs a proportionality coefficient determined by the number of eigenvectors, usually 95%, and V_TIs the sum of all the features. Namely:

V_T＝∑λ_i

any such shape vector used for training can be represented as:

<math> <mrow> <msub> <mi>&alpha;</mi> <mi>i</mi> </msub> <mo>&ap;</mo> <mover> <mi>&alpha;</mi> <mo>&OverBar;</mo> </mover> <mo>+</mo> <msub> <mi>P</mi> <mi>s</mi> </msub> <msub> <mi>b</mi> <mi>s</mi> </msub> </mrow> </math> formula (5)

In the above formula, b_sIs a vector containing t parameters, where,

<math> <mrow> <msub> <mi>b</mi> <mi>s</mi> </msub> <mrow> <mo>(</mo> <mi>i</mi> <mo>)</mo> </mrow> <mo>=</mo> <msup> <mi>P</mi> <mi>T</mi> </msup> <mo>&CenterDot;</mo> <mrow> <mo>(</mo> <msub> <mi>&alpha;</mi> <mi>i</mi> </msub> <mo>-</mo> <mover> <mi>&alpha;</mi> <mo>&OverBar;</mo> </mover> <mo>)</mo> </mrow> </mrow> </math>

in addition, to ensure the balance of_sThe shape generated by the change of (b) is similar to the shape in the training set, and the pair b is required_sSubject to limitations that

<math> <mrow> <msup> <msub> <mi>D</mi> <mi>m</mi> </msub> <mn>2</mn> </msup> <mo>=</mo> <msubsup> <mi>&Sigma;</mi> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>t</mi> </msubsup> <mrow> <mo>(</mo> <mfrac> <msup> <mrow> <msub> <mi>b</mi> <mi>s</mi> </msub> <mrow> <mo>(</mo> <mi>i</mi> <mo>)</mo> </mrow> </mrow> <mn>2</mn> </msup> <msub> <mi>&lambda;</mi> <mi>i</mi> </msub> </mfrac> <mo>)</mo> </mrow> <mo>&le;</mo> <msup> <msub> <mi>D</mi> <mi>max</mi> </msub> <mn>2</mn> </msup> </mrow> </math>

Wherein D_maxUsually 3 if b is in the update process D_m＞D_maxThen use

<math> <mrow> <msub> <mi>b</mi> <mi>s</mi> </msub> <mrow> <mo>(</mo> <mi>k</mi> <mo>)</mo> </mrow> <mo>=</mo> <msub> <mi>b</mi> <mi>s</mi> </msub> <mrow> <mo>(</mo> <mi>k</mi> <mo>)</mo> </mrow> <mo>&times;</mo> <mrow> <mo>(</mo> <mfrac> <msub> <mi>D</mi> <mi>max</mi> </msub> <msub> <mi>D</mi> <mi>m</mi> </msub> </mfrac> <mo>)</mo> </mrow> </mrow> </math>

To b is_sAnd (4) applying constraint.

2. Constructing local features for each feature point

In order to find a new position for each feature point in each iteration, local features need to be established for each feature point. For the ith feature point, the process of creating the local features is shown in fig. 7, m pixels are respectively selected along a direction perpendicular to a connecting line of two feature points before and after the ith feature point on two sides of the ith feature point on the ith training image to form a vector with the length of 2m +1, and the gray value of the pixel contained in the vector is differentiated to obtain a local texture g_ijTo other training setsThe ith feature point on the training sample image is operated in the same way, so that n local textures g of the ith feature point can be obtained_i1，g_i2，...，g_in. Then, the mean of them is found:

<math> <mrow> <mover> <msub> <mi>g</mi> <mi>i</mi> </msub> <mo>&OverBar;</mo> </mover> <mo>=</mo> <mfrac> <mn>1</mn> <mi>n</mi> </mfrac> <munderover> <mi>&Sigma;</mi> <mrow> <mi>j</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>n</mi> </munderover> <msub> <mi>g</mi> <mi>ij</mi> </msub> </mrow> </math> formula (6)

And variance:

<math> <mrow> <msub> <mi>S</mi> <mi>i</mi> </msub> <mo>=</mo> <mfrac> <mn>1</mn> <mi>n</mi> </mfrac> <munderover> <mi>&Sigma;</mi> <mrow> <mi>j</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>n</mi> </munderover> <msup> <mrow> <mo>(</mo> <msub> <mi>g</mi> <mi>ij</mi> </msub> <mo>-</mo> <mover> <msub> <mi>g</mi> <mi>i</mi> </msub> <mo>&OverBar;</mo> </mover> <mo>)</mo> </mrow> <mi>T</mi> </msup> <mrow> <mo>(</mo> <msub> <mi>g</mi> <mi>ij</mi> </msub> <mo>-</mo> <mover> <msub> <mi>g</mi> <mi>i</mi> </msub> <mo>&OverBar;</mo> </mover> <mo>)</mo> </mrow> </mrow> </math> formula (7)

Thus, the local feature of the ith feature point is obtained. The local feature of each feature point can be obtained by performing the same operation on all other feature points. Thus, the similarity measure between the new feature g of a feature point and its trained local feature can be represented by mahalanobis distance:

<math> <mrow> <msub> <mi>f</mi> <mi>sim</mi> </msub> <mo>=</mo> <mrow> <mo>(</mo> <mi>g</mi> <mo>-</mo> <mover> <msub> <mi>g</mi> <mi>i</mi> </msub> <mo>&OverBar;</mo> </mover> <mo>)</mo> </mrow> <msubsup> <mi>S</mi> <mi>i</mi> <mrow> <mo>-</mo> <mn>1</mn> </mrow> </msubsup> <msup> <mrow> <mo>(</mo> <mi>g</mi> <mo>-</mo> <mover> <msub> <mi>g</mi> <mi>i</mi> </msub> <mo>&OverBar;</mo> </mover> <mo>)</mo> </mrow> <mi>T</mi> </msup> </mrow> </math> formula (8)

Two, ASM search

After an ASM model is obtained by training a sample set and is established, ASM search can be carried out, firstly, affine transformation is carried out on an average shape to obtain an initial model:

X＝M(s，θ)[α_i]+X_cformula (9)

The above equation shows scaling the average shape by θ with its center rotated counterclockwise and then translated by X_cAn initial model X is obtained.

And searching the target shape in the new image by using the initial model, wherein the characteristic points in the searched final shape are closest to the corresponding real characteristic points, and the searching process is mainly realized by affine transformation and the change of the parameter b. The specific algorithm can be realized by repeating the following two steps:

2.1 calculating the New position of each feature Point

The initial ASM model is first overlaid on the image, as shown in figure 8,

for the ith characteristic point in the model, 1(1 > m) pixel is respectively selected on two sides by taking the ith characteristic point as the center in the direction vertical to the connecting line of the front characteristic point and the rear characteristic point, then the gray value derivative of the 1 pixel is calculated and normalized to obtain a local characteristic which comprises 2(1-m) +1 sub local characteristics, then the Mahalanobis distance between the sub local characteristics and the local characteristic of the current characteristic point is calculated by using the previous formula, so that the center of the sub local characteristic with the minimum Mahalanobis distance is the new position of the current characteristic point, and thus, a displacement is generated. Find their new positions for all feature points and make their displacements into a vector:

dX＝(dX₁，dX₂，...dX_k)

2.2 updating of parameters and b in affine changes

And (3) the position X of the current feature point is closest to the corresponding new position X + dX by affine transformation and adjusting the parameters of the affine transformation.

After affine transformation, the variable d of the affine transformation parameter can be obtained_s∶d_θ∶And is obtained by the formula (9):

M(s(1+ds)，(θ+dθ))[α_i+dα_i]+(X_c+dX_c) Formula (10)

And X can be represented by (9), so the above formula can be represented as follows:

M(s(1+ds)，(θ+dθ))[α_i+da_i]＝M(S，θ)[α_i]+dX+Xc-(Xc＝dX_C) Formula (11)

Also obtained from formula (9):

M^-1(s，θ)＝M(s^-1theta) formula (12)

The following can be obtained from formulas (11) and (12):

dα_i＝M(s(1+ds)^-1，-(θ+dθ))[M(S，θ)+dX-dXc]-alpha formula (13)

Meanwhile, the formula (5) can obtain:

<math> <mrow> <msub> <mi>&alpha;</mi> <mi>i</mi> </msub> <mo>+</mo> <mi>d</mi> <msub> <mi>&alpha;</mi> <mi>i</mi> </msub> <mo>&ap;</mo> <mover> <mi>&alpha;</mi> <mo>&OverBar;</mo> </mover> <mo>+</mo> <mi>P</mi> <mrow> <mo>(</mo> <mi>b</mi> <mo>+</mo> <mi>db</mi> <mo>)</mo> </mrow> </mrow> </math> formula (14)

Subtracting formula (5) from formula (14) gives:

dα_ip × db type (15)

Namely:

db＝P^-1dα_iformula (16)

db＝P^Tdα_iFormula (17)

Db can be obtained by combining formula (17) and formula (13). Therefore, the above parameter updating process is:

the affine transformation parameters and b can be updated as follows:

X_c＝X_c+w_tdX_c，Y_c＝Y_c+w_tdY_cθ＝θ+w_θdθ，s＝s(1+w_sds)，b＝b+w_bdb type (18)

In the above formula w_t，w_θ，w_s，w_bAre weights used to control parameter changes. Thus, a new shape can be obtained from the formulas (5) and (9). The search process ends when the parameters of the affine transformation and b do not vary much or the number of iterations reaches a specified threshold. The detection results are shown in fig. 9.

And (3) fitting the detected regions of five sense organs and the contour of the cheek to an existing human face 3-dimensional model, so that the human face 3-dimensional model can automatically present different visual angles, illumination and expression changes according to the setting of parameters. The specific implementation method comprises the following steps:

a 'BJUT-3D Face Database' three-dimensional Face library is selected, and data of about 60000 points and 120000 triangular plates of 100 males and 100 females per person are selected as a dense Face sample set through preprocessing such as resampling, smoothing, coordinate correction and the like. Then 60 three-dimensional feature points of each person are selected through manual interaction to serve as a sparse corresponding sample set, and the average model of the 200 persons is used as a general model.

The reconstruction is divided into the following four steps, as shown in fig. 10:

a. face feature points are detected by the ASM template. A modified ASM algorithm is employed. Automatically extracting 60 characteristic points;

b. and acquiring the depth information of the characteristic points by using the sparse deformation model. And optimally approaching the three-dimensional characteristic point sample set to the two-dimensional characteristic points of the photo by using the prior three-dimensional face statistical knowledge through plane projection and linear combination, thereby obtaining the three-dimensional coordinates corresponding to the characteristic points of the photo.

c. And deforming the general face model according to the displacement of the three-dimensional feature points to obtain the specific three-dimensional face. Selecting a thin-plate spline interpolation algorithm (TPS), see in particular: books teinfl. principllewarps: the thin-platestrines and the decoding position of the transformation (IEEETranson PAMI198911 (6): 567-.

d. And reconstructing the color information of the model through texture mapping. And performing affine transformation on the photo texture and then orthogonally projecting the photo texture onto the surface of the three-dimensional model.

Further, after the adjusting and replacing of the original character model into the image of the target character, the method may further include:

and adding props to the replaced original person, wherein the props comprise glasses, hats, clothes, backpacks and shoes.

Specifically, after the automatic editing system replaces the first person specified by the user with the picture or the image sequence of the second person, a prop can be further added to the replaced first person. The props can be glasses, hats, clothes, backpacks and the like.

Further, adjusting and replacing the original character model into the image of the target character may further include:

and according to the two-dimensional feature points detected by the ASM, all feature points of the texture mapping fall within the face area for adjustment.

Further, all the feature points of the texture mapping may further include:

the used feature points are corrected by a skin color model.

For example, the two-dimensional feature points detected by the ASM are used in the model reconstruction process, and the feature points used by texture mapping need to be corrected based on the skin color model, so that all the feature points fall within the face region, and the loss of side textures during texture mapping is avoided.

1) Skin tone spot determination

The method for reducing the influence of illumination on image quality by taking YUV and YIQ spaces as a basis and adding Gamma correction can be specifically seen in the following steps: automatic3D face model recovery using, by CHEN Lu, YANG Jie, to detect skin color information.

In YUV space, U and V are two mutually orthogonal vectors in a plane, the chrominance signal (i.e., the sum of U and V) is a two-dimensional vector, called chrominance signal vector, and each color corresponds to a chrominance signal vector whose saturation is represented by a modulus Ch and hue is represented by a phase angle θ:

$\mathrm{CH}=\sqrt{{\left|U\right|}^2+{\left|V\right|}^2}$ formula (19)

<math> <mrow> <mi>&theta;</mi> <mo>=</mo> <msup> <mi>tan</mi> <mrow> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mrow> <mo>(</mo> <mfrac> <mrow> <mo>|</mo> <mi>U</mi> <mo>|</mo> </mrow> <mrow> <mo>|</mo> <mi>V</mi> <mo>|</mo> </mrow> </mfrac> <mo>)</mo> </mrow> </mrow> </math> Formula (20)

Transforming the pixel P of the color image from RGB space to YUV space if the condition theta is satisfied_p∈[105，150]And P is the flesh tone spot. In the YIQ space, the I component represents a hue from orange to cyan, and the smaller the I value, the more yellow is contained and the less cyan is contained. The I value of the skin color in the YIQ space can be determined to be [20, 90 ] through experiments and statistical analysis]And (4) changing. The R, G, B three components are respectively subjected to Gamma correction, and the corrected values are respectively marked as Rgamma, Ggamma and Bgamma:

U＝-0.147×R_gamma-0.289×G_gamma+0.436×B_gammaformula (21)

V＝-0.615×R_gamma-0.515×G_gamma-0.100×B_gammaFormula (22)

<math> <mrow> <mi>&theta;</mi> <mo>=</mo> <msup> <mi>tan</mi> <mrow> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mrow> <mo>(</mo> <mfrac> <mrow> <mo>|</mo> <mi>U</mi> <mo>|</mo> </mrow> <mrow> <mo>|</mo> <mi>V</mi> <mo>|</mo> </mrow> </mfrac> <mo>)</mo> </mrow> </mrow> </math> Formula (23)

I＝0.596×R_gamma-0.274×G_gamma-0.322×B_gammaFormula (24)

And judging the pixel point as a skin color point according to the obtained sum value. If it satisfies

<math> <mfenced open='{' close='}'> <mtable> <mtr> <mtd> <mi>&theta;</mi> <mo>&Element;</mo> <mo>[</mo> <mn>105,150</mn> <mo>]</mo> </mtd> </mtr> <mtr> <mtd> <mi>I</mi> <mo>&Element;</mo> <mo>[</mo> <mn>20,90</mn> <mo>]</mo> </mtd> </mtr> </mtable> </mfenced> </math> Formula (25)

Then the pixel point is judged to be a skin color point.

2) Correcting feature points

Because the ASM template adopts a symmetrical template, the extraction of the face features of the incomplete front face can cause the out-of-bounds of side feature points on one side, and further cause the loss of side information for the reconstruction of the texture on the back. And judging the skin color point of the side feature point, if the side feature point is not the skin color point, falling outside the face, and retracting the point to the center of the face until all the side feature points become the skin color point.

3) Texture mapping using corrected feature points

And (4) calculating three-dimensional characteristic points by using the two-dimensional characteristic points before correction as symmetrical characteristic points are required to be used for model reconstruction, and finally obtaining a model. And the corrected feature points are used for mapping the three-dimensional feature points during mapping, so that the texture loss of the side surface is effectively avoided.

The three-dimensional face model with different postures, illumination and expressions generated by the model also has better sense of reality. As shown in fig. 11, the original input Facial image and the generated three-dimensional model of the face, in order to synthesize rich Facial expressions, 44 basic Action Units (AUs) are set up based on a Facial motion Coding System (FACS), and each AU can control the displacement of one or several Facial feature points in three-dimensional space. Different AUs are combined to generate various expressions such as joy, anger, sadness and the like. And (3) carrying out interpolation deformation on the three-dimensional feature points by using the TPS to realize expression change, as shown in figure 12, by using a simulated expression example.

And deducing related characteristic information of the second person through the detected areas of the five sense organs and the contour of the cheek. The characteristic information includes a change in viewing angle, illumination, and expression, and the like. As shown in FIG. 13, the specific embodiment is as follows.

When the feature points of the people in the picture or image sequence have been identified, we can easily find the AU units that have been defined in advance (as described above). The AUs can describe the expression of one face in a very detailed way, and the expression of the face of the person can be determined by the algorithm as long as the specific positions of the feature points and the specific forms of the AUs are determined.

As for the estimation of the pose of the head of a person from the face feature points in the two-dimensional image, the present algorithm utilizes the POSIT method.

1. The basic idea is as follows: the algorithm is divided into two parts

(1) Orthogonal projection transformation SOP (Standard Operation Procedure) with a proportionality coefficient is used for solving a rotation matrix and a translation vector according to a linear equation set;

(2) and updating a Scale Factor (Scale Factor) according to the obtained rotation matrix and translation vector coefficient, and updating the original point by the Scale Factor to perform iteration.

2. The algorithm process is as follows:

(1) assuming a rotation matrix $R=\left[\begin{array}{ccc}{R}_{11}& R_{12}& R_{13}\\ R_{21}& R_{22}& R_{23}\\ R_{31}& R_{32}& R_{33}\end{array}\right]=\left[\begin{array}{c}{R_1}^T\\ {R_2}^T\\ {R_3}^T\end{array}\right]$ And translation vector $T=\left[\begin{array}{c}{T}_x\\ T_y\\ T_z\end{array}\right],$ f is the focal length; in perspective projective transformationWhereas in the case of the SOP, the,wherein the scale factor is $s=\frac{f}{Z_0};$

(2) Performing basic perspective projection transformation, and changing the 3D point a to (a)_x，a_y，a_z)^TPerspective projection is carried out on an image plane to obtain homogeneous coordinates m ═ w, wy and w^TThe transformation process is <math> <mrow> <mfenced open='[' close=']'> <mtable> <mtr> <mtd> <mi>wx</mi> </mtd> </mtr> <mtr> <mtd> <mi>wy</mi> </mtd> </mtr> <mtr> <mtd> <mi>w</mi> </mtd> </mtr> </mtable> </mfenced> <mo>=</mo> <mfenced open='[' close=']'> <mtable> <mtr> <mtd> <msup> <msub> <mi>fR</mi> <mn>1</mn> </msub> <mi>T</mi> </msup> </mtd> <mtd> <mi>f</mi> <msub> <mi>T</mi> <mi>x</mi> </msub> </mtd> </mtr> <mtr> <mtd> <mi>f</mi> <msup> <msub> <mi>R</mi> <mn>2</mn> </msub> <mi>T</mi> </msup> </mtd> <mtd> <mi>f</mi> <msub> <mi>T</mi> <mi>y</mi> </msub> </mtd> </mtr> <mtr> <mtd> <msup> <msub> <mi>R</mi> <mn>3</mn> </msub> <mi>T</mi> </msup> </mtd> <mtd> <msub> <mi>T</mi> <mi>z</mi> </msub> </mtd> </mtr> </mtable> </mfenced> <mo>&CenterDot;</mo> <mfenced open='[' close=']'> <mtable> <mtr> <mtd> <msub> <mi>a</mi> <mi>x</mi> </msub> </mtd> </mtr> <mtr> <mtd> <msub> <mi>a</mi> <mi>y</mi> </msub> </mtd> </mtr> <mtr> <mtd> <msub> <mi>a</mi> <mi>z</mi> </msub> </mtd> </mtr> <mtr> <mtd> <mn>1</mn> </mtd> </mtr> </mtable> </mfenced> </mrow> </math> Since m is a homogeneous coordinate, the right side of the equation is divided by T_zWithout being affected, then obtain <math> <mrow> <mfenced open='[' close=']'> <mtable> <mtr> <mtd> <mi>wx</mi> </mtd> </mtr> <mtr> <mtd> <mi>wy</mi> </mtd> </mtr> <mtr> <mtd> <mi>w</mi> </mtd> </mtr> </mtable> </mfenced> <mo>=</mo> <mfenced open='[' close=']'> <mtable> <mtr> <mtd> <msup> <msub> <mi>fR</mi> <mn>1</mn> </msub> <mi>T</mi> </msup> </mtd> <mtd> <mi>f</mi> <msub> <mi>T</mi> <mi>x</mi> </msub> </mtd> </mtr> <mtr> <mtd> <mi>f</mi> <msup> <msub> <mi>R</mi> <mn>2</mn> </msub> <mi>T</mi> </msup> </mtd> <mtd> <mi>f</mi> <msub> <mi>T</mi> <mi>y</mi> </msub> </mtd> </mtr> <mtr> <mtd> <msup> <msub> <mi>R</mi> <mn>3</mn> </msub> <mi>T</mi> </msup> <mo>/</mo> <msub> <mi>T</mi> <mi>z</mi> </msub> </mtd> <mtd> <mn>1</mn> </mtd> </mtr> </mtable> </mfenced> <mo>&CenterDot;</mo> <mfenced open='[' close=']'> <mtable> <mtr> <mtd> <msub> <mi>a</mi> <mi>x</mi> </msub> </mtd> </mtr> <mtr> <mtd> <msub> <mi>a</mi> <mi>y</mi> </msub> </mtd> </mtr> <mtr> <mtd> <msub> <mi>a</mi> <mi>z</mi> </msub> </mtd> </mtr> <mtr> <mtd> <mn>1</mn> </mtd> </mtr> </mtable> </mfenced> <mo>,</mo> </mrow> </math> Wherein s ═ f/T_zTo obtain <math> <mrow> <mfenced open='[' close=']'> <mtable> <mtr> <mtd> <mi>wx</mi> </mtd> </mtr> <mtr> <mtd> <mi>wy</mi> </mtd> </mtr> </mtable> </mfenced> <mo>=</mo> <mfenced open='[' close=']'> <mtable> <mtr> <mtd> <msup> <msub> <mi>fR</mi> <mn>1</mn> </msub> <mi>T</mi> </msup> </mtd> <mtd> <mi>f</mi> <msub> <mi>T</mi> <mi>x</mi> </msub> </mtd> </mtr> <mtr> <mtd> <mi>f</mi> <msup> <msub> <mi>R</mi> <mn>2</mn> </msub> <mi>T</mi> </msup> </mtd> <mtd> <mi>f</mi> <msub> <mi>T</mi> <mi>y</mi> </msub> </mtd> </mtr> </mtable> </mfenced> <mo>&CenterDot;</mo> <mfenced open='[' close=']'> <mtable> <mtr> <mtd> <msub> <mi>a</mi> <mi>x</mi> </msub> </mtd> </mtr> <mtr> <mtd> <msub> <mi>a</mi> <mi>y</mi> </msub> </mtd> </mtr> <mtr> <mtd> <msub> <mi>a</mi> <mi>z</mi> </msub> </mtd> </mtr> <mtr> <mtd> <mn>1</mn> </mtd> </mtr> </mtable> </mfenced> <mo>,</mo> </mrow> </math> Wherein <math> <mrow> <mi>w</mi> <mo>=</mo> <mfrac> <mrow> <msub> <mi>R</mi> <mn>3</mn> </msub> <mo>&CenterDot;</mo> <mi>a</mi> </mrow> <msub> <mi>T</mi> <mi>z</mi> </msub> </mfrac> <mo>+</mo> <mn>1</mn> <mo>;</mo> </mrow> </math>

(3) Now the conversion process is $\left[\begin{array}{cccc}X& Y& Z& 1\end{array}\right]\left[\begin{array}{cc}s{R}_1& s{R}_2\\ s{T}_x& s{T}_t\end{array}\right]=\left[\begin{array}{cc}\mathrm{wx}& \mathrm{wy}\end{array}\right],$ I.e. the system of equations

$\left\{\begin{array}{c}\mathrm{Xs}{R}_{11}+\mathrm{Ys}{R}_{12}+\mathrm{Zs}{R}_{13}+s{T}_x=\mathrm{wx}\\ \mathrm{Xs}{R}_{21}+\mathrm{Ys}{R}_{22}+\mathrm{Zs}{R}_{23}+s{T}_y=\mathrm{wy}\end{array}\right.,$ The initial value of w is 1;

(4) let K₁＝(sR₁₁ sR₁₂ sR₁₃ sT_x)^T，K₂＝(sR₂₁ sR_２２ sR₂₃ sT_y)^T，

<math> <mrow> <mi>A</mi> <mo>=</mo> <mfenced open='[' close=']'> <mtable> <mtr> <mtd> <msub> <mi>X</mi> <mn>0</mn> </msub> </mtd> <mtd> <msub> <mi>Y</mi> <mn>0</mn> </msub> </mtd> <mtd> <msub> <mi>Z</mi> <mn>0</mn> </msub> </mtd> <mtd> <mn>1</mn> </mtd> </mtr> <mtr> <mtd> <msub> <mi>X</mi> <mn>1</mn> </msub> </mtd> <mtd> <msub> <mi>Y</mi> <mn>1</mn> </msub> </mtd> <mtd> <msub> <mi>Z</mi> <mn>1</mn> </msub> </mtd> <mtd> <mn>1</mn> </mtd> </mtr> <mtr> <mtd> <mo>&CenterDot;</mo> </mtd> <mtd> <mo>&CenterDot;</mo> </mtd> <mtd> <mo>&CenterDot;</mo> </mtd> <mtd> <mo>&CenterDot;</mo> </mtd> </mtr> <mtr> <mtd> <msub> <mi>X</mi> <mi>n</mi> </msub> </mtd> <mtd> <msub> <mi>Y</mi> <mi>n</mi> </msub> </mtd> <mtd> <msub> <mi>Z</mi> <mi>n</mi> </msub> </mtd> <mtd> <mn>1</mn> </mtd> </mtr> </mtable> </mfenced> <mo>,</mo> </mrow> </math> A is an (n +1) × 4 matrix, <math> <mrow> <msub> <mi>b</mi> <mn>1</mn> </msub> <mo>=</mo> <mfenced open='[' close=']'> <mtable> <mtr> <mtd> <msub> <mi>x</mi> <mn>0</mn> </msub> </mtd> </mtr> <mtr> <mtd> <msub> <mi>x</mi> <mn>1</mn> </msub> </mtd> </mtr> <mtr> <mtd> <mo>&CenterDot;</mo> </mtd> </mtr> <mtr> <mtd> <msub> <mi>x</mi> <mi>n</mi> </msub> </mtd> </mtr> </mtable> </mfenced> <mo>,</mo> <msub> <mi>b</mi> <mn>2</mn> </msub> <mo>=</mo> <mfenced open='[' close=']'> <mtable> <mtr> <mtd> <msub> <mi>y</mi> <mn>0</mn> </msub> </mtd> </mtr> <mtr> <mtd> <msub> <mi>y</mi> <mn>1</mn> </msub> </mtd> </mtr> <mtr> <mtd> <mo>&CenterDot;</mo> </mtd> </mtr> <mtr> <mtd> <msub> <mi>y</mi> <mi>n</mi> </msub> </mtd> </mtr> </mtable> </mfenced> <mo>,</mo> </mrow> </math> the initial set of equations then becomes $\left\{\begin{array}{c}{\mathrm{AK}}_1=b_1\\ A{K}_2=b_2\end{array}\right.,$ Applying least squares to obtain solutions $\left\{\begin{array}{c}{K}_1={\left(A^{T}A\right)}^{-1}{A}^T{b}_1\\ K_2={\left(A^{T}A\right)}^{-1}{A}^T{b}_2\end{array}\right.;$

(5) Obtaining K1 and K2 by dividing the obtained K1 and K2 points by a known constant value s to obtain R1, R2, Tx and Ty, obtaining R3 which is R1 × R2, and normalizing R1, R2 and R3 into a unit vector;

(6) then updatedBecause for different pairs of 2D-3D points, s ═ f/T_zIs a constant value, f is the focal length, is a known constant parameter, T_zThe parameter is also a known fixed value parameter and can be regarded as an average value of Z coordinates of all 3D points; for different 3D points, a is different, so w is different, thus changing the original 2D point into (wx, wy)^T；

(7) Starting from the step (2), solving an equation set by using the original 3D point and the updated 2D point by using a least square method to obtain new K1 and K2; updating w and updating the 2D point coordinate;

3. and (3) solving:

(1) given the initial position of the camera: focal length f, image coordinate center, i.e. (c)_x，c_y) Image range, i.e. a reasonable range of 2D coordinate values.

(2) The total number of the unknowns is 8, and at least 4 2D-3D point pairs are needed;

(3) the first 2D-3D point pair must be (0, 0) - (0, 0, 0);

(3) the algorithm execution stop conditions are as follows: and limiting the iteration times, and setting a threshold value of the size (accuracy) of the variation value of each 2D point.

In implementation, when the display of the face of the first person is adjusted in the model of the first person according to the feature information, the feature information may be one of the following parameters or a combination thereof: the three-dimensional 3D pose of the face of the second person, the state of the basic motion unit AU of the face of the second person, the ratio of the length to the width of the outline of the face of the second person, and the degree of lightness and darkness of the skin around the feature point of the face of the second person.

Specifically, the first character model may further replace the second character as follows:

a. according to the related characteristic information of the second person, adjusting the model of the first person to enable the model to be similar to the related characteristic of the second person; if can be divided into:

a.1, adjusting the posture of the 3D model of the first person according to the estimated posture of the 3D face of the second person;

a.2, adjusting the expression of the 3D model of the first person according to the estimated state of the AU of the second person;

a.3, adjusting the face shape of the 3D model of the first person according to the contour of the face of the second person, mainly the length-width ratio;

and a.4, adjusting the brightness of the face around the corresponding characteristic points of the first person according to the brightness of the skin around all the characteristic points of the face of the second person.

b. Wiping off the face area of the second person in each frame of image through the detected five sense organs and cheek outlines;

c. in each frame of image, placing the adjusted model of the first person in the face area of the second person;

based on the same inventive concept, the embodiment of the present invention further provides an image processing system, and as the principle of the problem solved by the system is similar to that of an image processing method, the implementation of the systems can refer to the implementation of the method, and repeated details are not repeated.

Fig. 14 is a schematic diagram of an image processing system, as shown in fig. 14, which may include:

a model simulation module 1401 for simulating a model of a first person based on at least one image including the first person;

a target image determination module 1402 for determining a target image containing a second person;

a feature information determination module 1403 for determining feature information of displaying the second person in the target image;

an adjustment display module 1404 for adjusting the display of the first persona in the model of the first persona based on the characteristic information;

a person replacing module 1405, configured to replace the second person with the display-adjusted first person in the target image.

In implementation, model simulation module 1401 may include:

a first detection unit configured to detect a region of a face of a first person;

a first determination unit configured to determine, in the detected face region, the region of the five sense organs and the contour of the cheek;

and the fitting unit is used for fitting the detected regions of the five sense organs and the contour of the cheek to the existing human face 3D model to obtain the simulated model of the face of the first person.

In implementation, the target image determination module 1402 may include:

a second detection unit configured to detect a region of a face of a second person;

a second determination unit configured to determine, in the detected region of the face, a region of the five sense organs and a contour of the cheek;

and the characteristic unit is used for determining characteristic information of the face of the second person displayed in the target image according to the detected areas of the five sense organs and the contour of the cheek.

In implementation, the feature information determination module 1403 is further configured to replace the face of the second person with the face of the first person after the adjustment according to the area of the face of the first person and the area of the face of the second person.

In practice, the adjustment display module 1404 is further configured to adjust the display of the face of the first person in the model of the first person based on the feature information of one or a combination of the following parameters: the 3D pose of the face of the second person, the state of the AU of the face of the second person, the aspect ratio of the outline of the face of the second person, and the degree of lightness and darkness of the skin around the feature points of the face of the second person.

In implementation, the adjustment display module 1404 is further configured to determine the contour of the facial region and cheek of the five sense organs using a face recognition algorithm in the detected facial region.

In the implementation, the method further comprises the following steps:

and the prop adding module is used for replacing the second person with the first person after the adjusted first person is displayed in the target image and adding an image to the first person in the target image.

In the technical scheme provided by the embodiment of the invention, the model of the original character is simulated, the characteristic information of the target character is comprehensively considered, and the model of the original character is adjusted and replaced into the image of the target character. The problems that the figure image is not consistent with the replaced shooting visual angle image and the representation of the figure is unchangeable when the image is replaced in a simulation mode are solved. Can be applied in many scenarios such as: when the faces of friendship, love, paternity and karaoke are changed, one person is personally brought to the environment where the other person is located; one character can also be virtualized to do something instead of another character; people who do not like in the photo can be replaced by people who like the photo.

By adopting the technical scheme provided by the embodiment of the invention, a user only needs one picture to replace or edit the face of any person in any photo or video; after the replacement, the shooting angle of the face of the first person may be changed according to the change of the shooting angle of the target person; after the replacement, the expression of the face of the first character may be changed according to the change of the expression of the target character; the first person can be personally brought into the world where the target person is located.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention.

Claims (14)

1. An image processing method, characterized by comprising the steps of:

simulating a model of a first person based on at least one image containing the first person;

determining a target image containing a second person;

determining feature information of a second person displayed in the target image;

adjusting the display of the first persona in the model of the first persona according to the characteristic information;

and replacing the second person with the first person after the display adjustment in the target image.

2. The method of claim 1, wherein simulating the model of the face of the first person based on the at least one image containing the first person comprises:

detecting a region of a face of a first person;

determining the areas of five sense organs and the contour of a cheek in the detected area of the face;

and fitting the detected five sense organ regions and the contour of the cheek to the existing three-dimensional human face 3D model to obtain the simulated model of the first person face.

3. The method according to claim 1 or 2, wherein in determining that the feature information of the face of the second person is displayed in the target image, it includes:

detecting a region of a face of a second person;

determining the areas of five sense organs and the contour of a cheek in the detected area of the face;

feature information of the face of the second person displayed in the target image is determined based on the detected regions of the five sense organs and the contour of the cheek.

4. The method of claim 3, wherein the replacing of the second person with the displaying of the adjusted first person is performed by replacing the face of the second person with the displaying of the adjusted face of the first person based on the area of the face of the first person and the area of the face of the second person.

5. The method of claim 4, wherein the feature information is one of the following parameters or a combination thereof when adjusting the display of the face of the first person in the model of the first person based on the feature information: the 3D pose of the face of the second person, the state of the basic motion unit AU of the face of the second person, the ratio of the length to the width of the outline of the face of the second person, and the degree of lightness and darkness of the skin around the feature point of the face of the second person.

6. A method as claimed in claim 3, characterized in that in the detected region of the face, a face recognition algorithm is used to determine the region of the five sense organs and the contour of the cheek.

7. The method of any of claims 1 to 6, wherein after replacing the second person with the display-adjusted first person in the target image, further comprising:

an image is added for a first person in the target image.

8. An image processing system, comprising:

the model simulation module is used for simulating a model of a first person according to at least one image containing the first person;

a target image determination module for determining a target image containing a second person;

the characteristic information determining module is used for determining characteristic information of a second person displayed in the target image;

an adjustment display module for adjusting the display of the first persona in the model of the first persona according to the characteristic information;

and the person replacing module is used for replacing the second person with the first person after the display adjustment in the target image.

9. The system of claim 8, wherein the model simulation module comprises:

a first detection unit configured to detect a region of a face of a first person;

a first determination unit configured to determine, in the detected face region, the region of the five sense organs and the contour of the cheek;

and the fitting unit is used for fitting the detected regions of the five sense organs and the contour of the cheek to the existing human face 3D model to obtain the simulated model of the face of the first person.

10. The system of claim 8 or 9, wherein the target image determination module comprises:

a second detection unit configured to detect a region of a face of a second person;

a second determination unit configured to determine, in the detected region of the face, a region of the five sense organs and a contour of the cheek;

and the characteristic unit is used for determining characteristic information of the face of the second person displayed in the target image according to the detected areas of the five sense organs and the contour of the cheek.

11. The system of claim 10, wherein the characteristic information determination module is further configured to replace the face of the second person with the display of the adjusted face of the first person based on the area of the face of the first person and the area of the face of the second person.

12. The system of claim 11, wherein the adjustment display module is further configured to adjust the display of the face of the first person in the model of the first person based on the feature information for one or a combination of the following parameters: the 3D pose of the face of the second person, the state of the AU of the face of the second person, the aspect ratio of the outline of the face of the second person, and the degree of lightness and darkness of the skin around the feature points of the face of the second person.

13. The system of claim 10, wherein the adjustment display module is further configured to determine the facial region and the cheek contour using a face recognition algorithm in the detected facial region.

14. The system of any of claims 8 to 13, further comprising:

and the prop adding module is used for replacing the second person with the first person after the adjusted first person is displayed in the target image and adding an image to the first person in the target image.