JPH0473698A - Shape control method based on audio signal - Google Patents

Abstract

PURPOSE:To easily control the shape of the jaw and lips more in real by obtaining the opening distance between the chin and the lips performing linear transformation and nonlinear transformation on the formant center frequency of an input audio signal. CONSTITUTION:The opening distance D of the chin and the opening distance L in the lateral direction of the lips of a face can be obtained by finding the center frequency of a formant frequency(for example, first and second formants H1, H2) representing the peak of the spectrum envelope of the input audio signal, and performing the linear and nonlinear transformation of the formant center frequency. In other words, the motion of the jaws i.e. the opening distance D of the chin, the motion of the operating positions P1-P5 of a tongue and/or the motion (a-e) of the tip shape of the tongue can be found by performing the linear transformation of the formant center frequency, and the opening distance L in the lateral direction of the lips can be obtained by performing the nonlinear transformation of the motion of the tongue. Thereby, it is possible to easily reproduce the motion of the lips and the chin in real time.

Description

DETAILED DESCRIPTION OF THE INVENTION (Field of Industrial Application) The present invention relates to a shape control method based on an audio signal, which controls the shape of the chin and lips of a face of an image or a doll based on the audio signal.

[Summary of the invention]

The present invention provides an audio signal that allows the shapes of the jaw and lips to be controlled more realistically by linearly and non-linearly transforming the formant center frequency of the input audio signal to obtain the degree of opening of the lower jaw and lips. The present invention provides a shape control method based on the following.

[Prior Art] In conventional animations, for example, the movements of the lips, jaws, etc. when a person in the animation performs a conversation are made by the creator of the animation image, for example, by adjusting the movements of the lips, etc., to match the conversation. I try to make decisions by making guesses based on past experience. The same applies to cases in which the lips, jaws, etc. of large so-called robots or dolls are moved in accordance with conversation.

[Problem to be solved by the invention]

By the way, in recent years, the above animations or robots,
There is a need for dolls and the like to be able to move their lips, jaws, etc. more realistically in accordance with conversation.

However, as described above, in the past, movements of the lips, etc. in accordance with conversation have been estimated based on experience, so it is difficult to say that the movements are realistic. In addition, for example, methods have been considered to calculate the movements of the lips, etc. using computers, etc., but the reality is that it requires a huge amount of calculation and it is not possible to easily obtain more realistic movements of the lips, etc. . Furthermore, conventionally, it is not possible to move the lips or the like in real time.

Therefore, the present invention was proposed in view of the above-mentioned circumstances, and it is possible to more realistically control the shape of the jaw and lips of the face of an image or a doll, and furthermore, it can be controlled in real time. It is an object of the present invention to provide a shape control method based on an audio signal that can also be controlled.

[Means to solve the problem]

The shape control method based on an audio signal of the present invention has been proposed to achieve the above-mentioned object, and is capable of determining the center frequency of the interformant wave number indicating the peak of the spectral envelope of the human input audio signal from the input audio signal. By performing linear and non-linear transformation on this formant center frequency, for example, the degree of opening of the lower jaw and the degree of opening of the lips in the lateral direction of the face shape of three video dolls (such as Robono Hi) are obtained. In other words, the movement of the jaw and the tongue are obtained by linearly transforming the formant center frequency, and the degree of opening of the subcontractor and the degree of lateral opening of the lips are determined by nonlinearly transforming this tongue movement. I'm trying to get what I want.

[Effect]

According to the present invention 7'1. For example, a simple linear transformation operation is performed on the formant center frequency of an input audio signal based on actual speech, and a simple nonlinear transformation operation is further performed to obtain the degree of opening of the lower jaw and lips. You can easily reproduce the movements of your lips and lower jaw in real time.

Embodiments] Hereinafter, embodiments to which the present invention is applied will be described with reference to the drawings.

FIG. 1 shows, for example, an animated face image to which the shape control method based on an audio signal according to an embodiment of the present invention is applied.

In the face image of this embodiment shown in FIG. 1, formants (for example, formants 1 and 2) indicating the peaks of the spectral envelope of the input audio signal as shown in FIGS. By finding the center frequency of the formant H,, Hl) frequency and performing linear and nonlinear transformation on this formant center frequency, we can calculate the lower jaw opening D (cm) and the lateral lip opening L (cm). ). That is, in this example, by linearly converting the formant center frequency using (1) 2, which will be described later, the movement of the jaw, that is, the degree of opening of the lower jaw, and each movement of the tongue shown in FIG. The movements at positions P1 to P (Fig. 6) and/or the movements of the tip shape of the tongue shown in Fig. 7 are determined, and the movements of the tongue are nonlinearly transformed using equation (2) described later. This is to obtain the degree of lateral opening of the lips.

Here, FIG. 2 shows the spectral envelope of the audio signal when the sound "A2" is produced, for example, FIG. For example, the spectral envelope of ``Tsu'' is shown.The peak part of the spectral enchelobe shown in Figures 2 to 4 is usually called a formant, and the formant of this audio signal is It is defined as the damped sinusoidal component of the acoustic impulse response of a road.

There are typically 3-4 formants within 3 kHz and 4-5 formants within 5 kHz for an average vocal tract of approximately 17 cm in length.

In voiced sounds, the first three formants are the most important;
Generally, the peak that appears at the lowest frequency is called the first formant (Hl), and the peak that appears after this first formant is called the second formant (Hl), and henceforth, the third formant, the fourth formant, etc.・It continues. These formants can be obtained based on, for example, so-called cepstral analysis or linear predictive analysis.
For example, by using the linear predictive analysis, it can be determined with a small amount of calculation.

As mentioned above, for example, each vowel ``A2.'I.

``tsu'', 'technique 1. - Find the formant frequency of -o. The fifth field shows the positional relationship when the center frequencies of the first formant H1 and the second formant H2 of the formant frequencies of each cough vowel are plotted on a plane. In this Figure 5, the positional relationship of each vowel is such that ``ozu'' is located between 175 and 1, and 1 knee is located between ``i'' and ``a.'' I can confirm that there is.

By the way, speech and tongue movements are, for example, shown in Figures 6 and 7.
It is known that there is a relationship as shown in the figure. In Figure 6, there is one vowel for each vowel ``a4.1i'' J + r ,...
Each operating position P1 to P of the tongue shown in FIG. 8 corresponding to [O]
, the curvature function (tongue shape curvature) C(crrr') is shown, and FIG. 7 shows the tip shape of the tongue corresponding to each vowel. Here, in FIG. 8, each operating position P1 to P,
is a position on the center line on the front side of the tongue, the operating position P is, for example, 10 mm from the tip of the tongue, and the operating position P2 is a position further 51 mm from the above operating position P, hereinafter referred to as operating position. P x, P 4. In the order of P s , the positions are shown in 5 mm increments. That is, based on the eighth area, FIG. 6 shows the curvature function of the tongue shape in a range of 5 mm above each of the operating positions P1 to P when each vowel is uttered at each of the operating positions P1 to P. (tongue shape curvature) C, and FIG. 7 shows the tongue shape curvature C as shown in FIG. 6 converted into the shape of the tip of the tongue for each vowel.

Further, the tongue shape curvature C at each action position P, -P for each vowel and the degree of opening of the lower jaw of the face have a relationship as shown in FIGS. 9 to 13.

That is, FIG. 9 shows the relationship between the degree of opening of the lower jaw of the face when each vowel is uttered and the tongue shape curvature C when each vowel is uttered at the above-mentioned operating position P1, and FIG. Similarly, FIG. 11 shows the relationship between the degree of opening of the mandible of the face and the tongue shape curvature C when each vowel is uttered at the above operating position P2. The degree of mandibular opening and the operating position P
4. FIG. 13 shows the relationship between the degree of mandibular opening and the tongue shape curvature C at the operating position P5. From these figures 9 to 13, the relationship between the mandibular opening degree and the tongue shape curvature C is as follows. It can be seen that it is located between ``. It can be confirmed that the positional relationships among these vowels are all the same in FIGS. 9 to 13 above.

From the above-mentioned Figures 9 to 13 and the above-mentioned Figure 5, the positional relationship between the mandibular opening degree and tongue shape curvature C and each vowel in the first and second Holmans) H, H2 is as follows. Similarly to the above, it can be confirmed that -o is located between -a and -u, and -work is located between near and carp.

That is, it can be confirmed that the formant frequency, the positional relationship between the tongue shape curvature C and the mandibular opening degree substantially match.

For this reason, as shown in FIG.
From the formant frequency of the second formant H1, Hz, it is relatively easy to derive a function that approximately maps the tongue shape curvature C and the degree of mandibular opening as shown in FIGS. 9 to 13. You will be able to do this.

In this embodiment, the approximate mapping function is linear. In this case, the linear transformation can be expressed as. However, in formula (1), F.

is the formant frequency (Hz) of the first formant H4, and F2 is the formant frequency (Hz) of the second formant H2.
z). Also, .

At this time, in formula (1), A and B are the points rx, P, and q that indicate the respective positions of "a", "i", and "tsu" of the formant frequency as shown in FIG. 14, for example, At the tongue shape curvature C and the degree of mandibular opening as shown in Fig. 16, each point rll+ Py+ Q
It is obtained by performing a linear transformation to y.

Note that since the formant frequency has individual differences depending on the person speaking, this individual difference is removed by normalization. As for this normalization, for example, "A" in Fig.
, by converting the triangle whose vertices are ``ico'' and ``tsu'' into an equilateral triangle as shown in Figure 15.
It becomes possible to normalize each vowel. This conversion to an equilateral triangle will be described later.

Furthermore, in the embodiment of the present invention, the tongue shape curvature C is converted to the degree of lateral opening of the lips. The transformation at this time is performed using nonlinear transformation. As the nonlinear transformation, for example, L = (C + d +)"" = d t
Use (2). However, in formula (2), d , d
2 is a constant. This nonlinear transformation can enhance the naturalness of mouth movements.

In this example, the tongue shape curvature C8 is converted to the mandibular opening degree by linear transformation of the formant frequency as described above, and the tongue shape curvature C is converted to the lateral opening degree of the lips by nonlinear transformation. By performing the conversion operation, it becomes possible to estimate the degree of opening of the lower jaw and the degree of lateral opening of the lips from the audio signal. Therefore, by using the shape control method of this embodiment, it is possible to more realistically control the shape of the chin and lips of the face of a doll or the like, not just videos such as animations. Furthermore, since individual differences in vocalization are removed by normalization, more accurate shape control is possible.

Here, a method of converting an arbitrary triangle used in the above normalization into an equilateral triangle or inversely converting it will be described. That is, as shown in FIG. 17, the triangle pq in the χY plane
Considering r, and using the following procedure, we can create an equilateral triangle p(“) qi61 r with side length 1 as shown in Figure 23.
(61, Convert. First, in the triangle pqr of FIG. 17, point p(χ+, )' 1) is translated in parallel so that it moves to the origin (FIG. 18).

Point q(1) of triangle p(1)q(1)r(1) in FIG.
), and the X coordinate (y3(1)) of point r(1) becomes 1 (Figure 19). Point q′tl in Figure 19
The triangle pftl q121 rItT is rotated by the angle θ so that the X coordinate (y, (Z)) of is O (FIG. 20). Triangle pf31 9fi in Figure 20
X coordinate of point q(3) of lr(3) (x2(ff+
) and the x, X coordinates are scale-transformed so that the point r
(4) X coordinate (x, +41) and X = 0.5
The difference between the two points is set as a, and the X coordinate of the point rf41 is scale-transformed using the straight line Y=X/a (FIG. 22). The scale of the X coordinate is converted so that the X coordinate of the triangle pf5)q(51rTS) in FIG. 22 becomes 3""/2 (FIG. 23).

・From the above procedure t, any triangle pqr can be converted into an equilateral triangle. Also, this 11117! Inverse transformation is also possible by tracing backwards.

〔Effect of the invention〕

In the shape control method based on the audio signal of the present invention, the formant center frequency of the input audio signal is linearly and non-linearly transformed to obtain the degree of opening of the lower jaw and lips. It has become possible to easily and more realistically control the shape of the jaws and lips of the faces of dolls, robots, etc., and it has also become possible to control them in real time.

[Brief explanation of the drawing]

FIG. 1 is a diagram showing a face image according to an embodiment of the present invention, and FIGS.
The figure is a characteristic diagram showing the spectral envelope of the audio signal.
Fig. 5 is a diagram for explaining the formant frequency of the audio signal, Fig. 6 is a diagram showing the tongue shape curvature, Fig. 7 is a diagram showing the shape of the tip of the tongue, and Fig. 8 is a diagram showing the operating position. Figures 9 to 13 are diagrams for explaining the tongue shape curvature and lower jaw opening degree at each operating position of the tongue. Figures 17 to 23 are diagrams for explaining the conversion from an arbitrary triangle to an equilateral triangle. D・・・・・・Degree of mandibular opening

Claims (1)

[Claims] From the input audio signal, find the center frequency of the formant frequency that indicates the peak of the spectral envelope of the input audio signal, and linearly and non-linearly transform this formant center frequency to determine the opening degree of the lower jaw and the lips of the face shape. A shape control method based on an audio signal, characterized in that the degree of expansion in the lateral direction is obtained.