JPH11120364A - Feature point tracking device - Google Patents

Abstract

(57) [Summary] [PROBLEMS] To improve the tracking accuracy in a feature point tracking device that tracks the movement of feature points of an object in a video, and relax the conditions of points that can be tracked. SOLUTION: The feature point tracking means is constituted by a plurality of different feature point tracking means 11 to 13 using a plurality of different matching degree evaluation means 211 to 232, respectively. The tracking control means 3 performs a plurality of feature point tracking means 1 for each feature point of the target video 5 in accordance with the contents written in the feature point recording means 4 in advance.
One of 1 to 13 is selected for tracking. As described above, each feature point tracking unit uses a plurality of different matching degree evaluation units, thereby enabling tracking of similar points and improving tracking accuracy. On the other hand, by selecting and using a plurality of different feature point tracking means, a feature point that cannot be tracked by a certain feature point tracking means can be tracked by another feature point tracking means. As a result, the tracking accuracy is improved, and the conditions for the traceable points are relaxed.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a feature point tracking apparatus for tracking a motion of an object by tracking a feature point on the object in an image.

[0002]

2. Description of the Related Art A feature point is tracked by providing a square window centered on the feature point and, when a new frame of the target video is given, finding a point at which the residual in the window between the frames is minimized. Has been done. In addition, regarding the said technique, for example, Tomasi an
d Kanade, "Shape and Moti
on from Image Streams: a F
activationMethod-Part 3
Detection and Tracking o
f Point Features "(Technic
al Report of School of Co
mputer Science, Carnegie M
ellon University, CMU-CS-9
1-132, 1991).

[0003]

However, in the above-mentioned prior art, there is a possibility that the feature points arranged on the contour line may slightly deviate from the contour line, and the tracking accuracy is not sufficient. In addition, since points that can be tracked are limited to points having geometrical characteristics such as corners, conditions to be satisfied are severe, and there is a problem that only a small number of characteristic points to be tracked can be arranged.

Accordingly, an object of the present invention is to provide a feature point tracking apparatus which improves the tracking accuracy of feature points and relaxes the conditions for trackable points.

[0005]

According to a first aspect of the present invention, to solve the above-mentioned problems, an object image obtained by photographing an object and an initial value of a feature point arranged on the object are set. A feature point tracking device that, as an input, tracks the movement of a feature point in accordance with the movement of an object in the target video, wherein a plurality of different feature point tracking means using outputs from a plurality of matching degree evaluation means; A feature point recording unit that records a current coordinate of a point and a feature point tracking unit to be used when tracking the feature point; and the current coordinates of the feature point and the feature point tracking unit from the feature point recording unit. The process of reading, selecting one of the feature point tracking means, obtaining new coordinates of the feature point, and writing this in the feature point recording means, is performed for all the feature points. Feature points that repeat every frame Characterized by comprising a mark control means.

[0006] A second invention according to the present invention also comprises:
As the matching degree evaluating means, there is at least a color distribution matching degree evaluating means, an edge matching degree evaluating means, and a contour deformation degree evaluating means, and as the feature point tracking means, the color distribution matching degree evaluating means Feature point tracking means using the edge matching degree evaluation means,
The apparatus further includes at least a feature point tracking unit using the edge matching degree evaluation unit and the contour deformation degree evaluation unit.

In the present invention, the feature point tracking means is constituted by a plurality of different feature point tracking means, and each of the feature point tracking means uses the output of the plurality of different matching degree evaluation means to perform feature point tracking for each feature point. Track by specifying means. The above two points are the main features of the present invention. In the present invention, the point that the conventional technique uses a single feature point tracking unit and a matching degree evaluation unit is improved in the present invention.

[0008] In the first invention according to the present invention, the feature point tracking means uses outputs from a plurality of matching degree evaluation means. Therefore, similar points can be tracked, and tracking accuracy is improved. On the other hand, a plurality of different feature point tracking means are prepared, and when the tracking control means selects one of them, a feature point which cannot be tracked by a certain feature point tracking means can be changed to another feature point tracking means. Means allow tracking. Therefore, it is possible to improve the tracking accuracy of the feature points and relax the conditions of the traceable points, which are the objects of the present invention.

In the second invention according to the present invention, one of the feature point tracking means uses the output from the color distribution matching degree evaluating means and the edge matching degree evaluating means. Therefore, it is possible to track similar points of the color distribution on the edge, and the tracking accuracy is improved. Another of the feature point tracking means uses the output from the edge matching degree evaluation means and the contour deformation degree evaluation means. Therefore, it is possible to track a point on the edge where deformation from the initial state is reduced as much as possible, and it is also possible to track a point having no characteristic in shape. Therefore, the condition regarding the feature point to be tracked can be relaxed. As described above, a plurality of different feature point tracking means are prepared, and by the tracking control means selecting one of them, a feature point which cannot be tracked by a certain feature point tracking means can be changed by another feature point tracking means. It is possible to track using another method. Therefore, it is possible to improve the tracking accuracy of the feature points and relax the conditions of the traceable points, which are the objects of the present invention.

[0010]

Embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 1 is a block diagram of a first embodiment showing the principle of the present invention.

In FIG. 1, 11 is a feature point tracking means (1), 12 is a feature point tracking means (2), 13 is a feature point tracking means (3), and 211 is a matching evaluation degree means used in the feature point tracking means 11. (11) and 212 are different matching degree evaluation means (12) and 22 used in the feature point tracking means 11.
1 is a matching degree evaluation means (21) used in the feature point tracking means 12, 222 is another matching degree evaluation means (22) used in the feature point tracking means 12, 231 is a matching degree evaluation means (231) used in the feature point tracking means 13 31) and 232 are different matching degree evaluation means (32) used in the feature point tracking means 13; 3 is a tracking control means; 4 is a feature point recording means for recording feature point tracking points; Represents a video.

In this embodiment, the feature point tracking means comprises a plurality of different feature point tracking means such as a feature point tracking means (1), a feature point tracking means (2),... Each feature point tracking means uses outputs of a plurality of different matching degree evaluation means such as a matching degree evaluation means (11), a matching degree evaluation means (12),.... The tracking point control means 3 is provided for each of the characteristic points of the target video 5 based on the characteristic point tracking means written in the characteristic point recording means 4. Select one and track.

In this embodiment, the feature point tracking means uses outputs from a plurality of matching degree evaluation means.
Therefore, similar points can be tracked, and tracking accuracy is improved. On the other hand, a plurality of feature point tracking means are prepared, and when the tracking control means 3 selects one of them, a feature point which cannot be tracked by a certain feature point tracking means is separated by another feature point tracking means. Can be tracked by using the method described above. Therefore, it is possible to improve the tracking accuracy of the feature points and relax the conditions of the traceable points, which are the objects of the present invention.

FIG. 2 is a block diagram of a second embodiment showing the principle of the present invention.

In FIG. 2, 11 is a feature point tracking means (1), 12 is a feature point tracking means (2), 21 is a matching evaluation degree means (1), 22 used in the feature point tracking means 11.
Is another matching degree evaluation means (2) used in the feature point tracking means 11 or the feature point tracking means 12, 23 is another matching degree evaluation means (3) used in the feature point tracking means 12, 3 is a tracking control means, Reference numeral 4 denotes feature point recording means for recording tracking points of the feature points, and reference numeral 5 denotes a target video composed of a plurality of frames.

In this embodiment, two feature point tracking means 11 and 12 are provided as feature point tracking means. A color distribution matching degree evaluating means (matching degree evaluating means 21) for comparing the state of color distribution around a specific point of two images as matching degree evaluating means, and a specific point is placed on the edge. Edge that represents the degree of
Matching degree evaluation means (matching degree evaluation means 22)
And a contour deformation degree evaluation means (matching degree evaluation means 23) indicating the degree of deformation from the initial state. The feature point tracking means 11 uses the outputs of the color distribution matching degree evaluating means 21 and the edge matching degree evaluating means 22. The feature point tracking means 12 includes an edge matching degree evaluation means 22
And the output of the contour deformation degree evaluation means 23 are used.

In this embodiment, one of the feature point tracking means uses the output from the color distribution matching degree evaluating means and the edge matching degree evaluating means. Therefore, it is possible to track similar points of the color distribution on the edge, and the tracking accuracy is improved. Another one of the feature point tracking means in the embodiment uses the output from the edge matching degree evaluation means and the contour deformation degree evaluation means. Therefore, it is possible to track a point on the edge where deformation from the initial state is reduced as much as possible, and it is also possible to track a point having no characteristic in shape. Therefore, the condition regarding the feature point to be tracked can be relaxed. As described above, a plurality of different feature point tracking means are prepared, and when the tracking control means 3 selects one of them, a feature point which cannot be tracked by a certain feature point tracking means can be changed to another feature point tracking means. Allows tracking using other methods. Therefore,
The object of the present invention is to improve the tracking accuracy of the feature points and relax the conditions of the traceable points.

FIG. 3 is a diagram for explaining a third embodiment of the present invention.

In FIG. 3, reference numeral 11 denotes a feature point tracking means (1), reference numeral 12 denotes a feature point tracking means (2), reference numeral 21 denotes a matching evaluation degree means (1), 22 used in the feature point tracking means 11.
Is another matching degree evaluation means (2) used in the feature point tracking means 11 or the feature point tracking means 12, 23 is another matching degree evaluation means (3) used in the feature point tracking means 12, 3 is a tracking control means, Reference numeral 4 denotes feature point recording means for recording tracking points of the feature points, and reference numeral 5 denotes a target video composed of a plurality of frames. In this embodiment, an example is described in which the movement of a mouth is extracted from a face image by feature point tracking.

FIG. 4 shows an example of a feature point to be tracked in this case.
Shown in In this example, there are 12 feature points to be tracked. The coordinates (x, y) are recorded in the feature point recording means 4. Also, it records which feature point tracking means tracks each feature point. In the case of the example of FIG. 4, points 1, 4, 7, 10: feature point tracking means (1) Points 2, 3, 5, 6, 8, 9, 11, 12: feature point tracking means (2) I do.

The matching degree evaluation means (1) 21 evaluates the degree of matching of the color distribution, and includes an image I ₁ , a point (x, y), an image I ₂ , a point (u, v), a tap on one side of a window. Take length n as input. As shown in FIG. 5, a square window having a side length (2n + 1) is formed on the image I ₁ and the image I ₂ , and the total value of the difference between the RGB values in the two windows is calculated according to the following equation. Find and output.

D ₁ (I ₁ , x, y, I ₂ , u, v) = Σ
_{i = -n} ⁿ [｛I _1R (x + i, y + j) -I _2R (u + i, v
+ J)｝ ² + ｛I _1G (x + i, y + j) −I _2G (x +
i, y + j)｝ ² + ｛I _1B (x + i, y + j) -I
_2B (x + i, y + j)｝ ² ] where I _1R (x, y), I _1G (x, y), I _1B (x, y
y) represents the luminance values of red, green, and blue at point (x, y) in image I ₁ by I _2R (u, v), I _2G (u, v), I 2
_2B (u, v) is the red color of the point in image I ₂ (u, v), green, and represent respectively the luminance values of the blue. It can be said that the smaller the value d ₁ , the smaller the difference in color distribution between the two windows. Incidentally, notation Σ _{i = -n} ⁿ in the formula, i =
The sum from −n to i = n is shown (the same applies hereinafter).

The matching degree evaluation means (2) 22 represents the degree of the point on the image on the edge.
The coordinates (x, y) and the tap length m on one side of the window are input. First, an edge image E is obtained from the image I. Then, according to the following equation, the sum of the edge luminances at points around the point (x, y) is obtained and used as the output of the evaluation means.

D ₂ (I, x, y) = Σ _{i = −m} ^m Σ _{j = −m} ^m E
(X + i, y + j) The larger this value is, the higher the point on the edge. Note that E (x, y) represents the luminance of the edge image at the point (x, y).

The matching degree evaluation means 3 indicates the degree of deformation from the initial position, and receives the coordinates of a plurality of points as input.
In this embodiment, the number of input points is four.
The current coordinates of the points _i (x i, y i) to (i = 1, ...,
4). As shown in FIG. 6, distances l ₁ , l ₂ , and l ₃ between points and angles θ ₁ and θ ₂ formed by line segments are defined, and their initial values are l ₁₀ ,
_{1 20, 1 30, θ 10} , and theta _20. Considering an elastic model in which points are connected by a spring, elastic energy due to deformation from the initial state is d ₃ (x ₁ , y ₁ , x ₂ , y ₂ , x ₃ , y ₃ , x ₄ , y ₄ ) = Σ
_{i = 1} ³ _{λ i} is given by ^{_{(l i -l i0) 2 +}} Σ j = 1 2 μ j (θ j -θ j0) 2, and the output of the evaluation means this value. It can be said that the smaller this value is, the less the deformation from the initial state is. Note that λ _i and μ _j are elastic coefficients and are determined in advance.

The feature point tracking means (1) 11 includes a matching degree evaluating means (1) 21 and a matching degree evaluating means (2) 2
Use the output of 2. Given the current image frame I _cur , the current coordinates (x, y) of the tracking point, and the next image frame I _next , the new coordinates (x + Δx, y +
Δy). Therefore, the search range of Δx and Δy is set to −
Let l ≦ Δx ≦ l, −l ≦ Δy ≦ l, and find Δx and Δy that minimize the following equation.

J ₁ (Δx, Δy) = d ₁ (I _cur , x,
y, I _next , x + Δx, y + Δy) + wd
₂ (I _next , x + Δx, y + Δy) Here, it can be said that the larger the value of d ₂ , the higher the value is on the edge, so that w is a negative constant and J ₁ is minimized. w is a weighting coefficient of the two evaluation scales and is determined in advance.

The feature point tracking means (2) 12 includes a matching degree evaluating means (2) 22 and a matching degree evaluating means (3) 2
Use the output of # 3. The coordinates (x _i , y _i ) (i = 1,..., 4) of the current image frame I _cur and the current tracking point, and the new coordinates (x ₁ ) of the next image frame I _next and the tracking points 1 and 4
+ Δx ₁ , y ₁ + Δy ₁ ), (x ₄ + Δx ₄ , y ₄ + Δy ₄ )
Is given, the new coordinates (x ₂ +
Δx ₂ , y ₂ + Δy ₂ ) and (x ₃ + Δx ₃ , y ₃ + Δy ₃ ) are obtained. Therefore, the search range of Δx _i and Δy _i is set to −l ′.
_{≦ Δx i ≦ l ', -} l' and ≦ Δy _i ≦ l ', seek Δx _i, Δy _i that minimizes the following equation.

J ₂ (Δx ₂ , Δy ₂ , Δx ₃ , Δy ₃ ) =
^{_{w '· Σ i = 1 2}} d 2 (I cur, x i, y i, I next, x i + Δ
_{x i, y i + Δy i} ) + d 3 (x 1 + Δx 1, y 1 + Δy 1, x
₂ + Δx ₂ , y ₂ + Δy ₂ , x ₃ + Δx ₃ , y ₃ + Δy ₃ , x ₄
+ Δx ₄ , y ₄ + Δy ₄ ) Here, it can be said that the larger the value of d ₂ , the higher the value is on the edge, so that w ′ is a negative constant to minimize J ₂ .
w 'is a weighting coefficient of the two evaluation scales and is determined in advance.

In order to operate the embodiment, first, the initial position of each feature point and the tracking method are set in the feature point recording means 4. The tracking control means 3 checks the tracking means part of the feature point recording means 4 and selects one using the feature point tracking means (1). In this case, points 1, 4, 7, and 10 correspond. For these points, the current image frame is frame 0, the value of the current tracking point is the value written in the feature point recording means 4,
With the next image frame as frame 1, new coordinates of the tracking point are obtained by the feature point tracking means (1). The value of the feature point recording means 4 is updated based on this result.

Next, the tracking control means 3 selects a feature point using the feature point tracking means (2). In this case, point 2,
3, 5, 6, 8, 9, 11, and 12 correspond. First, the feature point tracking means (2) is applied to points 1, 2, 3, and 4. Since the new coordinates of the points 1 and 4 have already been obtained by the feature point tracking means (1), the new coordinates of the points 2 and 3 are obtained by the feature point tracking means (2). Similarly, new coordinates of points 5 and 6 are obtained by applying the feature point tracking means (2) to points 4, 5, 6, and 7. Further, new coordinates of points 8, 9, 11, and 12 are also obtained.

In this way, new coordinates of each feature point on the frame 1 are obtained. Further, by setting frame 1 as the current image frame and frame 2 as the next image frame, new coordinates of each feature point on frame 2 are obtained. By repeating this, tracking of each feature point is performed.

With such a configuration, points 1, 4, 7, 1
By using the matching degree evaluation means (1) and (2) for 0, it is possible to accurately track the contour. The remaining points cannot be tracked by the feature point tracking means (1), but can be tracked by preparing the feature point tracking means (2).

As is apparent from these results, there are improvements that the accuracy of tracking is improved and that the number of tracking points can be increased as compared with the conventional technique.

[0036]

As described above, according to the present invention,
Since the tracking accuracy of the feature points is improved and more tracking points can be arranged, there is an effect that the tracking of the object can be performed more accurately.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a first embodiment illustrating the principle of the present invention.

FIG. 2 is a configuration diagram of a second embodiment illustrating the principle of the present invention.

FIG. 3 is a configuration diagram showing a third embodiment of the present invention.

FIG. 4 is a diagram showing an example of feature points to be tracked in the third embodiment.

FIGS. 5A and 5B are diagrams for supplementing the description of the matching degree evaluation means (1) in the third embodiment.

FIG. 6 is a diagram for supplementing the description of the matching degree evaluation means (3) in the third embodiment.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 11 ... Feature point tracking means (1) 12 ... Feature point tracking means (2) 13 ... Feature point tracking means (3) 21 ... Matching degree evaluation means (1) 22 ... Matching degree evaluation means (2) 23 ... Matching degree evaluation Means (3) 211 ... Matching degree evaluation means (11) 212 ... Matching degree evaluation means (12) 221 ... Matching degree evaluation means (21) 222 ... Matching degree evaluation means (22) 231 ... Matching degree evaluation means (31) 232 ... Matching degree evaluation means (32) 3 ... Tracking control means 4 ... Feature point recording means 5 ... Target video

Claims (2)

[Claims] 1. An object image obtained by capturing an object and an initial value of a feature point arranged on the object are input, and the movement of the feature point is tracked according to the movement of the object in the object image. In the feature point tracking device, a plurality of different feature point tracking means using outputs from a plurality of matching degree evaluation means, and a current coordinate of the feature point and a feature point tracking means to be used when tracking the feature point Reading the current coordinates of the feature point and the feature point tracking means from the feature point recording means, and selecting one of the feature point tracking means to obtain new coordinates of the feature point. And a feature point tracking control means for repeating a process of writing the same into the feature point recording means for each new frame of the target video for all the feature points. Tracking device. 2. The image processing apparatus according to claim 1, further comprising: a color distribution matching degree evaluating unit, an edge matching degree evaluating unit, and a contour deformation degree evaluating unit as the matching degree evaluating unit; Characteristic point tracking means using the matching degree evaluation means and the edge matching degree evaluation means, and feature point tracking means using the edge matching degree evaluation means and the contour deformation degree evaluation means, at least, The feature point tracking device according to claim 1.