JPH0771940A - Three-dimensional information input method and three-dimensional information input device using the same - Google Patents

Abstract

(57) [Summary] [Purpose] To efficiently determine three-dimensional information. [Structure] The object 19 is formed by at least three or more cameras.
One of the images obtained by capturing the image is used as a reference image, and a reference that passes through a feature point in the reference image corresponding to one focused three-dimensional feature point on the object and the lens center of the camera that obtained the image Define backprojection line. A reference backprojection line passing through each feature point on the epipolar line obtained by projecting the standard backprojection line on the image by each of the other cameras and the lens center of the camera is obtained. The coordinate position of the intersection of the standard back projection line and the reference projection line is obtained, and the number of intersections of the reference back projection line at each intersection is counted. The intersection point with the maximum number of intersections is determined as the position of the three-dimensional feature point. The same process is performed for all the feature points on the reference image to determine the positions of the three-dimensional feature points of the corresponding object.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to measuring the position, shape and movement of a three-dimensional moving body, and is useful for inputting three-dimensional information of a moving body, recognition and expression (CG), and other three-dimensional information input. The present invention relates to a method and a three-dimensional information input device using the method.

[0002]

2. Description of the Related Art Conventional three-dimensional information input techniques can be classified into three types. The first technique is stereo vision.
This is a scene corresponding to a pair of left and right images captured by two cameras by associating points and lines on the images with each other from corresponding point / line pairs of the camera position and the left and right images. This is a technique for estimating the position of a point or line in space. Second
The method of is a restoration method from motion. This is a technique for estimating the position of a point in the scene space corresponding to each feature point by tracking each feature point on the image in a continuous image taken while moving the camera. The third method is the back projection method. This is a technique for restoring the three-dimensional structure of the scene space by back-projecting the image onto the scene space.

The first method is (eg: Kanade T. and Okutom
i M. and Nakahara T .: “A multiplebaseline stereo me
thod ”, Proc. Image Understanding Workshop, pp. 409-
426, 1992, or U.S. Pat.No. 4,654,872), it is very difficult to accurately locate the feature points between the left and right images because it is easy to create a large number of occluded parts for the line of sight of either camera due to the unevenness of the object. It is difficult to obtain good three-dimensional information. The second method is (ex: Bolles RC,
Baker HH and Marimont DH: “Epipolar-plane im
age analysis: an apprproach to determining structu
re from motion ”, IJCV, Vol.1, No.1, pp.7-55, 1987),
Not applicable to moving objects, because the object must not move while taking images while moving the camera. Recently, a method that can simultaneously extract the three-dimensional shape and motion of an object from a series of images taken by a single camera was announced (Tomasi C. and Ka.
nade T .: “Shape and motion from image streams unde
rorthography: a factorization method ”, IJCV, Vol.9,
No.2, pp. 137-154, 1992), this method basically obtains three-dimensional information of feature points by tracking feature points between images. Very vulnerable to partial occlusion. Because such a blockage causes the feature point of interest on the image to disappear or appear, the trace of the feature point between the images is interrupted, and tracking becomes difficult. Therefore, this method is difficult to adapt to the animal body. The third method is the silhouette projection method (eg Ahuja N. an
d Veenstra J .: “Generation octree from object silh
ouettes in orthographic views ”, IEEE Trans.PAMI, V
ol. 11, No.2, pp.137-149,1989), it is extremely difficult and unstable to generate silhouette images, so it is difficult to acquire accurate three-dimensional information.

[0004]

On the other hand, as another example of the third method, an edge on an image is extracted, and using this, the edge of a three-dimensional object is restored by voting in the scene space. There is a method (Hamano T. and Yasuno
T. and Ishii K .: “Direct estimation of structure f
rom non-linear motion by voting algorithm with-out
tracking and matching ”, Proc. of ICPR, Vol.1, pp.
505-508, 1992, and S. Kawato: “3D ShapeRecovering
by Octree Voting Technique ”, PROCEEDINGS of SPLE
-The International Society for Optical Engineerin
g, 15-16 Nov. 1992), since such a method tries to extract a plurality of feature points at the same time, the processing for each feature point interferes with each other, which causes a false feature point to be extracted. is there. To solve the problem, a large number of images are needed. In the case of a moving object, it takes time to take a large number of images with one camera, and it is very expensive to construct a simultaneous photographing system with a large number of cameras.

An object of the present invention is to solve the above-mentioned problems and to obtain a three-dimensional object information input method capable of acquiring three-dimensional object information from an object image taken by a small number of cameras, and a method using the same.
It is to provide a dimensional information input device.

[0006]

The three-dimensional information input method according to the first aspect of the present invention is (a) n frames, n is an integer of 3 or more, and images of an object are picked up by n frames, respectively. And output those images as image data,
(b) Given the image data, extract feature points of the object on the image of the n frames, respectively, and (c) n
One of the cameras is selected as a reference camera, the image generated by the reference camera is used as a reference image, the other cameras are used as reference cameras, and the images generated by them are used as reference images. With the lens center,
A backprojection line that passes through the feature points in the reference image selected corresponding to the three-dimensional feature point of interest on the object is defined as a reference backprojection line, and (d) the above-mentioned reference image is obtained on the reference image by the reference camera. An epipolar line is defined by projecting a standard backprojection line, and (e) a backprojection line that passes through the feature point on the epipolar line on the reference image by each of the reference cameras and the lens center of the reference camera. Is defined as the reference backprojection line, and (f) the coordinate positions of the intersections of the standard backprojection line and the reference backprojection line are obtained, respectively, and the intersection of the standard backprojection line and the reference backprojection line at each intersection. Count the number, determine the intersection of the maximum count value as the position of the three-dimensional feature point of interest, and (g) attach the series of steps (c) to (f) to each feature point on the reference image. By repeating the above, Obtaining a position of the three-dimensional feature point, respectively as three-dimensional information of the object.

The three-dimensional information input method according to the second aspect of the present invention is as follows: (a) n objects, n is an integer of 3 or more, images the objects respectively to obtain n frame images, and The image is output as image data, (b) the image data is given, and the feature points of the object on the image of the n frames are extracted, respectively, and (c) 1 of the n cameras is selected.
One as a reference camera, the image generated by the reference camera as a reference image, another camera as a reference camera, the image generated by them as a reference image, the lens center of the reference camera, and the object The backprojection line passing through the feature points in the reference image selected corresponding to the above-mentioned three-dimensional feature point of interest is defined as the reference backprojection line, and (d)
Projecting the standard backprojection line on the reference image by the reference camera to define an epipolar line, respectively, (e) the feature point on the epipolar line on the reference image by each of the reference cameras, and the reference The backprojection line that passes through the center of the camera lens is defined as the reference backprojection line, and (f) the coordinate positions of the intersections of the reference backprojection line and the reference backprojection line are obtained, respectively, along the reference backprojection line. Filter processing for emphasizing the density of the intersection distribution is performed on the intersection distribution by convolution processing, and the position at which the filtered intersection distribution is maximized is determined to be the position of the noted three-dimensional feature point, (g) A series of the above steps (c) to (f)
By repeating for each feature point on the reference image, the position of each three-dimensional feature point on the object is obtained as the three-dimensional information of the object.

A three-dimensional information input device according to a third aspect of the present invention is arranged at mutually different positions and outputs an image obtained by picking up an image of an object as image data.
And n is an integer greater than or equal to 3, and image data from the n camera means are given, and image information processing means for obtaining three-dimensional information of the object from feature points in each image, The image information processing means includes: a feature extraction means for extracting a feature point in the image, one of the n cameras as a reference camera, and an image generated by the reference camera as a reference image, Other cameras are used as reference cameras, images generated by them are used as reference images, and the lens center of the standard camera and the feature points in the standard image selected corresponding to the three-dimensional feature point of interest on the object are described. A standard backprojection line generating means for calculating a backprojection line passing through the reference backprojection line and an epipolar line for projecting the standard backprojection line on the reference image by the reference camera to determine an epipolar line. Generating means, reference backprojection line generating means for obtaining a backprojection line passing through the feature point on the epipolar line on the reference image by each of the reference cameras and the lens center of the reference camera as a reference backprojection line, ,
The coordinate positions of the intersections of the standard backprojection line and the reference backprojection line are obtained, and the number of intersections of the standard backprojection line and the reference backprojection line at each intersection is counted, and the maximum intersection of the count values is calculated. 3D feature extraction means for determining the position of the 3D feature point of interest, and for each feature point on the reference image, the position of the corresponding 3D feature point on the object is determined. Obtained as 3D information of the object.

A three-dimensional information input device according to a fourth aspect of the present invention is arranged at mutually different positions and outputs an image obtained by picking up an image of an object as image data.
And n is an integer greater than or equal to 3, and image data from the n camera means are given, and image information processing means for obtaining three-dimensional information of the object from feature points in each image, The image information processing means includes: a feature extraction means for extracting a feature point in the image, one of the n cameras as a reference camera, and an image generated by the reference camera as a reference image, Other cameras are used as reference cameras, images generated by them are used as reference images, and the lens center of the standard camera and the feature points in the standard image selected corresponding to the three-dimensional feature point of interest on the object are described. A standard backprojection line generating means for calculating a backprojection line passing through the reference backprojection line and an epipolar line for projecting the standard backprojection line on the reference image by the reference camera to determine an epipolar line. Generating means, reference backprojection line generating means for obtaining a backprojection line passing through the feature point on the epipolar line on the reference image by each of the reference cameras and the lens center of the reference camera as a reference backprojection line, ,
Obtaining the coordinate positions of the intersections of the standard backprojection line and the reference backprojection line, respectively, a filtering process for emphasizing the density of the intersection distribution by convolution processing on the distribution of the intersections along the standard backprojection line. Each of the feature points on the reference image includes a filtering processing unit that performs the filtering, and a three-dimensional feature extracting unit that determines the position where the distribution of the filtered intersections is maximum as the position of the three-dimensional feature point of interest. The position of each corresponding three-dimensional feature point on the object is obtained as the three-dimensional information of the object.

[0010]

In the method and apparatus for inputting three-dimensional moving object information of the present invention, the backprojection region is significantly narrowed down by incorporating the projective geometric constraint, so that the interference of backprojection is reduced to a minimum, and a small number of images are used. It is possible to acquire three-dimensional information by backprojection of. Therefore, when three-dimensional information of a moving object is obtained by simultaneous photographing with a plurality of cameras, it can be realized with a smaller number of cameras than in the conventional case.

[0011]

Embodiments of the present invention will now be described in detail with reference to the drawings. FIG. 1 is a configuration block diagram of an embodiment of a three-dimensional information input device of the present invention. The three-dimensional information input device includes an image input unit 10 and an image information processing unit 20. Image input unit 10 n (n is an integer of 3 or more)
Image pickup unit (for example, TV camera, hereinafter simply referred to as camera)
111 to 11n, A / D converters 121 to 12n, frame memories 131 to 13n, a camera control unit 14,
It is composed of a movable base 15A, a drive unit 15B, and a drive control unit 16. The cameras 111 to 11n are mounted on a common movable base 15A so that their heights are different from each other, and their relative positions are held. The movable table 15A is the drive controller 1
Under the control of No. 6, it can be moved three-dimensionally by a drive unit 15B composed of a robot arm.

The cameras 111 to 11n simultaneously image the object 19 at a constant cycle (frame cycle), and output analog image signals. If the object is not moving, it is not always necessary to take images at the same time. Analog image signal is A / D
The converters 121 to 12n convert the signals into digital signals, and the frame memories 131 to 1 are converted into image data for each frame.
It is stored in 3n. The operations of the camera, A / D converter, and frame memory are controlled by the camera control unit 14. The image information processing unit 20 includes frame memories 131 to 13
Given the image data read from n,
3 of the object 19 by performing the processing described with reference to
Obtain dimensional information.

FIG. 2 is a functional block diagram showing the processing procedure of the image information processing unit 20 composed of a computer. Camera 1
The position / orientation information of the camera at the time of inputting the image of the object 19 by 11 to 11n is transferred from the drive controller 16 to the image information processor 20.
Is input to the camera information input unit 21 of. On the other hand, the image data read from the frame memories 131 to 13n is supplied to the feature extraction unit 22. The feature extraction unit 22 extracts feature points (for example, iso-concentration points / regions of a normal image, isothermal points / regions of a thermal image, etc.) from an n-frame image input from the image input unit 10. The reference back-projection line generation unit 23 selects any one image from the n-frame images from which the feature points are extracted as a reference image, and the feature points on the reference image and the reference input by the camera information input unit 21. A reference backprojection line passing through the lens center of the camera that generated the image (referred to as a reference camera) is obtained. The epipolar line generation unit 24 respectively projects the reference backprojection lines generated by the reference backprojection line generation unit 23 onto an image (referred to as reference image) plane generated by a camera (referred to as a reference camera) other than the reference camera. Obtain each of the obtained epipolar lines. The reference backprojection line generation unit 25 uses the camera information input unit 21 to detect each feature point existing on the epipolar line on each reference image generated by the epipolar line generation unit 24 or within a certain distance range on both sides of the epipolar line. All reference backprojection lines passing through the input camera and the lens center of the reference camera corresponding to those epipolar lines are obtained. The intersection extraction unit 26 obtains the coordinates of all intersections of the reference backprojection line generated by the reference backprojection line generation unit 23 and the reference backprojection line generated by the reference backprojection line generation unit 25. The intersection counting unit 27 counts the number of intersections at the intersections or the intersection regions between each reference backprojection line extracted by the intersection extracting unit 26 and the reference backprojection line along the reference backprojection line. Filtering unit 2
Reference numeral 8 filters the cross count distribution with the reference back projection line along the standard back projection line counted by the cross counting unit 27 along the standard back projection line. The three-dimensional feature extraction unit 29 determines the intersection closest to the peak position where the filtered intersection distribution exceeds a certain threshold as the position of the three-dimensional feature point. The 3D motion extraction unit 2A obtains the movement of each 3D feature point / region based on each 3D feature point / region information at each time extracted by the 3D feature extraction unit 29.

Next, the operation of the apparatus of the present invention thus constructed will be described with reference to FIGS. FIG. 3 shows the arrangement of the cameras 111 to 11n and an inverted image formed on each focal plane by each camera, for convenience sake, as an orthoimage image 32 by back-projecting it to the front of the lens. First, the camera control unit 14 simultaneously operates the n cameras 111 to 11n from the input time t, and converts the obtained analog image signal into a digital image signal by the n A / D converters 121 to 12n. Frame memory 1
Image data of one frame is obtained in each memory by taking in 3 1 to 13 n. The camera information input unit 21 obtains camera information at time t by calibration (for example, J. Weng and P. Cohen and M. Herniou: "Calibration o.
f stereocameras using a non-linear distortion mode
l ”, Proc. ICCV, pp.246-253, 1990).

The feature extraction unit 22 extracts feature points on each image 32 shown in FIG. 3 (for example, isodensity points / regions of images, contour lines / regions of differential images, isothermal points / regions of thermal images, etc.). . Here, for example, Canny's filter (Canny JF:
“A Computational approachto edge detection”, IEE
E Trans. PAMI, Vol.8, No.6, pp.679-698, 1986) extracts the edge of the image and sets it as a feature point. Next, the reference back-projection line generation unit 23 selects an arbitrary one-frame image 32 as the reference image 32s, and one feature point 33s on the reference image 32s and the image input by the camera information input unit 21 are selected. A backprojection line 34s passing through the center 31s of the lens of the corresponding camera (reference camera) is obtained.

The coordinates (x _b ,
y _b , z _b ) is the equation

[0017]

[Equation 1] To be satisfied. Here, (x _p , y _p , z _p ) and (x _c , y _c , z _c ) are the coordinates of the feature point of interest on the image 32s and the lens center 31s of the reference camera, respectively. Such back projection line 34s is used as a reference back projection line. An arbitrary point E = (x on the epipolar line 35 obtained by projecting the standard backprojection line onto the image of the reference camera
_e , y _e ),

[0018]

[Equation 2] There is a relational expression such as. Here, R ^{T ′} and T ′ are two-dimensional representations of a transposed matrix defining a rotation of a camera optical axis with respect to a coordinate system and a vector defining a parallel movement, respectively. B ′ = (x _b , y _b ) ^T is a two-dimensional representation of an arbitrary point on the back projection line. As is apparent from FIG. 3, the reference back projection line 34
s lies in the plane containing the center 31 of each reference camera lens and the epipolar line 35 from that reference camera. This plane is called the epipolar plane by the camera. All three-dimensional feature points on the object 19 existing in the extension plane of the epipolar plane of each camera are projected on the epipolar line 35 of that camera. Next, the reference backprojection line generation unit 25 generates all the feature points 33e on the epipolar line 35 thus obtained.
Find out. These feature points 33 on the epipolar line 35
One of e corresponds to the feature point 33s of interest. In reality, since there is a quantization error in the image, the epipolar line 3
The feature points that should be above 5 may deviate from the epipolar line. Therefore, the feature points existing within a certain distance from the epipolar line are defined as the feature points 33e of the epipolar line 35.
To consider. Next, a backprojection line 34 that passes through each feature point 33e on the epipolar line 35 of the image 32 by each reference camera and the center 31 of the lens of that reference camera is obtained, and such backprojection line 34 is referred to by backprojection. Let it be a line. Next, the intersection extraction unit 26 determines the coordinates of the intersections 36 of the reference backprojection line 34s generated by the reference backprojection line generation unit 23 and the reference backprojection line 34 generated by the reference backprojection line generation unit 25, respectively. Ask. At this time, there are many cases where the standard back projection line 34s and the reference back projection line 34 do not actually intersect in the three-dimensional space because of the feature extraction error and the camera lens center error on the image 32. In this embodiment, the reference backprojection line 3
4s and each reference backprojection line 34, the shortest distance between them is calculated.
It is assumed that 4s intersects with the reference backprojection line 34.

That is, if two arbitrary points B _b1 = (x _b1 , y _b1 , z _b1 ) ^T and B _b2 = (x _b2 , y _b2 , z _b2 ) ^T on the reference backprojection line 34 s, the reference inverse A matrix composed of two arbitrary points on the projection line B _r1 = (x _r1 , y _r1 , z _r1 ) ^T and B _r2 = (x _r2 , y _r2 , z _r2 ) ^T

[0020]

[Equation 3] If the rank of is 1, then these two lines are not parallel in space. In this case, two points (x _bs , y _bs , z _bs ) ^T and (x _rs , y _rs , z which are on the standard back projection line and the reference back projection line and give the shortest distance between the two lines, respectively. _rs ) ^T

[0021]

[Equation 4] Given in. _{Here, K b = (G r H} -F r G b) / (F b F r -H 2) K r = (F b G r -G b H) / (F b F r -H 2) F _b = (x _b2 -x _b1 ) ² + (y _b2 -y _b1 ) ² + (z _b2 -z _b1 ) ² F _r = (x _r2 -x _r1 ) ² + (y _r2 -y _r1 ) ² + ( z _r2 -z _r1 ) ² G _b = (x _b2 -x _b1 ) (x _b1 -x _r1 ) + (y _b2 -y _b1 ) (y _b1 -y _r1 ) + (z _b2 -z
_b1 ) (z _b1 -z _r1 ) G _r = (x _r2 -x _r1 ) (x _r1 -x _b1 ) + (y _r2 -y _r1 ) (y _r1 -y _b1 ) + (z _r2 -z
_r1 ) (z _r1 -z _b1 ) H = (x _b2 -x _b1 ) (x _r1 -x _r2 ) + (y _b2 -y _b1 ) (y _r1 -y _r2 ) + (z _b2 -z
_b1 ) (z _r1 -z _r2 ) If the distance d between these two points is the threshold λ
If the condition d = [(x _bs -x _rs ) ² + (y _bs -y _rs ) ² + (z _bs -z _rs ) ² ] ^1/2 <λ (6) is satisfied, these 2 It is considered that the standard backprojection line 34s and the reference backprojection line 34 existing at each point intersect, and the coordinates of the position of the intersection point 36 are the midpoint of the distance between these two points x = (x _bs + x _rs ) / 2 y = ( y _bs + y _rs ) / 2 z = (z _bs + z _rs ) / 2 (7). Next, the intersection counting unit 27 counts the number of intersections with the reference backprojection lines 34 from all the reference cameras at each position on the standard backprojection line 34s.

By the way, in the case of using images photographed by two (n = 2) cameras, a plurality of reference backprojection lines 34 with respect to one standard backprojection line 34s are provided at a plurality of different positions. It is possible for the projection line 34s to intersect. In that case, it is impossible to uniquely determine which intersection the three-dimensional feature point corresponding to the feature point 33s through which the reference backprojection line 34s passes corresponds to. Therefore, we consider as follows.

For the sake of simplicity, the number of characteristic points on the epipolar line of each of the other cameras defined by the straight line 34s back-projected from one camera to the characteristic point of interest is set equal to M for simplicity. Only one of the M feature points on the epipolar line on the image by each camera corresponds to the feature point of interest, or none of them. Here, it is assumed that one of the M feature points on each epipolar line corresponds to the feature point of interest. Therefore, M reference backprojection lines are obtained by one reference camera, and the probability β that any one of M intersections of these M reference backprojection lines and the standard backprojection line is the feature point of interest is β = 1 / M. The same applies to other reference cameras. Therefore, the reference back projection line 34
The probability α that one reference backprojection line from each reference camera intersects at any intersection on s is α = β ^n-1
Becomes When there is only one feature point on the epipolar line of the reference camera (M = 1), that feature point corresponds to the feature point of interest, and the reference backprojection line passing through that feature point and the center of the camera lens is of interest. There is no problem because it intersects with the feature points. The problem is when the number M of reference backprojection lines from each reference camera is 2 or more, so that the value of β = 1 / M from 0 to 0.5 should be considered.

FIG. 4 shows that (n-1) reference backprojection lines from (n-1) reference cameras are on the standard backprojection line for various values of β in the range of 0 <β≤0.5. Probability α at one point of
The calculated value of is shown in the graph. As is clear from this graph, the number n of all cameras (the number of viewpoints) is 6 even when β = 0.5.
If it is above, the probability α that five reference backprojection lines from the reference camera intersect at one point on the standard backprojection line is (0.5) ⁵ = 0.03
And it becomes small enough to be ignored. Even when n = 5 (the number of reference cameras is 4), α is as small as 0.06. That is, the possibility that a plurality of reference backprojection lines from the respective reference cameras accidentally intersect at the feature point of interest decreases rapidly with the number n of cameras. On the contrary, if there are a plurality of reference backprojection lines (for example, four or more) that actually intersect at a certain point on the standard backprojection line, it is very likely that the intersection is a feature point of interest. I mean. Therefore, as described above, the intersection counting unit 27 counts the number of reference backprojection lines that intersect at each point on the standard backprojection line 34s (simply called the number of intersections), and the three-dimensional feature point extraction unit 29 determines the maximum number. The position of the three-dimensional feature point of interest can be uniquely determined by obtaining the position given the number of intersections. In this way, any three-dimensional feature point on the object 19 can be associated with one intersection on the reference backprojection line.

From the above, although the value of n is selected to be 3 or more according to the principle of the present invention, in one preferred embodiment, 6 different viewpoints, that is, 6 cameras are used. With this configuration, when the object 1 is observed from a certain camera,
The three-dimensional feature points in the part that is not observed due to the occlusion of 9 itself are more likely to be observed by the other cameras, so that the positions of such three-dimensional feature points can be determined. In the actual measurement, as shown in FIGS. 5A and 5B, the feature points 33 on the image 32 and the camera lens center 3
The back projection line may deviate from the true direction due to the position error of 1. For example, since the image 32 formed by the image data obtained by each camera has been subjected to the quantization processing by the pixels of the image, the position of each feature point on the image includes the quantization error due to the pixel. Therefore, as shown in FIG. 5A, the true feature points 33 on the image 32 by the reference camera are
If the position P of the reference back projection line 34 shifts to P'due to the quantization error, the reference back projection line 34 shifts around the lens center 31 by rotation, and the true feature point B on the reference back projection line 34s becomes B '( It is not always on the reference back projection line). Alternatively, as shown in FIG. 5B, if the lens center measured and determined with respect to the true camera lens center L is deviated to L ', the true three-dimensional feature point B on the reference backprojection line 34s becomes B ". It is judged that there is.

Reference backprojection line 3 due to the influence of these two errors
Due to the deviation of 4, the reference backprojection line 34 passing through the feature point 33 captured in each image from the same 3D feature point on the object and the corresponding camera lens center 31 has this 3D feature point truly present. There is a possibility of crossing out from where you do.
As a result, the deviated points B ′ or B ″ become
If it is within the threshold distance from s, when the number of intersections is counted by the intersection counting unit 27, there is a possibility that it will be counted at an intersection position different from the true 3D feature point B of interest on the reference backprojection line 34s. is there. In such a case, if the position of the intersection having the maximum count value exceeding the predetermined count value is directly extracted by the threshold processing in the three-dimensional feature point extraction unit 29 as the position of the three-dimensional feature point, the three-dimensional feature point of interest is obtained. 3D feature points other than points may be extracted as pseudo 3D feature points. However, since the directions and magnitudes of these backprojection line errors are random, in the actually obtained discrete and spread count distribution, the central portion where the intersections are concentrated has the true three-dimensional focus. It can be presumed that the feature points are most likely to exist. Therefore, filtering is performed along the reference backprojection line 34s for which intersections are obtained by the filtering unit 28, and the intersection count distribution is emphasized where the intersections are concentrated to reduce erroneous extraction of pseudo three-dimensional feature points. Can be done. Examples of filters used include square filter, triangular filter, Gaussian filter, Laplacian-
Gaussian filters, etc. are possible.

To filter the discrete distribution P (s) of the intersection count 51 along the reference backprojection line 34s as shown in FIG. 6A by the function f (t), the following equation P '(s ) = ∫P (st) · f (t) dt (8) The convolution operation may be performed. Here, P (s) and P '(s) are distributions of backprojection line crossing counts before and after convoluting the filter function f (t), respectively, and s represents a position along the reference backprojection line 34s. . As the filter function f (t), various filter functions as described above can be used. In this embodiment, a Laplacian-Gaussian filter (called a ² G filter) 52 having the characteristics shown in FIG. 6B is convoluted with the distribution P (s) of the intersection count 51 along the reference backprojection line 34s. To do. ▽ The ² G filter has the characteristic of emphasizing continuous waveform peaks while suppressing high-frequency discrete noise. Therefore, this convolution process suppresses discrete crossing counts and concentrates crossing counts. Peaks are highlighted as shown in Figure 6C. Therefore, next, the three-dimensional feature extraction unit 29 detects the intersection count peak 51p on the reference back projection line 34s, and determines that the intersection closest to the peak position is the position of the three-dimensional feature point of interest. It is possible to prevent erroneous extraction of pseudo three-dimensional feature points.

As described above, the intersections with the reference backprojection line 34 from each reference camera are defined on the standard backprojection line 34s, and the number of intersections at each intersection is counted. As described above, because of the lens center error or the pixel quantization error of the feature point, the intersection of the reference backprojection line 34 from each reference camera and the reference backprojection line 34s corresponding to the true three-dimensional feature point is strictly speaking. Not 1 point. Therefore, in order to count the number of intersections at each intersection, the reference backprojection line 34s is divided into unit lengths in the length direction with a predetermined fixed minute length Δs as a unit length, and each unit length Δs is divided. You only have to count the number of intersections. In this case, the smaller the unit length Δs is selected, the higher the resolution with respect to the intersection is. Therefore, the unit length Δ of the same position on the reference backprojection line 34s is increased.
The number of intersections contained in s will be reduced. If the unit length Δs is reduced to such an extent that all the intersections are separated, the maximum number of intersections at each intersection becomes 1, and it becomes impossible to uniquely identify the three-dimensional feature point of interest. However, even in this case, for example, as shown in FIG. 6A, the intersections are close to the three-dimensional feature point of interest.
If the convolution processing by the above-described filter is performed on the discrete distribution of the intersections, a peak is obtained in the center of the dense area of the intersections, and the intersection closest to the position of the peak is identified by the true 3
The position of the dimensional feature point can be determined.

All other feature points 33 on the reference image 32s
By repeating the above process for each of s,
The coordinates of all the three-dimensional feature points 36t corresponding to all the feature points 33s on the reference image 32s can be determined.
At that time, since backprojection lines that intersect each other at the position determined to correspond to the focused 3D feature point are associated with only the focused 3D feature point, immediately before the other 3D feature points are obtained, By removing the feature points 33, 33s on the images 32, 32s corresponding to the three-dimensional feature points whose coordinates are determined from the respective images, it is possible to reduce the interference of backprojection and the amount of calculation. Further, by selecting another image as a reference image and repeating the above processing until all the images are selected, the positions of the three-dimensional feature points corresponding to the feature points on the images obtained from all the viewpoints can be obtained. it can.

Finally, the three-dimensional motion extraction unit 2A extracts the three-dimensional motion information of the moving object by obtaining the temporal change of the three-dimensional feature point information obtained from the three-dimensional feature extraction unit 29. As a concrete method of this, a two-dimensional motion information extraction method (for example, R. Agarwal and J. Sklansky: Optical flo
w estimation by the clustering of constraint sets
in velocity space, Technical Report TP-91-2, Departm
ent of Electrical and Computer Engineering, Patter
n Recognition and Image Modeling Project, Univers
City of Irvine, Irvine, CA, USA) can be realized by a method that is expanded in three dimensions.

Further, the motion of the three-dimensional feature points extracted by the three-dimensional motion extracting unit 2A is given to the drive control unit 16 of the image input unit 10, the next viewpoint is planned, the drive unit 15 is operated, and a plurality of drive points are operated. Simultaneously with moving the image pickup units 111 to 11n, the movement amount of the driving unit 15 at the input time t + w is measured and given to the camera information input unit. By repeating such operations, it is possible to trace the moving body.

[0032]

As described above, the present invention significantly narrows the backprojection region by adopting the projective geometric constraint, so that the interference of backprojection is reduced to the minimum, and it is possible to reduce the number of images in a small number of images. Three-dimensional information can be acquired by back projection. Therefore, it is possible to reconstruct the three-dimensional shape of the moving animal body by capturing images with a small number of cameras simultaneously.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of an embodiment of a three-dimensional information input device according to the present invention.

FIG. 2 is a processing block diagram for explaining details of an image processing unit 20 in FIG.

FIG. 3 is a back projection diagram for explaining the principle of the present invention.

FIG. 4 is a three-dimensional graph for explaining the probability that reference back projection lines intersect at the same point.

5A is a perspective view illustrating a pixel quantization error of a feature point in an image, and FIG. 5B is a perspective view illustrating an error of a lens center.

6A is a diagram showing a cross distribution on a reference backprojection line, FIG. 6B is a diagram showing an example of filter characteristics, and FIG. 6C is a diagram showing an example of a cross distribution subjected to filter processing.

[Explanation of symbols]

 10 image input unit 111 to 11n camera 15A movable base 15B driving device 19 object 31 lens center 32 image 33 feature point 34 back projection line 35 epipolar line 36 intersection

Continuation of the front page (51) Int.Cl. ⁶ Identification number Office reference number FI technical display location G06T 1/00 5/20 9191-5L G06F 15/68 400 A

Claims (22)

[Claims] 1. An object is captured by a plurality of cameras to obtain an image, and three-dimensional feature points on the object are associated with feature points in each of the images to obtain three-dimensional information of the object. A method for inputting dimensional information, including the following steps: (a) n objects, n is an integer of 3 or more, images each object to obtain n frame images, and those images are used as image data. (B) The image data is given, and the feature points of the object on the image of the n frames are extracted, and (c) one of the n cameras is selected as a reference camera, An image generated by a camera is used as a standard image, another camera is used as a reference camera, and an image generated by them is used as a reference image, which corresponds to a lens center of the reference camera and a three-dimensional feature point of interest on the object. Select The backprojection line that passes through the selected feature points in the reference image is defined as the reference backprojection line, and (d) the epipolar line is defined by projecting the reference backprojection line on the reference image by the reference camera. Then, (e) the back projection line passing through the feature point on the epipolar line on the reference image by each of the reference cameras and the lens center of the reference camera is defined as a reference back projection line, (f) Obtaining the coordinate position of the intersection of the standard backprojection line and the reference backprojection line, and counting the number of intersections of the standard backprojection line and the reference backprojection line at each intersection,
By determining the maximum intersection of the count values as the position of the three-dimensional feature point of interest, (g) repeating the series of steps (c) to (f) for each feature point on the reference image, The position of each 3D feature point on the object is obtained as 3D information of the object. 2. An object is imaged by a plurality of cameras to obtain an image, and three-dimensional feature points on the object are associated with feature points in each of the images to obtain three-dimensional information of the object. A method for inputting dimensional information, including the following steps: (a) n objects, n is an integer of 3 or more, images each object to obtain n frame images, and those images are used as image data. (B) The image data is given, and the feature points of the object on the image of the n frames are extracted, and (c) one of the n cameras is selected as a reference camera, An image generated by a camera is used as a standard image, another camera is used as a reference camera, and an image generated by them is used as a reference image, which corresponds to a lens center of the reference camera and a three-dimensional feature point of interest on the object. Select The backprojection line that passes through the selected feature points in the reference image is defined as the reference backprojection line, and (d) the epipolar line is defined by projecting the reference backprojection line on the reference image by the reference camera. Then, (e) the back projection line passing through the feature point on the epipolar line on the reference image by each of the reference cameras and the lens center of the reference camera is defined as a reference back projection line, (f) The coordinate position of the intersection of the standard backprojection line and the reference backprojection line is obtained, and the distribution of the intersections along the standard backprojection line is filtered by convolution processing to emphasize the density of the intersection distribution. , The position where the distribution of the filtered intersections is maximum is determined to be the position of the above-mentioned three-dimensional feature point of interest, and (g) the series of steps (c) to (f) are performed for each feature on the reference image. Repeat for points By returning, the position of each three-dimensional feature point on the object is obtained as the three-dimensional information of the object. 3. The three-dimensional information input method according to claim 1 or 2, wherein the position on the reference backprojection line is defined in the length direction by a unit length that is determined in advance, and the intersection is defined. Is the number of intersections in the unit length of the position on the standard backprojection line through which the reference backprojection line passes. 4. The three-dimensional information input method according to claim 3, wherein any reference backprojection line intersecting with the reference backprojection line has a distance from the reference backprojection line within a predetermined range. It is a reference backprojection line. 5. The three-dimensional information input method according to claim 1, wherein the step of counting the number of intersections is performed by convolution processing the distribution of the intersections along the reference backprojection line with a density of the distributions of the intersections. Is a step of performing a filtering process for emphasizing the above, and determining the position where the distribution of the filtered intersections is the maximum as the position of the above-mentioned three-dimensional feature point of interest. 6. The three-dimensional information input method according to claim 2 or 5, wherein the filter processing is processing by a Laplacian-Gaussian filter. 7. The three-dimensional information input method according to claim 2 or 5, wherein the filtering process is a square filter process. 8. The three-dimensional information input method according to claim 2 or 5, wherein the filter processing is processing by a triangular filter. 9. The three-dimensional information input method according to claim 2 or 5, wherein the filter processing is processing by a Gaussian filter. 10. The three-dimensional information input method according to claim 1 or 2, wherein the step (g) of repeating the series of steps executes the series of steps to determine a feature point corresponding to the three-dimensional feature point of interest. Each time identification is performed, the step of removing the identified feature point from the processing target in the repetition of the above series of steps is included. 11. The three-dimensional information input method according to claim 1 or 2, wherein the number n of the cameras is 5 or more and 8 or less. 12. The three-dimensional information input method according to claim 1 or 2, wherein each time step (g) is completed, one of the reference cameras is set as a new reference camera, and the reference camera is set as a new reference camera. As a camera, a new step (g)
To repeat for all the above cameras. 13. The three-dimensional information input method according to claim 1 or 2, wherein the image pickup of the object by the n number of cameras is simultaneously performed in a constant cycle, and the object 3 is read according to the series of steps. The step of obtaining dimensional information is included. 14. Camera means of n, n being an integer of 3 or more, which are arranged at mutually different positions and output images obtained by imaging an object as image data, respectively. Image information processing means for obtaining three-dimensional information of the object from feature points in each image, the image information processing means: feature extracting means for extracting feature points in the image; And one of the n cameras is the reference camera, the image generated by the reference camera is the reference image, the other cameras are the reference cameras, and the images generated by them are the reference images. A reference backprojection line generation method that obtains a backprojection line that passes through the lens center of the object and the feature point in the reference image selected corresponding to the three-dimensional feature point of interest on the object as a reference backprojection line. And an epipolar line generating means for obtaining an epipolar line by projecting the standard backprojection line on the reference image by the reference camera, and the feature point on the epipolar line on the reference image by each reference camera, , A reference backprojection line generating means for obtaining a backprojection line passing through the lens center of the reference camera as a reference backprojection line, and coordinate positions of intersections of the reference backprojection line and the reference backprojection line, respectively, and Three-dimensional feature extraction means for counting the number of intersections of the reference back-projection line and the reference back-projection line at the intersection, and determining the intersection having the maximum count value as the position of the three-dimensional feature point of interest. For each feature point on the reference image,
A three-dimensional information input device for obtaining positions of corresponding three-dimensional feature points on the object as three-dimensional information of the object. 15. Camera means of n, n being an integer of 3 or more, which are arranged at mutually different positions and output images obtained by imaging an object as image data, respectively. Image information processing means for obtaining three-dimensional information of the object from feature points in each image, the image information processing means: feature extracting means for extracting feature points in the image; And one of the n cameras is the reference camera, the image generated by the reference camera is the reference image, the other cameras are the reference cameras, and the images generated by them are the reference images. A reference backprojection line generation method that obtains a backprojection line that passes through the lens center of the object and the feature point in the reference image selected corresponding to the three-dimensional feature point of interest on the object as a reference backprojection line. And an epipolar line generating means for obtaining an epipolar line by projecting the standard backprojection line on the reference image by the reference camera, and the feature point on the epipolar line on the reference image by each reference camera, , A reference backprojection line generating means for obtaining a backprojection line passing through the lens center of the reference camera as a reference backprojection line, and coordinate positions of intersections of the reference backprojection line and the reference backprojection line, respectively, and the reference The filter processing means for performing a filtering process for emphasizing the density of the distribution of the intersections by the convolution process with respect to the distribution of the intersections along the back projection line, and the position where the distribution of the filtered intersections is maximum is noted above. A three-dimensional feature extraction means for determining the position of the three-dimensional feature point, for each feature point on the reference image, on the object. The position of each of the three-dimensional feature point to respond inputting device for obtaining a three-dimensional information of the object. 16. The three-dimensional information input device according to claim 14, wherein the three-dimensional feature extraction means obtains the density of the intersection distribution by convolution processing on the intersection distribution along the reference backprojection line. A position where the distribution of the filtered intersections is maximized is determined as the position of the above-mentioned three-dimensional feature point of interest, which includes a filter processing unit for emphasizing. 17. The three-dimensional information input device according to claim 15 or 16, wherein the filter processing unit is a Laplacian-Gaussian filter processing unit. 18. The three-dimensional information input device according to claim 15 or 16, wherein the filter processing means is a processing means using a square filter. 19. The three-dimensional information input device according to claim 15 or 16, wherein the filter processing means is a processing means using a triangular filter. 20. The three-dimensional information input device according to claim 15 or 16, wherein the filter processing unit is a Gaussian filter processing unit. 21. The three-dimensional information input device according to claim 14 or 15, wherein the number n of the cameras is 5 or more and 8 or less. 22. The three-dimensional information input device according to claim 14 or 15, wherein the image pickup of the object by the n number of cameras is simultaneously performed in a constant cycle, and image information processing is performed every time image data is obtained. A camera control means is provided for supplying the means.