JPH0949714A - 3D shape extraction device - Google Patents

Abstract

(57) 【Abstract】 PROBLEM TO BE SOLVED: To easily and highly accurately obtain a three-dimensional shape of an object. SOLUTION: Image data # ₁ and # imaged from different viewpoints
_In the image preprocessing unit 61, the image of the body to be measured and the feature point group 2 are extracted. The parameter estimation unit 62 estimates and corrects the parameters from the parameter P (1,2) indicating the displacement of each image data and the feature point group, and the distance distribution extraction unit 63 sets two sets of each measurement object image and the above estimation. Two sets of distance distributions at the overlapping portion of each image are obtained from the obtained parameters. The distance information integration unit 21 integrates the two sets of distance distributions according to one parameter, and the three-dimensional shape extraction unit 65 determines the three-dimensional shape of the measured object from the integrated distance distribution and the parameters at the time of integration. Extract.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a three-dimensional shape extraction device for obtaining three-dimensional information of an object or environment necessary for creating a three-dimensional shape model for CG or CAD from a two-dimensional image.

[0002]

2. Description of the Related Art Conventionally, as a technique for obtaining a three-dimensional shape of an object, for example, Journal of Television Society, Vo1.45,
No. 4 (1991), pp. 453-460. As described in the above document, methods for obtaining the three-dimensional shape of an object are roughly classified into a passive method and an active method. A typical passive method is a stereo image method in which triangulation is performed using two cameras. In this method, a place where the same object is shown is searched from the left and right images, and the three-dimensional position of the subject is measured from the amount of displacement. A typical active method is a light-laser range finder that measures the time it takes to project light, reflect it, and return to it, or project a slit-shaped light pattern on the subject. There is a slit light projection method for measuring a three-dimensional shape from the displacement of the pattern shape shown in FIG.

[0003]

However, in the above-mentioned stereo image method, there is a problem that the apparatus becomes large in size and the positioning accuracy of the two cameras is strict, which makes setting difficult. Moreover, since it is not possible to obtain a three-dimensional shape for a region that is not shown in the image, such as the back surface of an object, it is necessary to perform troublesome setting again. In addition, since an active method irradiates an object with some energy such as light, when the distance to the object is long, it is necessary to increase the intensity of the energy, which is dangerous.

Therefore, a first object of the present invention is to provide a three-dimensional shape extraction device which can accurately obtain the three-dimensional shape of an object by a passive method without making a troublesome setting.

A second object of the present invention is to provide a three-dimensional shape extraction device capable of accurately obtaining the three-dimensional shape of an object even when the positioning accuracy of the camera is poor. A third object of the present invention is to provide a three-dimensional shape extraction device that can more accurately determine the three-dimensional shape of an object using the correspondence information of the images taken by the camera.

[0006]

According to the present invention, an image pickup means for picking up an image of an object to be measured, a detecting means for detecting a change in the position and orientation of the image pickup means, and at least an image pickup position obtained from the image pickup means are provided. Parameter estimating means for estimating an imaging parameter from a plurality of different and partially overlapping images and the output of the detecting means, and the image capturing means and the measured object based on the plurality of images and the output of the parameter estimating means. Distance calculation means for calculating the distance to the object is provided.

[0007]

The change of the position and the posture of the image pickup means is detected by the detection means, and the image pickup parameter is estimated by the parameter estimation means from the images of the plurality of objects to be measured taken at different positions and the output of the detection means. The distance calculation means calculates the distance between the imaging means and the measured object based on the estimated parameters and the above-mentioned images.

[0008]

1 shows the concept and basic configuration of three-dimensional shape extraction in a three-dimensional shape extraction device according to a first embodiment of the present invention. As shown in FIG. 1, a user inputs an image while scanning around the measured object 3 with a camera 2 connected to a computer 1 that performs graphics, CAD, and the like. FIG. 2 shows the basic configuration. The computer 1 includes a processing device 11 for performing calculation and graphics processing, a display device 12 for displaying a CRT, an input device 13 for inputting a command such as a keyboard and a mouse, an IC.
A storage device 14 such as a memory or a hard disk is provided, and a camera-dedicated processing device 5 that directly connects to the camera 2 is installed in the computer 1. The camera-dedicated processing device 5 may be provided in the camera 2. In addition, the internal processing program of the processing device 11 allows the camera-specific processing device 5
You may make it perform the process of.

The three-dimensional shape is input by activating an image input program of the computer 1. At this time, the user first places the measured object 3 on the correction pad 4. Then, when the image input program of the computer 1 is started by command input from the input device 13, a window corresponding to the viewfinder of the camera is displayed on the display device 12 of the computer 1 (hereinafter referred to as a viewfinder window).
Is generated. Then switch on camera 2,
The image captured by the camera 2 is displayed in the window.
While observing the displayed image, the user performs framing so that the measured object 3 is located at the substantially center of the screen,
Press the shutter. Then, the object 3 to be measured is scanned as it is, and the image data of the object 3 to be measured is acquired. By processing the image data by the camera-dedicated processing device 5 and the processing device 11, the 3 of the measured object 3 is processed.
Dimensional data is obtained.

FIG. 3 is a block diagram showing the structure of the camera 2. Reference numeral 21 denotes an imaging lens for imaging the measured object 3, 22
Is an image sensor such as a CCD that captures an image as an electric signal, and 23 is a sample hold circuit that holds the electric signal from the image sensor 22 and outputs it as an image signal s1. The center of the detected image area of the image sensor 22 substantially intersects with the optical axis of the imaging lens 21.
The detection surface of is adjusted so as to be substantially orthogonal to the optical axis of the imaging lens 21. Reference numeral 24 denotes a focus detection device used for auto-focusing of a camera, which projects infrared light from an LED (not shown) onto the object 3 to be measured, receives reflected light returning to the camera by a line sensor (not shown), and receives the light. The focus position is detected by. Reference numeral 25 denotes a lens driving unit such as a pulse motor, which drives the imaging lens 21 in a direction parallel to the optical axis to focus the image. 26 is a lens drive circuit,
The lens drive unit 25 is controlled according to the focus position detection signal from the focus detection device 24, and the lens position signal s2
Is output.

Reference numeral 27 denotes a camera displacement detection device composed of an acceleration sensor (not shown), an angular velocity sensor such as a vibration gyro, and an integrator. The camera displacement detection device 27 detects displacement of the camera 2 in three-dimensional position and orientation and outputs it as a camera displacement signal s3. To do.
The axis of the detected rotation angle of the camera displacement detection device 27 is adjusted so as to substantially match the optical axis of the imaging lens 21 and the horizontal and vertical axes of the image sensor 22. In addition, this 3
The origin of the two coordinate axes is approximately the center position of the camera displacement detection device 27. A shutter 28 outputs a shutter signal (on / off signal) s4. Reference numeral 29 is a camera-dedicated processing device 5 for parallelizing the output signals s1 to s4 from the camera 2.
Interface for sending to.

FIG. 4 is a block diagram showing the configuration of the camera-dedicated processing device 5. Reference numeral 51 denotes an input interface unit, which is an image signal s1 and a lens position signal s sent from the camera 2.
2. The camera displacement signal s3 and the shutter signal s4 are received and sent to each section in the apparatus. 52 is an auto gain control circuit which automatically controls the gain of the image signal s1. Reference numeral 53 is a gamma correction circuit, which performs gradation correction of the output signal of the automatic gain control circuit 52. An A / D converter 54 performs analog-digital conversion on the output signal of the gamma correction circuit 53 and outputs digital image data. A lens information storage unit 55 stores information unique to the imaging lens 21.

Reference numeral 56 denotes a viewpoint position calculation circuit, which uses the lens position signal s2 and the data stored in the lens information storage section 55 to determine the viewpoint position (the image side principal point of the image pickup lens 21 corresponding to the lens position (hereinafter, referred to as the image side principal point). The position from the center of the digital image obtained by the image sensor 22)
The optical arrangement of the image pickup system including the image pickup lens 21 and the image sensor 22 is calculated and output. Reference numeral 57 denotes a viewpoint displacement calculation circuit, which uses the camera displacement signal s3 and the data stored in the lens information storage unit 55 to capture the image pickup lens 21.
And the amount of rotation displacement of the optical axis is calculated and output. Reference numeral 58 denotes an imaging data writing control unit, which calculates the digital image data which is the output signal of the A / D converter 54 and the viewpoint position data which is the output of the viewpoint position calculating circuit 56, the viewpoint and the viewpoint displacement when the shutter signal s4 is on. Control is performed to write the data of the viewpoint movement amount and the optical axis rotation amount output from the circuit 57 to the storage device 14.

Next, the operations of the camera 2 and the camera-dedicated processor 5 when the camera 2 scans the object 3 to be measured and inputs an image will be described. First, when the image input program of the computer 1 is started, the computer 1
The processing device 11 therein generates a finder window and displays it on the display device 12. Then, with the power switch (not shown) of the camera 2 turned on, the image captured by the image pickup lens 21 is passed through the image sensor 22, the sample hold circuit 23, and the interface unit 29 to obtain the image signal s.
1 is sent to the camera-dedicated processing device 5. The image signal s1 is received by the input interface unit 51, and the auto gain control circuit 52, the gamma correction circuit 53,
It is output as digital image data via the A / D converter 54. This digital image data is written in a graphic memory (not shown) of the processing device 11 in the computer 1, and the processing device 11 displays this image in the finder window of the display device 12.

During this time, the focus detection device 24 constantly detects the focus shift amount in the area corresponding to the substantially center of the screen of the image sensor 22, and the lens drive circuit 26 causes the focus detection device 24.
The lens drive unit 25 is controlled according to the signal detected by
The lens driving unit 25 drives the imaging lens 21 to focus the image. At this time, the lens drive circuit 26 sends a signal for controlling the lens drive unit 25 to the camera-dedicated processing device 5 via the interface unit 29 as a lens position signal s2.
The lens position signal s2 sent at this time represents the amount of extension of the imaging lens 21 from the reference position. By the above operation, the image captured by the imaging lens 21 is displayed as it is in the finder window.

Next, the user views the image, performs framing, etc., and presses the shutter 28 when the composition is determined. When the shutter 28 is pressed, the lens drive circuit 26 controls the lens drive unit 25 not to drive the image pickup lens 21, and the focus position of the image pickup lens 21 is fixed.
Further, a shutter signal s4 from the shutter 28 is sent to the camera-dedicated processing device 5 via the interface unit 29.
On the other hand, the camera displacement detection device 27 constantly detects the displacement of the position and orientation of the camera in a state where the power switch (not shown) of the camera 2 is turned on, and the camera displacement signal s3 receives the camera-dedicated processing via the interface unit 29. Sent to the device 5. The lens position signal s2 sent to the camera-dedicated processor 5 is sent to the viewpoint position calculation circuit 56 via the input interface unit 51.

The viewpoint position calculation circuit 56 reads the viewpoint position correction table stored in the lens information storage unit 55 according to the value of the lens position signal s2, and obtains the viewpoint position.
That is, in the viewpoint position correction table stored in the lens information storage unit 55, the image sensor 2 at the extended position of the imaging lens 21 according to the value of the lens position signal s2.
The viewpoint position data from the center of the digital image obtained in 2 is stored. This viewpoint position data is taken by the imaging lens 2
The optical axis direction component f of 1 and the image sensor 2 orthogonal to it
It has three components of two horizontal and vertical components (Δx, Δy). Ideally, f is almost equal to the focal length of the imaging lens 21, and (Δx, Δy) is almost equal to 0.

On the other hand, the camera displacement signal s3 sent to the camera-dedicated processor 5 is sent to the viewpoint displacement calculation circuit 57 via the input interface section 51. The camera displacement signal s3 sent from the camera 2 represents information on the positional displacement and the angular displacement in the coordinate system which can be determined by the three orthogonal coordinate axes having the center of the camera displacement detection device 27 as the origin.
The viewpoint displacement calculation circuit 57 performs matrix calculation using the viewpoint conversion data stored in the lens information storage unit 55, and the image pickup lens 21 whose origin is an intersection of the detection surface of the image sensor 22 and the optical axis of the image pickup lens 21. Of the optical axis and the horizontal and vertical axes of the image sensor 22 in the coordinate system (hereinafter referred to as a viewpoint movement amount) and angular displacement (hereinafter referred to as an optical axis rotation amount) are obtained. That is, by the processing of the viewpoint displacement calculation circuit 57, the displacements of the origins and coordinate axes of the camera displacement detection device 27 and the imaging lens 21 are corrected. Therefore, the lens information storage unit 55 stores, as viewpoint conversion data, a conversion matrix that converts the camera displacement signal s3 detected by the camera displacement detection device 27 into the viewpoint movement amount and the optical axis rotation amount.

Next, when the shutter signal s4 sent to the camera-dedicated processing unit 5 via the input interface unit 51 is in the ON state, the image pickup data writing control unit 58 outputs A /
The digital image data output from the D converter 54, the viewpoint position data output from the viewpoint position calculation circuit 56, the viewpoint movement amount and the optical axis rotation amount data output from the viewpoint displacement calculation circuit 57 are received, and these data are received. The computer 1 is controlled to write to the storage device 14 of the computer 1. This data writing is performed until the shutter signal s4 is turned off. When the shutter signal s4 is turned off, the input processing of the digital image data, the viewpoint position data at the time of input, the viewpoint movement amount, and the optical axis rotation amount data is completed.

In this embodiment, the input process has been described assuming that the image pickup lens 21 is an ideal image forming system. However, for example, if the optical characteristic of the image pickup lens 21 is large, the A / D converter is used. It is more preferable to provide a processing unit that performs distortion correction on the digital image data output from 54 and write the output image data to the storage device 14. In that case, the accuracy deterioration due to the distortion of the imaging lens 21 can be corrected in the subsequent three-dimensional shape extraction processing.

Further, when the light quantity distribution characteristics as the optical characteristics of the image pickup lens 21 are greatly different between the central portion and the peripheral portion of the image, the light quantity correction is performed on the digital image data output from the A / D converter 54. It is more preferable to provide a processing unit and write the output image data to the storage device 14. Furthermore, for digital image data output from the A / D converter 54, for example, JPEG, MPEG
If a processing unit for compressing the image data is provided and the image data is compressed and then written in the storage device 14, the image data is expanded and processed in the subsequent three-dimensional shape extraction process. 14 capacity can be saved.

Next, the processing for extracting the three-dimensional shape of the measured object 3 from the data stored in the storage device 14 by the above-described input processing will be described. The three-dimensional shape is extracted by inputting the three-dimensional shape extraction program of the computer 1 into the input device 1.
It is performed by starting up by command input from 3. FIG. 5 shows a block diagram of the three-dimensional shape extraction processing.
The data used for the processing are N sets of digital image data, viewpoint position data, viewpoint movement amount, and optical axis rotation amount data. Note that in FIG. 5, the viewpoint position data, the viewpoint movement amount, and the optical axis rotation amount data are collectively shown as parameters. The parameter indicating the displacement between the image data # ₁ and the image data # ₂ is shown as P (1,2).

Reference numeral 61 denotes an image preprocessing unit which extracts the object to be measured and the feature point group from the image data. A parameter estimating unit 62 estimates and corrects the parameters from the two sets of feature point groups and parameters output from the image preprocessing unit 61. 63
Is a distance distribution extraction unit, which is an output of the image preprocessing unit 61.
From the pair of measured object images and the parameter output from the parameter estimation unit 62, the distance distribution of the overlapping portion between the two measured object images is obtained. Reference numeral 64 denotes a distance information integration unit, which matches the distance distribution extraction unit 6 according to one of the two sets of parameters output from the parameter estimation unit 62.
The two sets of distance distributions, which are the outputs of 3, are integrated. Reference numeral 65 is a three-dimensional shape extraction unit that extracts a three-dimensional shape from the integrated distance distribution and the parameters at the time of integration and stores it in the storage device 14.

Next, detailed operation of the processing by the above configuration
Will be described. First, the concept of three-dimensional shape input in FIG.
Explain the meaning of images and parameters at the time of input using figures
I will tell. In FIG. 6, time t₁And t₂Turtle at two times
The state of LA 2 is shown. Time t₁, T₂From the perspective
Each O₁, O₂, Coordinate system is X₁Y₁Z₁, X₂Y₂Z
₂And image data#₁Is the time t₁Imaged with
data#₂Is the time t₂Is imaged. Also the parameters
Of these, the viewpoint position data is the same as f (optical axis direction component) in FIG.
(Δx, Δy) (horizontal / vertical direction of the image sensor 22)
Component) and does not change during imaging. (O in FIG. 6_iIs a picture
Also shows the image center) and parameter #₂The amount of viewpoint movement is
6 is a vector indicated by T, and O₁From O₂Move to
The rotation amount of the optical axis is indicated by R in FIG.
System X₁Y₁Z₁To X₂Y₂Z ₂Represents a rotation to. processing
For the operation of, the grid pattern shown in FIG.
Place the ball as the measured object 3 on the correction pad 4
The case where it has been explained.

FIG. 7 shows a block diagram of the image preprocessing unit 61. Reference numeral 611 denotes a measured object region dividing unit, which detects edges of input image data and divides it into small regions, and then performs region integration using features such as brightness level and texture to measure the measured object region and the background pattern. Divide into areas and. And
A measured object image of only the measured object area and a background image of only the background area are generated. FIG. 8 shows an example of the measured object image and the background image.
(B) and (c). FIG. 8A is an input image. 6
Reference numeral 12 is a feature point extraction unit, which is a measured object region dividing unit 611.
The background image which is the output of is differentiated and the peak position is extracted as the feature point coordinates. As for the spatial coordinates of the point on the image sensor 22, the Z coordinate is constant in the coordinate system with the viewpoint as the origin,
As the characteristic point group, the horizontal and vertical directions with the image center of the image sensor 22 as the origin are defined as X and Y coordinates, and the plurality of Xs are defined as
Output a set of Y coordinates.

FIG. 9 shows a block diagram of the parameter estimation unit 62. Reference numeral 621 denotes a corresponding point search unit that associates the feature point groups extracted by the image preprocessing unit 61 from the two sets of image data with the viewpoint position and viewpoint movement stored in the storage device 14 from the camera-dedicated processing device 5. Amount and optical axis rotation amount parameters are used. This operation will be described with reference to FIG. 10 (a) and 10 (b) show images in which the characteristic points are displayed in the image areas, respectively. For example, when searching for a feature point corresponding to the feature point p in (a), first, a corresponding area (area indicated by A in the drawing) in (b) for the point p is calculated. Generally, the feature points of two sets of images taken by a specific camera are associated one-to-one with the viewpoint position, the amount of movement of the viewpoint, the amount of rotation of the optical axis, and the object distance.

This relationship is shown in FIG. The object point P corresponding to the point p is obtained by the viewpoint position f and (Δx, Δy) and the object Zp, and the object point P is coordinate-converted by the viewpoint movement amount T and the optical axis rotation amount R to obtain the viewpoint position f and ( Δx, Δy), object Z
Another image point p'is determined again by p '.
The corresponding point searching unit 621 calculates corresponding points by giving an approximate lower limit value and an upper limit value as the object distance, the viewpoint movement amount, and the optical axis rotation amount, and determines the applicable area A according to the range. Then, the feature points therein are corresponded.
At this time, if there is no feature point in the available area A, there is no corresponding point. If most feature points do not have corresponding points, expand the range of the above lower limit and upper limit. When there are a plurality of feature points, a plurality of feature points are stored, the above processing is performed for all the feature points, and then the matching of the feature points is performed so that the one-to-one correspondence can be performed. Outputs a pair of corresponding points having no contradiction in the corresponding relationship between them.

Reference numeral 622 is a parameter optimization unit. In the parameter optimization unit 622, the viewpoint movement amount T and the optical axis rotation amount R stored in the storage device 14 are used as initial values, and the viewpoint position f and (Δx, Δy), and two sets of outputs of the corresponding point search unit 621 are output. From the pair of corresponding points of the feature points of the image, the viewpoint movement amount T, the optical axis rotation amount R, and the formula of the plane of the background pattern portion which is a plane are estimated. The estimation method will be briefly described below. The image plane coordinate system and the spatial coordinate system of image # ₁ are x ₁ y ₁ ,
X ₁ Y ₁ Z ₁ , the image plane coordinate system and the spatial coordinate system of the image # ₂ are x ₂ y ₂ and X ₂ Y ₂ Z ₂ , respectively, and the viewpoint moving amount T and the optical axis rotation amount R given as initial values are set. Each (Δ
X ₀ , ΔY ₀ , ΔZ ₀ ), (Δα ₀ , Δβ ₀ , Δγ ₀ )
And However, each component of the viewpoint movement amount ΔX ₀ , ΔY ₀ , Δ
Z ₀ is the movement amount in the X ₁ Y ₁ Z ₁ direction, and each component Δα ₀ , Δβ ₀ , and Δγ ₀ of the optical axis rotation amount is X ₁ Y ₁ Z, respectively.
_It represents the rotation angle around the center of _one axis. Further, the n sets of corresponding point pair data are (x _1i , y _1i ): (x _2i , y _2i ), i = 1,
2, ..., N. Here, the formula of the plane of the background pattern part in the coordinate system X ₁ Y ₁ Z ₁ is

[0029]

[Equation 1]

Assume that At this time, the relation between the image plane coordinate system of image # ₁ and the spatial coordinate system is

[0031]

[Equation 2]

The relationship between the image plane coordinate system of image # ₂ and the spatial coordinate system is

[0033]

(Equation 3)

And the spatial coordinate systems X ₁ Y ₁ Z ₁ and X ₂ Y
The relationship of ₂ Z ₂

[0035]

(Equation 4)

It becomes Here, as the initial estimated value (ΔX,
ΔY, ΔZ) = (ΔX ₀ , ΔY ₀ , ΔZ ₀ ), (Δα,
Δβ, Δγ) = (Δα ₀ , Δβ ₀ , Δγ ₀ ) is given by Equation (4)
To the image # ₁ from these equations (1) to (4).
The point (x _1i, y _1i) estimate of a point of the image # ₂ corresponding to the _{(x 2i ', y 2i'} ) leads to the estimated value _{(x 2i ', y 2i'} )
And the parameters (a, b, c) of the equation (1) of the plane are calculated so that the total sum of n sets of squares of the distance between the point and the corresponding point (x _2i , y _2i ) is minimized. Next, the plane equation obtained here,
Substituting the parameters a, b, and c of equation (1) into equation (1),
From these equations (1) to (2), the estimated value (x _2i ′, y _2i ′) of the point of image # ₂ corresponding to the point (x _1i , y _1i ) of image # ₁ is _derived , and the estimated value (X _2i ′, y _2i ′)
And the point (x _2i , y _2i ) corresponding to the parameter (ΔX, Δ)
Y, ΔZ), (Δα, Δβ, Δγ).

These processes are repeated several times to obtain the estimated amount of the viewpoint movement amount T, the optical axis rotation amount R and the formula of the plane of the background pattern portion which is a plane. The parameter estimation unit 62 outputs the viewpoint movement amount and the optical axis rotation amount and the inputted viewpoint position data. In the present embodiment, by providing this parameter estimation unit 62, even if the accuracy of the camera displacement signal output from the camera displacement detection device 27 is poor, the three-dimensional part of the measured object 3 in which the background pattern portion is flat. Since the parameter estimation unit 62 corrects the amount of movement of the viewpoint and the amount of rotation of the optical axis using known shape information, highly accurate output data can be obtained in the subsequent three-dimensional shape extraction processing.

In this embodiment, the viewpoint movement amount and the optical axis rotation amount are explained as the parameters to be estimated. However, for example, when the accuracy of the viewpoint position data f or (Δx, Δy) is bad, The viewpoint position data can be estimated by the same method as the above estimation method. By estimating the parameters representing these optical arrangements, even when the extended position of the image pickup lens 21 cannot be accurately detected as the lens position signal s2 or when the image pickup lens 21 and the image sensor 22 cannot be accurately aligned. , High-precision output data can be obtained by the subsequent three-dimensional shape extraction processing. Further, in the present embodiment, the parameters are estimated using the known information that the background pattern portion is a flat surface, but it goes without saying that the parameters can be similarly estimated even if the background pattern portion has another shape such as a spherical surface.

FIG. 12 shows a block diagram of the distance distribution extraction unit 63. Reference numeral 631 denotes a matching processing unit, which correlates the measured object images extracted by the image preprocessing unit 61 from the two sets of image data by using the parameters of the viewpoint position, the viewpoint movement amount, and the optical axis rotation amount. At this time, the output of the parameter estimation unit 62 is used as the parameters of the amount of movement of the viewpoint and the amount of rotation of the optical axis. Then, the corresponding point search unit 62 of the parameter estimation unit 62
Similar to 1, the corresponding points are calculated by giving the lower limit value and the upper limit value as the object distance, the viewpoint movement amount, and the optical axis rotation amount, respectively, and the corresponding points are determined by the range to obtain the corresponding points. At this time, the lower limit value and the upper limit value of the parameters of the viewpoint movement amount and the optical axis rotation amount are set to be narrower than the lower limit value and the upper limit value of the parameter of the corresponding point searching unit 621.
This is because the parameter used in the matching processing unit 631 is estimated from the association of the feature points, and is therefore more accurate than the parameter used in the corresponding point searching unit 621.

Further, the lower limit value and the upper limit value of the object distance parameter are set to be sufficiently wider than the lower limit value and the upper limit value of the parameter of the corresponding point searching unit 621. This is because the object 3 to be searched for the corresponding points is a plane, whereas the shape of the object to be searched for by the matching processing unit 631 is unknown. At this time, first, a small area of the subject area of image # ₁ is extracted as a template, and the template is positionally offset and moved in parallel to perform template matching with the image data within the range of the applicable area of image # _2. , image # ₁ of the coordinates (x _1, y ₁₎ image # ₂ of coordinates corresponding to the (x _2,
y ₂ ) is detected. This template matching process is performed only for the image data of the subject region of image # ₁ , and is output as a pair of corresponding points in the measured body region.

Reference numeral 632 is a distance calculating unit, which is a viewpoint position, viewpoint moving amount, optical axis rotation amount parameter, and corresponding point searching unit 63.
The distance is calculated by the pair of corresponding points which is the output of 1. That is, the distance Z ₁ at each point in the subject area of the image # ₁ is obtained by the equations (2) to (3). At this time, the distance is obtained from the relational expressions of x and y in the equations (2) and (3), and x
The average of the distance obtained by the relational expression and the distance obtained by the relation of y is averaged. Then, X ₁ and Y ₁ are obtained from this Z ₁ and the equation (2), and the coordinates (X ₁ , Y ₁ , Z ₁ ) are output by the number of corresponding point pairs.

FIG. 13 shows a block diagram of the distance information integration unit 64. Reference numeral 641 is a distance information memory. Here, if the parameters calculated by the parameter estimation unit 62 from the image # ₁ and the image # ₂ are expressed as P (1,2) and the distance distribution calculated by the distance distribution calculation unit 63 is expressed as Z (1,2), Distance distribution Z
When (1, 2) and the distance distribution Z (2, 3) are integrated, the distance distribution Z (1, 2) is stored in the distance information memory 641 as it is.

Reference numeral 642 is a coordinate conversion unit for the distance distribution Z
Parameter (2,3) is set using the parameter P (1,2).
The coordinate system of the data P (1,2) is converted. Ie distance
Coordinates (X which is data of distribution Z (2,3)₂, Y₂, Z
₂) To the coordinate (X₁, Y₁, Z
₁Get) Then, the transformed distance distribution Z (2,3)
Are stored in the distance information memory 641. 643 is a parameter
Data integration unit, which includes parameters P (1, 2) and P (2,
3) is integrated and the parameter P (1,3) is output.
From the formula, the spatial coordinate system, X₁Y₁Z₁And X_ThreeY_ThreeZ_ThreeSeki
The formula is (5). However, the parameters P (1,2),
The viewpoint movement amount of each of P (2, 3) is (ΔX₁, Δ
Y₁, ΔZ₁), (ΔX₂, ΔY₂, ΔZ₂),,optical axis
The rotation amount is (Δα ₁, Δβ₁, Δγ₁), (Δα₂, Δβ
₂, Δγ₂), And.

[0044]

(Equation 5)

Therefore, the parameter integration unit 643 calculates the sum of the second and third terms enclosed by [] in the equation (5) as the viewpoint movement amount, (Δα ₁ + Δα ₂ , Δβ ₁ + Δβ ₂ , Δγ ₁ + Δγ
₂ ) is output as the optical axis rotation amount. Similar processing is performed in integration of other distance information, and when N sets of image data are input, all distance distributions obtained from the input image are integrated in N-2 times of processing.

FIG. 14 shows a block diagram of the three-dimensional shape extraction unit 65. A central point extraction unit 651 obtains a range in which the surface of the measured object 3 is distributed from the integrated distance distribution output from the distance information integration unit 64 at the final stage, and outputs the central point of the range. An object model forming unit 652 uses the viewpoint position data of the integrated distance distribution output from the distance information integrating unit 64 for the integrated distance distribution, and uses the center point output from the center point extracting unit 651 as the origin. Is converted to polar coordinates.

For example, FIG. 15 shows a state of coordinate conversion when a ball is input as the measured object 3. That is,
Spatial coordinates (X ₁ , Y ₁ , and X ₁ Y ₁ Z ₁ coordinate system whose origin is O ₁ representing distance information as shown in FIG.
Z ₁ ) is converted to the spatial coordinates of the XYZ coordinate system with the center point O as the origin by only moving the origin as shown in FIG. 15B, and further converted into the spatial coordinates (γ, θ, φ) of the polar coordinate system. Convert. Then, the data converted into polar coordinates for each mesh-shaped area according to the resolution of the three-dimensional shape designated by the input device 13 is averaged and stored in the storage device 14 as the three-dimensional shape data of the measured object 3. Thus, in this embodiment, 3
Since the distance information is converted into polar coordinates by the three-dimensional shape extraction unit 65 and is output as three-dimensional shape data, the surface shape of the measured object 3 can be efficiently stored.

FIG. 16 shows the concept and basic configuration of three-dimensional shape extraction in the three-dimensional shape extraction device according to the second embodiment of the present invention.

As shown in FIG. 16 (a), the user inputs an image while the camera 7 scans the measured object 3 against the background of a flat surface such as a wall surface. That is, when an image is input outdoors, a flat background pattern is used instead of the correction pad 4 in the first embodiment. The captured image is processed by the camera 2 and stored as distance data in a removable storage medium 8 such as an IC card. Then, as shown in FIG. 16B, the storage medium 8 is detached from the camera 7 and attached to the computer 1. The computer 1 obtains the three-dimensional shape data of the measured object 3 from the distance data by the internal three-dimensional shape extraction program. The camera 7 is the image preprocessing unit 61, the parameter estimation unit 62, among the processes of the camera 2, the camera-dedicated processing device 5, and the processing device 11 of the computer 1 according to the first embodiment.
The computer 1 has the same functions as those of the distance distribution extraction unit 63 and the distance information integration unit 64, and the computer 1 has an internal three-dimensional shape extraction program equivalent to the three-dimensional shape extraction unit 65 of the processing of the processing device 11 of the computer 1. With function.

FIG. 17 is a block diagram showing the structure of the camera 7. 71 is an imaging lens, 72 is an image sensor, 7
Reference numeral 3 is a sample hold circuit, 74 is an automatic gain control circuit, 75 is a gamma correction circuit, and 76 is an A / D converter, and digital image data is formed by the respective units 71 to 76. Reference numeral 77 is a focus detection device, 78 is a lens drive unit, and 79 is a lens drive circuit.
The image is focused by 79. Reference numeral 80 denotes a camera displacement detection device that detects displacement of the position and orientation of the camera. Reference numeral 81 is a shutter. Reference numeral 82 denotes a lens information storage unit that stores information unique to the imaging lens 71. 83
Is a viewpoint position calculation circuit, 84 is a viewpoint displacement calculation circuit,
Parameters (viewpoint position, viewpoint movement amount, optical axis rotation angle) are output.

Reference numeral 90 is a control unit, which controls data such as image data and parameters. An image memory 85 stores digital image data together with parameters. An image preprocessing unit 86 extracts the measured object image and the feature point group from the image data. A parameter estimation unit 87 estimates and corrects the parameters from the two sets of feature point groups and parameters output from the image preprocessing unit 86. Reference numeral 88 is a distance distribution extraction unit, and the measured object 3 overlaps with two sets of measured object images from the two sets of measured object images output from the image preprocessing unit 86 and the parameters output from the parameter estimation unit 87. Find the distance distribution in the part where A distance information integration unit 89 integrates the two sets of distance distributions output from the distance distribution extraction unit 63 in accordance with either one of the two sets of parameters output from the parameter estimation unit 62. The integrated distance distribution and the parameters at the time of integration are stored in the storage medium 8.

The blocks having the same names as those in the first embodiment have the same functions as those in the first embodiment. For example, the inter-image processing unit 86, the parameter estimation unit 87, the distance distribution extraction unit 88, and the distance information integration unit 89 are the image preprocessing unit 61 of FIG. 7, the parameter estimation unit 62 of FIG. 9, and the distance distribution extraction unit 63 of FIG. 12, respectively. , And has the function shown in the distance information integration unit 64 in FIG.

[0053]

As described above, according to the first aspect of the present invention, the change of the position and the posture of the image pickup means is detected by the detection means, and the plurality of images of the object to be measured picked up by the image pickup means are detected. A parameter estimating means estimates an imaging parameter from the output of the detecting means, and a distance between the imaging means and the measured object is calculated from the images of the plurality of measured objects and the output of the parameter estimating means. Therefore, it is possible to easily extract the three-dimensional shape of the measured object without making a troublesome setting. Particularly, by providing the parameter estimating means, the three-dimensional shape of the measured object can be obtained with high accuracy.

According to the second aspect of the present invention, the image pickup parameter includes at least one of a change in the position and orientation of the image pickup means, which is an output of the detection means, and an optical arrangement of the image pickup means. Therefore, even if the positioning accuracy of the camera is poor, the three-dimensional shape of the measured object can be obtained with high accuracy.

According to the third aspect of the present invention, the parameter estimating means determines the correspondence among the plurality of images obtained by the image pickup means in the area where the three-dimensional shape of the measured object is known. Since the imaging parameter is estimated, the three-dimensional shape of the object can be obtained with higher accuracy.

[Brief description of drawings]

FIG. 1 is a configuration diagram showing a concept of a three-dimensional shape extraction device according to a first embodiment of the present invention.

FIG. 2 is a basic configuration diagram of a three-dimensional shape extraction device according to the first embodiment.

FIG. 3 is a block diagram showing a configuration of a camera according to the first embodiment.

FIG. 4 is a block diagram showing a configuration of a camera-dedicated processing device according to the first embodiment.

FIG. 5 is a block diagram showing a three-dimensional shape extraction process according to the first embodiment of the present invention.

FIG. 6 is a configuration diagram showing the concept of three-dimensional shape input.

FIG. 7 is a block diagram of an image preprocessing unit according to the first embodiment.

FIG. 8 is a configuration diagram illustrating subject area division processing.

FIG. 9 is a block diagram of a parameter estimation unit according to the first embodiment.

FIG. 10 is a configuration diagram illustrating corresponding point search processing.

FIG. 11 is a configuration diagram showing a relationship between an image point and an object point.

FIG. 12 is a block diagram of a distance distribution extraction unit according to the first embodiment.

FIG. 13 is a block diagram of a distance information integration unit according to the first embodiment.

FIG. 14 is a block diagram of a three-dimensional shape extraction unit according to the first embodiment.

FIG. 15 is a configuration diagram illustrating coordinate conversion in a three-dimensional shape extraction process.

FIG. 16 is a configuration diagram showing the concept and configuration of a three-dimensional shape extraction device according to a second embodiment of the present invention.

FIG. 17 is a block diagram showing a configuration of a camera according to a second embodiment.

[Explanation of symbols]

 1 Computer 2 Camera 3 Object to be Measured 4 Correction Pad 5 Processing Device for Camera 11 Processing Device 24, 77 Focus Detection Device 27, 80 Camera Displacement Detection Device 56, 83 Viewpoint Position Calculation Circuit 57, 84 Viewpoint Displacement Calculation Circuit 63, 87 Parameters Estimator 65 Three-dimensional shape extractor

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Masakazu Yukazu 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inc. (72) Inventor Nao Kurahashi 3-30-2 Shimomaruko, Ota-ku, Tokyo Kya Non-Incorporated (72) Inventor Motohiro Ishikawa 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inc.

Claims (3)

[Claims] 1. An image pickup unit for picking up an image of an object to be measured, a detection unit for detecting a change in the position and orientation of the image pickup unit, and at least an image pickup position obtained from the image pickup unit is different and a part of an image is overlapped. Parameter estimation means for estimating an imaging parameter from the plurality of images and the output of the detection means, and distance calculation for calculating the distance between the imaging means and the measured object from the plurality of images and the output of the parameter estimation means. And a three-dimensional shape extraction device including means. 2. The image pickup parameter includes at least one of a change in position and orientation of the image pickup means, which is an output of the detection means, and an optical arrangement of the image pickup means. 3D shape extraction device. 3. The parameter estimating means selects 3 of the measured objects from among the plurality of images obtained from the imaging means.
The three-dimensional shape extraction device according to claim 1, wherein the imaging parameter is estimated based on correspondence in a region having a known three-dimensional shape.