JP2000307947A - Image processing apparatus and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To correct the distortion of an image photographed with an image pickup surface and an object surface inclined to each other and also to easily prepare a synthetic image whose direction is corrected by imaging the object surface and correcting distortion due to inclination photographing of an image picked up by an image pickup means on the basis of the direction of the measured object surface. SOLUTION: A preprocessing means 16 performs preprocessing such as amplification and clamp of an analog video signal outputted by a photoelectric element 15 and subsequently, converts the analog video signal into a digital video signal. The recording of an image signal read from a frame memory 7 is performed by such a manner that a signal processing means 3 performs signal processing such as image compression of the image signal and then the image signal is stored in an external storing means 9 through an interface 6. A plane measuring means 10 measured the direction of an object surface to an image processor 1 when respective images are photographed, and an image distortion correcting means 11 corrects image distortion.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to an image processing apparatus and a method for correcting distortion caused by oblique photographing of an image of a subject surface, which is used in a digital still camera or the like.

[0002]

2. Description of the Related Art With the rapid progress of computer networks, business modes have diversified, and the need to quickly obtain important information in all aspects has arisen. Accordingly, there is an increasing demand for simple and highly accurate input of a planar document or photograph using a portable input device. At present, there is an application example in which an image input device such as a digital still camera is used to utilize image information obtained by photographing a subject surface such as a paper document of about A4 size.

However, in this application example, when the image is taken in a state where the imaging surface of the device is not parallel to the paper surface,
There is a problem in that the captured image is distorted due to the perspective effect.

On the other hand, Japanese Patent Application Laid-Open No. Hei 3-94383 discloses a method of assuming a known fixed shape, arranging an input image in the fixed shape, and examining the distortion of the fixed shape.
Find the transformation parameters that will be the shape on the original plane,
A technique for correcting image distortion due to oblique shooting has been disclosed.

In Japanese Patent Application Laid-Open No. 5-101221,
A technique has been disclosed in which an orthogonal lattice is introduced into an object plane, spatial coordinates of each lattice point are obtained, and orthogonal transformation is performed on a coordinate system provided on an imaging surface and projected to correct image distortion.

In Japanese Patent Application Laid-Open No. 9-289600, there are provided angle setting means for inputting an inclination angle between an imaging surface of a camera and a subject surface, and distance measuring means for measuring a distance to a subject.
A technique is known in which a subject is photographed in a state in which the tilt angle is input using an angle setting unit, and the subject is corrected to an image viewed from the front based on the tilt angle and the subject distance detected by the distance measuring unit. It has been disclosed.

In Japanese Patent Application Laid-Open No. 10-79878, a combined image obtained by joining divided images obtained by periodically scanning a subject by scanning an imaging surface by two-dimensional coordinate transformation processing such as affine transformation is the same. Label each component, classify the connected components of the labeled image into straight lines, characters and figures according to their size, evaluate the slope of the classified straight line or character base line, and distance between the circumscribed rectangle of the character and the straight line A still image capturing apparatus that obtains an image with high resolution by rotating the straight line by an amount corresponding to the evaluated inclination and correcting the corrected inclination is disclosed.

[0008]

SUMMARY OF THE INVENTION However, Japanese Patent Laid-Open No. 3-9438
According to the technique disclosed in Japanese Patent Publication No. 3 (1995), it is necessary to dispose an object to be photographed in a fixed figure having a known shape, and a cumbersome operation is required to photograph a sheet.

In the technique disclosed in Japanese Patent Application Laid-Open No. 5-101221, an orthogonal grid is superimposed on a subject plane, and two grid points of each grid point are overlapped.
An operation of manually inputting the dimensional coordinates is required, and it is extremely difficult to photograph the subject surface with a simple operation.

Furthermore, in the technique disclosed in Japanese Patent Application Laid-Open No. 9-289600, it is very difficult to manually input the inclination angle between the imaging surface and the object surface by operating the angle setting means. As a result, it becomes difficult to correct image distortion with high accuracy.

In the above-mentioned application example, there is also a disadvantage that it is extremely difficult to read information on a large area such as a newspaper or information drawn on a panel or a wall with a high resolution by the above-mentioned apparatus.

Therefore, these large subjects are divided and photographed by a portable image input device, and a plurality of obtained images are pasted together to form one composite image to obtain a high-resolution image. There is technology. An example is described in
This is a technique disclosed in Japanese Patent Publication No. -79878.

However, in general, the orientation of the image pickup apparatus with respect to the document when each divided image is picked up is different, so that the inclination of the straight line or the character base line varies depending on the position of the composite image on the screen. For this reason, it is necessary to perform the labeling, the evaluation of the inclination, and the rotation correction everywhere in the composite image, and the processing becomes very complicated.

An object of the present invention is to correct distortion of an image photographed in a state where an imaging surface and a subject surface are inclined with respect to each other, and to easily create a composite image with the distortion corrected from these images. is there.

[0015]

According to a first aspect of the present invention, there is provided an image pickup means for picking up an image of an object surface, a plane measuring means for measuring the orientation of the object surface with reference to the image pickup means, An image processing apparatus comprising: an image distortion correction unit configured to correct distortion caused by oblique imaging of an image captured by the imaging unit based on a direction of a subject surface.

Therefore, even when an arbitrary object plane is photographed at an arbitrary viewpoint and posture, the distortion of the image can be easily corrected by measuring the plane.

According to a second aspect of the present invention, there is provided an image pickup means for picking up an image of a subject from a plurality of viewpoints so that a part of the image is overlapped with the previously picked-up image; And a plane measuring means for measuring a certain image as a reference image, and when an image having a portion overlapping with the reference image is used as a reference image, a plurality of feature points are extracted in the reference image, and in the reference image, Correspondence detection means for detecting a corresponding point indicating the same point as the feature point, and coordinates of the reference image on the reference image based on a relationship between the feature point and the corresponding point detected by the correspondence detection means. An image pasting unit for pasting the images by conversion, and oblique photographing of the image pasted by the image pasting unit based on the orientation of the object plane measured by the plane measuring unit. A synthetic image distortion correction means for correcting only an image processing apparatus which comprises a.

Therefore, by combining a plurality of images obtained by photographing an arbitrary object plane with an arbitrary viewpoint and posture and using the measured orientation of the object plane, coordinate conversion is performed, so that a composite image without distortion can be easily obtained. Can be created.

According to a third aspect of the present invention, there is provided an image pickup means for picking up an object from a plurality of viewpoints so that a part of the image is overlapped with the previously picked up image, and an orientation of the object plane with respect to the image pickup means. Plane measuring means for measuring the image orientation, image distortion correcting means for correcting distortion due to oblique photographing of the image taken by the imaging means based on the measured orientation of the subject surface, and an image of the corrected plurality of images Is used as a reference image, and when an image having a portion overlapping with the reference image is used as a reference image, a plurality of feature points are extracted in the reference image, and the same point as the feature point is indicated in the reference image. Correspondence detection means for detecting a point, and image conversion by transforming the reference image onto the reference image based on a relationship between the feature point and the corresponding point detected by the correspondence detection means. And be bonded image joining means is an image processing apparatus which comprises a.

Therefore, even when an arbitrary object plane is photographed at an arbitrary viewpoint and posture, the distortion of each image can be easily corrected by measuring the plane.

According to a fourth aspect of the present invention, in the image processing apparatus according to any one of the first to third aspects, the plane measuring means also measures a distance of the object plane with respect to the imaging means. The image distortion correcting means and the composite image distortion correcting means correct distortion caused by oblique photographing of an image using an enlargement coefficient proportional to the distance.

Therefore, by correcting the distortion of each image obtained by photographing the object plane using the distance information obtained by measuring the plane, the size of the object image in each distortion corrected image can be made substantially equal. it can.

According to a fifth aspect of the present invention, there is provided the image processing apparatus according to any one of the first to fourth aspects, wherein an image corrected by the image distortion correcting means or the composite image distortion correcting means is displayed on a screen. Is provided.

Therefore, it is possible to correct the distortion and inclination of the subject image in each of the photographed images and the bonded images.

According to a sixth aspect of the present invention, there is provided a method of imaging an object plane by an imaging means, a step of measuring an orientation of the object plane with respect to the imaging means, and a step of measuring the orientation of the measured object plane. Correcting the distortion of the image captured by the image capturing means due to oblique shooting.

Therefore, even when an arbitrary object plane is photographed at an arbitrary viewpoint and posture, the distortion of the image can be easily corrected by measuring the plane.

According to a seventh aspect of the present invention, there is provided a method of imaging a subject from a plurality of viewpoints so that a part of the image is overlapped with an image previously taken by the imaging means, Measuring the orientation of the, and a certain image as a reference image, when an image having a portion overlapping with the reference image is a reference image, while extracting a plurality of feature points in the reference image, in the reference image A step of detecting a corresponding point indicating the same point as the feature point, and, based on a relationship between the feature point and the corresponding point detected in this step, by performing coordinate conversion on the reference image on the reference image, An image bonding step, and a step of correcting distortion due to oblique photographing of the image bonded in the bonding step based on the orientation of the subject surface measured in the measurement step. It is a processing method.

Therefore, by combining a plurality of images obtained by photographing an arbitrary object plane with an arbitrary viewpoint and posture and using the measured orientation of the object plane, coordinate conversion is performed, so that a composite image without distortion can be easily obtained. Can be created.

The invention according to claim 8 is a step of taking an image of a subject from a plurality of viewpoints such that a part of the subject is overlapped with an image previously taken by the image taking means, Measuring the orientation of the subject, correcting the distortion of the image captured by the imaging unit due to oblique shooting based on the measured orientation of the subject surface, and setting a certain image of the corrected plurality of images as a reference image. When an image having a portion overlapping with the reference image is used as a reference image, a plurality of feature points are extracted from the reference image, and a corresponding point indicating the same point as the feature point is detected in the reference image. Performing a coordinate conversion of the reference image onto the reference image based on the relationship between the feature points and the corresponding points detected in this step, and bonding the images together. The image processing method comprising at.

Therefore, even when an arbitrary object plane is photographed at an arbitrary viewpoint and posture, the distortion of each image can be easily corrected by measuring the plane.

According to a ninth aspect of the present invention, in the image processing method according to any one of the sixth to eighth aspects, the measuring step also measures a distance of the object plane with respect to the imaging means. In the correcting step, distortion due to oblique photographing of an image is corrected using an enlargement coefficient proportional to the distance.

Therefore, by correcting the distortion of each image obtained by photographing the object plane using the distance information obtained by measuring the plane, the size of the object image in each distortion corrected image can be made substantially equal. it can.

The invention according to claim 10 is the invention according to claims 6 to 9
The image processing method according to any one of the above, further comprising a step of correcting a tilt on a screen of the image corrected in the correction step.

Therefore, it is possible to correct the distortion and inclination of the subject image in each of the photographed images and the bonded images.

[0035]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [First Embodiment of the Invention] The configuration and operation of an image processing apparatus 1 for correcting an image distortion generated when a subject surface as shown in FIG. 1 is photographed will be described below. FIG. 2 is a block diagram illustrating a configuration of the image processing apparatus 1 according to the first embodiment of the present invention.

That is, the image processing apparatus 1 includes
A signal processing unit 3 for processing an image signal obtained by the imaging unit 2; a memory control unit 4 to which an output of the signal processing unit 3 is supplied; a main control unit 5; and an interface (hereinafter abbreviated as I / F) 6, a frame memory 7 for storing image signals according to a command from the memory control means 4, an I / F 6
A display unit 8 for displaying an image signal via the I / F 6
And an external storage means 9 for reading and writing various signals including an image signal via the CPU. In addition, a plane measuring unit 10 for measuring the orientation of the subject plane based on the imaging unit 2, and an image for correcting distortion due to oblique photographing in the photographed image based on the orientation of the subject plane output from the plane measuring unit 10. It also has a distortion correction means 11.

The details of each block will be described below. The imaging means 2 includes a lens 12, an aperture 13, a shutter 14, a photoelectric conversion element 15, and a preprocessing means 16.
The photoelectric conversion element 15 includes, for example, a charge coupled device (CCD).
vice) is used. Further, the pre-processing means 16 includes an analog signal processing such as a preamplifier and an AGC (auto gain control) or an analog-digital converter (hereinafter abbreviated as an A / D converter).
After performing preprocessing such as amplification and clamping on the analog video signal output from the photoelectric conversion element 15, the analog video signal is converted into a digital video signal.

The signal processing means 3 is composed of a digital signal processor (DSP processor) and the like. The digital video signal obtained from the imaging means 2 is subjected to various kinds of color separation, white balance adjustment, γ correction and the like. Perform image processing. The memory control means 4 stores the processed image signal in the frame memory 7 and reads the image signal stored in the frame memory 7 on the contrary. The frame memory 7 can store at least one image, and generally uses VRAM, SRAM, DRAM, or the like. The recording of the image signal read from the frame memory 7 is performed by the signal processing unit 3.
After the image signal is subjected to signal processing such as image compression in, the image signal is stored in the external storage unit 9 via the I / F 6. The external storage means 9 can use an IC memory card, a magneto-optical disk, or the like.
An image signal may be directly transmitted to a remote recording medium via a network using a DN card. vice versa,
Reading of the image signal recorded in the external storage means 9 is performed according to I
This is performed by transmitting an image signal to the signal processing unit 3 via / F6 and performing image expansion in the signal processing unit 3. On the other hand, the display of the image signal read from the external storage unit 9 and the frame memory 7 is performed by the signal processing unit 3 by performing digital-analog conversion (hereinafter referred to as D / A
After performing signal processing such as conversion) and amplification, the I / F 6
Is transmitted to the display means 8 via the.
The display unit 8 is configured by, for example, a liquid crystal display device installed in a housing of the image processing apparatus 1.

The plane measuring means 10 measures the direction of the object plane with respect to the image processing apparatus 1 when each image is photographed. As an example, the plane measuring means 10 includes a spot light source 21, a light receiving element 22, and a three-dimensional coordinate calculating means 2 as shown in FIG.
3. It is composed of a plane calculating means 34. Spot light source 2
Reference numeral 1 denotes a light source 25 such as a light emitting diode or a semiconductor laser, a scanning mirror 26 such as a polygon mirror, and a scanning mirror 26.
The light source 25 comprises driving means 27 for controlling the movement of the light source.
This is a part for controlling the direction of light so that the generated spot light hits the object surface.
7 to be detected. Light receiving element 22
Is configured by a photoelectric conversion element such as a PSD (position sensitive detector) or a CCD installed at a location whose positional relationship with the spot light source 21 is known, and detects the direction of reflected light returning from the subject surface. The three-dimensional coordinate calculation means 23
An object based on the principle of triangulation based on the principle of triangulation from the direction of the spot light emitted by the spot light source 21, the positional relationship between the spot light source 21 and the light receiving element 22, and the direction of the reflected light detected by the light receiving element 22. 3D coordinates of the surface (X,
Y, Z) are calculated. The plane calculation unit 34 estimates a plane equation using the three-dimensional coordinates of three or more points that are not on the same straight line, calculated by the three-dimensional coordinate calculation unit 23. For example,
The required plane equation is

[0040]

(Equation 1) And the three-dimensional coordinates (X _i , Y _i , Z _i ) of N points (i =
1,..., N), the four parameters (a, b, c, d) of the above plane equation are calculated by the least square method shown by the following equation.

[0041]

(Equation 2) The image distortion correcting means 11 includes three parameters (a, b, c) representing the direction of the object plane output by the plane measuring means 10
Is used to convert the object plane into an image taken from the front, thereby correcting image distortion due to oblique shooting. In the present embodiment, as shown in FIG. 4, a plane parallel to the object plane measured by the plane measuring means 10 is set as a plane to be projected, and an image having an object image distorted toward the plane to be projected is projected. Correct image distortion. The image whose distortion has been corrected in this manner is hereinafter referred to as a distortion corrected image. Hereinafter, a method of creating a distortion-corrected image will be described in detail.

First, the direction of the line of sight corresponding to the image coordinates (x, y) is determined. As shown in FIG. 5, the optical system of the imaging means 2 is as follows: x-axis: positive to the right of the image surface y-axis: positive to the downward direction of the image plane z-axis: optical axis direction; positive to the target Origin 0: optics of the imaging means 2 The center is assumed to be a center projection model with f: focal length.
The coordinate system is called a device coordinate system. Then, the direction of the line of sight corresponding to (x, y) can be represented by the next three-dimensional vector P based on the device coordinate system.

[0043]

(Equation 3) Next, as shown in FIG. 6, a rotation matrix R for converting the z-axis of the apparatus coordinate system into a unit normal vector of the projection target surface is obtained.
In this case, the following relational expression holds.

[0044]

(Equation 4) Although there are many rotation matrices R satisfying the expression (4), here, the rotation matrix R is defined as the following expression.

[0045]

(Equation 5) This corresponds to rotating the device coordinate system in the following order as shown in FIG. (1) Rotate the device coordinate system by β around the y-axis. The coordinate system formed by this rotation is referred to as an x ₁ y ₁ z ₁ coordinate system. (2) the device coordinate system, when this is rotated by α in x ₁ ^{axis, a 2 + b 2 + c} 2 = 1 (c> 0), and (4) - (6)
Using the equations, the rotation angles α and β are derived as follows.

[0046]

(Equation 6) R can be uniquely determined by substituting the rotation angles α and β obtained above into the equation (5).

Finally, the image coordinates (x, y) are obtained by using the three-dimensional vector p, the rotation matrix R, and the three parameters (a, b, c) representing the direction of the object plane, which have been obtained by the above procedure. ) Is transformed into coordinates on the projection surface. That is, FIG.
, A point p that intersects with the projection plane when the three-dimensional vector P corresponding to one point on the image plane is extended is defined as image coordinates after coordinate conversion. Assuming that a three-dimensional vector corresponding to the point p with respect to the apparatus coordinate system is P, P is represented by the following equation.

[0048]

(Equation 7) Here, k is an enlargement coefficient representing a distance from the optical center 0 of the imaging means 2 to the projection surface. Since distortion is corrected by projecting an image toward the projection surface as described above, k represents the size of the distortion-corrected image to be created. Further, P is a vector representing the point p with reference to the apparatus coordinate system at the time of image capturing. By performing coordinate conversion of P using the rotation matrix R as in the following equation, the image processing apparatus 1 is moved to the object plane. Is converted to a three-dimensional vector P 'at the time of being directly opposed to.

[0049]

(Equation 8) Therefore, the x coordinate of the three-dimensional vector P ′ in equation (10), y
By using the coordinates as the image coordinates of the point (x, y) after the coordinate conversion, the image coordinates (x, y) after the correction conversion are obtained. The rearrangement is as follows.

[0050]

(Equation 9) According to the above procedure, the arbitrary coordinate (x, y) on the image plane is converted into (X, Y) by using the equation (11), whereby a distortion-corrected image can be created. The image coordinates (x, y) before transformation corresponding to the image coordinates (X, Y) of the distortion-corrected image are not generally integers.
An image may be created by performing interpolation using an existing method such as the B-spline interpolation method.

Regarding the image processing method as described above,
This will be described with reference to the flowchart of FIG. First, at the same time as imaging the subject surface by the imaging means 2, three parameters representing the orientation of the subject surface at the time of shooting are obtained (step S1). Next, based on the obtained (a, b, c), a rotation matrix R is calculated by the above-described procedure (step S2).
Based on (a, b, c) and the rotation matrix R obtained by the above-described procedure, the coordinate transformation of Expression (11) is performed to create a distortion-corrected image (step S3), and the process ends.

[Second Embodiment of the Invention] In the second to fourth embodiments of the invention described below, as shown in FIG. 9, the image processing apparatus 1 is used to form still images of the same subject plane from different places. The operation in the case where images are taken from K points from the viewpoint 1 to the viewpoint K such that a part of the image overlaps will be described.
Images taken at viewpoint 1 are Im 1, images taken at viewpoint 2 are Im
Let's call it 2 and so on. For the sake of simplicity, it is assumed that there is an overlap area between Imj and Imj + 1 photographed in adjacent order, such as Im1 and Im2, and Im2 and Im3.

FIG. 10 is a configuration diagram of an image processing apparatus 1 according to Embodiment 2 of the present invention. In FIG. 10, FIG.
The members having the same reference numerals as those described above are the same members as in the first embodiment of the present invention, and thus detailed description is omitted.

The image processing apparatus 1 extracts correspondence points between images having mutually overlapping areas and corresponding points, and detects correspondence points between the feature points and corresponding points output by the correspondence detection section 31 based on the relation between the corresponding points and corresponding points. Image combining means 32 for combining images having overlapping regions by coordinate transformation, a combined image storage means 33 for storing a combined image created by the image combining means 32, and the output of the plane measuring means 10 A composite image distortion correcting unit that corrects distortion of the combined image stored in the combined image storage unit 33 due to oblique photographing based on the orientation of the subject surface.

[0055] Correspondence detecting means 31, image combining means 3
2. The configuration and operation of the combined image storage unit 33 and the composite image distortion correction unit 34 will be described below. The correspondence detecting means 31 detects a portion where the same part is photographed in two images having overlapping areas. Here, a method using a correlation operation will be described. As shown in FIG. 11, the correspondence detecting means 31 includes the feature point setting means 3.
5. It is composed of a correlation operation means 36. The previously captured image (reference image) and a reference image having an area overlapping the reference image are stored in the frame memory 7 of FIG. The feature point setting means 35 determines a position of the feature point in the reference image, extracts a shading pattern centered on the feature point, and creates a correlation window. The position of the feature point is a corner (corne
It is determined by extracting a portion where the density pattern of the image is characteristic as in r). The correlation calculating means 36
A portion that substantially matches the shading pattern of the correlation window created in the reference image is detected by a correlation operation in the reference image, and is determined as a corresponding point. Here, an example in which a corresponding point is detected by block matching by a correlation operation will be described. As shown in FIG. 12, (2N + 1) ( 2P + 1) in block matching using the correlation window of, i-th feature point in the reference image and (x _i0, y _i0), the point in the reference image (x _i0 + dx _i, y The cross-correlation value S _i of ( _{i 0} + dy _i ) is calculated by the following equation.

[0056]

(Equation 10) For each feature point, by determining the point the maximum value of the cross-correlation value S _i is the threshold value than the predetermined, the corresponding point in the reference image _{(x i0 + dx i, y} i0 + dy i) is calculated. If the maximum value of S _i is equal to or less than the threshold, it is determined that no corresponding point exists.

The image combining means 32 combines the images by performing coordinate conversion of the reference image into the reference image based on the relationship between the feature points and the corresponding points output from the correspondence detecting means 31. Here, description will be made assuming that projective transformation is used for coordinate transformation. The projective transformation means that, when two object planes are photographed as shown in FIG. 13 and one is used as a reference image and the other is used as a reference image, the object image shown in the reference image as shown in FIG. Refers to conversion to the appearance of When this is expressed by a mathematical expression, a feature point (x
_s, the corresponding point in the y _s) and the reference image (x _r, between y _r), the following relation is established.

[0058]

[Equation 11] In equation (13), since there are eight unknown parameters, if four or more pairs of feature points and corresponding points are obtained, the coefficients b _{1 to} b ₈ can be obtained by a calculation method such as the least square method. In the above procedure, the image was bonded by calculating the coefficients b ₁ ~b _8, it will be referred to as a projection transformation image.

In the above, the method of creating a projection conversion image when photographing from two viewpoints has been described. However, it is possible to sequentially create projection conversion images by iterative processing and paste three or more images together. Hereinafter, an image obtained by projecting and transforming Im 1 to Im j−1 with respect to Im j will be referred to as PIm j (1 ≦ j ≦ K, where PIm 1 = Im 1). In PIm j, Im1 to Imj-1
Are converted into the appearance at the time of photographing Imj, and are pasted to Imj. The iterative process for creating PIm j is described below.

The following processing is repeated for j = 2,..., K. Let PIm j-1 be the reference image and Im j be the reference image. Feature points are extracted from the reference image, and corresponding points are detected from the reference image. Using four or more pairs of feature points and corresponding points detected in
The coefficients b _{1 to} b ₈ in the expression 3) are obtained, and PIm j−1 is projectively transformed to Imj to create PIm j. Increment j by 1, and return to.

By the above iterative process, finally Im 1
An image PIm K obtained by projecting and converting Im K-1 to Im K is obtained.
That is, one bonded image can be created from Im1 to ImK. The combined image thus created is stored in the combined image storage unit 33.

The composite image distortion correction means 34 corrects the distortion in the composite image PImK created by the image composite means 32. Image bonding means 3
As described above, the composite image PIm K created by Step 2 is an image that has been converted to the appearance at the time of photographing the reference image and then laminated, so if the subject image is distorted in the reference image, the composite image is naturally distorted. Contains. Therefore, in FIG. 15, when the reference image (= Im K) is taken, the direction of the subject plane output by the plane
Using the three parameters (a, b, c), the distortion of the bonded image PImK is corrected by the processing of the equations (3) to (11), as in the first embodiment of the invention.

With respect to the above-described image processing method,
This will be described with reference to the flowchart of FIG. First, a subject surface is photographed by the imaging means 2, and this image is stored as a reference image, and at the same time, three parameters (a, b, c) indicating the orientation of the subject surface at the time of photographing are obtained (step S11). Subsequently, the viewpoint and the attitude of the image processing apparatus 1 are changed so that a part thereof overlaps with the reference image, and an image of the object surface is photographed by the imaging means 2, and this image is stored as a reference image. 3 representing the direction of
(A, b, c) (Step S1)
2). A feature point is extracted from the reference image, and a corresponding point indicating the same place as the feature point is detected in the reference image (step S13). Based on the relationship between the feature points and the corresponding points thus obtained, an image is created by performing coordinate conversion of the reference image with respect to the reference image by the method described above (step S1).
Four). After that, if the number of images to be synthesized by taking more images is increased (step S15), the combined image created by the above processing is replaced with the reference image (step S1).
6) The operation from step S2 to step S14 is repeated. If not, copy the last created image
Object surface parameters (a, b, c) at the time of photographing the reference image
Is used to correct the distortion of the bonded image by performing the coordinate transformation of Expression (11) (step S17), and the processing is ended.

[Third Embodiment of the Invention] FIG. 17 is a configuration diagram of an image processing apparatus 1 according to a third embodiment of the present invention. In FIG. 17, members having the same reference numerals as those in FIG. 2 are the same as those in Embodiment 1 of the present invention, and thus detailed description is omitted.

The image processing apparatus 1 is provided with an image distortion correcting means 41 for correcting distortion of each image due to oblique photographing, based on the orientation of the object plane output from the plane measuring means 10.

The configuration and operation will be described focusing on the procedure of processing by the image processing apparatus 1. The plane measuring means 10 measures the direction of the subject plane with respect to the image processing apparatus 1 when each image is taken. The image distortion correction unit 41 outputs the direction (a, b,
Using c), the distortion due to the oblique shooting of each image is corrected in the procedure described above. The correspondence detecting unit 31 detects a portion where the same portion is photographed in two images having regions that are overlapped with each other and the distortion of which is corrected by the image distortion correcting unit 41. For example, in the case where the association is performed using the above-described correlation calculation, in the present embodiment, the correlation between the images corrected for the distortion due to the oblique imaging is obtained, so that the association can be performed more easily and with higher accuracy. Reference image obtained by correcting distortion by the above, the relationship between the reference image and the feature points and corresponding points are attached to each reference image in the reference image in search of the coefficient b ₁ ~b ₈ (13) below.

Regarding the above-described image processing method,
This will be described with reference to the flowchart of FIG. First, the object plane is photographed by the imaging means 2 and this image is stored as a reference image, and at the same time, three parameters (a, b, c) representing the direction of the object plane at the time of photographing are obtained (step
S21), using these parameters (a, b, c),
The distortion in the reference image is corrected by the processing procedure of Expression (11), and the reference image is replaced with the corrected distortion (step S2).
2). Subsequently, the viewpoint and the attitude of the image processing apparatus 1 are changed so that a part of the image is overlapped with the reference image, the object surface is photographed by the image pickup means 2, and this image is stored as a reference image. The three parameters (a, b, c) representing the orientation of the surface are obtained (step S23),
Using the parameters (a, b, c), the distortion in the reference image is corrected by the processing procedures of equations (3) to (11), and the reference image is replaced with the corrected distortion (step S24).
A feature point is extracted from the reference image, and a corresponding point indicating the same location as the feature point is detected in the reference image (step S25). Based on the obtained relationship between the feature points and the corresponding points, an image is created by performing coordinate conversion of the reference image with respect to the reference image by the method described above (step S26). Thereafter, when the number of images to be captured and combined is increased (step S27), the combined image created by the above processing is replaced with the reference image (step S28), and the operations from step S23 to step S26 are repeated. . If not, the process ends.

In this embodiment, since the images obtained by correcting the image distortion due to the oblique photographing are pasted together, the affine transformation of the following equation can be used instead of the projective transformation represented by the equation (13). A laminated image can be obtained.

[0069]

(Equation 12) The image pasting means 32 performs the projective transformation of the expression (13) or (1).
It goes without saying that the present invention is not limited to the configuration and operation based on the affine transformation of the expression 4).

[Fourth Embodiment] In the fourth embodiment of the present invention, the size of the distortion-corrected image created by using the distance to the object plane in the first to third embodiments of the present invention is described. And adjusting the size of each subject image so as to be substantially equal. For example, as shown in FIG. 19A, when the reference image and the reference image are photographed in a state where the distance to the subject surface and the direction of the optical axis (z-axis) of the imaging unit 2 are different, as shown in FIG. In some cases, the size of the subject image in each distortion-corrected image may be different. Therefore, the image distortion correcting unit 41 is configured to adjust the size of the distortion corrected image using the output of the plane measuring unit 10. An example will be described below.

The plane measuring means 10 according to the first embodiment of the present invention
And outputs the parameters (a, b, c, d) representing the equations of the object plane and the three-dimensional coordinates of each point with reference to the apparatus coordinate system. Image distortion correction means 41
Based on the (a, b, c, d) output from the plane measuring means 10 and the three-dimensional coordinates of each point, the distortion of each image is corrected by the procedures of equations (3) to (11), and the equations (11) are used. Calculate the optimal expansion coefficient k used for the coordinate transformation of. For example, in FIG. 19A, the parameters of the object plane output by the plane measuring means 10 at the time of photographing the reference image are (a, b, c, d), and the parameters of the object plane output by the plane measuring means 10 at the time of photographing the reference image. Let the parameters be (a ', b', c ', d'). Reference image,
If the enlargement coefficients used to create the distortion-corrected image of the reference image are k and k ′ (k> 0, k ′> 0),

[0072]

(Equation 13) Then, as shown in FIG. 19 (b), a distortion-corrected image in which the size of each subject image is substantially equal is obtained.

[Fifth Embodiment] In the fifth embodiment of the present invention, in the first to fourth embodiments of the present invention, three parameters (a,
The distortion of each image is corrected by converting the image processing apparatus 1 into an image taken with the image processing apparatus 1 facing the subject plane using b) and c). Means to pseudo-generate an image obtained as a result of performing a coordinate transformation such that is made to match the direction of the object plane. However, this coordinate conversion leaves a degree of freedom around one axis with the normal vector of the subject plane as the rotation axis. That is, depending on the attitude of the image processing apparatus 1 at the time of shooting, a distortion-corrected image in which the subject image is inclined on the image plane is created as shown in FIG. In the present embodiment, it is an object to provide a tilt correction means 51 in the first to fourth embodiments of the present invention and obtain a distortion-corrected image having almost no tilt as shown in FIG.

As shown in FIG. 21, the inclination correcting means 51
It comprises a rotation angle setting means 52, a rotation center setting means 53, and a rotation calculation means 54. The rotation angle setting means 52 and the rotation center setting means 53 are constituted by switches and the like, and are arranged, for example, on the housing of the image processing apparatus 1 at positions adjacent to the display means 8. The rotation angle setting means 52 specifies the rotation angle θ on the screen, and the rotation center setting means 53 specifies the rotation center coordinates (x ₀ , y ₀ ) on the screen. The rotation calculation means 54 is configured by a calculation circuit or the like built in the housing, and has a rotation angle θ.
And the rotation center coordinates (x ₀ , y ₀ ), the inclination of the distortion-corrected image on the screen is corrected by the following coordinate conversion formula.

[0075]

[Equation 14] An image whose inclination has been corrected by applying the coordinate transformation of the expression (16) to the distortion corrected image can be confirmed on the display unit 8.

Needless to say, the description of each embodiment of the present invention is merely an example of the present invention.
The present invention should not be limited or reduced in accordance with the above-described embodiments of the present invention. For example, the configuration of the plane measuring unit 10 is not limited to the range described in the above-described embodiment, but may be a wave (light,
Other configurations such as irradiating the target with radio waves and sound waves and measuring the propagation time and phase difference of the reflected wave from the target to determine the distance to the target, and obtaining the distance from the amount of defocus on the subject surface But it doesn't matter. Further, it has been described that the correspondence detecting means 31 detects the corresponding point by the density matching by the correlation method. However, another method such as a spatio-temporal differentiation method may be used.

[0077]

According to the first aspect of the present invention, even when an arbitrary object plane is photographed at an arbitrary viewpoint and posture, the distortion of the image can be easily corrected by measuring the plane.

According to a second aspect of the present invention, an image obtained by bonding a plurality of images obtained by photographing an arbitrary object plane at an arbitrary viewpoint and posture is coordinate-converted by using the measured direction of the object plane. A composite image without distortion can be easily created.

According to the third aspect of the present invention, even when an arbitrary object plane is photographed at an arbitrary viewpoint and posture, the distortion of each image can be easily corrected by measuring the plane. Further, since the correspondence is detected using the image in which the distortion has been corrected, the accuracy of the association is further improved as compared with the image processing apparatus according to claim 2, and as a result, a higher definition composite image can be easily formed. Obtainable.

According to a fourth aspect of the present invention, there is provided the image processing apparatus according to any one of the first to third aspects, wherein each image of the object surface is photographed using distance information obtained by measuring a plane. By correcting the image distortion, the size of the subject image in each distortion-corrected image can be made substantially equal. In addition, when the invention described in claim 3 is applied to the invention described in claim 3, the correspondence is detected using the distortion correction image in which the size of the subject image is substantially equal, so that the accuracy of the correspondence is further improved, As a result, a higher-definition composite image can be easily obtained.

According to a fifth aspect of the present invention, there is provided the image processing apparatus according to any one of the first to fourth aspects, wherein the distortion and inclination of the subject image in each of the photographed images and the combined images are corrected. be able to. In addition, if the invention described in claim 3 is applied to the invention described in claim 3, the correspondence is detected by using the distortion-corrected image in which the inclination of the subject image is substantially equal, so that the accuracy of the correspondence is further increased, and As a result, a higher-definition composite image can be easily obtained.

According to the sixth aspect of the present invention, even when an arbitrary object plane is photographed at an arbitrary viewpoint and posture, the distortion of the image can be easily corrected by measuring the plane.

According to a seventh aspect of the present invention, an image obtained by pasting a plurality of images obtained by photographing an arbitrary object plane at an arbitrary viewpoint and posture is coordinate-converted by using the measured direction of the object plane. A composite image without distortion can be easily created.

According to the eighth aspect of the invention, even when an arbitrary object plane is photographed at an arbitrary viewpoint and posture, the distortion of each image can be easily corrected by measuring the plane. Further, since the correspondence is detected using the image in which the distortion has been corrected, the accuracy of the association is further improved as compared with the invention according to claim 7, and as a result, a higher-definition composite image can be easily obtained. Can be.

According to a ninth aspect of the present invention, in the image processing method according to any one of the sixth to eighth aspects, each image of the object plane is photographed using distance information obtained by measuring a plane. By correcting the image distortion, the size of the subject image in each distortion-corrected image can be made substantially equal. Further, if the invention described in claim 8 is applied to the invention described in claim 8, the correspondence is detected using a distortion-corrected image in which the size of the subject image is substantially the same, so that the accuracy of the association is further improved, As a result, a higher-definition composite image can be easily obtained.

The invention according to claim 10 is the invention according to claims 6 to 9
In the image processing method according to any one of the above, it is possible to correct the distortion and inclination of the subject image in each of the captured images and the combined image. In addition, when the invention described in claim 8 is applied to the invention described in claim 8, the correspondence is detected using the distortion-corrected image in which the inclination of the subject image is substantially equal, so that the accuracy of the association is further improved, and result,
It is possible to easily obtain a higher-definition composite image.

[Brief description of the drawings]

FIG. 1 is a conceptual diagram illustrating correction of image distortion generated when an image of a subject surface is captured.

FIG. 2 is a block diagram illustrating a configuration of the image processing apparatus according to the first embodiment of the present invention;

FIG. 3 is a block diagram illustrating a configuration of a plane measuring unit of the image processing apparatus.

FIG. 4 is a conceptual diagram illustrating the operation of the image processing apparatus.

FIG. 5 is a conceptual diagram illustrating an operation of the image processing apparatus.

FIG. 6 is a conceptual diagram illustrating an operation of the image processing apparatus.

FIG. 7 is a conceptual diagram illustrating the operation of the image processing apparatus.

FIG. 8 is a flowchart illustrating an operation of the image processing apparatus.

FIG. 9 is a diagram illustrating a still image of the same subject surface from a different place using an image processing device, such that a part of the image is overlapped.
FIG. 8 is a conceptual diagram in a case where shooting is performed from K points from viewpoint 1 to viewpoint K.

FIG. 10 is a block diagram illustrating a configuration of an image processing apparatus according to a second embodiment of the present invention.

FIG. 11 is a block diagram illustrating a configuration of a correspondence detection unit of the image processing apparatus.

FIG. 12 is a conceptual diagram illustrating the operation of the image processing apparatus.

FIG. 13 is a conceptual diagram illustrating the operation of the image processing apparatus.

FIG. 14 is a conceptual diagram illustrating an operation of the image processing apparatus.

FIG. 15 is a conceptual diagram illustrating the operation of the image processing apparatus.

FIG. 16 is a flowchart illustrating an operation of the image processing apparatus.

FIG. 17 is a block diagram illustrating a configuration of an image processing apparatus according to a third embodiment of the present invention.

FIG. 18 is a flowchart illustrating an operation of the image processing apparatus.

FIG. 19 is a conceptual diagram illustrating an operation of the image processing apparatus according to the fourth embodiment of the present invention.

FIG. 20 is a conceptual diagram illustrating an operation of the image processing apparatus according to the fifth embodiment of the present invention;

FIG. 21 is a conceptual diagram illustrating the operation of the image processing apparatus.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Image processing apparatus 2 Imaging means 10 Plane measurement means 11 Image distortion correction means 31 Correspondence detection means 32 Image bonding means 34 Synthetic image distortion correction means 51 Tilt correction means

──────────────────────────────────────────────────の Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) H04N 5/232 G06F 15/66 350A 5/91 H04N 5/91 J F-term (Reference) 5B057 BA02 CD12 CE10 5C021 YC01 YC02 YC03 5C022 AB00 AB28 AB51 AB68 AC69 5C023 AA10 AA31 AA37 AA38 DA04 5C053 FA27 KA03 KA25 LA01 LA06

Claims (10)

[Claims] 1. An image pickup means for picking up an image of an object surface, a plane measuring means for measuring the direction of the object surface based on the image pickup means, and an image picked up by the image pickup means based on the measured direction of the object surface. An image processing apparatus comprising: an image distortion correction unit configured to correct distortion due to oblique imaging of an image. 2. An image pickup means for picking up an image of a subject from a plurality of viewpoints so that a part of the image is overlapped with a previously picked-up image, and a plane measuring means for measuring the direction of the object plane with reference to the image pickup means. When a certain image is used as a reference image and an image having a portion overlapping with the reference image is used as a reference image, a plurality of feature points are extracted in the reference image, and the same location as the feature point is used in the reference image. A correspondence detecting means for detecting a corresponding point indicating the following, and pasting an image by performing coordinate transformation of the reference image on the reference image based on a relationship between the feature point and the corresponding point detected by the correspondence detecting means. Image combining means for combining, and a composite image for correcting distortion caused by oblique photographing of the image combined by the image combining means based on the orientation of the subject surface measured by the plane measuring means. The image processing apparatus includes a distortion correcting unit. 3. An image pickup means for picking up an object from a plurality of viewpoints so that a part of the image is overlapped with a previously picked up image, and a plane measuring means for measuring the direction of the object plane with reference to the image pickup means. Image distortion correction means for correcting distortion due to oblique photographing of the image captured by the imaging means based on the measured orientation of the object surface; and setting a certain image as a reference image among the plurality of corrected images, Correspondence detecting means for extracting a plurality of feature points in the reference image and detecting a corresponding point indicating the same point as the feature point in the reference image when an image having a portion overlapping with the image is used as the reference image. And an image pasting unit that transforms the reference image onto the reference image based on the relationship between the feature points and the corresponding points detected by the correspondence detection unit, and pastes the images together. Image processing apparatus comprising: means combined, the. 4. The plane measuring means also measures a distance of the object plane with respect to the imaging means. The image distortion correcting means and the composite image distortion correcting means
The image processing apparatus according to any one of claims 1 to 3, wherein the distortion caused by oblique photographing of the image is corrected using an enlargement coefficient proportional to the distance. 5. The image processing apparatus according to claim 1, further comprising a tilt correction unit configured to correct a tilt of an image corrected on the screen by the image distortion correction unit or the composite image distortion correction unit. Image processing device. 6. A step of capturing an image of a subject surface by an image capturing means, a step of measuring an orientation of the subject surface with respect to the image capturing means, and an image captured by the image capturing means based on the measured orientation of the subject surface. Correcting the distortion due to oblique shooting of the image;
An image processing method comprising: 7. A step of imaging a subject from a plurality of viewpoints so that a part of the image is overlapped with an image previously captured by an imaging unit; and a step of measuring an orientation of the subject surface with respect to the imaging unit. When an image is used as a reference image and an image having a portion overlapping with the reference image is used as a reference image, a plurality of feature points are extracted from the reference image, and the same location as the feature point is used in the reference image. Detecting the corresponding point indicating, based on the relationship between the feature point and the corresponding point detected in this step, the step of bonding the image by performing coordinate transformation on the reference image on the reference image, Correcting the distortion caused by oblique photographing of the image bonded in the bonding step based on the orientation of the object plane measured in the measuring step. 8. A step of capturing an object from a plurality of viewpoints so that a part of the image is overlapped with an image previously captured by an image capturing unit; and a step of measuring an orientation of the object surface with respect to the image capturing unit. Correcting the distortion caused by oblique imaging of the image captured by the imaging unit based on the measured orientation of the subject surface; and setting a certain image as a reference image among the corrected plurality of images and overlapping the reference image. A step of extracting a plurality of feature points in the reference image and detecting a corresponding point indicating the same place as the feature point in the reference image, in a case where an image having a portion to be used as a reference image, Bonding the images by performing coordinate transformation on the reference image on the reference image based on the relationship between the detected feature points and the corresponding points. 9. The measuring step also measures a distance of the object plane with respect to the imaging unit, and the correcting step uses an enlargement coefficient proportional to the distance to obtain a distortion due to oblique photographing of an image. The image processing method according to any one of claims 6 to 8, wherein the image processing is corrected. 10. The image processing method according to claim 6, further comprising a step of correcting a tilt of the image corrected in the correction step on a screen.