JP2012248004A - Image processing system, image recognition device and method - Google Patents

Abstract

An image processing apparatus having a wide application range and effective in automating stereo calibration necessary for stereo image recognition processing is provided.
According to this embodiment, an image processing apparatus includes an image acquisition unit, a feature amount extraction unit, a correspondence detection unit, and a calculation unit. The image acquisition unit acquires image data corresponding to the common shooting area from a plurality of cameras. The feature amount extraction unit calculates at least two on-plane patterns based on the same real image existing in the common photographing region as feature amount information of the common photographing region from the image data of each camera. The correspondence detection unit detects a correspondence between the image data for each camera in the two on-plane patterns. The calculation unit calculates image conversion parameter information between images for each camera based on the correspondence relationship information calculated by the correspondence detection unit.
[Selection] Figure 1

Description

  Embodiments described herein relate generally to an image processing apparatus that performs calibration of a stereo camera.

  In recent years, a stereo image recognition technique using a stereo camera having two or more cameras and using an image obtained from each camera (hereinafter sometimes referred to as a camera image) has been developed. This stereo image recognition technology can be applied to systems such as a safe driving support system, a vehicle detection sensor, the pedestrian detection sensor, a road management optimization system, a station home monitoring system, a human detection sensor, and a crossing monitoring system. .

  By the way, when performing image recognition based on each camera image from the stereo camera, image processing for performing calibration (stereo calibration) between the camera images is necessary. Specifically, for example, image processing that realizes automation of stereo calibration (adjustment work between camera images) by calculating a pair of corresponding straight lines between camera images and calculating image conversion parameter information (projection transformation matrix) A device has been proposed.

Japanese Patent No. 3977776

  Stereo calibration can be automated by a conventional image processing apparatus. However, for example, when realizing a monitoring system that applies a stereo image recognition technique on a road or a train track, obtain at least four pairs of corresponding straight lines from a common shooting area of each camera that is the monitoring target. Is often difficult. For this reason, the application range is narrow in the conventional stereo calibration method, and as a result, it is not effective in realizing automation of stereo calibration (adjustment work between camera images).

  Therefore, an image processing apparatus having a wide application range and effective for automating stereo calibration necessary for stereo image recognition processing is provided.

  According to the present embodiment, the image processing apparatus includes an image acquisition unit, a feature amount extraction unit, a correspondence detection unit, and a calculation unit. The image acquisition means acquires image data corresponding to the common shooting area from a plurality of cameras. The feature amount extraction unit calculates at least two on-plane patterns based on the same real image existing in the common photographing region as feature amount information of the common photographing region from the image data of each camera. The correspondence detection means detects the correspondence between the image data for each of the cameras in the two on-plane patterns. The calculation means calculates image conversion parameter information between images for each camera based on the correspondence relationship information calculated by the correspondence detection means.

FIG. 3 is a block diagram for explaining a main part of the image processing apparatus according to the first embodiment. The block diagram for demonstrating the principal part of the stereo image recognition apparatus regarding 1st Embodiment. The figure for demonstrating operation | movement of the image acquisition part regarding 1st Embodiment. The figure for demonstrating operation | movement of the feature-value extraction part regarding 1st Embodiment. The figure for demonstrating operation | movement of the detection object area | region calculation part regarding 2nd Embodiment. 6 is a flowchart for explaining the operation of the image processing apparatus according to the first embodiment. The block diagram for demonstrating the principal part of the image processing apparatus regarding 2nd Embodiment. The figure for demonstrating operation | movement of the detection object area | region calculation part regarding 2nd Embodiment. The block diagram for demonstrating the principal part of the image processing apparatus regarding 3rd Embodiment.

  Embodiments will be described below with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram for explaining a configuration of an image processing apparatus 10 according to the present embodiment.

  As shown in FIG. 1, an image processing apparatus 10 according to the present embodiment is applied to, for example, a road monitoring system 1 that uses stereo image recognition, and includes components that execute a so-called stereo calibration process. That is, the image processing apparatus 10 includes an image acquisition unit 11, a feature amount extraction unit 12, a correspondence detection unit 13, an image conversion matrix calculation unit 14, and an image conversion matrix output unit 15. The image processing apparatus 10 is mainly composed of computer hardware and software.

  The image acquisition unit 11 receives an image (video) output from the stereo camera 20, converts it into digital image data, and stores it in an internal memory. In the present embodiment, the stereo camera 20 has two left and right cameras 20L and 20R including a charge coupled device (CCD), and the left and right images output from the cameras 20L and 20R (in the case where they are expressed as camera images). Output) at the same time.

  As shown in FIG. 2, the stereo camera 20 is installed at a position where the cameras 20L and 20R can photograph the monitoring area 200 of the road monitoring system 1, for example. The monitoring area 200 is a common imaging area for the cameras 20L and 20R, and includes a detection target area for detecting the vehicle 30 traveling on the road surface 31. The road monitoring system 1 inputs each camera image output from the cameras 20L and 20R and transmitted by, for example, a wireless communication type communication device 21. The stereo camera 20 may have a configuration having three or more cameras.

  The feature amount extraction unit 12 extracts the feature amount of each camera image (left and right images) from the digital image data acquired and stored by the image acquisition unit 11. The correspondence detection unit 13 compares the feature amounts of the camera images obtained by the feature amount extraction unit 12, and obtains a correspondence relationship between the camera images.

  The image conversion matrix calculation unit 14 calculates a projective conversion matrix H, which is an image conversion parameter between camera images, from the correspondence relationship (four sets of corresponding points described later) detected by the correspondence detection unit 13. The image conversion matrix output unit 15 outputs the projective transformation matrix H calculated by the image conversion matrix calculation unit 14 as a stereo calibration result.

(Stereo calibration process)
Hereinafter, the stereo calibration process of the image processing apparatus 10 of the present embodiment will be described with reference to the flowcharts of FIGS. 3, 4, and 6.

  First, the image acquisition unit 11 inputs an image (video) output from the stereo camera 20 and acquires digital image data (step S1). Here, as shown in FIG. 2, the stereo camera 20 captures an image of a road plane (road surface 31 in a three-dimensional space), and outputs left and right camera images simultaneously from the cameras 20L and 20R. That is, as shown in FIG. 2, each of the cameras 20 </ b> L and 20 </ b> R takes an image of a common shooting area 200 on the road surface 31 where no object having a height such as the vehicle 30 exists. The common photographing area 200 corresponds to a road monitoring area from the road monitoring system 1.

  FIG. 3A shows a left camera image from the camera 20L. FIG. 3B shows a right camera image from the camera 20R. In the present embodiment, the left camera image is used as a reference image, and the right camera image is handled as an image corresponding to the reference image. Each camera image includes white line or yellow line straight areas 300 and 301 which are road signs painted on the road surface 31.

  The feature amount extraction unit 12 extracts the feature amount of each camera image (left and right images) acquired by the image acquisition unit 11 (steps S2 and S3). Specifically, the feature quantity extraction unit 12 detects, as a feature area, a straight line region in which pixels having a large luminance gradient are connected in a straight line from each camera image.

  Here, the luminance gradient indicates the degree of luminance change in the vicinity of the pixel, and in the case of the monitoring region 200 shown in FIG. 2, the luminance gradient is large at the boundary in the image corresponding to the boundary of the white line on the road surface 31. It is an edge image showing a value. The feature amount extraction unit 12 obtains an edge image by a method of superimposing a filter such as a Sobel operator on the image (step S2).

  Hereinafter, a specific method for obtaining an edge image will be described.

The luminance at the position (x, y) in the image is set to I (x, y), and each element Fx (i, j), Fy (i, j) of the filter is set to a predetermined value. The following relational expression (1) is obtained when the values obtained by superimposing on Gx and Gy are obtained.

Since the relational expression (1) indicates the degree of the luminance change in the horizontal direction and the vertical direction in the vicinity of each position (x, y), the luminance gradient strength G (x, y) is expressed by the following mathematical expression (2). It can be calculated.

  Therefore, an edge pixel can be obtained by a filter process that extracts only pixels whose luminance gradient intensity is greater than a preset threshold value. 4A and 4B show examples of edge images 400 and 401 obtained from the camera images when the cameras 20L and 20R are installed as shown in FIG. In addition, there are various filters such as Laplacian, Robinson, Canny and the like that can be used for obtaining the luminance gradient, and any of them can be applied.

Next, after obtaining the edge image (edge pixel), the feature amount extraction unit 12 detects a straight line (straight line region) passing through the edge pixel using, for example, the Hough transform method (step S3). Specifically, the position (x, y) of the edge pixel is converted into a trigonometric function curve in the (ρ, θ) space using the following relational expression (3).

  Since the intersection of a plurality of curves generated from a plurality of edge pixels corresponds to a straight line, the feature quantity extraction unit 12 expresses the relational expression (3) and the coordinates of the intersection when the straight line is expressed by the expression “ax + by + c = 0”. The straight line parameters (a, b, c) are calculated using the values (ρ, θ).

  For example, a robust estimation method such as RANSAC (RANdom SAmple Consensus) is also applicable as a method for detecting a straight line (straight line region) other than the Hough transform method. In this method, a straight line fitting process is executed on an edge pixel group by a least square method or the like using a plurality of randomly selected edge pixels as initial values. In this case, the process of removing pixels having a large outlier (distance) from the straight line and the line fitting process are again performed in the same manner by newly including an edge pixel having a small outlier. By repeating the above-described straight line fitting process until the maximum outlier falls within the allowable range, a straight line (straight line region) passing through the edge pixels arranged on the straight line can be detected.

  As described above, as shown in FIGS. 4A and 4B, even when the edge pixel groups 400 and 401 are discontinuous or are bent and continued in a U-shape, a straight line (straight line region) is correctly obtained. ). The feature amount extraction unit 12 performs straight line detection processing on each camera image, and obtains two or more straight lines (straight line regions) for each camera image.

  In the present embodiment, it is assumed that each camera image is an image obtained by parallel equalization. A parallel-equalized image is an image adjusted so that the optical axes of the cameras 20L and 20R are parallel and have no lens distortion.

  Next, the correspondence detection unit 13 compares the feature amounts (straight line regions) of the camera images obtained by the feature amount extraction unit 12, and executes correspondence detection processing for obtaining a correspondence relationship between the camera images (Step S1). S4). Here, the correspondence relationship means a set of straight lines (straight line areas) included in each camera image based on the same real image existing in the common shooting area of each camera 20L, 20R. Specifically, it means a set of corresponding points (at least four sets) located on an epipolar line that is a straight line.

  As described above, in the present embodiment, it is assumed that each camera image is an image obtained by parallel equalization. In the case of an image parallelized, any point on the reference image (left camera image) is located on the same y coordinate on one image (right camera image) due to the epipolar constraint condition. There is. For this reason, the correspondence detection unit 13 executes a corresponding point search process using a one-dimensional search method. Hereinafter, a correspondence detection method by the correspondence detection unit 13 will be described.

  Specifically, the correspondence detection unit 13 detects a correspondence relationship between the camera images by applying a method based on region matching using, for example, an image feature near the target pixel. The image feature means a physical quantity such as luminance, color, and motion that can be extracted from an image that can distinguish a detected region near a straight line from another region.

The region matching method is a method of calculating the degree of coincidence (E _R ) between the comparison source image region and the comparison destination image region. For example, a calculation method using the SAD (Sum of Absolute Difference) method will be described.

Here, when two image regions to be compared are represented by R as the same shape and the position in the image region (R) is (x, y), the degree of coincidence (E _R ) can be calculated from the following relational expression (4). .

From the relational expression (4), the smaller the matching degree (E _R ), the higher the degree of matching between the two image regions. In addition, when there are a plurality of straight lines having similar edges, the distance between the straight lines is the same in any image on the same y coordinate in an image that is parallel-equalized. For this reason, it is possible to improve the reliability of the correspondence relationship by using information indicating the degree of coincidence of the image regions (R).

Next, the image conversion matrix calculation unit 14 calculates a projective conversion matrix H, which is an image conversion parameter between camera images, from the correspondence detected by the correspondence detection unit 13 (step S5). Specifically, the image conversion matrix calculation unit 14 is an image conversion parameter between camera images from at least four sets of corresponding points as the correspondence calculated by the correspondence detection unit 13 in the parallel-equalized image. A projective transformation matrix H is calculated. This projective transformation matrix H and corresponding points on the image coordinate system have a relationship as shown in the following relational expression (5).

Note that m ^T means a transposed matrix, and m ⁻¹ means an inverse matrix.

In this case, the projective transformation matrix H can be described as a 3 × 3 matrix as shown in the following relational expression (6).

  Here, the projective transformation matrix H is eight unknown parameters h0 to h7, and one set of corresponding points gives two equations. Therefore, if there are four or more pairs of corresponding points, this projective transformation matrix H can be obtained by the method of least squares. The image conversion matrix output unit 15 outputs the projective transformation matrix H calculated by the image conversion matrix calculation unit 14 as a stereo calibration result.

  As described above, according to the present embodiment, the feature amount (straight region) is detected from each camera image acquired from the left and right cameras of the stereo camera, and the degree of coincidence of the comparison target image region (R) from these feature amounts. For example, four corresponding points (corresponding relationships) are detected based on (position). Image conversion parameter information (projective transformation matrix H) between the camera images is calculated from these four corresponding points. This image conversion parameter information (projection conversion matrix H) can be used as information for executing calibration between the camera images of the stereo camera.

  In this embodiment, image conversion parameter information (matrix H) between camera images can be calculated by detecting, for example, four corresponding points from at least two linear regions from each camera image. Therefore, as a common shooting area of each camera, for example, a road marking pattern such as a white line is acquired as each camera image from the road surface of one lane, and if at least two straight line areas are detected, image conversion parameter information (projection conversion) is obtained. Matrix H) can be calculated.

  Here, in the conventional stereo calibration method in which, for example, a mark or the like is provided in advance on a road and at least four pairs of corresponding straight lines are detected, a road surface of one lane is monitored as a monitoring area (a common shooting area for each camera). ), The required number of straight lines is often insufficient, so the applicable range is narrow. On the other hand, with the method of the present embodiment, for example, if a camera image of a road marking pattern such as a white line can be acquired from the road surface of one lane, calibration between each camera image can be executed, so the applicable range is A wide range of stereo calibration can be automated.

  In the present embodiment, the case of applying a method of extracting a straight line (straight line region) as a feature amount in the feature amount extraction unit 12 has been described. However, the present invention is not limited to this, and a correspondence relationship is detected from the following feature amount It is possible to do it. For example, a road marking pattern such as numbers and characters painted on the road surface may be used as a planar pattern on the road surface. Moreover, the pattern on a plane corresponding to road incidental facilities, such as a road joint and a manhole, may be sufficient. Furthermore, in addition to the planar pattern on the road surface, a planar pattern corresponding to the train track line may be used. Further, the planar pattern may be sunlight projected on the road surface, a boundary between illumination light and a shadow portion, or a boundary portion of spot light projected at night.

[Second Embodiment]
5, FIG. 7, and FIG. 8 are diagrams for explaining an image processing apparatus according to the second embodiment.

  As shown in FIG. 7, the image processing apparatus 10 according to the present embodiment is applied to the road monitoring system 1 that uses, for example, stereo image recognition, as in the first embodiment, and includes an image acquisition unit 11, A feature amount extraction unit 12, a correspondence detection unit 13, an image conversion matrix calculation unit 14, and an image conversion matrix output unit 15 are included. Furthermore, the image processing apparatus 10 according to the present embodiment includes a detection target area calculation unit 16 and a detection target area output unit 17.

  Hereinafter, the operation of the image processing apparatus 10 of the present embodiment will be described. The operations of the image acquisition unit 11, the feature amount extraction unit 12, the correspondence detection unit 13, the image conversion matrix calculation unit 14, and the image conversion matrix output unit 15 are the same as those in the first embodiment, and thus will be described. Is omitted.

  The detection target region calculation unit 16 calculates the detection target region for the feature amount (for example, a straight region) associated with the correspondence detection unit 13, particularly for a lane on a road surface or a rail on a train track. Specifically, as illustrated in FIGS. 5A and 5B, the detection target region calculation unit 16 detects a region included in the lane for the feature amount representing the lanes 500 and 501 on the road surface. Calculate as a region. As shown in FIG. 8, the detection target area is an area 800 indicated by hatching. Here, FIG. 8 is an enlarged view of a part of the area included in the lanes 500 and 501 shown in FIGS. The detection target area output unit 17 outputs the area information (800) calculated by the detection target area calculation unit 16 as a calculation result.

  According to this embodiment, when the road to be detected is, for example, two lanes, the detection target area calculation unit 16 converts the area surrounded by the two outermost lines into a stereo image recognition target detection area (shown in FIG. 2). 200).

[Third Embodiment]
FIG. 9 is a diagram for explaining an image processing apparatus according to the third embodiment.

  As shown in FIG. 9, the image processing apparatus 10 of the present embodiment is applied to the road monitoring system 1 that uses, for example, stereo image recognition, as in the first embodiment, and includes an image acquisition unit 11, A feature amount extraction unit 12, a correspondence detection unit 13, an image conversion matrix calculation unit 14, and an image conversion matrix output unit 15 are included. Furthermore, the image processing apparatus 10 according to the present embodiment includes a detection target area calculation unit 18 and a detection target area output unit 19.

  Hereinafter, the operation of the image processing apparatus 10 of the present embodiment will be described. The operations of the image acquisition unit 11, the feature amount extraction unit 12, the correspondence detection unit 13, the image conversion matrix calculation unit 14, and the image conversion matrix output unit 15 are the same as those in the first embodiment, and thus will be described. Is omitted.

  The detection target area calculation unit 18 converts one camera image from the reference camera image using the projection conversion matrix (image conversion parameter information between camera images) calculated by the image conversion matrix calculation unit 14, Compare between camera images. The detection target area calculation unit 18 calculates an area existing on the same plane as the detection target area based on the difference between the camera images. In other words, if there is no difference between the camera images for the area existing on the plane, it can be estimated that the area is the same plane, so that the area existing on the same plane can be calculated as the detection target area. The detection target area output unit 19 outputs the area information calculated by the detection target area calculation unit 16 as a calculation result.

  According to this embodiment, the detection target region calculation unit 18 calculates the detection target region of the monitoring target based on the comparison result (difference) between the two camera images using the image conversion parameter information between the camera images. it can. Therefore, it is possible to improve the reliability when estimating the monitoring target detection area.

  Furthermore, the position of the detection target can be obtained by executing stereo monitoring image processing based on the calculated detection target region. It is also possible to correct the detection target area based on the calculated position information. In other words, according to the present embodiment, in each camera image, a matching region having no difference between the camera images is determined as a detection target region, and conversely, a non-matching region having a difference is excluded from the detection target region. Processing is possible.

[Application example]
By applying the image processing apparatus 10 described in the first to third embodiments as described above, stereo calibration in stereo image recognition can be realized. Therefore, it is possible to construct a road monitoring system 1 that monitors a vehicle traveling on a road surface using stereo image recognition.

  Further, by applying the image processing apparatus 10 described in the first to third embodiments, as a monitoring system using stereo image recognition, for example, a monitoring system in which a common area including a station track and a platform is a monitoring area. It is possible to build In such a monitoring system, since there is a step between the track and the home, the stereo image recognition device to which the image processing device 10 is applied can distinguish each region of the track and the home at the edge of the home. Using such stereo image recognition, the monitoring system can recognize, for example, a person who has fallen on the track from each camera image, and can also recognize a person at the end of the home.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 1 ... Road monitoring system, 10 ... Image processing apparatus, 11 ... Image acquisition part,
12 ... feature amount extraction unit, 13 ... correspondence detection unit, 14 ... image conversion matrix calculation unit,
15 ... image conversion matrix output unit, 16, 18 ... detection target region calculation unit,
17, 19 ... detection target region output unit, 20 ... stereo camera,
20L ... Left camera, 20R ... Right camera.

Claims (9)

Image acquisition means for acquiring image data corresponding to a common shooting area from a plurality of cameras;
Feature amount extraction means for calculating at least two on-plane patterns based on the same real image existing in the common photographing region as feature amount information of the common photographing region from the image data for each camera;
Correspondence detection means for detecting a correspondence relationship between the image data for each camera in the two on-plane patterns;
An image processing apparatus comprising: a calculation unit that calculates image conversion parameter information between images for each camera based on the correspondence relationship information calculated by the correspondence detection unit. The feature amount extraction means includes:
The image processing apparatus according to claim 1, wherein at least two linear regions based on the same real image are detected as the planar pattern. The correspondence detecting means includes
The image processing apparatus according to claim 2, wherein the image processing apparatus is configured to detect a set of a plurality of corresponding points of the at least two linear regions as the correspondence relationship. The feature amount extraction means includes:
As the feature amount information, at least 2 corresponding to any one of a road sign pattern included in the common imaging region, a planar structure attached to the road, a train route track, or a shadow or light projected on a plane. The image processing apparatus according to claim 1, wherein the image processing apparatus is configured to calculate two on-plane patterns.   4. The method according to claim 1, further comprising a detection target area calculation unit that calculates a detection target area of a stereo image recognition target based on the feature amount information associated by the correspondence detection unit. The image processing apparatus according to item 1.   The detection target area calculating means for calculating a common imaging area existing on the same plane as a detection target area using the image conversion parameter information calculated by the calculation means. 4. The image processing apparatus according to any one of items 3.   The detection target area calculation means compares the image data for each camera, determines a matching area having no difference in the comparison result as a detection target area, and conversely, a non-matching area having a difference is detected from the detection target area. The image processing apparatus according to claim 6, wherein the image processing apparatus is configured to exclude the image processing apparatus.   A stereo image recognition unit that performs stereo image recognition processing using a stereo camera using image conversion parameter information output from the image processing apparatus according to claim 1. An image recognition device. A method of performing a stereo calibration applied to stereo image recognition using a stereo camera,
Processing for acquiring image data corresponding to a common shooting area from a plurality of cameras included in the stereo camera;
Processing for calculating at least two on-plane patterns based on the same real image existing in the common imaging area as feature amount information of the common imaging area from image data for each camera;
A process of detecting a correspondence between the image data for each camera in the two on-plane patterns;
And a process of calculating image conversion parameter information between images for each camera based on the correspondence information calculated by the correspondence detection means.

Images