JPH09142208A - Vehicle periphery monitoring device - Google Patents

Abstract

[PROBLEMS] To provide a vehicle periphery monitoring device capable of eliminating erroneous detection of an object due to smear. A vehicle having a CCD camera (11) for imaging a predetermined range around the vehicle, and providing a driver with information on an object existing around the vehicle based on image information from the CCD camera (11). In the peripheral monitoring device,
A smear detecting means (41a) for detecting a linear smear image appearing on an image picked up by the CCD camera (11) by a high-intensity light source, and a smear removing means (41b) for removing the smear image detected by the smear detecting means. Have.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a vehicle periphery monitoring device, and more particularly, it is effectively applied to monitor the periphery of a vehicle such as an automobile to assist the driver's safety confirmation in driving the vehicle. The present invention relates to a vehicle surroundings monitoring device.

[0002]

2. Description of the Related Art Conventionally, a vehicle periphery monitoring device for monitoring the periphery of a vehicle such as an automobile has been proposed and put into practical use. This vehicle surroundings monitoring device uses a video camera attached to the vehicle to capture an image of the area (monitoring area) that is difficult for the driver to visually recognize, such as the rear or side of the vehicle, or the area in front of the vehicle. The captured image is displayed on a display mounted on the dashboard of the driver's seat or the like to monitor the monitoring area.

As such a vehicle periphery monitoring device, there is a device proposed by the present applicant in Japanese Patent Application No. 6-42433 (hereinafter referred to as a prior application device). This prior application device includes two CCD cameras installed at a predetermined distance in a vehicle, a frame memory for individually storing and holding the image information, a data processing unit including a CPU, a CC, and the like.
A display for displaying an image from the D camera or a graphic image processed by the data processing unit;
It has an alarm unit composed of a buzzer or the like.

In this prior application device, all the image information recorded in one of the frame memories has a height of 0.
Assuming that this image information is captured by the other camera, the projection image information obtained is created, and the image on the road surface is calculated from the difference between the created projection image information and the image information recorded in the other memory. Remove. Further, the edge of the object is detected based on the horizontal differential value of the data recorded in the other frame memory and the image information from which the road surface image has been removed. Then, for this object edge, the position of the object is calculated based on the image information recorded in the frame memory, and an alarm is output based on the calculated position information of the object.

Such a prior application device generates an edge image of an object by removing a background image having a height of "0", that is, a road surface image, based on image information captured by a CCD camera, and generates the edge image. Since it is configured to calculate the relative position between the object and the vehicle based on the edge image, there is a feature that unnecessary calculation processing can be omitted and the processing can be speeded up.

[0006]

By the way, as described above, in this prior application device, since the CCD camera is adopted as the image pickup means, an image of extremely high brightness such as a reflection surface or a light source due to sunlight in the image pickup area is provided. Is present,
From this high-luminance image, a high-luminance linear image that linearly extends over the entire area in the vertical direction of the screen, that is, an image called a smear appears. This smear is the two CCDs mentioned above.
In the image information captured by the camera, it appears on both image information or on one image information. That is, various patterns appear according to the imaging conditions.

When such a smear appears, the smear may be erroneously recognized as an edge point of the object, and thus an object that does not originally exist may be detected. There was room for Therefore, in view of such improvements, it is an object of the present invention to provide a vehicle periphery monitoring device that can eliminate erroneous detection of an object due to smear.

[0008]

In order to solve the above-mentioned problems, a vehicle periphery monitoring device according to the present invention is a CCD camera (11) for picking up an image of a predetermined area around the vehicle as shown in the basic configuration diagram of FIG. ), And the CCD camera (1
In a vehicle periphery monitoring device that gives a driver information about objects existing around the vehicle based on image information from 1), a linear smear image that appears on an image captured by a CCD camera (11) by a high-intensity light source. And a smear removing means (41) for removing the smear image detected by the smear detecting means.
It is characterized by having b). (Claim 1)

The smear detecting means (41a) is
Edge image generating means (41c) for generating an edge image based on image information from the CCD camera (11),
The coordinate value acquisition means (41d) for acquiring the coordinate value of the pixel of interest of the pixels forming the edge image, and the CCD camera (1) based on the coordinate value of the pixel of interest.
First brightness value acquisition means (41e) for acquiring the brightness value of the corresponding pixel corresponding to the pixel of interest in the image information from 1).
And based on the luminance value of the corresponding pixel from the first luminance value acquisition means (41e), when the luminance value of the corresponding pixel is equal to or more than a specified value, refer to the image information from the CCD camera (11) Then, a second brightness value acquisition means (41f) for acquiring the brightness value of the pixel group existing on the straight line including the corresponding pixel.
And a smear determining means for determining whether or not the pixel group existing on the straight line including the corresponding pixel is a smear image based on the brightness value of the pixel group acquired by the second brightness value acquiring means (41f). And (41 g). (Claim 2)

The smear removing means (41b) is
Third brightness value acquisition means (41h) for acquiring the brightness value of the pixel group on the boundary line adjacent to the pixel group judged as the smear image by the smear judgment means (41g), and the third brightness value acquisition means (41h) ) An image that calculates the luminance value of the pixel group determined to be the smear image based on the luminance value of the pixel group on the boundary obtained by the above and interpolates the pixel group determined to be the smear image based on the calculated luminance value And an interpolating means (41i). (Claim 3)

The coordinate value acquisition means (41d) is
It is characterized in that an edge point is searched for in an arbitrary column in the edge image generated by the edge image generating means (41c). (Claim 4)

Further, it has two CCD cameras (11R, 11L) which are installed at a predetermined distance in the vehicle and which image a predetermined range around the vehicle.
R, 11L), in a vehicle periphery monitoring device that provides the driver with information about objects existing around the vehicle based on image information from the CCD camera (1
1) smear detecting means (41a) for detecting a linear smear image appearing on the captured image, smear removing means (41b) for removing the smear image detected by the smear detecting means (41a), and the smear Removal means (41
Regarding the captured image from which the smear image has been removed in b),
It is assumed that all the images captured by one CCD camera (11R) have a height of 0, and projection image data obtained by capturing this image by the other camera is created. Object edge detection means (41j) for detecting the edge of the object by removing the image on the road surface from the difference with the image data recorded in the memory, and the object detected by the object edge detection means (41j). Object position calculating means (41k) for calculating the position of the object by the edge
And an alarm means (60) for outputting an alarm based on the position of the object calculated by the object position calculating means (41k). (Claim 5)

According to the structure of claim 1, the smear detecting means (41a) detects the linear smear image appearing by the high-intensity light source, and the smear removing means (41b) detects the detected smear image. Remove.

That is, means for detecting a smear image,
Since a means for removing the detected smear image is provided,
Erroneous detection of an object due to smear can be eliminated.

Then, as a concrete structure of the smear detecting means (41a), the structure shown in claim 2 is applied. And each structure of the said claim performs the following effects. That is, the edge image generation means (41c) generates an edge image which is an edge image of the object based on the image information from the CCD camera (11), and the coordinate value acquisition means (41d).
Acquires the coordinate value of the pixel of interest of the pixels forming the edge image.

The first brightness value acquisition means (41e) measures the brightness value of the corresponding pixel corresponding to the pixel of interest by the CCD camera (11).
The second brightness value acquisition means (41f) acquires the image information from the CCD camera (11) when the brightness value of the corresponding pixel is equal to or more than a specified value. The brightness value of the pixel group existing on the straight line including the corresponding pixel is acquired with reference. Then, the smear determination means (41g) determines whether or not this pixel group is a smear image based on the acquired luminance value of the pixel group.

As described above, the smear is treated in the same manner as an object to obtain an edge image including the edge image of the smear from the image picked up by the CCD camera, and the smear appears in a straight line. Means for acquiring the brightness value of a pixel group existing on a straight line including the smear candidate point, that is, the brightness value of the smear image based on the brightness value of the acquired pixel group, and determining whether or not the smear candidate point belongs to the smear Since the and are provided, the processing can be simplified and the smear can be detected at high speed and reliably.

Further, as a concrete constitution of the smear removing means (41b), the constitution shown in claim 3 is applied.
And each structure of the said claim performs the following effects. That is, the third brightness value acquisition means (41h) acquires the brightness value of the pixel group on the boundary line adjacent to the pixel group determined as the smear image by the smear determination means (41g), and the image interpolation means (41i). Calculates the luminance value of the pixel group determined to be the smear image based on the luminance value of the pixel group on the boundary line and interpolates the pixel group determined to be the smear image based on the calculated luminance value.

As described above, the brightness value of the pixel group determined to be smear is calculated based on the brightness value of the pixel existing on the boundary line of the captured image adjacent to the area (straight line) determined to be smear, and this calculation is performed. Since the smear image is configured to be interpolated based on the luminance value, the area determined as smear is interpolated as a pixel having a luminance value close to the luminance value of the pixels in the surrounding area. An image with less discomfort can be obtained.

Further, in the structure of the above-mentioned claim 4, when searching the edge point by the coordinate value acquisition means (41d) according to the claim 2, the search is performed for an arbitrary column in the edge image. I am configuring.

That is, in the structure of claim 4, the characteristic that the smear appears in a straight line over the entire area of the image is used, and only one arbitrary column is targeted for the search. Therefore, the processing is simple and high speed. Can be done.

Further, in the structure of the above-mentioned claim 5, the smear detecting means (41a) detects the linear smear image appearing by the high brightness light source, and the smear removing means (41b).
Removes the detected smear image.

The object edge detection means (41j) assumes that all the images picked up by one CCD camera (11R) of the picked-up image from which the smear image has been removed have a height of 0, and this image is taken as the other image. Create the projection image data obtained by capturing with the camera, and detect the edge of the object by removing the image on the road surface from the difference between the created projection image data and the image data recorded in the other memory . The object position calculation means (41k) calculates the position of the object from this object edge, and the alarm means (60) outputs an alarm based on the calculated position of the object.

That is, according to the fifth aspect of the invention, the smear detecting means (41a) and the smear removing means (4).
Regarding the captured image from which the smear image has been removed by 1b), the edge image of the object is generated by removing the background image having a height of “0”, that is, the road surface image, and based on the generated edge image, Since the relative position and the like are calculated, the relative position and the like between the object and the vehicle can be calculated even in the image in which the smear appears.

[0025]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 2 is a block diagram illustrating a configuration of an apparatus to which the present invention is applied. In FIG. 2, 10 is an image capturing unit that captures an image around the vehicle, 20 is a steering angle of a steering wheel or a steering angle of steered wheels, A rudder angle detection unit that outputs this rudder angle information as turning information, and 30 is the imaging unit 1
A storage unit that holds captured image information or vehicle shape information obtained from 0, 40 is a data processing unit configured by a computer that operates according to a prestored program, 50 is a display unit including a display 51, and 60 is a display unit.
Is an alarm unit that generates an alarm sound or voice guidance.

The image pickup section 10 is composed of a CCD camera 11. And this CCD camera 11
It is composed of a right CCD camera 11R and a left CCD camera 11L. And these CCD cameras 11
R and 11L are attached to the vehicle in parallel with each other with a predetermined distance. For example, as shown in FIG.
The vehicle 100 is mounted at a position of a height H at the rear center of the vehicle 100 at a predetermined distance toward the rear of the vehicle in parallel with each other at a depression angle θ _S. As a result, the image pickup unit 10 is provided with a monitoring area 100a [100a (11R) and 10a behind the vehicle.
0a (11L)] is imaged.

The steering angle detecting section 20 includes a steering angle sensor for detecting the amount and direction of rotation of the steering wheel, a steering angle sensor for detecting the steering angle of the steered wheels (generally the front wheels) (neither is shown), and the like. The turning direction of the vehicle is output as turning information based on detection signals from these sensors.

The storage unit 30 is a right frame memory 31.
R, left frame memory 31L, right corrected image memory 3
2R, left side correction image memory 32L, difference image memory 3
3, a differential image memory 34 and an edge image memory 35. The right frame memory 31R and the left frame memory 31L temporarily store image information captured by the right CCD camera 11R and the left CCD camera 11L of the image capturing unit 10.

The right-side corrected image memory 32R and the left-side corrected image memory 32L store the corrected image information obtained by performing a predetermined correction process on the image information stored in the right-side frame memory 31R and the left-side frame memory 31L, respectively. Retained. The differential image memory 33, the differential image memory 34, and the edge image memory 35 hold a differential image, a differential image, and an edge image, which will be described later, respectively.

Each of these memories (frame memory 31 to edge image memory 35) has a structure of m rows × n columns, and is configured as a memory of 512 pixels (m) × 512 pixels (n), for example. Each pixel forming this memory has, for example, 256 gradations (“0” to “25”).
5 ") or two-gradation (" 0 "or" 1 ") luminance value data is stored.

The data processing section 40 has a CPU 41 which operates according to an operation program, a ROM 42 which stores this operation program, and a RAM 43 which temporarily stores information required when the CPU 41 operates. The display unit 50 has a display 51, and based on the display image signal output from the CPU 41, the display 51 displays the relative position of the object with respect to the vehicle or displays a message to the driver. Alarm unit 60
Outputs the buzzer sound or the voice guidance from the speaker 61 based on the buzzer sound signal or the voice guidance signal output from the CPU 41.

Next, the operation of the specific example having the above-described structure will be described with reference to the flow chart. In this specific example, first, step S in the flowchart of FIG.
At 110, the captured images from the right CCD camera 11R and the left CCD camera 11L of the imaging unit 10 are stored in the corresponding right frame memory 31R and left frame memory 31L as right image information and left image information, respectively.

That is, in this step S110, for example, as shown in FIG. 6, a road surface 400 on which two white lines 300 are drawn and the road surface 400 are set upright, and the surface is made a reflective surface such as a metallic glossy surface. An image of a pole-shaped object 200 (hereinafter, simply referred to as the pole 200) and a linear shape extending over the entire area in the vertical direction (V direction) that appears when strong light such as sunlight is reflected by the pole 200. The image of the smear 500, which is light, is stored in the right frame memory 31R and the left frame memory 31L.
Then, when the process of step S110 ends, the process proceeds to step S120.

In step S120, smear removal processing is performed. This smear removal processing is specifically performed according to the flowchart of FIG. The smear removing process will be described below with reference to the flowchart of FIG.

In this smear removal processing, first, in step S210, an image to be processed is selected. As described above, the smear appears in various patterns on both image information of one of the image information captured by the two CCD cameras or on one of the image information depending on the image capturing conditions. Therefore, it is necessary to perform the smear detection and removal processing on both captured images stored in the right frame memory 31R and the left frame memory 31L. Then, in this step S210, it is selected whether the captured image to be processed is the captured image stored in the right frame memory 31R or the captured image stored in the left frame memory 31L.

In the following step S220, the image pickup image selected in the above step S210 is subjected to differential processing to create a differential image. More specifically, in this step S220, regarding the captured image stored in the frame memory 31R or 31L on the detection target side, the brightness value I _{m, n} of the image data of m rows and n columns is set in the horizontal direction (see FIG. (Corresponding to the H direction in 6) and the differential image is generated by the calculation of the following equation (1). | I _{m, n + 1} −I _{m, n} | ≧ E ₀ , E _{m, n} = 1 | I _{m, n + 1} −I _{m, n} | <E ₀ , E _{m, n} = 0. (1) In the above formula, E ₀ is a threshold value

When this differential processing is performed on the picked-up image, an edge image, that is, a value "1" is given to a portion having a difference in brightness in the horizontal direction (H direction in FIG. 6), that is, brightness. Is extracted as a processed image. The differential image thus created is stored in the differential image memory 34. After storing the differential image, the process proceeds to step S230.

In step S230, an edge point search is performed. In this edge point search processing, step S220
For the pixel group of any one row in the differential image (that is, the edge image of the object) created in step 1, the process of detecting the edge point while scanning the pixel group in the horizontal direction, that is, the pixel to which the value “1” is added is The detection process is performed.

In the subsequent step S231, a process of acquiring the brightness value of the pixel in the target image (captured image) is performed.
Specifically, in this step S231,
In the pixel group searched in 230, the coordinate value of the detected pixel, that is, the pixel to which the value “1” is given, is acquired, and the corresponding coordinate value is obtained in the target image, that is, the captured image stored in the frame memory 31R or 31L. The brightness value of the pixel of is acquired. And this step S2
When the process of 31 is completed, the process proceeds to step S232.

In step S232, the above step S2
Referring to the brightness value of the pixel in the captured image acquired in 31,
It is determined whether or not this brightness value is equal to or greater than the specified value. The prescribed value in this determination process is given as a high-luminance predetermined value for determining smear, and is given as a value “200”, for example.

Then, in the determination processing of step S232, based on the brightness value of the pixel acquired in step S231, if this brightness value exceeds the specified value (value "200") (Y), this The pixel is determined to be a smear candidate point, and the process proceeds to subsequent step S240. On the other hand, when the brightness value of this pixel is equal to or less than the specified value (value “200”) (N), this pixel is determined to be a normal visible image, and the process proceeds to step S230 described above to search for an edge point. Then, the next edge point is detected.

In step S240, the brightness value of the pixel group existing in the vertical direction is acquired. In this luminance value acquisition processing, for the pixel determined as the smear candidate point in step S232, the luminance value of the pixel group existing on the straight line extending in the vertical direction from the pixel is referred to by referring to the frame memory 31R or 31L. Processing to acquire is performed.

In a succeeding step S241, it is determined based on the brightness value of the pixel group acquired in the above step S240 whether or not the brightness values of all the pixels forming this pixel group are equal to or more than a specified value. The prescribed value in this determination process is given as a prescribed value of high brightness for judging smear, like the prescribed value of step S232, and is given as a value "200", for example.

In this determination process, when the luminance value of the pixel group to be determined is equal to or more than this specified value, that is, the pixel group extending in the vertical direction including the smear candidate point is composed of high-luminance pixels. If it is (Y), the process proceeds to step S242, and the pixel group to be determined is recognized as a smear image in step S242. Then, the information of the recognized smear image, for example, the coordinate information indicating the smear image is stored and held in the RAM 43 of the data processing unit 40.

On the other hand, if there is a pixel whose luminance value is equal to or less than the above-mentioned specified value in the pixel group to be judged (N), it is determined that the pixel group to be judged is not a smear image and the above-mentioned steps are performed. In step S230, an edge point is searched for and the next edge point is detected. Then, when the above-described processing of step S242 ends, step S2
Move to 50.

In step S250, it is determined whether or not the search for edge points has been completed. That is, this step S
In step 250, it is determined whether or not the process of detecting the edge point while scanning the pixel group in the horizontal direction described in step S230 is executed over the entire horizontal direction. This determination process is performed based on the coordinate value of the pixel that is the scan target at that time, and is determined whether or not this coordinate value is the coordinate value corresponding to the horizontal end.

If the search ends in this determination process, that is, the coordinate value of the pixel to be scanned is the coordinate value corresponding to the horizontal end (Y), step S260.
If the search is still in progress (N), the process moves to step S230 described above to search for the remaining edge points.

In step S260, the presence / absence of a smear image is determined. The presence / absence determination of the smear image is made based on the information of the smear image stored and held in the RAM 43 of the data processing unit 40. Then, when the information indicating the presence of the smear image is stored in the RAM 43, it is determined that "smear exists (Y)", the process proceeds to step S270, and the information indicating the presence of the smear image is not stored. In this case, it is determined that "no smear (N)" and the process proceeds to step S280.

In step S270, a brightness value acquisition process for the pixel group on the boundary is performed. Specifically, this step S27
In 0, the coordinate value of the pixel group determined to be the smear image in step S260 is acquired by referring to the RAM 43, and the brightness value of the pixel group on the boundary line adjacent to this smear image is determined based on the acquired coordinate value. get. The brightness value acquisition process in step S270 is performed for the pixel groups on the boundary lines on both sides of the smear image.

In the following step S271, based on the brightness value of the pixel group on the boundary line obtained in step S270,
The brightness value of the pixel group determined to be the smear image is calculated, and the pixel group determined to be the smear image is interpolated based on the calculated brightness value. That is, in this interpolation processing, the captured image stored in the frame memory 31R or 31L to be processed is referred to, and based on the brightness values of the pixel groups on the boundary line existing on both sides of the smear image acquired in step S270, The luminance values of pixels on both sides of the pixel to be interpolated are averaged, and the average value is replaced with the luminance value of the pixel to be interpolated to perform interpolation on the captured image. Then, when the interpolation processing in step S271 is completed, the process proceeds to step S280.

In step S280, it is determined whether or not the series of processes described above has been performed for both picked-up images stored in the right frame memory 31R and the left frame memory 31L. Then, in step S280, when a series of smear removal processing is performed on both captured images (Y), the smear removal processing is terminated,
The process moves to step S130 in the flowchart of FIG. Further, when the series of smear removal processing is performed only on one captured image (N), the step S2 is performed again.
Then, the processing shifts to 10, and a series of processing is executed on the unprocessed captured image.

Here, the actual operation of the series of smear removal processing described above will be described with reference to the schematic diagrams of FIGS. 6 to 9. In this smear removal processing, first, a captured image to be processed is set (S210), and
The differential processing described in the above equation (1) is executed on the captured image to be processed (S220).

For example, when this differential processing is performed on the picked-up image illustrated in FIG. 6, as shown in FIG.
Images of the edges of the two white lines 300 indicated by reference numerals ED1 and ED2, ED7 and ED8, images of the vertical edges of the poles 200 indicated by reference numerals ED3 and ED4, and edges of the smear 500 indicated by reference numerals ED5 and ED6. Image is extracted as an edge image, that is, an image to which a value "1" is added for this edge image and a value "0" is added to the other images. Then, the created differential image is stored in the differential image memory 3 of the storage unit 30.
Stored in 4.

Next, the leftmost pixel H indicated by reference numeral HL in FIG.
An edge point is searched by sequentially scanning from the left end pixel H1 in the horizontal pixel row extending from 1 to the right end pixel H6. By this search, in the example of the figure, first, the edge point indicated by the symbol H2 is detected (S230). Then, the brightness value of the pixel in the captured image corresponding to the edge point H2, that is, the brightness value of the pixel indicated by the symbol H2 ′ in FIG. 6 is acquired (S231), and the brightness value of this pixel H2 ′ determines smear. Judgment whether the value exceeds the specified value of.

Assuming that the specified value is the brightness value "200" as shown in FIG. 8 and the brightness value of the pixel H2 'is the value "1".
If it is "80", it is determined that this edge point H2 is not a smear candidate point (S232). And again,
The edge point H3, which is the next detection point, is detected by performing the search for the horizontal pixel row (S230). Also for this edge point H3, the brightness value of the pixel in the corresponding captured image is acquired, and thereby the brightness value of the pixel indicated by the symbol H3 ′ in FIG. 6 is acquired (S231).

Then, the luminance value of this pixel H3 'is the value "2".
If it is 10 ", this edge point H3 exceeds the above-mentioned specified value" 200 ", so this edge point H3
Is determined as a smear candidate point (S232). In response to this determination, the luminance value of the corresponding pixel group in the captured image, that is, the pixel group existing in the vertical direction including the pixel H3 ′ is acquired. More specifically, reference numerals V11 ′ and V12 shown in FIG.
The brightness value of the vertical pixel group existing on the vertical straight line VL1 ′ connecting the upper and lower end pixels indicated by 'is acquired (S240).

Next, it is determined whether or not the obtained brightness value of the pixel group exceeds the specified value for determining the smear described with reference to FIG. In this case, since the vertical pixel group on the straight line VL1 ′ includes the image of the road surface 400 having a relatively low luminance, the pixels forming the vertical pixel group include pixels having a predetermined value or less. It is determined that the edge point H3 is not a smear image (S24).
1).

As a result, the horizontal pixel row is searched again, the edge point H4 which is the next detection point is detected, and the pixel in the corresponding picked-up image, that is, the pixel indicated by the symbol H4 'in FIG. 6 is detected. The brightness value is acquired (S230,
S231).

Then, the luminance value of the pixel H4 'is the value "2".
40 ", this edge point H4 is determined to be a smear candidate point (S232). As a result, the pixel H4
The luminance value of the pixel group existing in the vertical direction including ‘′, that is, the vertical pixel group existing on the straight line VL2 ′ in the vertical direction connecting the upper and lower end pixels denoted by reference signs V21 ′ and V22 ′ in FIG. 6 is acquired (S240). ).

Next, a determination is made as to the brightness value of the acquired pixel group. In this case, since the vertical pixel group on the straight line VL2 ′ is the image of the smear 500, all of the vertical pixel groups exceed the specified value “200” described above. As a result, the edge point H4 is determined to be a smear image (S241, S242), and the vertical pixel group on the straight line VL2 ′ including the edge point H4, that is, the straight line VL in FIG.
Information regarding No. 2 is stored in the RAM 43 as smear image information.

Then, the horizontal pixel row is searched again, the edge point H5 which is the next detection point is detected, and the pixel in the corresponding picked-up image, that is, the symbol H in FIG.
The luminance value (value "240") of the pixel indicated by 5'is acquired (S230, S231). Also for this pixel H5 ',
Similar to the previous pixel H4 ′, it is determined as a candidate point based on the luminance value (S232), and the vertical pixel group, that is, VL3.
The luminance value of the vertical pixel group existing on the ‘′ is acquired (S2
40).

Further, a determination is made as to the obtained brightness value of the vertical pixel group. This vertical pixel group is an image of the smear 500 as in the vertical pixel group of the edge point H4 described above, and thus is determined to be a smear image (S241,
S242), information about the vertical pixel group on the straight line VL3 ′ including the edge point H5, that is, the straight line VL3 in FIG. 7 is stored in the RAM 43 as smear image information.

Then, after confirming that the search has been completed up to the rightmost pixel H6 in FIG. 7 (S250), the presence or absence of smear is determined (S260). In the example above,
The edge image by the smear 500 (VL2 in FIG. 7,
Since VL3) has been detected, it is determined that "smear is present", and processing for removing this smear image is performed.

In this smear image removal process, first, a process of acquiring the brightness value of the pixel group on the boundary line adjacent to the edge image detected as the smear is performed. For example, in FIG.
As shown in (a), a matrix-shaped frame memory configuration is provided, which includes pixels in α, β and γ rows and pixels in a, b, c and d columns, and the column c is a pixel group corresponding to a smear image. Then, as shown in the figure, the pixels αc, ..., γ
Each of the values c has a brightness value of “230” and high brightness.

In this case, columns b (pixels αb, ..., γb) and columns d (pixels α) on both sides of the smear image.
The brightness value of the pixel group of d, ..., γd) is acquired (S27).
0). Then, based on the acquired brightness value of the pixel group,
A brightness value for interpolating the pixel group determined as the smear image is calculated (S271). The calculation of the brightness value is performed by calculating the average value of the pixels on both sides of the pixel determined to be the smear image.

Then, as shown in FIG. 9B, in the α row, the average value “105” of the luminance value “100” of the pixel αb and the luminance value “110” of the pixel αd is set to the pixel αc of the smear image.
Brightness value. Similarly, in the β row, since the brightness value of the pixel βb and the brightness value of the pixel βd are the same value “120”, this value “120” is set as the brightness value of the pixel βc of the smear image,
In the γ row, the average value “100” of the luminance value “110” of the pixel γb and the luminance value “90” of the pixel γd is set to the pixel γ of the smear image.
With the luminance value of c, the image is interpolated by each luminance value.

Further, as shown in FIG. 9C, when the smear image extends over a plurality of columns (columns b to d), column a on both sides of the smear image is used as in the above-described example. (Pixel α
a, ..., γa) and the brightness value of the pixel group of the e column (pixels αe, ..., γe), and the brightness value of each pixel is stepwise based on the acquired brightness value. It is changing. For example,
As shown in FIG. 9D, the pixel αa has a luminance value of “10”.
0 ”and the pixel αe has a luminance value of“ 120 ”,
Regarding this α row, the pixel αb has a luminance value “105”, the pixel αc has a luminance value “110”, and the pixel αd has a luminance value “11”.
For example, a luminance value that is changed stepwise such that the amount of change in the luminance value is substantially equal is given.

As described above, the brightness value of the pixel group determined to be smear is calculated based on the brightness value of the pixel existing on the boundary line of the captured image adjacent to the area (straight line) determined to be smear, and this calculation is performed. Since the smear image is configured to be interpolated based on the luminance value, the area determined as smear is interpolated as a pixel having a luminance value close to the luminance value of the pixels in the surrounding area. An image with less discomfort can be obtained.

In the above-described series of smear removal processing, the smear 500 is treated in the same manner as an object to obtain C.
An edge image including a smear edge image is acquired from a captured image from the CD camera 11, and a pixel group existing on a straight line including a smear candidate point, that is, a smear image, based on the property that the smear appears in a straight line. Since it is configured to acquire the luminance value of and to determine whether or not this smear candidate point belongs to the smear based on the acquired luminance value of the pixel group, the processing can be simplified and the smear speed can be increased. In addition, it can be detected reliably.

Further, since the smear appears linearly over the entire area of the image and only one arbitrary column is the object of the search, the processing is simple and can be performed at high speed.

Next, the processing of step S130 and thereafter, which is performed following the processing of step S120 described above, will be described. In step S130, with respect to the captured image that has been subjected to the smear removal processing in step S120 described above,
Performs distortion correction processing. That is, the above step S
The imaged image of 120 is an image including the distortion aberration of the lens included in the CCD camera 11. This distortion aberration causes an error in detecting the position of the object, which will be described later.

For example, considering a case where a lattice-shaped pattern is imaged, when the lens of the CCD camera 11 has no aberration, the lattice-shaped pattern is directly stored in the frame memory 31 as shown in FIG. It is stored.
Further, when the lens has an aberration, FIG.
As shown in Fig. 10, the image at the center of the optical axis is distorted into a barrel shape bulging outward, or Fig. 10 (c).
As shown in FIG. 5, the image at the center of the optical axis is distorted in the shape of a pincushion which is curved inward.
Is stored in

Then, in this step S130, the correction image information is generated by performing the correction processing on the distorted image as shown in FIG. 10 (b) or FIG. 10 (c), and the correction image information is generated. Right side correction image memory 32
R and the left corrected image memory 32L. The process of correcting the distortion of the image will be described below.

As shown in FIG. 10D, the point P (x ₀ , y ₀ ) when there is no distortion is point P'due to the distortion.
Assuming that an image is formed at (x, y), the aberration amount D can be expressed by the following equation (2). _{D = [(x 0 -x)} 2 + (y 0 -y) 2] 0.5 ··· (2)

Then, FIG. 10B or FIG.
In the distorted image as shown in (c), the aberration amount D increases in proportion to the cube of the distance from the optical axis. That is, since it is proportional to the cube of the distance from the center point of the lens to the point P, the right side of the above equation (2) can be expressed by the following equation (3). [(X ₀ −x) ² + (y ₀ −y) ² ] ^0.5 = k ₁ [(x ₀ ² −y ₀ ² ) ^0.5 ] ³ (3) In the above formula, k ₁ is determined by the lens. Proportional constant

Therefore, when the lens has distortion, the distortion can be corrected by, for example, performing the calculation of the following equation (4). x ₀ ≈x (1 + k ₁ (x ² + y ² )) y ₀ ≈y (1 + k ₁ (x ² + y ² )) (4)

By the way, when the distortion of the image is corrected by the calculation of the equation (4), the distance between the adjacent pixels may be increased and the image may be omitted. The data processing unit 40 interpolates the missing image based on the information of the adjacent pixels. Then, the image information that has undergone such correction processing and interpolation processing is stored as corrected image information in the right side corrected image memory 32R and the left side corrected image memory 32L, respectively, and the distortion aberration correction processing in step S130 ends.

In the subsequent step S140, road surface image removal processing is performed. Before describing this road surface image removal processing, first, the position calculation processing of the object in the imaging area (monitoring area 100a) in the optical system described in FIG. 3 will be described.

As described with reference to FIG. 3, the CCD camera 11 is mounted at a predetermined position on the rear side of the vehicle at an installation depression angle θ _S and images the monitoring area 100a. Therefore C
Since the image captured by the CD camera 11 is an image with this predetermined position as a reference, the position calculated from this captured image is also a position with this predetermined position as a reference. Therefore, in order to facilitate the following description, as shown in FIG. 11, the coordinates based on this predetermined position, that is, the camera installation position are represented by X ', Y', and Z ', and the coordinates based on the road surface are represented by X'. , Y, Z.

In the X ', Y', Z'coordinate system with the camera installation position as a reference, the Z'axis is defined as the lens optical axis of the CCD camera 11, as shown in FIG. , The X'axis is defined as an axis parallel to the road surface, and the Y'axis is defined as an axis orthogonal to both the Z'axis and the X'axis. Therefore, the right CCD camera 1
The 1R and the left CCD camera 11L are arranged such that their lens optical axes coincide with the Z ′ axis. Also,
With respect to the X'-axis, the center points of the respective lenses are arranged on the X'-axis and the distances dxa from each other.
Are spaced apart. In order to facilitate the following description, the origin O of the X ′, Y ′ and Z ′ axes will be defined as the center of the lens of the left CCD camera 11R.

CCD camera 11 installed as described above
Point P (X _P ′, Y _P ′, Z _P ′) imaged by
Is held as P _R (x _RP , y _RP ) in the right side correction information memory 32R, and P is stored in the left side correction information memory 32L.
It is held as _L (x _LP , y _LP ).

The Z ′ coordinate Z _P ′ of this point P is shown in FIG.
As shown in the schematic view in the X′Z ′ plane of 2 (b),
It can be determined by the similarity of triangles. That is, this distance Z _P ′ can be expressed by the following equation (5). Z _P ′ = dxa · f / (x _LP −x _RP ) ... (5) In the above formula, dxa is the distance between both lenses and f is the focal length of the lens.

Similarly, for the X ′ coordinate X _P ′ of the point P,
As shown in the schematic view of the X′Z ′ plane of FIG. 12B, it can be obtained by similarity of triangles. And Y'of point P
The coordinates Y _P ′ can also be obtained in the same manner by assuming a Y′Z ′ plane (not shown). In other words, these points P X 'coordinate X _P' and Y 'coordinate Y _P'
Can be obtained by the following equations (6) and (7), respectively. _{X P '= Z P' x} LP / f = dxa · x LP / (x LP -x RP) ··· (6) Y P '= Z P' y LP / f = dxa · y LP / (x LP -X _RP ) ... (7) Regarding the coordinate X _P ′, the reference coordinate is the right CC
When the distance is between the D camera 11R and the left CCD camera 11L, the difference of 1/2 of the coordinate X _P ′ calculated by the above equation (5) and the distance dxa between both lenses may be taken.

Point P (X _P ′, Y _P ′, in the X′Y′Z ′ coordinate system calculated by the above equations (5) to (7)
Since Z _P ′) is the coordinate value of the coordinate system with the camera installation position as the reference as described above, it is necessary to convert this coordinate value into the coordinate value in the XYZ coordinate system with the road surface as the reference. In this case, if the depression angle of the image pickup unit 10 (CCD camera 11) is θ _S , the coordinates (X _P ′, Y _P ′, Z _P ′) of the point P at the X′Y′Z ′ coordinates and the road surface. XY based on
The relationship with the Z coordinate is as shown in FIG. 11 described above.

Therefore, it is calculated by the above equations (5) to (7).
Coordinate (Z_P′, Y_P′, X_P′) Is the following formula (8) -formula
By executing (10), the coordinates of point P in the XYZ coordinate system
(X _P, Y_P, Z_P) Can be converted. X_P= X_P′ ・ ・ ・ (8) Y_P= H-Z_P′ Cos θ_S+ Y_P′ Sin θ_S (9) Z_P= Z_P′ Sin θ_S+ Y_P′ Cos θ_S ···(Ten)

Next, the removal processing of the road surface image (image of height "0") in step S140 will be described. FIG. 13A shows the right-side corrected image that has been captured by the right-side CCD camera 11R, has been subjected to the above-described distortion correction, and is stored in the right-side corrected image memory 32R. In the figure, 300 is a white line drawn on the road surface, and 200 is a pole-shaped object. Here, it is assumed that all the right-side corrected images stored in the right-side corrected image memory 32R are background images having a height of "0", that is, images drawn on the road surface. Then, based on the right-side corrected image assumed in this way, conversion processing is performed so that the right-side corrected image becomes an image (projection image) taken at the image pickup position of the left CCD camera 11L [FIG. 13].
(B)].

The conversion process of the projected image will be described below. Here, the point of the projection image corresponding to the point P _R (x _RP , y _RP ) of the right image is P _L ′ (x _LP ′, y _LP ′).
As shown in FIG. 11, the X ′ axis of the camera coordinates and the X axis of the road surface coordinates are parallel, and the x axes of the scanning lines imaged by the camera (x _L axis and x _R axis in FIG. 12) are also parallel. , Y of the captured image when the same object is captured.
_{The L} and y _R values match. If all the images are on the road surface, the value of Y _P shown in the above equation (9) is zero.
From the above, the following equations (11) and (12) can be derived. Then, by substituting the Z _P 'and (7) of Y _P' of the formula (5) in the Z _P 'and Y _P' of formula (12), as shown in equation (13), x _LP ' Can be asked.

Y _LP ′ = y _RP (11) 0 = H _P −Z _P ′ cos θ _S + Y _P ′ sin θ _S (12) x _LP ′ = (dxa · fcos θ _S −dxa Y _RP sin θ _S ) / H + x _RP (13) Then, the data processing unit 40 (CPU 41) performs the calculation of the above equations (11) and (13), and the projected image [FIG. b)] is created.

Then, the projection image thus created is superimposed on the left side corrected image as shown in FIG. 13 (c). That is, when the image captured by the right CCD camera 11R is projected, the pattern such as a white line drawn on the road surface matches the pattern captured by the left CCD camera 11L in position and brightness, and the object is higher than the road surface. The greater the difference, the greater the difference.

Therefore, by taking the difference between the left image data and the projected image data, the luminance value of the pixels forming the road surface other than the pixels forming the object having the height is close to the value "0" or "0". It becomes a value. If the value equal to or smaller than the predetermined threshold value is set to "0", all values will be "0".

In this way, by obtaining the difference between the left image data and the projection image data, the difference image is shown in FIG.
As shown in (d), the road surface image (background image of height "0") is removed, and only the portion having the height is extracted as a value other than the value "0". Then, this difference image is stored in the difference image memory 33 of the storage unit 30.

As described above, this step S14
At 0, the road surface image of height “0” in the captured image is removed,
Only the image with height, that is, the image of the object is extracted. Then, when the road surface image removal processing in step S140 is completed, the process proceeds to step S150 and the object position calculation processing is performed.

In the object position calculation processing in step S150, edge extraction processing is first performed on the extracted object image. The object edge extraction processing will be described below. This object edge extraction processing is performed on the image information stored in the left side corrected image memory 32L. That is, with respect to the left-side corrected image stored in the left-side corrected image memory 32L, the brightness value I of the image data in the m-th row and the n-th column
_{The m, n} are scanned in the horizontal direction, that is, the X'-axis direction in FIG. 12, and a differential image is generated by the calculation of the following expression (14). | I _{m, n + 1} −I _{m, n} | ≧ E ₀ , E _{m, n} = 1 | I _{m, n + 1} −I _{m, n} | <E ₀ , E _{m, n} = 0. (14) In the above formula, E ₀ is the threshold value

As shown in FIG. 14B, this differential image is an image in which the vertical edge portion of the character or the like drawn on the object or the road surface is "1" and the other portions are "0". Becomes Then, this differential image is overwritten and held in the differential image memory 34 of the storage unit 3.

The differential image thus obtained [FIG.
(B)] and the difference image generated by the above-described road surface image removal processing and held in the difference image memory 33 [FIG. 13 (d)]
Then, when an AND is performed, the image of the object edge shown in FIG. 14C in which only the edge part of the object is extracted is generated. Then, this object edge image is stored in the edge image memory 35 as information indicating the object.

The data processing unit 40 (CPU 41)
Recognizes an object based on the object edge image stored in the edge image memory 35, performs the three-dimensional position coordinate calculation operation described with reference to FIGS. 11 and 12 for each coordinate point of the object edge image. The relative position between the recognized object, that is, the object edge and the vehicle is calculated by calculating the three-dimensional coordinate value of the edge image.

In the following step S160, it is determined whether or not there is an obstacle. That is, in this step S160, 3 of the edge image calculated in the above step S150.
The imaging region 100 described with reference to FIG. 3 based on the dimensional coordinate values.
It is determined whether or not there is an obstacle in a. When it is determined that there is no obstacle in this step S160,
The process moves to step S110, and the processing operation of the next operation cycle is executed again. If it is determined in step S160 that there is an obstacle, the process proceeds to step S170.

In step S170, a vehicle course prediction process is performed. In the vehicle course prediction process of step S170, the turning information from the steering angle detection unit 20 (see FIG. 2) is read, the course of the vehicle is predicted based on the read turning information, and the shape of the vehicle is further added to the predicted course. The predicted trajectory of the vehicle is calculated by taking the information into consideration. Then, the calculated predicted trajectory of the vehicle is stored in the RAM 43 of the data processing unit 40.
To be stored.

In the following step S180, the prediction locus calculated in step S170 and the step S1
Based on the three-dimensional coordinate value of the obstacle calculated in 50, it is determined whether there is a possibility of collision between the obstacle and the vehicle. When it is determined that “there is a possibility of collision”, the speaker of the alarm unit 60 generates an alarm sound to notify the driver of that. Then, the series of processing operations are ended by the end of the alarm processing in step S180, and the step S180 is performed again.
In step 110, the processing operation in the next operation cycle is executed.

As is clear from the above description, in this specific example, the edge image of the object is removed by removing the background image having the height "0", that is, the road surface image, from the captured image from which the smear image has been removed. Since the relative position between the object and the vehicle is calculated based on the generated edge image, the relative position between the object and the vehicle can be calculated even in the image in which the smear appears.

Further, since the smear removal processing is performed before the processing for correcting the lens distortion aberration, the smear image is not curved due to the lens distortion correction. This allows the smear to be easily removed.

The flowcharts showing the basic configuration of the present invention and the operation of the specific example have the following correspondence.
That is, the smear detecting means (41a) corresponds to steps S210 to S250 in the flow chart of FIG. 5, and the smear removing means (41b) similarly performs step S27.
It corresponds to 0 to S271.

The edge image generating means (41c) proceeds to step S220, the coordinate value acquiring means (41d) similarly to step S230, the first brightness value acquiring means (41e) similarly to step S231 in the same flowchart,
The second brightness value acquisition means (41f) similarly performs step S24.
0, the smear determination means (41g) is the same as step S2.
It corresponds to 41.

The third luminance value acquisition means (41h) corresponds to step S270 in the same flowchart, and the image interpolation means (41i) also corresponds to step S271.

The object edge detecting means (41j) corresponds to step S140 in the flowchart of FIG.
The object position calculating means (41k) similarly performs step S150.
It corresponds to.

[0106]

As described above, the vehicle periphery monitoring device of the present invention has the following effects. That is, since the means for detecting the smear image and the means for removing the detected smear image are provided, erroneous detection of the object due to the smear can be eliminated.

Based on the fact that the smear is treated in the same manner as an object to obtain an edge image including the edge image of the smear from the image picked up by the CCD camera, and that the smear appears linearly. A pixel group existing on a straight line including a smear candidate point, that is, a brightness value of a smear image is acquired, and a means for determining whether or not the smear candidate point belongs to a smear based on the brightness value of the acquired pixel group. Since it is provided, the processing can be simplified and the smear can be detected at high speed and reliably.

Further, the brightness value of the pixel group judged as smear is calculated based on the brightness value of the pixel existing on the boundary line of the picked-up image adjacent to the area (straight line) judged as smear,
Since the smear image is interpolated by the calculated brightness value, the area determined as smear is interpolated as the pixel having the brightness value close to the brightness value of the pixel in the surrounding area. It is possible to obtain an image with less discomfort regarding the image.

Further, since the smear appears linearly over the entire area of the image and only one arbitrary column is the object of the search, the processing is simple and can be performed at high speed.

Further, with respect to the picked-up image from which the smear image has been removed by the smear detecting means and the smear removing means, the edge image of the object is generated by removing the background image having the height "0", that is, the road surface image. Since the relative position between the object and the vehicle is calculated based on the edge image, the relative position between the object and the vehicle can be calculated even in the image in which the smear appears.

[Brief description of the drawings]

FIG. 1 is a diagram showing a basic configuration of the present invention.

FIG. 2 is a block diagram illustrating a configuration of an apparatus to which the present invention is applied.

FIG. 3 is an explanatory diagram of a mounting mode of a CCD camera 11 which constitutes the image pickup unit 10.

FIG. 4 is a flowchart illustrating the operation of a specific example.

FIG. 5 is a flowchart illustrating smear removal processing.

FIG. 6 is a diagram illustrating a captured image including smear.

FIG. 7 is a diagram illustrating a differential image generated from a captured image including smear.

FIG. 8 is a schematic diagram illustrating a specified value for determining smear in a specific example.

FIG. 9 is a diagram illustrating a process of interpolating a smear image.

FIG. 10 is an explanatory diagram of lens aberration correction in a specific example.

FIG. 11 is an explanatory diagram of depression angle correction of a CCD camera in a specific example.

FIG. 12 is an explanatory diagram of three-dimensional position measurement in a specific example.

FIG. 13 is an explanatory diagram of road surface image removal in a specific example.

FIG. 14 is an explanatory diagram of object edge detection in a specific example.

[Explanation of symbols]

 10 image pickup unit 11, 11R, 11L CCD camera 20 steering angle detection unit 30 storage unit 31R, 31L frame memory 34 differential image memory 35 edge image memory 40 data processing unit 41 CPU 42 ROM 43 RAM 50 display unit 60 alarm unit 100 vehicle 200 Obstacle 500 Smear

Claims (5)

[Claims] 1. A vehicle periphery monitoring device having a CCD camera for capturing an image of a predetermined area around a vehicle, and providing a driver with information on an object existing in the periphery of the vehicle based on image information from the CCD camera, A vehicle periphery characterized by having a smear detecting means for detecting a linear smear image appearing on an image picked up by a CCD camera by a high-intensity light source, and a smear removing means for removing the smear image detected by the smear detecting means. Monitoring equipment. 2. The smear detection means acquires edge image generation means for generating an edge image based on image information from the CCD camera, and coordinate values of a pixel of interest of pixels forming the edge image. Coordinate value acquisition means, first brightness value acquisition means for acquiring the brightness value of the corresponding pixel corresponding to the pixel of interest in the image information from the CCD camera based on the coordinate value of the pixel of interest, 1 Based on the luminance value of the corresponding pixel from the luminance value acquisition means, when the luminance value of the corresponding pixel is equal to or more than a specified value, the image information from the CCD camera is referred to and a straight line including the corresponding pixel is displayed. Second brightness value acquisition means for acquiring the brightness value of the existing pixel group, and pixels existing on a straight line including the corresponding pixel based on the brightness value of the pixel group acquired by the second brightness value acquisition means There vehicle periphery monitoring device according to claim 1, characterized in that it has a smear determining means for determining whether a smear image. 3. The smear removing means obtains a luminance value of a pixel group on a boundary line adjacent to the pixel group judged to be a smear image by the smear judging means.
A brightness value acquisition unit, and a brightness value of the pixel group determined to be the smear image based on the brightness value of the pixel group on the boundary line acquired by the third brightness value acquisition unit and the calculated brightness value. The vehicle periphery monitoring device according to claim 2, further comprising image interpolation means for interpolating a pixel group determined to be a smear image. 4. The vehicle periphery monitoring device according to claim 1, wherein the coordinate value acquisition unit searches for an edge point in an arbitrary row in the edge image generated by the edge image generation unit. . 5. An object existing around the vehicle based on image information from the CCD cameras, the two CCD cameras being installed at a predetermined distance from the vehicle and imaging a predetermined range around the vehicle. In a vehicle periphery monitoring device that provides information to a driver, a smear detecting means for detecting a linear smear image appearing on an image picked up by a CCD camera by a high-intensity light source, and a smear image detected by the smear detecting means With respect to the smear removing means and the captured image from which the smear image has been removed by the smear removing means, all the images captured by one CCD camera are assumed to have a height of 0, and this image is captured by the other camera. The projection image data obtained as is created, and the road surface is calculated from the difference between the created projection image data and the image data recorded in the other memory. An object edge detection unit that removes the above image and detects the edge of the object, an object position calculation unit that calculates the position of the object from the object edge detected by the object edge detection unit, and an object position calculation unit A vehicle periphery monitoring device comprising: an alarm unit that outputs an alarm based on the calculated position of the object.