JPH0478924B2 - - Google Patents

Description

[Detailed description of the invention]

[Purpose of the invention] (Field of industrial application) This invention images an object from two directions, and
The present invention relates to a stereo visual device that is used to view an object three-dimensionally (so-called stereo viewing) using images from different directions and detect the position of an object such as an obstacle. (Prior art) For example, when a robot is to perform extreme work such as maintenance and inspection of a nuclear power plant in consideration of safety against radiation, etc., the robot must be able to move around obstacles to prevent it from colliding with them. In order to detect the position of objects such as .71, No.7,
PP.872-884, “The Standard” by H.P. Moravec, published in 1983.
(see "Cart and the CMV Rover"), it is necessary to detect obstacles and move to the destination while avoiding them. Conventional methods for obtaining three-dimensional positional information of an object, that is, methods for obtaining positional information of an object by stereoscopically viewing the object, include a method using ultrasound, a method of projecting a laser beam pattern, etc. Many methods have been proposed, but it has become possible to use them in general environments, and by using images, it has become possible to recognize and understand the entire moving environment visually or three-dimensionally. Stereo vision method, i.e. 2
An object is imaged from two directions using a camera on a stand, and corresponding points are found in the two left and right images from these two directions. From these corresponding points and the placement of the camera, the three-dimensional position of the object and the object are determined using the principle of triangulation. Stereo vision methods for determining the distance between the
(See "Binocular Stereoscopic Vision for Object Recognition" by Yasue and Shirai, Vol. 12, pp. 1101-1119, 1974). In this conventional stereo vision method, feature points (object points) in objects in the two left and right images are detected from both the left and right images, and the principle of triangulation is applied to these detected corresponding points. Although the position of the point is detected, since we are trying to associate the object points,
This point is affected by shadows and noise, and the correspondence is incorrect due to failures in extraction, overlapping objects, differences in the field of view of the left and right cameras, etc., and the position of the object cannot be calculated accurately.
Reliability is significantly reduced. To solve this problem, another method has been proposed that uses DP matching to maintain consistency over a wide range (for example, Journal of the Institute of Electronics and Communication Engineers, No. J68- D, No.4, No.
(See "Segment Correspondence Method for Stereo Images Using Dynamic Programming" by Ota, Masai, and Ikeda, pp. 554-561, April 1985). (Problems to be Solved by the Invention) In the conventional stereo vision method described above, necessary image information is easily lost due to the effects of shadows and noise, and due to overlapping objects and differences in camera field of view, etc. If the correspondence is incorrect, there are problems such as a significant drop in reliability, and other methods have problems in determining the consistency index. The present invention has been made in view of the above, and its purpose is to provide a stereo visual device that can accurately and reliably detect the three-dimensional position of an object to be imaged without error. [Structure of the Invention] (Means for Solving the Problems) In order to solve the above problems, the object to be imaged is imaged from two directions, and the object to be imaged is imaged from two directions. The present invention is a stereo visual device that measures a three-dimensional position by viewing stereoscopically, and the present invention includes a feature detecting means for detecting a feature from the first image information, and a feature detecting means for detecting a feature from the first image information, and a first mask window forming means for forming one mask window; and a second mask window forming means for forming a second mask window at a position shifted from the first mask window by a parallax corresponding to the measurement distance with respect to the first image.
The present invention is characterized in that it has a second mask window forming means for forming the image on the image, and a similarity index calculating means for calculating the similarity index in the mask window corresponding to each of the first and second images. (Function) In the stereo visual apparatus of the present invention, a characteristic part is detected from a first image of an object to be imaged from two directions, and a first mask window of a predetermined area is formed for the characteristic part, A second mask window is formed that is shifted by a parallax with respect to the first mask window, and a similarity index of corresponding first and second images in the first and second mask windows is calculated. (Example) Hereinafter, an example of the present invention will be described using the drawings. FIG. 1 is a block diagram showing the configuration of essential parts of a stereo visual apparatus according to an embodiment of the present invention. This stereo vision device is built into a robot that moves within a nuclear power plant to perform extreme work such as maintenance and inspection inside the nuclear power plant, and detects objects such as obstacles that exist in the direction in which the robot is moving. It is used to control robots so that they do not collide with objects. This stereo vision device has, for example, two left and right cameras mounted on the robot horizontally side by side, and uses these two cameras to detect objects existing in the direction in which the robot moves, that is, objects in the moving environment. Two left and right images are captured, and moving environment image information, which is the captured left and right images, is input to the stereo image input unit 3 as shown by arrow 1 in FIG. Of the pair of left and right original image information of the moving environment captured by the two cameras described above and input to the stereo image input unit 3, one of the images, for example, the left image information, is supplied to the feature edge extraction unit 5, Here, as will be described later, characteristic edge portions are extracted from the captured image. This extracted characteristic edge portion is supplied to the mask generation section 7. This mask generation section 7 creates a left mask window for the feature edge portion, and stores the left original image corresponding to this left mask window as a left window memory. The left original image of the characteristic edge portion stored as the left window memory in the mask generation unit 7 is supplied to the similarity index calculation/judgment unit 9, which generates a right mask centered on a position shifted to the left by the amount of parallax with respect to the left window memory. A window is created, and the right original image for this right mask window is stored as a right window memory. Then, the similarity index values of the images stored in the left and right window memories respectively stored in this way are calculated, and it is determined from this similarity index value whether or not there is an object such as an obstacle. This is output as information on the presence or absence of the object. Next, with reference to FIGS. 2 to 4, the details of the configuration of each part of the feature edge extraction section 5, mask generation section 7, and similarity index calculation determination section 9 will be explained in detail, and then the overall operation will be explained in the fifth section. The explanation will be made with reference to FIGS. 7 to 7. First, as shown in FIG.
9, 21, 23, 25 and similarly image bus 13
and address bus 15, and image bus 1
Connected to each image memory above through 3.
TV camera 27, spatial filtering circuit 29,
A spatial filter bank 3 includes a binarization circuit 33, a line thinning circuit 37, and a noise removal circuit 41, and is connected to the spatial filtering circuit 29.
1. Consisting of a threshold memory 35 connected to a binarization circuit 33, a logic filter bank 39 connected to a thinning circuit 37, and a threshold memory 43 connected to a noise removal circuit 41. . Note that the first image memory 17 is 1 in FIG.
Although only one image memory 1 is shown, this first image memory 1
Reference numeral 7 indicates a first left image memory 17L and a first right image memory 17 corresponding to the two left and right cameras.
The first left image memory 17L stores left original image information taken by the left camera, and the first right image memory 17R stores right original image information taken by the right camera. It's summery. Further, as shown in FIG. 3, the mask generation section 7 is connected to the image bus 13 and the address bus 15, and stores the positions of the characteristic edges stored in the characteristic edge extraction section 5 in the fifth image memory 25. An edge position detection circuit 45 to detect, a feature edge position memory 47 that stores position information of a feature edge detected by this edge position detection circuit 45, a mask window for the feature edge position information stored in this feature edge position memory 47, especially a left mask window creation circuit 49L that creates a left mask window;
The configuration includes a left window memory 51L that reads out and stores the left original image corresponding to the left mask window created by the left mask window creation circuit 49L from the first left image memory 17L. Furthermore, as shown in FIG. 4, the similarity index calculation/judgment unit 9 generates a right mask window generation circuit 49R and a right mask window generation circuit 49R and a right mask window generation circuit 49L and a left window memory 51L, respectively, which constitute the mask generation unit 7. A parallax designation circuit 53 having a window memory 51R and supplying parallax information S corresponding to the distance L to be measured to the right mask window creation circuit 49R, and the left window memory 5
A similarity index calculation circuit 55 that calculates similarity index values of the left image and right image output from the 1L and right window memories 51R, and a similarity index memory 57 that stores the similarity index values calculated by this similarity index calculation circuit 55. The configuration further includes: The operation of the stereo visual apparatus of this embodiment configured as described above will be explained with reference to FIGS. 5 to 7. First, left and right original image information captured by the two left and right cameras via the stereo image input section 3 is stored in the first left image memory 17L and first right image memory 17R, respectively. One of the left and right original image information, for example, the left original image information stored in the first left image memory 17L, is supplied from the first left image memory 17L to the spatial filtering circuit 29, Edge portions in the image of the object or moving environment shown in the left original image are emphasized, and this edge-enhanced left image information is stored in the second image memory 19.
In this spatial filtering circuit 29, the operation of obtaining edge-enhanced left image information is performed using image values supplied from the first left image memory 17L and stored in a spatial filter bank 31 connected to the spatial filtering circuit 29. This is achieved by performing a product-sum operation with a numerical string, and this is constructed by a multiplier and an adder. As an example, in the case of spatially filtering the left original image captured by the left camera as shown in Fig. 5a, the results are obtained by processing the left original image using, for example, a 3x3 spatial filter having the following weighting coefficient as the spatial filter. The edge-enhanced left image of is shown in FIG. 5b.

[Table] The image in Figure 5a shows structures such as walls, columns, and ceiling beams inside the building, but in Figure 5b, only the edges of these structures are highlighted. The edge-enhanced left image shown in FIG. The edge-enhanced left image information stored in the second image memory 19 is supplied to the binarization circuit 33 via the image bus 13, and based on the threshold value stored in advance in the threshold value memory 35, 0, 1] and stored in the third image memory 21. In this case, a fixed value stored in the threshold memory 35 may be used as the threshold for binarization, but it is also possible to dynamically determine the threshold by measuring the density distribution of the original image. It's okay. In this embodiment, a fixed value stored in the threshold value memory 35 is used. Figure 5c shows this binarized 2
It shows a value image, which is used as a threshold
100. The binary image of the edge-enhanced left image stored in the third image memory 21 is supplied to the thinning circuit 37 via the image bus 13, and the logical filter bank 39
According to the mask stored in , thinning processing is performed until the line width of the binary edge becomes one pixel, and this thinning image information is stored in the fourth image memory 23 . Further, the edge thinning image information stored in the fourth image memory 23 is supplied to the noise removal circuit 41 via the image bus 13, where the edges are labeled in order of their length, and the edge thinning image information is output from the threshold memory 43. Short ones that can be considered as noise below the threshold are removed, and the thinned image information of the remaining long edges is extracted as characteristic edge image information and stored in the fifth image memory 2.
Store in 5. Figure 5d shows this extracted feature edge image, where most of the short edge images present in the binary image shown in Figure 5c have been removed, and only long feature edge image information is shown. has been done. The characteristic edge image information for the left image stored in the fifth image memory 25 by the characteristic edge extraction section 5 as described above is sent to the edge position via the image bus 13 in the mask generation section 7 shown in FIG. The detection circuit 45 scans and detects the starting point coordinates (Xs, Ys) and edge length l of the characteristic edge group and stores them in the characteristic edge position memory 47. The start point coordinates and edge length of the feature edge stored in the feature edge position memory 47 are supplied to the left mask window creation circuit 49L, and a mask window for the feature edge is created based on this information, and a mask window corresponding to this mask window is created. original image,
That is, among the left original images written in the first left image memory 17L, the original image corresponding to the mask window is stored in the left window memory 51L configured with a high-speed RAM. FIGS. 6a and 6b are diagrams showing this mask window and the original image corresponding to the mask window. As shown by the line 61 in FIG. A mask window 63 indicated by a rectangle 63 for a feature edge 61 having a length l is formed as a rectangular area centered on a point on the feature edge 61 that is a certain pixel value below the starting point, and with the edge 61 as the central axis. be done. For the mask window 63 formed in this way, the sixth
As shown in FIG. b, the mask window corresponding original image 65 of the corresponding left original image (stored in the first left image memory 17L) is stored in the first left image memory 17L.
, and stored in the left window memory 51L. In addition, in FIG. 6b, the reference numeral 6
Reference numeral 7 indicates an object in the moving environment imaged by the left camera, that is, the above-mentioned structure. As described above, when the left mask window is formed in the left mask window creation circuit 49L by the processing operation of the mask generation unit 7, and the left mask window corresponding original image 65 is stored in the left window memory 51L,
A right mask window corresponding to the left mask window is formed by the right mask window creation circuit 49R of the similarity index calculation/judgment section 9. This right mask window has a distance L to be measured with respect to the left mask window.
The right mask window creation circuit 49R uses the start of the characteristic edge of the left mask window supplied from the characteristic edge position memory 47 of the mask generation section 7. A right mask window is formed based on the point coordinates and length information and the parallax information supplied from the parallax designation circuit 53, and the right original image corresponding to this right mask window, that is, the right mask window corresponding original image information, is It is read from the right image memory 17R and stored in the right window memory 51R. The right mask window corresponding original image information stored in the right window memory 51R and the left mask window corresponding original image information stored in the left window memory 51L are input to the similarity index calculation circuit 55, where the left and right windows are Similarity index values for both images in the memory are calculated and stored in the similarity index memory 57. And the similarity index memory 5
By comparing the similarity index value stored in 7 with a predetermined threshold value, it is accurately determined whether an obstacle exists, and the robot can move without colliding with the obstacle. Note that between the above-mentioned parallax S and distance L, the optical axes of the left and right cameras are made parallel, the mounting heights are set equal, and the distance between the left and right cameras is a, and the distance between the imaging surfaces at the center of the lens is Assuming that l is the pixel/distance conversion ratio and ε is the pixel/distance conversion ratio, there is the following relationship, and based on this relationship, the parallax designation circuit 53 can supply the parallax information S to the right mask window creation circuit 49R. S=εal/L...(1) Moreover, the above-mentioned similarity index value ψab is obtained using a function expressed by the following equation. ψab＝ 〓 { ^Aij } 〓 〓 { ^Bij }{(Aij−μa)/σ _a −(Bij−μb)/σ _b } ² ……
(2) In the above equation, Aij...the density value of each pixel of the image within the mask window of the left input image, μa...the average density of the image within the mask window of the left input image, σa...the mask of the left input image Standard deviation of the image within the window, Bij...The density value of each pixel of the image within the mask window of the right input image, μb...The average density of the image within the mask window of the right input image, σb...The density value of each pixel of the image within the mask window of the right input image The standard deviation of the image within the mask window is . Figure 7 shows the similarity index value 3× based on the above formula (2).
The figure shows the predicted position of a 10-pixel mask window in the horizontal direction. As can be seen from this figure, it is confirmed that the similarity index value is minimized at the position where the positions of both the left and right mask windows correctly correspond to the position of the obstacle. In addition,
In FIG. 7, characteristics for each edge indicated by d in FIG. 5 are shown. In the above embodiment, a case was explained in which an obstacle is detected by, for example, a robot, but the present invention is not limited to this, and the present invention is not limited to this, and the present invention is not limited to this, and the present invention is not limited to this. It is widely applicable to [Effects of the Invention] As described above, according to the present invention, a characteristic part is detected from a first image of an object to be imaged from two directions, and a first mask is applied to a predetermined area for the characteristic part. forming a second mask window that is shifted by a parallax with respect to the first mask window, and determining similarity indices of corresponding first and second images in the first and second mask windows. Since the similarity index is calculated, it is possible to detect an accurate and precise correspondence between the first and second images based on this similarity index, and the three-dimensional position of the imaged object can be determined using the principle of triangulation. Detection can be performed accurately and reliably without errors.

[Brief explanation of the drawing]

FIG. 1 is a block diagram showing the configuration of a stereo visual device according to an embodiment of the present invention, FIG. 2 is a block diagram of a feature edge extraction unit used in the device shown in FIG. 1, and FIG. 4 is a block diagram of the similarity index calculation and determination section used in the device shown in FIG. 1, and FIG. 5 is an image captured by the stereo visual device shown in FIG. 1. and a diagram showing each image as a result of processing this image, FIG. 6 is a diagram showing a mask window for the feature edge processed by the mask generation unit of FIG. 3, and an original image corresponding to the mask window, and FIG. is a graph showing similarity index values calculated by the similarity index calculation determination section of FIG. 4; 3... Stereo image input unit, 5... Feature edge extraction unit, 7... Mask generation unit, 9... Similarity index calculation determination unit, 17... First image memory, 19...
Second image memory, 21...Third image memory,
23...Fourth image memory, 25...Fifth image memory, 47...Feature edge position memory, 49L
...Left mask window creation circuit, 49R...Right mask window creation circuit, 51L...Left window memory, 51R...Right window memory, 55...Similarity index calculation circuit.

Claims (1)

[Scope of Claims] 1. A stereo visual device that images an object to be imaged from two directions, and measures the three-dimensional position by viewing the object three-dimensionally using first and second images from these two directions. a feature detection means for detecting a feature from the first image information; a first mask window forming means for forming a first mask window in a predetermined area with respect to the feature; a second mask window forming means for forming a mask window on a second image at a position shifted from the first mask window by a parallax corresponding to a measurement distance with respect to the image; 1. A stereo visual device comprising: similarity index calculation means for calculating similarity indexes within mask windows corresponding to two images. 2. The stereo visual apparatus according to claim 1, wherein the characteristic portion is an edge portion characteristically present in the first image information. 3 The second mask window forming means sets optical axes parallel to each other for imaging the object to be imaged from two directions, sets a distance to the object to be imaged by L, and sets a distance L between the imaging means for imaging the object to be imaged from two directions. When the distance a is the distance between the imaging surfaces of the lens center of the imaging means, l, and the pixel/distance conversion ratio is ε, the second mask window is converted to the first mask window based on the formula S=εal/L. 2. The stereo visual apparatus according to claim 1, further comprising a parallax calculation means for calculating a parallax S to be shifted relative to the stereo visual apparatus.