CN108564604A - Binocular vision solid matching method and device based on plane restriction and triangulation - Google Patents

Abstract

Embodiments of the present invention provide a binocular vision stereo matching method and device based on plane constraints and triangulation. The method includes: obtaining left and right images collected by two cameras, and using any one as a reference image; determining support points of the reference image and calculating parallax; segmenting the reference image in a preset manner; dividing each segmented area into the first category and/or Or the second type of segmentation area; each first type of segmentation area is divided twice; according to the support points and parallax of each sub-segmentation area, the disparity of non-support points in the area is determined; each second type of segmentation area is triangulated According to the vertices and disparity of each triangular area, determine the disparity search range of unsupported points in the area; match the corresponding points in the range with unsupported points pixel by pixel, and determine the disparity of unsupported points. The present invention can more accurately match occluded areas and large-area non-textured areas. In addition, the parallax search range of the small-area segmentation area is reduced, and the matching efficiency is improved.

Description

Binocular Vision Stereo Matching Method and Device Based on Plane Constraint and Triangulation

technical field

The invention relates to the technical field of computer vision, in particular to a binocular vision stereo matching method and device based on plane constraints and triangulation.

Background technique

Binocular stereo vision mainly includes four steps: binocular camera calibration, acquisition of two-dimensional image pairs, image correction, and binocular vision stereo matching. Among them, binocular vision stereo matching includes: first, use a binocular camera or two parallel cameras to shoot the same physical scene at the same time to obtain left and right images; then find the projection point of the same target in the scene in the left and right images, which is called the corresponding point ;Finally, a disparity map is generated according to the disparity of the corresponding point, that is, the offset of the corresponding point in the u-axis direction of the pixel coordinate system. The actual distance between the target and the camera in the captured physical scene can be further calculated according to the generated disparity map and the principle of similar triangles. This process is also called stereo vision depth calculation. Stereo vision depth calculation is widely used in practical applications such as 3D scene reconstruction and autonomous navigation of mobile robots. At the same time, it is also widely used in medical imaging, industrial inspection and other fields.

Currently commonly used binocular vision stereo matching algorithms include: local algorithms based on support windows, global algorithms based on image cutting or dynamic programming, non-local matching algorithms based on minimum spanning trees, etc.

Since the left and right images are obtained from different shooting angles, it is inevitable that there will be occluded areas or large non-textured areas. Among them, the occluded area refers to an area that is only visible in one image and has no corresponding pixel information in another image, which is usually caused by different shooting angles. Large areas without texture usually refer to those areas without obvious feature points, such as white walls in indoor areas, etc. However, the above-mentioned stereo matching algorithms are all matched pixel by pixel. Therefore, for the lack of pixel information in occluded areas and the ambiguity in matching large areas without texture, the above-mentioned stereo matching algorithms cannot well solve the problem of occluded areas or large areas. The matching of areas with no texture will result in higher matching error rate and lower matching efficiency.

Contents of the invention

The purpose of the embodiments of the present invention is to provide a binocular vision stereo matching method and device based on plane constraints and triangulation, so as to reduce the matching error rate of binocular vision stereo matching and improve matching efficiency. The specific technical scheme is as follows:

In order to achieve the above object, in the first aspect, the implementation of the present invention provides a binocular vision stereo matching method based on plane constraints and triangulation, the method comprising:

Obtain the left image and the right image of the same shooting scene collected by two parallel cameras at the same time, and use any one of them as a reference image and the other as an image to be matched;

Determining the support points in the reference image, and calculating the parallax of each of the support points; wherein, the support points are pixels in the reference image that have unique and correct matching points in the image to be matched ;

According to the preset image segmentation method, the reference image is segmented to obtain a plurality of segmented areas; each of the segmented areas is divided into the first type of segmented area whose segmented area is larger than the first threshold, and/or the segmented area is not larger than the second type of segmentation region of the first threshold;

For each first-type segmented area, perform: perform a second segment on the first-type segmented area, divide the first-type segmented area into a plurality of first-type sub-segmented areas; The support points in the region and the parallax of the support points are fitted to the first plane equation of the first type of sub-segmentation area; according to the first plane equation, the parallax of the non-support points in the first type of sub-segmentation area is determined; wherein , the plurality of first-type sub-divided regions respectively correspond to different physical planes in the shooting scene;

For each second-type segmented area, execute: triangulate the second-type segmented area, divide the second-type segmented area into a plurality of triangular areas; for each non-support point in each triangular area, Determine the first parallax search range corresponding to the unsupported point according to the vertices of the triangular area and the parallax of each vertex; match the points to be matched corresponding to each parallax in the first parallax search range with the unsupported point pixel by pixel. , to determine the disparity of the non-support point; wherein, each of the vertices is a support point.

Optionally, the determining the support points in the reference image includes:

Taking the pixel point at the upper left corner of the reference image as a starting point, along the u-axis and the v-axis of the pixel coordinate system corresponding to the reference image, construct a grid with a step size of a preset step size in the reference image; In the above grid, except the intersection point at the edge of the reference image as a candidate support point; wherein, the u axis is rightward, and the v axis is downward;

For each candidate support point, calculate the distance between the vector of the point to be matched and the vector of the candidate support point corresponding to each parallax in the second parallax search range corresponding to the candidate support point that is preset; if the calculated distance is small Among the distances of the second threshold, if the number of the smallest distance is 1, then the candidate support point is used as a support point; wherein, the vector of the point to be matched corresponding to each parallax in the second parallax search range, and the The vectors of the candidate support points are all calculated according to the sobel operator of the preset size.

Optionally, the reference image is segmented according to a preset image segmentation method to obtain a plurality of segmented regions, including:

The reference image is segmented according to the Meanshift image segmentation manner to obtain at least one segmented region.

Optionally, performing a second segmentation on the first type of segmented area, dividing the first type of segmented area into multiple first type sub-segmented areas, including:

Fit the plane equation of the physical plane corresponding to the plurality of support point subsets in the first type of segmented area according to the preset plane fitting rule; wherein, the plurality of support point subsets respectively correspond to different physical plane;

Among the fitted physical planes, if multiple physical planes satisfy the preset merging condition, then according to the support points in the multiple physical planes and the parallax of the support points, the physical plane after fitting is obtained by fitting. Plane equation; according to the plane equation of the merged physical plane and the plane equation of the unmerged physical plane, calculate the intersection line between each merged physical plane and the unmerged physical plane, and determine the plurality of first-type sub-planes Segmenting an area; wherein, the preset merging condition is: among the plurality of physical planes, the distance between any physical plane and at least one other physical plane is less than a third threshold;

Among the fitted physical planes, if any two of the physical planes do not satisfy the preset merging condition, the intersection line between the physical planes is calculated according to the plane equations of the physical planes, The plurality of first-type sub-divided regions are determined.

Optionally, according to the preset plane fitting rules, fitting the plane equations of the physical planes corresponding to the plurality of support point subsets in the first type of segmented area includes:

S1. Based on the random sampling consistency RANSAC algorithm, select a target support point subset O from the current support point set Y in the first type of segmented area; wherein, the O is the support with the largest number of support points in the Y set of ideas;

S2. Fitting the plane equation of the physical plane corresponding to the O according to the support points of the O and the parallax of the support points;

S3. Judging whether the number of support points of the currently unfit support point subset Y-O is less than the number of support points in the first type of segmented area of the preset ratio;

If the number of support points of the Y-O is not less than the number of support points in the first-type segmented area of the preset ratio, then use the Y-O as the current support point set in the first-type segmented area, and return to execute Step S1;

If the number of support points of the Y-O is less than the number of support points in the first-type segmented area of the preset ratio, the plane equation of the currently fitted physical plane is used as the final fitting result.

Optionally, fitting the first plane equation of the first type of sub-segmentation area according to the support points in each first type of sub-segmentation area and the disparity of the support points includes:

Set the first plane equation of the first type of sub-division area as:

d(u,v)=au+bv+c

Wherein, d(u, v) represents the disparity of the pixel point (u, v) in the first type of sub-division area, a, b, c are the parameters to be solved of the first plane equation, and the a is solved by the following formula ,b,c:

Among them, u _i represents the coordinates of the u-axis of the i-th support point in the pixel coordinate system corresponding to the reference image in the first type of sub-segmentation area, and v _i represents the i-th support point in the first type of sub-segmentation area. The coordinates of the v axis of the pixel coordinate system corresponding to the reference image, the u axis is right, the v axis is down, i=1,..., m, m represents the number of support points in the first type of sub-division area, d _i represents the disparity of the i-th support point in the sub-segmentation area of the first type.

Optionally, for each unsupported point in each triangular area, determining the first parallax search range corresponding to the unsupported point according to each vertex of the triangular area and the disparity of each vertex includes:

For each triangular area, according to each vertex in the triangular area and the parallax of each vertex, determine the second plane equation of the plane where the triangular area is located;

determining an estimated parallax value of the unsupported point according to the second plane equation;

The first disparity search range includes: the disparity of each vertex, the disparity obtained by adding and subtracting a preset value from the disparity of each vertex, the estimated disparity value, and the preset neighborhood of the non-support point The disparity of the target support point; wherein, the absolute value of the difference between the disparity of the target support point and the estimated value of the disparity of the target support point determined according to the second plane equation does not exceed a fourth threshold.

Optionally, performing pixel-by-pixel matching of the points to be matched corresponding to each parallax in the first parallax search range with the non-support point to determine the parallax of the non-support point includes:

The non-support point is determined by the following formula Parallax for:

in, for the non-support point The disparity, d _n represents the disparity estimated value of the non-support point, Represents each disparity within the first disparity search range, N represents the number of disparities within the first disparity search range, and S represents the support point set of the reference image; express The posterior probability of , calculated by the following formula:

in, Indicates that the point to be matched corresponding to d _n is the non-support point The probability of the correct matching point of , Indicates that the matching points corresponding to each disparity within the first disparity search range are the non-support points The probability of the correct matching point of .

In the second aspect, an embodiment of the present invention provides a binocular vision stereo matching device based on plane constraints and triangulation, the device comprising:

The obtaining module is used to obtain the left image and the right image of the same shooting scene collected by two parallel cameras at the same time, and any one of them is used as a reference image, and the other is used as an image to be matched;

A determining module, configured to determine the support points in the reference image, and calculate the disparity of each of the support points; wherein, the support points in the reference image are unique and correct in the image to be matched Pixels of matching points;

A segmentation module, configured to segment the reference image according to a preset image segmentation method to obtain a plurality of segmented areas; divide each segmented area into a first type of segmented area whose segmented area is greater than a first threshold, and/or or a second type of segmented area whose segmented area is not greater than the first threshold;

The first execution module is configured to, for each first-type segmented area, perform: performing a second segment on the first-type segmented area, and dividing the first-type segmented area into a plurality of first-type sub-segmented areas; The support points and the parallax of the support points in each first type of sub-segmentation area are fitted to the first plane equation of the first type of sub-segmentation area; according to the first plane equation, determine the first type of sub-segmentation area Parallax of non-support points; wherein, the plurality of first-type sub-segments respectively correspond to different physical planes in the shooting scene;

The second execution module is configured to, for each second-type segmented area, execute: perform triangulation on the second-type segmented area, and divide the second-type segmented area into a plurality of triangular areas; for each triangular area For each non-support point, determine the first parallax search range corresponding to the non-support point according to each vertex of the triangular area and the parallax of each vertex; the points to be matched corresponding to each parallax in the first parallax search range and the The non-support point is matched pixel by pixel to determine the disparity of the non-support point; wherein, each of the vertices is a support point.

In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete communication with each other through the communication bus;

memory for storing computer programs;

The processor is configured to implement the method steps for binocular vision stereo matching based on plane constraints and triangulation as described in the first aspect when executing the program stored in the memory.

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes the plane-based Method steps for binocular stereo matching with constraints and triangulation.

In the fifth aspect, the embodiment of the present invention provides a computer program product containing instructions, which, when run on a computer, enables the computer to perform binocular vision stereo matching based on plane constraints and triangulation as described in the first aspect above method steps.

In the binocular vision stereo matching method and device based on plane constraints and triangulation provided by the embodiments of the present invention, after the reference image is segmented, the segmented area with a larger area is segmented a second time to determine the The sub-segmentation area; further fitting the plane equation of the sub-segmentation area according to the support points of the sub-segmentation area and the disparity of the support points, and determining the disparity of the non-support points in the sub-segmentation area according to the plane equation. At the same time, triangulate the smaller segmented area and divide it into multiple triangles; further determine the first parallax search range of non-support points in the triangle according to the apex of the triangle and the parallax of the apex, and correspond to each parallax in the range The point to be matched is matched pixel by pixel with the non-support point, so as to determine the disparity of the non-support point.

It can be seen that in the embodiment of the present invention, for each sub-segmented area of a larger-area segmented area, based on the idea of plane constraints, the parallax of all non-support points in each sub-segmented area can be determined through the plane equation of each sub-segmented area, and the parallax can be realized. Occluded areas with missing pixel information and large textureless areas with matching ambiguity are more accurately and robustly matched, thereby reducing the matching error rate. And in the embodiment of the present invention, triangulation is carried out for the smaller segmented area, and the parallax search range of the unsupported points in the triangle is determined according to the vertices of the triangle and the parallax of the vertices, which greatly reduces the parallax search range of the unsupported points and improves the matching efficiency.

Of course, implementing any product or method of the present invention does not necessarily need to achieve all the above-mentioned advantages at the same time.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings that are required in the description of the embodiments or the prior art.

Fig. 1 is the flow chart of the binocular vision stereo matching method based on plane constraint and triangulation provided by the embodiment of the present invention;

Fig. 2 is a specific flow chart for realizing the second segmentation of the first-type segmented area in step S104 of the embodiment shown in Fig. 1, and dividing the first-type segmented area into a plurality of first-type sub-segmented areas ;

Fig. 3 is a kind of specific flowchart of step S201 in the embodiment shown in Fig. 2;

Fig. 4 is the simulation experiment result figure that the embodiment of the present invention provides;

5 is a structural diagram of a binocular vision stereo matching device based on plane constraints and triangulation provided by an embodiment of the present invention;

FIG. 6 is a structural diagram of an electronic device provided by an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

In binocular vision stereo matching, in order to reduce the matching error rate of occluded areas and large areas without texture, and improve the matching efficiency, as shown in Figure 1, the present invention provides a binocular vision system based on plane constraints and triangulation A visual stereo matching method, the method comprising:

S101. Obtain the left image and the right image of the same shooting scene respectively collected by two parallel cameras at the same time, and use any one of them as a reference image and the other as an image to be matched.

In order to imitate the three-dimensional stereoscopic vision produced by human eyes when watching a scene, in computer vision, the left and right images of the same shooting scene collected by two parallel cameras at the same time can be obtained to further find the shooting scene. The projected points of the target in the left and right images, also known as corresponding points, generate a disparity map. Then, the disparity map can be converted into a depth map containing the depth information between the camera and each object in the shooting scene, that is, the actual distance between the camera and each object in the shooting scene, for further three-dimensional reconstruction.

In this embodiment, the left and right images of a shooting scene can also be collected simultaneously by two camera units of a binocular camera.

Either the left image or the right image collected by the camera can be used as a reference image, if one of them is used as a reference image, then the other is the image to be matched. For example, if the left image is used as a reference image, the purpose of the present invention is to determine the disparity of each pixel in the left image and generate a disparity map of the left image. In this embodiment, for example, each step is described in detail by taking the left image as a reference image.

S102. Determine the support points in the reference image, and calculate the disparity of each of the support points; wherein, the support points are those in the reference image that have unique and correct matching points in the image to be matched pixel.

The present invention is based on the observation that a certain point in the actual scene is usually not isolated, but coexists with other points on the same physical plane. Correspondingly, a certain pixel in the reference image is usually not isolated, but Together with some other pixels, it corresponds to the same physical plane in the shooting scene. Then, based on the idea of plane constraints, if the physical plane corresponding to each pixel in the reference image can be accurately determined, the disparity value can be filled with the plane as the smallest calculation unit, that is, the physical plane corresponding to the pixel can be Plane equation to determine the disparity of the pixel. Therefore, the disparity determination of pixels in occluded regions and large texture-free regions is robust.

Based on the above observation, from another perspective, the reference image must correspond to at least one physical plane in the shooting scene. In order to determine the plane equation of the physical plane corresponding to the reference image, in this embodiment, some support points in the reference image can be determined in advance. The difference has better robustness and can be used to fit the plane equation in subsequent calculations.

It can be understood that there must be many pixels in the reference image that have unique and correct matching points in the image to be matched, but in this embodiment, it is not necessary to determine all such pixels, but can be selected A part of such pixel points is determined as support points.

In an implementation manner, determining the support points in the reference image may include the following steps:

S11, using the upper left pixel point of the reference image as a starting point, constructing a grid with a preset step size in the reference image along the u-axis and v-axis of the pixel coordinate system corresponding to the reference image; Taking the intersection points in the grid except at the edge of the reference image as candidate support points; wherein, the u axis is rightward, and the v axis is downward.

It can be understood that by constructing a grid in the reference image, using the intersection points in the grid except at the edge of the reference image as candidate support points, and further determining the support points from the candidate support points, the support points can be evenly distributed. distributed in the reference image. When fitting the plane equation through the support points and the parallax of the support points, the distribution of the support points in each plane is relatively uniform, and the density of the support points of different planes is relatively similar, so that the fitted plane equation can be more accurate .

For example, the size of the reference image is 256×256, the upper left pixel point is (0, 0), the preset step size is 5 pixels, and the grid is constructed along the u-axis and v-axis of the pixel coordinate system corresponding to the reference image, Then the pixel points at the intersection points in the grid include: the first row of the grid: (0, 0), (0, 5), (0, 10), ..., (0, 255), the second row of the grid: ( 5, 0), (5, 5), (5, 10), ..., (5, 255), ..., grid last row: (255, 0), (255, 5), (255, 10), ..., (255, 255). In the above intersection points, after removing the intersection points at the edge of the reference image, the pixels at the remaining intersection points are candidate support points: the second row of the grid: (5, 5), (5, 10), ..., (5, 255) , the third row of the grid: (10, 5), (10, 10), ..., (10, 255), ..., the last row of the grid: (255, 5), (255, 10), ..., (255 , 255).

The preset step size can be determined according to actual needs or experience.

S12. For each candidate support point, calculate the distance between the vector of the point to be matched and the vector of the candidate support point corresponding to each disparity in the second disparity search range corresponding to the candidate support point that is preset; if the calculated Among the distances that are less than the second threshold, the number of the smallest distance is 1, and the candidate support point is used as a support point; wherein, the vector of the point to be matched corresponding to each parallax in the second parallax search range, And the vector of the candidate support point is calculated according to the sobel operator of the preset size.

Parallax is the difference between the abscissas of the two projection points of the target in the shooting scene on the left and right images. For example, the projection point x _l of the target on the left image is (5, 10), and the projection point x _r on the right image is (5, 8), then the parallax d(x _l )=2 of x _l .

The second disparity search range includes a plurality of disparity values. For a candidate support point, each disparity value in the corresponding second disparity search range is consistent with a pixel to be matched in the image to be matched (referred to as corresponding to the point to be matched). For example, for the candidate support point (5, 255), the corresponding disparity within the second disparity search range is 2, and the point to be matched corresponding to the disparity 2 is (5, 253).

In this embodiment, the minimum value of the second disparity search range may be 0, and the maximum value may be set as required, such as half the width of the reference image. For example, if the width of the reference image is 256, then the second disparity search range can be set as 0-128. Then for the candidate support point (5, 255), the points to be matched corresponding to each parallax in the corresponding second parallax search range are: pixel points (5, 137) in the image to be matched, ..., (5, 255 ). Of course, it can be understood that the maximum value of the second disparity search range corresponding to different candidate support points may be different.

The preset size of the sobel operator can be set according to actual needs, for example, set to 3×3. The second threshold can also be set according to experience, for example, set to 2. In step S12, after the vectors of the candidate support points and the vectors of the points to be matched are obtained through the calculation of the sobel operator, the distances between the vectors of the candidate support points and the vectors of the points to be matched can be calculated to determine the candidate support points and the vectors of the points to be matched. The distance between the points to be matched, for example, the distance can be 0, 1, 2, etc. The smaller the distance, the more similar the candidate support point is to the point to be matched. Further, if the number of the smallest distance among the calculated distances is 1, it means that the candidate support point has the only correct matching point, and the candidate support point can be used as a support point. Conversely, if the number of the smallest distance among the calculated distances is greater than 1, it means that there are multiple points to be matched that are similar to the candidate support points, that is, there is ambiguity in the matching of the candidate support points, then the candidate support points Will not be used as a support point.

For example: the candidate support point L1 has 5 corresponding points to be matched, namely R1-R5, and the second threshold is 2. If the distances between the candidate support point L1 and the points R1-R5 to be matched are: 0, 1, 3, 2, 1, then among the distances smaller than the second threshold (0, 1, 1 in sequence), the smallest If the number of distance 0 is 1, then the candidate support point L1 has the only correct matching point R1, and the candidate support point can be used as a support point. If the distances between the candidate support point L1 and the points to be matched R1-R5 are respectively: 1, 1, 2, 2, 2, then among the distances smaller than the second threshold (1, 1 in sequence), the minimum distance 1 The number of is 2, then the matching of the candidate support point L1 is ambiguous, and the candidate support point will not be used as a support point.

If a candidate support point is used as a support point, the disparity of the support point is the difference between the abscissa of the candidate support point and the abscissa of the matching point.

S103. Segment the reference image according to a preset image segmentation method to obtain a plurality of segmented regions; divide each segmented region into a first-type segmented region whose segmented area is greater than a first threshold, and/or segmented area The second type of segmented regions not greater than the first threshold.

The purpose of segmenting the reference image is to segment the segmented regions corresponding to different physical planes in the reference image as accurately as possible. In an implementation manner, the reference image may be segmented according to a Meanshift image segmentation manner to obtain multiple segmented regions. Specifically, during the segmentation process, the color radius and space radius in the Meanshift segmentation can be adjusted empirically for the size of the reference image and/or other attributes of the image, so as to make the segmentation result as accurate as possible. Normally, when adjusting the color radius and space radius for the attribute of image size, if the image is relatively large, the color radius is relatively small and the space radius is relatively large. For example, for an image with a size of 450×375, you can set the color radius to 30 and the space radius to 30; for an image with a size of 128×128, you can set the color radius to 40 and the space radius to Set at 20.

However, in actual segmentation, segmentation errors may also occur, that is, different pixel points that do not correspond to the same plane in the shooting scene may be segmented into the same segmented area. The larger the size of the segmented region, the more likely this error will occur. Therefore, in this embodiment, a first threshold can be set based on experience, so that the first type of segmented area whose area is larger than the first threshold is subjected to secondary segmentation, and the second type of segmented area whose area is not larger than the first threshold is considered to be The pixels all correspond to the same plane. Specifically, when Meanshift is used to segment the reference image, the setting of the first threshold is related to the color radius and space radius in the Meanshift segmentation. Take the above example as an example: when the image size is 450×375, the color radius is 30, and the space radius is set to 30, the first threshold can be set to 800 pixels, that is, the number of pixels in a segmented area If there are more than 800, the segmented area is the first type of segmented area.

Generally, when the space radius is constant, if the color radius is relatively large, the first threshold is relatively small; when the color radius is constant, if the space radius is relatively large, the first threshold is relatively large.

In this embodiment, other image segmentation methods may also be used to segment the reference image, which is not limited in the present invention.

S104. For each first-type segmented area, perform: perform a second segment on the first-type segmented area, and divide the first-type segmented area into a plurality of first-type sub-segmented areas; The support points and the parallax of the support points in the sub-segmentation area are fitted to the first plane equation of the first type of sub-segmentation area; according to the first plane equation, the parallax of the non-support points in the first type of sub-segmentation area is determined ; Wherein, the plurality of sub-divided regions of the first type respectively correspond to different physical planes in the shooting scene.

Because there is a segmentation error in setting the first type of segmented area, that is, different segmented areas corresponding to multiple planes are all segmented into the same first type of segmented area. Therefore, a second division can be performed on the first-type segmented area to obtain multiple first-type sub-segmented areas, so that the pixels in each first-type sub-segmented area correspond to the same plane.

In one implementation, fitting the first plane equation of the first type of sub-segmentation area according to the support points and the disparity of the support points in each first type of sub-segmentation area in step S104 may include:

Set the first plane equation of the first type of sub-division area as:

d(u,v)=au+bv+c

Wherein, d(u, v) represents the disparity of the pixel point (u, v) in the first type of sub-division area, a, b, c are the parameters to be solved of the first plane equation, and the a is solved by the following formula ,b,c:

Among them, u _i represents the coordinates of the u-axis of the i-th support point in the pixel coordinate system corresponding to the reference image in the first type of sub-segmentation area, and v _i represents the i-th support point in the first type of sub-segmentation area. The coordinates of the v axis of the pixel coordinate system corresponding to the reference image, the u axis is right, the v axis is down, i=1,..., m, m represents the number of support points in the first type of sub-division area, d _i represents the disparity of the i-th support point in the sub-segmentation area of the first type.

Based on the first plane equation of each first type of sub-division area, the disparity of non-support points in the first type of sub-division area can be determined. The non-support point here is a concept opposite to the support point, and specifically refers to the pixel point whose parallax is to be determined.

S105, for each second-type segmented area, perform: perform triangulation on the second-type segmented area, and divide the second-type segmented area into a plurality of triangular areas; point, according to each vertex of the triangular area and the parallax of each vertex to determine the first parallax search range corresponding to the non-support point; the points to be matched corresponding to each parallax in the first parallax search range are compared with the non-support point one by one Pixel matching, determining the disparity of the non-support point; wherein, each of the vertices is a support point.

In this embodiment, although it is considered that the pixels in the second type of segmented area whose area is not greater than the first threshold all correspond to the same plane. However, due to the small area of the second type of segmentation region, the number of support points contained in it is small, and there may not even be enough support points for plane fitting. Therefore, for the second type of segmented area, the present invention uses a triangulation method to calculate the disparity.

Triangulation, the construction of Delaunay triangles, is a fundamental research method in algebraic topology. After triangulating an image area, any two triangles either do not intersect, or intersect exactly at one edge. In this embodiment, since each vertex of the triangle is a support point, the disparity of the pixel points in the triangle is relatively similar to the disparity of each vertex of the triangle. Moreover, the parallax of the pixel points in the triangle is relatively similar to the parallax of the surrounding support points in a certain range. Matching the point to be matched corresponding to the disparity similar to the disparity of a pixel with the pixel can greatly reduce the disparity search range of the pixel, and improve the matching accuracy on the basis of ensuring the matching accuracy. Efficiency, in the specific computer implementation process, achieves faster matching calculations.

In an implementation of the embodiment shown in FIG. 1 , as shown in FIG. 2 , in step S104, the first type of segmented area is divided for the second time, and the first type of segmented area is divided into a plurality of first Class sub-segmentation areas, which can include:

S201. Fit the plane equation of the physical plane corresponding to the plurality of support point subsets in the first type of segmented area according to the preset plane fitting rule; wherein, the plurality of support point subsets respectively correspond to the shooting scene different physical planes.

S202. Among the fitted physical planes, if multiple physical planes satisfy the preset merging condition, according to the support points in the multiple physical planes and the parallax of the support points, fit the merged physical plane The plane equation of the plane; according to the plane equation of the merged physical plane and the plane equation of the unmerged physical plane, calculate the intersection line between each merged physical plane and the unmerged physical plane, and determine the plurality of first Class subdivided regions; wherein, the preset merging condition is: among the plurality of physical planes, the distance between any physical plane and at least one other physical plane is less than a third threshold;

In this embodiment, for the multiple planes of the first type of segmented area obtained in step S201, if the distance between two of the planes is relatively small, the two planes may be merged and regarded as one plane. It can be understood that the number of support points of the merged plane is more than that of the unmerged single plane, so fitting the plane equation through the support points of the merged plane is more accurate than the plane equation of a single plane. The above-mentioned third threshold can be set according to experience, for example, set to 2.

Specifically, calculating the distance between two planes can be achieved in the following ways:

Set the plane equation of plane 1 as: A ₁ *x+B ₁ *y+C ₁ *d+D ₁ =0, its center point is (x ₁ , y ₁ , d ₁ ), the plane equation of plane 2 is : A ₂ *x+B ₂ *y+C ₂ *d+D ₂ ＝0, its center point is (x ₂ , y ₂ , d ₂ ), then the distance between plane 1 and plane 2 is D=dis ₁ + dis ₂ , where,

Specifically, in step S202, the implementation process of "according to the support points in the multiple physical planes and the disparity of the support points, fitting to obtain the plane equation of the combined physical plane" can refer to the "according to each Support points in a class of sub-segmentation regions and the parallax of the support points, fitting the first plane equation of the first type of sub-segmentation regions” is the implementation process of this step.

In each merged plane and each unmerged plane, the pixel points on the intersection line between two adjacent planes satisfy the respective plane equations of the two adjacent planes. After determining the intersection lines between all adjacent planes, multiple first-type sub-segmentation areas of the first-type divisional area are also determined.

S203. Among the fitted physical planes, if any two of the physical planes do not meet the preset merging condition, calculate the intersection between the physical planes according to the plane equations of the physical planes. Lines to determine the plurality of first-type sub-divided regions.

Through the above steps S201-S203, the secondary segmentation of the first type of segmented area is realized, and each obtained first type of sub-segmented area corresponds to a different physical plane in the shooting scene. Therefore, in this embodiment, the plane equation of the first type of sub-segmentation area may be fitted based on the determined support points in the first type of sub-segmentation area.

In a specific implementation of the implementation shown in FIG. 2 , as shown in FIG. 3 , in step S201, according to the preset plane fitting rules, fitting the plurality of support point subsets corresponding to the first type of segmentation area The plane equation for the physical plane of , can include:

S301, based on the random sampling consistent RANSAC algorithm, select a target support point subset O from the current support point set Y in the first type of segmented area; wherein, the O is the support with the largest number of support points in the Y set of ideas;

The RANSAC algorithm can more accurately select the pixels corresponding to the same plane in the image. Because the present invention specifically fits the equation of a plane through the support points corresponding to the plane and the parallax of the support points. Therefore, in this embodiment, it is only necessary to select the support points corresponding to each plane in the first type of segmented area through the RANSAC algorithm. Specifically, the plane equations of the planes corresponding to each support point subset may be fitted one by one in the descending order of the number of support points in the support point subset.

S302, fitting the plane equation of the physical plane corresponding to the O according to the support points of the O and the parallax of the support points;

Specifically, the implementation process of step S302 can refer to the "according to the support points in each first type of sub-segmentation area and the parallax of the support points, fitting the first plane equation of the first type of sub-segmentation area" in step S104 One-step implementation process.

S303, judging whether the number of support points of the currently unfit support point subset Y-O is less than the number of support points in the first type of segmented area of the preset ratio;

It can be understood that if the number of support points in the current unfit support point subset Y-O is too small, or even the current unfit support point subset Y-O does not have enough support points for plane fitting, then in this case, it cannot be based on The support points of the support point subset Y-O are not currently fitted to fit their plane equations. Specifically, the aforementioned preset ratio can be set according to experience, for example, set to 10%.

S304, if the number of support points of the Y-O is not less than the number of support points in the first-type segmented area of a preset ratio, use the Y-O as the current support point set in the first-type segmented area, Return to step S301;

S305. If the number of support points of the Y-O is less than the number of support points in the first-type segmented region of the preset ratio, use the plane equation of the currently fitted physical plane as the final fitting result.

For the support points of the remaining unfit support point subset Y-O, they will be included in a certain first type of sub-division area during the process of determining the plane intersection line in step S202 or step S203 in the embodiment shown in FIG. 2 Inside. Specifically, these unfit support points may be included in different first-type sub-segmentation regions, or may be included in the same first-type sub-segmentation region.

Through the iteration of the above steps S301-S305, the plane equations of multiple different planes in the first type of segmented area can be obtained. Moreover, in a specific computer implementation process, the iterative processing of the above-mentioned steps S301-S305 can be performed on multiple first-type segmented regions at the same time, thus improving the matching efficiency.

In an implementation of the embodiment shown in FIG. 1 , for each unsupported point in each triangular area in step S105, the corresponding vertex of the unsupported point is determined according to the vertices of the triangular area and the parallax of each vertex. The first disparity search range can include:

For each triangular area, according to each vertex in the triangular area and the parallax of each vertex, determine the second plane equation of the plane where the triangular area is located;

determining an estimated parallax value of the unsupported point according to the second plane equation;

The first disparity search range includes: the disparity of each vertex, the disparity obtained by adding and subtracting a preset value from the disparity of each vertex, the estimated disparity value, and the disparity in the preset neighborhood to which the non-support point belongs. The parallax of the target support point; wherein, the absolute value of the difference between the parallax of the target support point and the estimated value of the parallax of the target support point determined according to the second plane equation does not exceed a fourth threshold.

In the above implementation, the implementation process of "for each triangular area, according to the vertices in the triangular area and the parallax of each vertex, determine the second plane equation of the plane where the triangular area is located" can refer to the "According to The implementation process of the step of fitting the support points and the parallax of the support points in each first type of sub-division area to the first plane equation of the first type of sub-division area".

Exemplarily, assuming that the determined second plane equation is: A*x+B*y+C*d+D=0, then the disparity estimation value of a non-support point (x ₀ , y ₀ ) in the triangle is

The aforementioned preset value can be one value or multiple values, if it is one value, it can be 1, if it is multiple values, it can be 1, 2, etc. For example: the parallax value of a vertex of a triangle is 2, and the default value is 1, then the parallax after adding and subtracting the preset value to the vertex parallax is: 1 and 3.

The aforementioned preset neighborhood to which the non-support point belongs may be: after dividing the reference image into a plurality of rectangular areas of a preset size, the rectangular area to which the non-support point belongs is determined according to the coordinates of the non-support point. More specifically, each rectangular area can be marked with a position, and the position mark of the rectangular area to which it belongs is determined according to the coordinates of the non-support point, and it is determined which rectangular area the rectangular area described by the non-support point is. For example: the reference image is a 40×40 image, which is divided into four small areas of 20×20, and these four rectangular areas are marked as (0,0), (0,1), (1,0) , (1, 1). The coordinates of a non-support point are (3, 5), then 3/20=0, 5/20=0, so it belongs to the area of (0, 0); the coordinates of another non-support point are (35, 2) , similarly 35/20=1, 2/20=0, then it belongs to the rectangular area (1,0). The above preset size can be set according to experience, for example, it is set to 20×20. Similarly, the above fourth threshold can also be set according to experience, for example, set to 2.

After determining the first parallax search range of a non-support point in the triangle, each point to be matched corresponding to each parallax of the first parallax search range can be matched with the non-support point one by one, so as to determine from each point to be matched Out of the matching point of the support point.

In an implementation of the embodiment shown in FIG. 1 , in step S105, the point to be matched corresponding to each disparity in the first disparity search range is matched pixel by pixel with the non-support point, and the non-support point is determined. Parallax, which can include:

The non-support point is determined by the following formula Parallax for:

in, for the non-support point The disparity, d _n represents the disparity estimated value of the non-support point, Represents each disparity within the first disparity search range, N represents the number of disparities within the first disparity search range, and S represents the support point set of the reference image; express The posterior probability of , calculated by the following formula:

in, Indicates that the point to be matched corresponding to d _n is the non-support point The probability of the correct matching point of , Indicates that the matching points corresponding to each disparity within the first disparity search range are the non-support points The probability of the correct matching point of .

the above as well as is a probabilistic representation. Specifically, in this embodiment, non-support points can be calculated The similarity with each point to be matched, the similarity can use non-support points The absolute value of the difference between the pixel value of and the pixel value of the point to be matched. If the pixel value of a point to be matched is the same as the non-support point The smaller the absolute value of the difference between the pixel values, it means that the point to be matched and the non-support point The more similar, the point to be matched is a non-support point The probability of the correct matching point is greater.

The binocular vision stereo matching method based on plane constraints and triangulation provided by the embodiment of the present invention, for each sub-segmented area of a larger-area segmented area, based on the idea of plane constraints, each sub-segmented area is determined by the plane equation of each sub-segmented area. The disparity of all non-support points in the sub-segmentation area can achieve more accurate and robust matching of occluded areas with missing pixel information and large areas of texture-free areas with matching ambiguities, thereby reducing matching errors Rate. And in the embodiment of the present invention, triangulation is carried out for the smaller segmented area, and the parallax search range of the unsupported points in the triangle is determined according to the vertices of the triangle and the parallax of the vertices, which greatly reduces the parallax search range of the unsupported points and improves the matching efficiency.

In order to verify the beneficial effects of the binocular vision stereo matching method based on plane constraints and triangulation provided by the embodiment of the present invention, the present invention conducts specific verification through simulation experiments, as shown in FIG. 4 . Among them, (a) is the left image in the Cones test set; (b) is the right image in the Cones test set; (c) is the real disparity map provided by the Middlebury platform with (a) as the reference image; (d) is the pair ( a) The result map after Meanshift segmentation; the black box in (e) indicates the first type of segmentation error in the Meanshift segmentation result map; (f) is a certain segment extracted from (e) to be divided twice The outline of the first type of segmented area; (g) is the second segmentation result obtained by fitting the first type of segmented area in (f) with RANSAC plane; (h) is obtained by triangulating (a) (i) is the disparity map obtained based on the matching method proposed by the present invention; (j) is the disparity map obtained by the non-local method based on the minimum spanning tree; (k) is based on LIBELAS (Library for Efficient Large-scale The disparity map obtained by the Stereo Matching algorithm; (1) is the disparity map obtained by the local method based on the support window.

In the simulation experiment, in order to improve the matching efficiency, the existing computing resources are used to perform two operations on the reference image at the same time, one is to perform Meanshift segmentation on the reference image, and the other is to directly triangulate the entire reference image, such as (h) shown.

It can be seen from the simulation results in Fig. 4 that the effect of the disparity map obtained based on the binocular vision stereo matching method based on the plane constraint and triangulation proposed by the present invention is significantly better than that obtained based on the other three methods.

In order to further verify the beneficial effects of the binocular vision stereo matching method based on plane constraints and triangulation provided by the embodiment of the invention, as shown in Table 1, the present invention is aimed at four sets of standard test diagrams provided by the Middlebury platform, specifically: Venus , Cones, Tsukuba and Teddy, compared the matching error rate of the binocular vision stereo matching method based on plane constraints and triangulation provided by this application with other three stereo matching algorithms when matching each set of standard test images. Specifically, for each set of standard test patterns, the matching error rates in three specific cases are compared. Among them, the other three stereo matching algorithms include: Guided filter (GF for short) algorithm; Non-Local average (NL for short) algorithm; LIBELAS algorithm. Three specific cases include: Nonocc: the case where there is no occlusion area; All: the case that includes all areas of the whole image; Disc: only the case where the depth discontinuity area is considered.

Table 1 The matching error rate of the inventive method and other three stereo matching algorithms

It can be seen from Table 1 that in most cases, the matching error rate of the binocular vision stereo matching method based on plane constraints and triangulation proposed by the present invention is lower than that of the other three stereo matching algorithms. Therefore, the stereo matching effect of the method of the present invention is better than the other three stereo matching algorithms.

FIG. 5 is a structural diagram of a binocular vision stereo matching device based on plane constraints and triangulation provided by an embodiment of the present invention. The device includes:

The acquisition module 501 is used to acquire the left image and the right image of the same shooting scene collected by two parallel cameras at the same time, and any one of them is used as a reference image, and the other is used as an image to be matched;

The determination module 502 is configured to determine the support points in the reference image, and calculate the disparity of each of the support points; wherein, the support points are unique and unique in the image to be matched in the reference image. Correctly match the pixels of the point;

A segmentation module 503, configured to segment the reference image according to a preset image segmentation method to obtain a plurality of segmented areas; divide each segmented area into a first type of segmented area whose segmented area is greater than a first threshold, and /or a second type of segmented area whose segmented area is not greater than the first threshold;

The first executing module 504 is configured to execute, for each segmented area of the first type: performing a second segment on the segmented area of the first type, and divide the segmented area of the first type into a plurality of sub-segmented areas of the first type; According to the support points and the disparity of the support points in each first type of sub-segmentation area, fitting the first plane equation of the first type of sub-segmentation area; according to the first type of sub-segmentation area, determining the first type of sub-segmentation area The parallax of the inner non-support point; wherein, the plurality of first-type sub-segmented regions respectively correspond to different physical planes in the shooting scene;

The second execution module 505 is configured to, for each second-type segmented area, execute: triangulating the second-type segmented area, dividing the second-type segmented area into a plurality of triangular areas; for each triangular area For each unsupported point in the triangular area, determine the first disparity search range corresponding to the unsupported point according to each vertex of the triangular area and the disparity of each apex; The unsupported point is matched pixel by pixel to determine the disparity of the unsupported point; wherein, each of the vertices is a support point.

The beneficial effect of the binocular vision stereo matching device based on plane constraint and triangulation provided by the embodiment of the present invention is the same as that of the method embodiment. For each sub-segmented area of a larger-area segmented area, based on the idea of plane constraints, the device determines the parallax of all non-support points in each sub-segmented area through the plane equation of each sub-segmented area, and can realize the occlusion of missing pixel information Regions and large textureless regions with matching ambiguity are more accurately and robustly matched, thereby reducing the matching error rate. And example of the present invention, triangulation is carried out for the segmented area lesser in area, determines the disparity search range of unsupported points in the triangle according to the apex of triangle and vertex disparity, greatly reduces the disparity search range of unsupported points, improves matching efficiency.

Optionally, the determination module 502 includes: a grid construction submodule and a determination submodule.

The grid construction sub-module is used to use the upper left pixel point of the reference image as a starting point, along the u-axis and v-axis of the pixel coordinate system corresponding to the reference image, and construct a step size of preset in the reference image A grid with a step size; the intersection points in the grid, except those located at the edge of the reference image, are used as candidate support points; wherein, the u axis is rightward, and the v axis is downward.

The first determination submodule is used to calculate, for each candidate support point, the distance between the vector of the point to be matched and the vector of the candidate support point corresponding to each parallax in the second parallax search range corresponding to the preset candidate support point; If the number of the smallest distance among the calculated distances smaller than the second threshold is 1, then use the candidate support point as a support point; wherein, each parallax corresponding to the second parallax search range The vectors of the points to be matched and the vectors of the candidate support points are calculated according to a sobel operator with a preset size.

Optionally, the segmentation module 503 is specifically configured to segment the reference image according to the Meanshift image segmentation method to obtain at least one segmented region.

Optionally, the first executing module 504 includes: a first fitting submodule, a second fitting submodule, a second determining submodule, and a third determining submodule.

The first fitting submodule is used to fit the plane equation of the physical plane corresponding to the plurality of support point subsets in the first type of segmented area according to the preset plane fitting rule; wherein, the plurality of support point subsets Corresponding to different physical planes in the shooting scene respectively; in each of the physical planes obtained by fitting, if a plurality of physical planes satisfy the preset merging condition, the second fitting submodule is triggered; after the fitting is obtained In each of the physical planes, if any two of the physical planes do not meet the preset merging condition, a third determining submodule is triggered; wherein the preset merging condition is: among the multiple physical planes, A distance of any physical plane from at least one other physical plane is less than a third threshold.

The second fitting submodule is used to fit the plane equation of the merged physical plane according to the support points in the multiple physical planes and the parallax of the support points, and trigger the second determination submodule;

The second determining submodule is used to calculate the intersection lines between the combined physical planes and the unmerged physical planes according to the plane equation of the combined physical plane and the plane equation of the unmerged physical plane, and determine the multiple A first-class sub-segmentation area.

The third determination sub-module is configured to calculate the intersection lines between the physical planes according to the plane equations of the physical planes, and determine the plurality of first-type subdivided regions.

Optionally, the first fitting submodule includes: a selection subunit, a fitting subunit and a judging subunit.

Selecting a subunit, used to select a target support point subset O from the current support point set Y in the first type of segmented area based on the random sampling consistency RANSAC algorithm; wherein, the O is the number of support points in the Y The subset with the largest number of support points.

The fitting subunit is configured to fit the plane equation of the physical plane corresponding to the O according to the support points of the O and the parallax of the support points.

A judging subunit, used to judge whether the number of support points of the current unfit support point subset Y-O is less than the number of support points in the first type of segmented area of the preset ratio; if the number of support points of the Y-O is not The number of support points in the first type of segmented area that is less than the preset ratio, then use the Y-O as the current support point set in the first type of segmented area to trigger the selection of subunits; if the support points of the Y-O If the number of support points in the first-type segmented region is less than the preset ratio, the plane equation of the currently fitted physical plane is taken as the final fitting result.

Optionally, the first execution module 504 is specifically used to set the first plane equation of the first type of sub-segmentation area as:

d(u,v)=au+bv+c

Wherein, d(u, v) represents the disparity of the pixel point (u, v) in the first type of sub-division area, a, b, c are the parameters to be solved of the first plane equation, and the a is solved by the following formula ,b,c:

Among them, u _i represents the coordinates of the u-axis of the i-th support point in the pixel coordinate system corresponding to the reference image in the first type of sub-segmentation area, and v _i represents the i-th support point in the first type of sub-segmentation area. The coordinates of the v axis of the pixel coordinate system corresponding to the reference image, the u axis is right, the v axis is down, i=1,..., m, m represents the number of support points in the first type of sub-division area, d _i represents the disparity of the i-th support point in the sub-segmentation area of the first type.

Optionally, the second executing module 505 includes: a fourth determining submodule and a fifth determining submodule.

The fourth determination sub-module is used for determining, for each triangular area, the second plane equation of the plane where the triangular area is located according to the vertices in the triangular area and the parallax of the vertices.

The fifth determination submodule is configured to determine the estimated parallax value of the non-support point according to the second plane equation.

The first parallax search range includes: the parallax of each vertex, the parallax obtained by adding and subtracting a preset value from the parallax of each vertex, the estimated value of parallax determined by the fifth determining submodule, and the non-supported The parallax of the target support point within the preset neighborhood centered on the point; wherein, the absolute value of the difference between the parallax of the target support point and the estimated value of the target support point’s parallax determined according to the second plane equation does not exceed the first Four thresholds.

Optionally, the second execution module 505 is specifically configured to determine the non-support point through the following formula Parallax for:

in, for the non-support point The disparity, d _n represents the disparity estimated value of the non-support point, Represents each disparity within the first disparity search range, N represents the number of disparities within the first disparity search range, and S represents the support point set of the reference image; express The posterior probability of , calculated by the following formula:

in, Indicates the prior probability of d _n , Indicates that the matching points corresponding to each disparity within the first disparity search range are the non-support points The probability of the correct matching point of .

The embodiment of the present invention also provides an electronic device, as shown in FIG. complete the mutual communication,

Memory 603, used to store computer programs;

When the processor 601 is used to execute the program stored on the memory 603, the following steps are implemented:

Obtain the left image and the right image of the same shooting scene collected by two parallel cameras at the same time, and use any one of them as a reference image and the other as an image to be matched;

Determining the support points in the reference image, and calculating the parallax of each of the support points; wherein, the support points are pixels in the reference image that have unique and correct matching points in the image to be matched ;

According to the preset image segmentation method, the reference image is segmented to obtain a plurality of segmented areas; each of the segmented areas is divided into the first type of segmented area whose segmented area is larger than the first threshold, and/or the segmented area is not larger than the second type of segmentation region of the first threshold;

For each first-type segmented area, perform: perform a second segment on the first-type segmented area, divide the first-type segmented area into a plurality of first-type sub-segmented areas; The support points in the region and the parallax of the support points are fitted to the first plane equation of the first type of sub-segmentation area; according to the first plane equation, the parallax of the non-support points in the first type of sub-segmentation area is determined; wherein , the plurality of first-type sub-divided regions respectively correspond to different physical planes in the shooting scene;

For each second-type segmented area, execute: triangulate the second-type segmented area, divide the second-type segmented area into a plurality of triangular areas; for each non-support point in each triangular area, Determine the first parallax search range corresponding to the unsupported point according to the vertices of the triangular area and the parallax of each vertex; match the points to be matched corresponding to each parallax in the first parallax search range with the unsupported point pixel by pixel. , to determine the disparity of the non-support point; wherein, each of the vertices is a support point.

In the electronic device provided by the embodiment of the present invention, when the processor executes the program stored in the memory, for each sub-segmented area of a larger-area segmented area, based on the idea of plane constraints, each sub-segmented area is determined by the plane equation of each sub-segmented area. The disparity of all non-support points in the sub-segmentation area can achieve more accurate and robust matching of occluded areas with missing pixel information and large areas of texture-free areas with matching ambiguities, thereby reducing matching errors Rate. And example of the present invention, triangulation is carried out for the segmented area lesser in area, determines the disparity search range of unsupported points in the triangle according to the apex of triangle and vertex disparity, greatly reduces the disparity search range of unsupported points, improves matching efficiency.

The communication bus mentioned in the above-mentioned electronic device may be a Peripheral Component Interconnect (PCI for short) bus or an Extended Industry Standard Architecture (EISA for short) bus or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the electronic device and other devices.

The memory may include a random access memory (Random Access Memory, RAM for short), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far away from the aforementioned processor.

The above-mentioned processor can be a general-purpose processor, including a central processing unit (Central Processing Unit, referred to as CPU), a network processor (Network Processor, referred to as NP), etc.; it can also be a digital signal processor (Digital Signal Processing, referred to as DSP) , Application Specific Integrated Circuit (ASIC for short), Field Programmable Gate Array (Field-Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

In yet another embodiment provided by the present invention, a computer-readable storage medium is also provided. Instructions are stored in the computer-readable storage medium. When the computer-readable storage medium is run on a computer, it causes the computer to execute any one of the above-mentioned embodiments. Stereo matching method for binocular vision based on plane constraints and triangulation.

The instructions stored in the computer-readable storage medium provided by the embodiments of the present invention, when running on a computer, for each sub-division area of a larger-area division area, based on the idea of plane constraints, pass the plane of each sub-division area The equation determines the disparity of all non-support points in each sub-segmentation area, which can achieve more accurate and robust matching of occluded areas with missing pixel information and large-area texture-free areas with matching ambiguity, thereby reducing match error rate. And example of the present invention, triangulation is carried out for the segmented area lesser in area, determines the disparity search range of unsupported points in the triangle according to the apex of triangle and vertex disparity, greatly reduces the disparity search range of unsupported points, improves matching efficiency.

In yet another embodiment provided by the present invention, a computer program product containing instructions is also provided, and when it is run on a computer, it causes the computer to perform any of the above-mentioned double-dimensional analysis based on plane constraints and triangulation. Visual stereo matching method.

The calculation program product containing instructions provided by the embodiment of the present invention, when it is run on a computer, for each sub-division area of a larger-area division area, based on the idea of plane constraints, each sub-division area is determined by the plane equation of each sub-division area. The disparity of all non-support points in the sub-segmentation area can achieve more accurate and robust matching of occluded areas with missing pixel information and large areas of texture-free areas with matching ambiguities, thereby reducing matching errors Rate. And example of the present invention, triangulation is carried out for the segmented area lesser in area, determines the disparity search range of unsupported points in the triangle according to the apex of triangle and vertex disparity, greatly reduces the disparity search range of unsupported points, improves matching efficiency.

In the above embodiments, all or part of them may be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the embodiments of the present invention are produced in whole or in part. A computer can be a general purpose computer, special purpose computer, computer network, or other programmable device. Computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, e.g. Coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server, a data center, etc. integrated with one or more available media. The available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk (SSD)).

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or order between them. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

Each embodiment in this specification is described in a related manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device/electronic equipment/storage medium/computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and for relevant parts, please refer to part of the description of the method embodiments.

The above are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present invention are included in the protection scope of the present invention.

Claims (10)

1. A binocular vision stereo matching method based on plane constraint and triangulation, it is characterized in that, comprising: Obtain the left image and the right image of the same shooting scene collected by two parallel cameras at the same time, and use any one of them as a reference image and the other as an image to be matched; Determining the support points in the reference image, and calculating the parallax of each of the support points; wherein, the support points are pixels in the reference image that have unique and correct matching points in the image to be matched ; According to the preset image segmentation method, the reference image is segmented to obtain a plurality of segmented areas; each of the segmented areas is divided into the first type of segmented area whose segmented area is larger than the first threshold, and/or the segmented area is not larger than the second type of segmentation region of the first threshold; For each first-type segmented area, perform: perform a second segment on the first-type segmented area, divide the first-type segmented area into a plurality of first-type sub-segmented areas; The support points in the region and the parallax of the support points are fitted to the first plane equation of the first type of sub-segmentation area; according to the first plane equation, the parallax of the non-support points in the first type of sub-segmentation area is determined; wherein , the plurality of first-type sub-divided regions respectively correspond to different physical planes in the shooting scene; For each second-type segmented area, execute: triangulate the second-type segmented area, divide the second-type segmented area into a plurality of triangular areas; for each non-support point in each triangular area, Determine the first parallax search range corresponding to the unsupported point according to the vertices of the triangular area and the parallax of each vertex; match the points to be matched corresponding to each parallax in the first parallax search range with the unsupported point pixel by pixel. , to determine the disparity of the non-support point; wherein, each of the vertices is a support point. 2. The method according to claim 1, wherein the determining the support points in the reference image comprises: Taking the pixel point at the upper left corner of the reference image as a starting point, along the u-axis and the v-axis of the pixel coordinate system corresponding to the reference image, construct a grid with a step size of a preset step size in the reference image; In the above grid, except the intersection point at the edge of the reference image as a candidate support point; wherein, the u axis is rightward, and the v axis is downward; For each candidate support point, calculate the distance between the vector of the point to be matched and the vector of the candidate support point corresponding to each parallax in the second parallax search range corresponding to the candidate support point that is preset; if the calculated distance is small Among the distances of the second threshold, if the number of the smallest distance is 1, then the candidate support point is used as a support point; wherein, the vector of the point to be matched corresponding to each parallax in the second parallax search range, and the The vectors of the candidate support points are all calculated according to the sobel operator of the preset size. 3. The method according to claim 1, wherein the said reference image is segmented according to a preset image segmentation method to obtain a plurality of segmented regions, comprising: The reference image is segmented according to the Meanshift image segmentation method to obtain a plurality of segmented regions. 4. The method according to claim 1, wherein the second segmentation of the first type of segmented area is carried out, and the first type of segmented area is divided into a plurality of first type sub-segmented areas, comprising: Fit the plane equation of the physical plane corresponding to the plurality of support point subsets in the first type of segmented area according to the preset plane fitting rule; wherein, the plurality of support point subsets respectively correspond to different physical plane; Among the fitted physical planes, if multiple physical planes satisfy the preset merging condition, then according to the support points in the multiple physical planes and the parallax of the support points, the physical plane after fitting is obtained by fitting. Plane equation; according to the plane equation of the merged physical plane and the plane equation of the unmerged physical plane, calculate the intersection line between each merged physical plane and the unmerged physical plane, and determine the plurality of first-type sub-planes Segmenting an area; wherein, the preset merging condition is: among the plurality of physical planes, the distance between any physical plane and at least one other physical plane is less than a third threshold; Among the fitted physical planes, if any two of the physical planes do not satisfy the preset merging condition, the intersection line between the physical planes is calculated according to the plane equations of the physical planes, The plurality of first-type sub-divided regions are determined. 5. The method according to claim 4, said fitting the plane equation of a physical plane corresponding to a plurality of support point subsets in the first type of segmented area according to the preset plane fitting rule, comprising: S1. Based on the random sampling consistency RANSAC algorithm, select a target support point subset O from the current support point set Y in the first type of segmented area; wherein, the O is the support with the largest number of support points in the Y set of ideas; S2. Fitting the plane equation of the physical plane corresponding to the O according to the support points of the O and the parallax of the support points; S3. Judging whether the number of support points of the currently unfit support point subset Y-O is less than the number of support points in the first type of segmented area of the preset ratio; If the number of support points of the Y-O is not less than the number of support points in the first-type segmented area of the preset ratio, then use the Y-O as the current support point set in the first-type segmented area, and return to execute Step S1; If the number of support points of the Y-O is less than the number of support points in the first-type segmented area of the preset ratio, the plane equation of the currently fitted physical plane is used as the final fitting result. 6. method according to claim 1, is characterized in that, described according to the parallax of support point and support point in each first type sub-segmentation area, fitting the first plane equation of this first type sub-segmentation area ,include: Set the first plane equation of the first type of sub-division area as: d(u,v)=au+bv+c Wherein, d(u, v) represents the disparity of the pixel point (u, v) in the first type of sub-division area, a, b, c are the parameters to be solved of the first plane equation, and the a is solved by the following formula ,b,c: Among them, u _i represents the coordinates of the u-axis of the i-th support point in the pixel coordinate system corresponding to the reference image in the first type of sub-segmentation area, and v _i represents the i-th support point in the first type of sub-segmentation area. The coordinates of the v axis of the pixel coordinate system corresponding to the reference image, the u axis is right, the v axis is down, i=1,..., m, m represents the number of support points in the first type of sub-division area, d _i represents the disparity of the i-th support point in the sub-segmentation area of the first type. 7. The method according to claim 1, wherein, for each unsupported point in each triangular area, the corresponding first point of the unsupported point is determined according to each vertex of the triangular area and the parallax of each apex. A parallax search range, including: For each triangular area, according to each vertex in the triangular area and the parallax of each vertex, determine the second plane equation of the plane where the triangular area is located; determining an estimated parallax value of the unsupported point according to the second plane equation; The first disparity search range includes: the disparity of each vertex, the disparity obtained by adding and subtracting a preset value from the disparity of each vertex, the estimated disparity value, and the disparity in the preset neighborhood to which the non-support point belongs. The parallax of the target support point; wherein, the absolute value of the difference between the parallax of the target support point and the estimated value of the parallax of the target support point determined according to the second plane equation does not exceed a fourth threshold. 8. The method according to claim 7, characterized in that, performing pixel-by-pixel matching between the points to be matched corresponding to each parallax in the first parallax search range and the non-support point, and determining the parallax of the non-support point ,include: The non-support point is determined by the following formula Parallax for: in, for the non-support point The disparity, d _n represents the disparity estimated value of the non-support point, Represents each disparity within the first disparity search range, N represents the number of disparities within the first disparity search range, and S represents the support point set of the reference image; express The posterior probability of , calculated by the following formula: in, Indicates that the point to be matched corresponding to d _n is the non-support point The probability of the correct matching point of , Indicates that the points to be matched corresponding to the disparities within the first disparity search range are the non-support points The probability of the correct matching point of . 9. A binocular vision stereo matching device based on plane constraints and triangulation, characterized in that it comprises: The obtaining module is used to obtain the left image and the right image of the same shooting scene collected by two parallel cameras at the same time, and any one of them is used as a reference image, and the other is used as an image to be matched; A determining module, configured to determine the support points in the reference image, and calculate the disparity of each of the support points; wherein, the support points in the reference image are unique and correct in the image to be matched Pixels of matching points; A segmentation module, configured to segment the reference image according to a preset image segmentation method to obtain a plurality of segmented areas; divide each segmented area into a first type of segmented area whose segmented area is greater than a first threshold, and/or or a second type of segmented area whose segmented area is not greater than the first threshold; The first execution module is configured to, for each first-type segmented area, perform: performing a second segment on the first-type segmented area, and dividing the first-type segmented area into a plurality of first-type sub-segmented areas; The support points and the parallax of the support points in each first type of sub-segmentation area are fitted to the first plane equation of the first type of sub-segmentation area; according to the first plane equation, determine the first type of sub-segmentation area Parallax of non-support points; wherein, the plurality of first-type sub-segments respectively correspond to different physical planes in the shooting scene; The second execution module is configured to, for each second-type segmented area, execute: perform triangulation on the second-type segmented area, and divide the second-type segmented area into a plurality of triangular areas; for each triangular area For each non-support point, determine the first parallax search range corresponding to the non-support point according to each vertex of the triangular area and the parallax of each vertex; the points to be matched corresponding to each parallax in the first parallax search range and the The non-support point is matched pixel by pixel to determine the disparity of the non-support point; wherein, each of the vertices is a support point. 10. An electronic device, characterized in that it comprises a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete mutual communication through the communication bus; memory for storing computer programs; When the processor is used to execute the program stored in the memory, it realizes the method steps described in any one of claims 1-8.