CN117190871A - Initial welding dry elongation measurement method and device based on binocular vision - Google Patents

Abstract

The invention relates to the technical field of robots and discloses an initial welding dry elongation measurement method and device based on binocular vision, which mainly comprise camera calibration, image acquisition and processing, three-dimensional matching acquisition of a parallax image and a depth image, positioning of a welding gun axis and identification of a welding gun key point, identification and positioning of a welding point, interception of a welding region ROI (region of interest), and determination of the welding point through an intersection point of an extension line of the welding gun axis and a welding line; and indirectly calculating the welding dry extension by utilizing the key points of the welding gun and the three-dimensional coordinates of the to-be-welded point. The method can be used for rapidly and stably detecting the welding dry extension before welding, and is beneficial to timely adjusting the welding dry extension according to the welding parameters or timely adjusting the welding parameters according to the welding dry extension. The invention not only can improve the welding quality and efficiency of the robot, but also can improve the automation degree of the welding of the robot and reduce the labor intensity of welders.

Description

An initial welding dry elongation measurement method and device based on binocular vision

Technical field

The invention relates to the field of robotic technology, and specifically to an initial welding dry elongation measurement method and device based on binocular vision.

Background technique

Welding is of great significance in industrial production, and welding dry elongation has a vital impact on welding quality. When the welding stem is stretched too long, the resistance voltage of the welding wire becomes larger, and the voltage at both ends of the welding arc becomes smaller, which reduces the arc heat and affects the welding quality. If the extension is too short, it is easy for the arc to retrace and burn out the contact tip, resulting in cracks in the weld due to the material doped with the contact tip. Normally, when the welding wire diameter is less than 3 mm, the extension length of the welding wire should be between 20 and 60 mm. Therefore, dry elongation needs to be detected and controlled according to welding requirements to ensure welding quality.

In the actual production environment, in order to facilitate the measurement of the extension length of the welding wire, the distance from the lower end of the contact tip to the surface of the weldment is usually used as a reference. However, manual measurement using tools such as rulers is inefficient and unsafe. Therefore, an efficient, contact-free method is needed for preliminary measurements of welding dry elongation.

Contents of the invention

In view of the above problems, the present invention proposes an initial welding dry elongation measurement method and device based on binocular vision, aiming to achieve automatic, fast, and non-contact measurement of the initial welding dry elongation. The invention can ensure rapid and accurate measurement of the initial welding dry elongation, reduce the labor intensity of welding personnel, and improve welding quality and efficiency. In order to solve the problems existing in the existing initial welding dry elongation measurement technology, the present invention adopts the following technical solutions:

An initial welding dry elongation measurement method based on binocular vision, the method includes the following steps:

Step 1, camera calibration; obtain the internal and external parameters of each camera module in the binocular camera and the structural parameters between the two camera modules.

Step 2, image acquisition and image processing; perform distortion correction and stereoscopic correction on the collected original RGB images to obtain distortion-free and coplanar row-aligned left and right perspective RGB images and grayscale images.

Step 3: Stereo matching and obtaining the disparity map and depth map; use the semi-global block matching algorithm SGBM to calculate the binocular disparity to obtain the point cloud map and depth map, and align the left view of the binocular camera with the depth map.

Step 4: Adaptively position the welding gun and the welding gun axis; by locating the point on the image at the end of the welding gun nozzle joint on the welding gun axis, the welding gun axis is positioned to provide a basis for positioning the welding area. Since the surface of the welding gun nozzle is not only smooth and textureless but also has specular reflection, it is difficult to match feature points. Therefore, it is difficult and unstable to directly identify the welding gun nozzle and calculate the three-dimensional coordinates of the welding gun nozzle vertex. Notice that there is a brass-colored standard nozzle connector above the nozzle that is coaxial with the welding gun, which can be more stably identified and positioned. Dry elongation is therefore measured indirectly by calculating the distance from the point at the end of the welding gun nozzle joint located on the axis of the welding gun to the point to be welded.

Step 5: Estimate the pixel position of the spot to be soldered in the image; in order to obtain the pixel coordinates of the spot to be soldered on the left-view image, intercept the welding area ROI (region of interest) and determine it by the intersection of the extension line of the welding gun axis and the weld seam. For the spot to be soldered, the coordinates of the solder spot on the cropped welding area ROI image are finally converted into pixel coordinates on the left-view image.

Step 6: Convert the key point coordinates and calculate the distance; convert the two-dimensional coordinates of the key points of the spot to be welded and the welding gun into three-dimensional coordinates, and calculate the Euclidean distance between the two points in the three-dimensional space. After excluding outliers, get The measurement average of every 40 frames was taken as the final result of the initial weld stem elongation measured by binocular vision.

Further, the binocular cameras are calibrated to obtain the internal and external parameters of each camera and the structural parameters between the dual cameras. Steps include:

First, the two cameras on the binocular camera module simultaneously capture images of the checkerboard calibration plate at different positions.

Next, use OpenCV for camera calibration and stereo calibration. Obtain the internal and external parameters, distortion parameters and structural parameters between the two cameras of each camera module, including the rotation matrix and translation vector from the right camera coordinate system to the left camera coordinate system.

Further, the steps of collecting images, performing distortion correction and stereoscopic correction on the collected images, and obtaining two images of left and right angles without distortion and coplanar row alignment include:

First, convert two three-channel RGB images from the left and right perspectives of the binocular camera into two single-channel grayscale images.

Then, the distortion parameters obtained by camera calibration are used to perform distortion correction on the two grayscale images.

Then, the two grayscale images are stereoscopically corrected using the internal camera parameters obtained by camera calibration and the structural parameters between the binocular cameras to obtain coplanar row-aligned left and right perspective binocular images.

Further, stereo matching is performed and the disparity map and depth map are obtained. Proceed as follows:

First, the semi-global block matching algorithm SGBM is used to calculate the disparity in binocular vision.

Then, a mapping map is obtained using the disparity map and the reprojection matrix obtained during stereoscopic correction. This map is a three-channel image with the same size as the disparity map. Each channel stores the value of the pixel position on the X-axis, Y-axis, and Z-axis in the camera coordinate system, that is, each pixel is in the camera coordinate system. The three-dimensional coordinates (x, y, z), where the value of z represents the distance from the object to the binocular camera plane, in millimeters (mm).

Further, the welding gun and the welding gun axis are adaptively positioned. Steps include:

First, the corrected left-view image is converted into an HSV color gamut image. By adjusting the H, S, and V thresholds, the nozzle joint part is segmented according to color and displayed on the binary image.

Next, small connected areas are filtered to remove noise, and then closed operations are used to make the segmented image of the nozzle joint more complete. A contour extraction algorithm is used to obtain the edge contour of the nozzle joint in the segmented image, and a minimum circumscribing rectangle is generated based on the contour.

Then, calculate the midpoint of the two sides of the minimum circumscribed rectangle that are perpendicular to the axis of the welding gun. The line connecting these two points is the straight line where the axis of the welding gun is located. Solve the equation of the straight line in the pixel coordinate system of the left view and intercept it for the next step. Provide basis for welding area. In addition, record the pixel coordinates of the midpoint closer to the welding point on the corrected left view.

Further, the welding area is extracted and cropped from the left-view image, and the pixel coordinates of the spots to be welded are estimated on the cropped welding area image, and finally the pixel coordinates are converted into pixel coordinates on the left-view image. Steps include:

First, draw the welding area ROI rectangle on the corrected left view. Because the relative position of the camera and the welding gun is fixed, the pixel coordinates of the welding gun vertex are fixed. On the left view image, draw a line segment of appropriate length perpendicular to the axis of the welding gun through the vertex of the welding gun as the width of the welding area ROI. In particular, the welding gun vertex is the midpoint of this line segment. Based on the width of the welding area ROI, the long side of the welding area ROI is parallel to the axis of the welding gun.

Next, cut the welding area ROI rectangle. Because the rectangle is a rotated rectangle, the left view is rotated so that the rotation angle of the welding area ROI is 0. Cut the welding area ROI. At this time, the vertical center line of the cut out welding area ROI is the axis of the welding gun.

Then, extract the spots to be soldered. Grayscale the welding area ROI image, use the canny algorithm to extract the welding seam, and record the intersection point between the welding seam and the vertical center line (welding gun axis) of the welding area ROI. This intersection point is the point to be welded.

Finally, calculate the pixel coordinates of the spot to be soldered. Using the opposite operation in the step of cropping the welding area ROI, convert the pixel coordinates of the to-be-soldered spot on the cropped welding area ROI image into the pixel coordinates on the corrected left view.

Furthermore, the two-dimensional coordinates of the key points of the spot to be welded and the welding gun are converted into three-dimensional coordinates, and the Euclidean distance of the two points in the three-dimensional space is calculated. After excluding outliers, the average measurement value of every 40 frames is taken as Final results of binocular vision measurements. Specific steps are as follows:

First, calculate the three-dimensional coordinates of the welding gun nozzle joint vertex and the spot to be welded in the camera coordinate system obtained in the previous steps.

Next, calculate the Euclidean distance between two points in three-dimensional space. The formula is as follows,

Among them, (x _a , y _a , z _a ) are the three-dimensional coordinates of the point at the end of the welding gun nozzle joint located on the axis of the welding gun in the camera coordinate system. (x _b ,y _b ,z _b ) are the three-dimensional coordinates of the spot to be welded in the camera coordinate system. The length is the visually measured Euclidean distance from the point (x _a , y _a , z _a ) at the end of the welding gun nozzle joint on the welding gun axis to the point to be welded (x _b , y _b , z _b ).

Then, the distance length from the end of the nozzle joint to the welding point minus the length of the welding gun nozzle len _weld is the distance len from the end of the nozzle to the point to be welded. This distance is also the dry extension of the welding wire and satisfies the following formula,

len=length-len _weld

Among them, len is the final calculated dry extension of the welding wire, and len _weld is the length of the nozzle.

Then, determine whether the distance len measured in the previous step is within a reasonable range (0 cm to 7.5 cm), otherwise jump directly to the next frame for measurement. After measuring 40 frames, take the average measurement value of these 40 frames as the final result of the binocular vision measurement.

The invention also provides an initial welding dry elongation measurement device based on binocular vision, including a frame-synchronous binocular camera module, camera fixture, computer, lighting module, welding gun and a device for moving the welding gun, wherein:

The frame synchronized binocular camera module has a focal length of 3.0mm, a baseline of 63mm, and an image resolution of 640×480, which is used for original image collection.

The camera fixture is used to fix the relative position between the binocular camera module and the welding gun, so that the baseline of the camera is approximately perpendicular to the axis of the welding gun on a certain projection plane, and to fix the camera on one side in the forward direction of the welding gun. This design reduces the number of parameters that need to be estimated separately, simplifies the measurement algorithm, improves the measurement accuracy, and speeds up the calculation speed of the algorithm.

The computer is used for real-time image acquisition, image processing, feature matching, stereo matching, and key point estimation and identification to complete the measurement of the initial welding dry elongation.

The lighting module is fixed on the device for moving the welding gun and is used to improve ambient lighting and improve measurement accuracy.

The welding gun model is Panasonic MIG/MAG welding gun. The outer diameter of the nozzle is 24mm and the total length is 73mm. The nozzle connector is a brass-colored standard nozzle connector. The welding gun is fixed together with the camera clamp on a device for moving the welding gun. The welding gun, camera, and lighting module keep their relative positions fixed.

The beneficial effects of the present invention are:

The invention can quickly and stably detect the welding dry elongation, thereby promptly adjusting the welding dry elongation according to the welding parameters or adjusting the welding parameters in time according to the dry elongation. This can not only improve the quality and efficiency of machine welding, but also improve the automation of robot welding and reduce the labor intensity of welders.

Description of the drawings

Figure 1 is a flow chart of the initial welding dry elongation measurement method based on binocular vision of the present invention;

Figure 2 is a flow chart of the adaptive positioning welding gun and the welding gun axis of the present invention;

Figure 3 is a flow chart of the present invention for estimating the pixel position of the spot to be soldered in the image;

Figure 4 is a flow chart in which the average value of every 40 measurement results is used as the final measurement result of the present invention;

Figure 5 is an overall structural diagram of the initial welding dry elongation measurement device based on binocular vision of the present invention;

Figure 6 is a side view of the initial welding dry elongation measurement device based on binocular vision of the present invention;

Figure 7 shows the experimental results using the adaptive positioning welding gun, welding gun axis and welding area of the present invention;

Figure 8 shows the experimental results of welding seam identification using the present invention; (a) display of welding seam area positioning and welding point positioning results; (b) welding area welding seam identification after rotating and cropping the welding area;

Figure 9 is an experiment using the present invention to verify the stability and accuracy of visual measurement when the dry extension of the welding wire is 2.0 cm; (a) The experimental results of 100 consecutive measurements when each frame measurement value is taken as a measurement result. (b) The experimental results of 50 measurements when the average of 40 consecutive frame measurement values is used as the measurement result;

Figure 10 is an experiment using the present invention to verify the stability and accuracy of visual measurement when the dry extension of the welding wire is 2.5 cm; (a) The experimental results of 100 consecutive measurements when each frame measurement value is taken as a measurement result. (b) The experimental results of 50 measurements when the average of 40 consecutive frame measurement values is used as the measurement result;

Figure 11 is an experiment using the present invention to verify the stability and accuracy of visual measurement when the dry extension of the welding wire is 3.0 cm; (a) The experimental results of 100 consecutive measurements when each frame measurement value is taken as a measurement result. (b) The experimental results of 50 measurements when the average of 40 consecutive frame measurement values is used as the measurement result;

Figure 12 is an experiment using the present invention to verify the stability and accuracy of visual measurement when the dry extension of the welding wire is 3.5 cm; (a) The experimental results of 100 consecutive measurements when each frame measurement value is taken as a measurement result. (b) The experimental results of 50 measurements when the average of 40 consecutive frame measurement values is used as the measurement result;

Figure 13 is an experiment using the present invention to verify the stability and accuracy of visual measurement when the dry extension of the welding wire is 4.0 cm; (a) The experimental results of 100 consecutive measurements when each frame measurement value is taken as a measurement result. (b) The experimental results of 50 measurements when the average of 40 consecutive frame measurement values is used as the measurement result;

Figure 14 is an experiment using the present invention to verify the stability and accuracy of visual measurement when the dry extension of the welding wire is 4.5 cm; (a) The experimental results of 100 consecutive measurements when each frame measurement value is taken as a measurement result. (b) The experimental results of 50 measurements when the average of 40 consecutive frame measurement values is used as the measurement result;

Figure 15 is an experiment using the present invention to verify the stability and accuracy of visual measurement when the dry extension of the welding wire is 5.0 cm; (a) The experimental results of 100 consecutive measurements when each frame measurement value is taken as a measurement result. (b) The experimental results of 50 measurements when the average of 40 consecutive frame measurement values is used as the measurement result;

Figure 16 is an experiment using the present invention to verify the stability and accuracy of visual measurement when the dry extension of the welding wire is 5.5 cm; (a) The experimental results of 100 consecutive measurements when each frame measurement value is taken as a measurement result. (b) The experimental results of 50 measurements when the average of 40 consecutive frame measurement values is used as the measurement result;

Figure 17 is an experiment using the present invention to verify the stability and accuracy of visual measurement when the dry extension of the welding wire is 6.0 cm; (a) The experimental results of 100 consecutive measurements when each frame measurement value is taken as a measurement result. (b) The experimental results of 50 measurements when the average of 40 consecutive frame measurement values is used as the measurement result;

Figure 18 shows the relationship between the measurement error, standard deviation and the number of frames used to calculate the average when the actual initial welding dry elongation is 2.5 cm in a certain measurement experiment;

Figure 19 shows the relationship between the calculation error and the time consumed when the actual initial welding dry elongation is 2.5 cm in a certain measurement experiment and the number of frames involved in calculating the average value increases by 10 frames;

The reference numbers are as follows: 1. Nozzle joint; 2. Nozzle; 3. Binocular camera module; 4. Lighting module; 5. Material to be welded; 6. Camera fixture; 7. Base; 8. Welding dry extension.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

Referring to Figure 1, the present invention provides an initial welding dry elongation measurement method based on binocular vision. The steps include:

Step one, camera calibration. For binocular cameras, it is necessary to obtain the internal and external parameters of each camera module and the structural parameters between the two camera modules.

First, the two imaging systems in the binocular camera module used in the present invention are on a plane. The calibration plate is placed in the overlapping field of view of the binocular camera. The image area of the calibration plate taken accounts for 1 of the entire image area. /3 or so, and ensure that both cameras can recognize all corners at the same time. The two cameras on the binocular camera module simultaneously capture 15 pictures of the checkerboard calibration plate (9×6, 19.5mm) at different positions (the checkerboard is imaged in the upper left, upper right, lower left, and lower corners of the camera field of view) with a fixed aperture and focal length. lower right, center) image.

Next, use OpenCV for camera calibration and stereo calibration. Obtain the internal and external parameters, distortion parameters and structural parameters between the two cameras of each camera module, including the rotation matrix and translation vector between the two coordinate systems.

Obtain the internal parameter K and distortion parameter d of the two cameras respectively,

d＝[k ₁ ,k ₂ ,p ₁ ,p ₂ ,k ₃ ]

Among them, f _x , f _y , c _x , c _y are the internal parameters of the camera linear model. f _x , f _y are the scale factors in the u-axis and v-axis directions of the pixel coordinate axes respectively, and c _x , cy _y are the optical centers. k _i (i=1,2,3) is the radial distortion coefficient, and p _i (i=1,2) is the tangential distortion coefficient.

Using the above camera parameters, stereo calibration obtains a rotation matrix R (3×3, an orthogonal matrix with determinant 1) and a translation vector t (3×1). When the three-dimensional coordinates (x ₁ , y ₁ , z ₁ ) of a certain point in the first camera coordinate system are given, the three-dimensional coordinates (x ₂ , y ₂ , z ₂ ) of the second camera coordinate system can be calculated ,

Step two, image acquisition and image processing. Perform distortion correction and stereoscopic correction on the collected original RGB images to obtain distortion-free and coplanar line-aligned left and right perspective RGB images and grayscale images.

First, convert two three-channel RGB images from the left and right perspectives of the binocular camera into two single-channel grayscale images.

GRAY←0.299·R+0.587·G+0.114·B

Among them, R, G, and B are the pixel brightness values of the three channels of the color image, and GRAY is the corresponding pixel gray value of the grayscale image.

Then, the distortion parameters obtained by camera calibration are used to perform distortion correction on the two grayscale images from the left and right viewing angles of the binocular camera, respectively, to obtain two distortion-free grayscale images. First calculate the point P on the undistorted image, whose coordinates are [u, v] ^T corresponding to the coordinates [u _distorted , v _distorted ] ^T on the distorted image. First convert the coordinates [u, v] ^T of point P in the undistorted image to the normalized plane to obtain the coordinates [x, y] ^T ,

Calculate the radial distortion and tangential distortion for the point [x, y] ^T on the normalized plane,

Among them, [x,y] ^T is the coordinate of point P on the normalized plane, [x _distorted ,y _distorted ] ^T is the normalized coordinate of the point after distortion, is the distance between point P and the origin of the coordinate system, k _i (i=1,2,3) is the radial distortion coefficient, and p _i (i=1,2) is the tangential distortion coefficient.

Project the normalized coordinates of the distorted point to the pixel plane through the internal parameter matrix, and obtain the point [u, v] ^T on the undistorted image corresponding to the coordinate [u _distorted ,v _distorted ] ^T on the distorted image,

Finally, the value of the distorted image located on [u _distorted , v _distorted ] ^T is assigned to the position of the undistorted image coordinates [u, v] ^T ,

image _undistorted (u,v)←image _distorted (u _distorted ,v _distorted )

Among them, [u,v] ^T is the correct position of point P on the image. [u _distorted ,v _distorted ] ^T represents the point P on the undistorted image corresponding to the coordinates on the distorted image. image _undistorted represents the distortion-free image after distortion correction, and image _distorted represents the original image.

Then, use the internal camera parameters obtained from the camera calibration in step 1 and the structural parameters between the binocular cameras to perform stereoscopic correction on the two grayscale images to obtain coplanar row-aligned left and right perspective binocular images. The left and right camera views of the binocular cameras used in this invention move primarily horizontally along the X-axis (possibly with a small vertical offset). After the coplanar row alignment corrects the image, the corresponding epipolar lines in the left and right viewing angles are horizontal, the optical axes are parallel, the left and right viewing angle imaging planes are coplanar, and the two pixel coordinate systems projected into the two camera image planes have the same y coordinate. At the same time, the disparity to depth mapping matrix is obtained in stereo correction,

Among them, T _x is the length of the binocular camera baseline, [c_x ₁ , c_y] is the origin of the left image, c_x ₂ is the origin of the X coordinate axis in the right view, and f is the focal length.

Step 3: Stereo matching and obtaining disparity map and depth map. The SGBM algorithm is used to calculate the binocular disparity to obtain the point cloud image and depth map, and the left view of the binocular camera is aligned with the depth map.

First, the semi-global block matching algorithm SGBM is used to calculate the disparity in binocular vision, in which the cv::StereoSGBM::create() function and the cv::StereoMatcher::compute() function are used.

Next, a mapping map is obtained using the disparity image and the reprojection matrix Q obtained in step 2. This map is a three-channel image with the same size as the disparity map. Each channel stores the value of the pixel position on the X-axis, Y-axis, and Z-axis in the camera coordinate system, that is, each pixel is in the camera coordinate system. The three-dimensional coordinates (x, y, z), where the value of z represents the distance from the object to the binocular camera plane, in millimeters (mm).

Step 4: Adaptively position the welding gun and the welding gun axis. Since the surface of the welding gun nozzle is not only smooth and textureless but also has specular reflection, it is difficult to match feature points. Therefore, it is difficult and unstable to directly identify the welding gun nozzle and calculate the three-dimensional coordinates of the welding gun nozzle vertex. Notice that there is a brass-colored standard nozzle connector above the nozzle that is coaxial with the welding gun, which can be more stably identified and positioned. Dry elongation is therefore measured indirectly by calculating the distance from the point at the end of the welding gun nozzle joint located on the axis of the welding gun to the point to be welded. Specific steps are as follows:

First, the corrected left-view RGB image is converted into an HSV color gamut image. By adjusting the H, S, and V thresholds, the nozzle joint part is segmented according to color and displayed on the binary image. The formula for converting RGB images into HSV color gamut images is as follows,

V＝max(R,G,B)

When H<0, H=H+360. Output V∈[0,1], S∈[0,1], H∈[0,360]. When displaying 8-bit images,

V＝255V

S＝255S

H＝H/2

Among them, R, G, and B are the pixel brightness of the R, G, and B channels of the color image respectively, and H, S, and V are the corresponding pixel values of the H, S, and V channels of the image in the HSV color gamut. By adjusting the HSV threshold, a binary image is generated with only the pixel value in the target area being 255 and the pixel values in the remaining non-target areas being 0.

Then, the binary image generated in the previous step still has many noise points in non-target areas, and small connected areas are filtered to remove the noise points. Then, the closed operation is used to first expand to eliminate the small holes and cracks with a pixel value of 0 in the target area of the binary image, and then the target area is restored to its original size by erosion. The segmented image of the nozzle joint is made more stable and complete through closed operation. Then, obtain the minimum circumscribed rectangle of the target area outline. The minimum circumscribed rectangle is a rotated rectangle, represented by ((x _center , y _center ), (width, height), angle). Among them, (x _center , y _center ) is the center coordinate of the rotated rectangle, (width, height) is the width and height of the rotated rectangle, angle is the angle of rotation of the horizontal rectangle around the center point, in addition, θ=angle*π/180, The unit is rad, then the rotation matrix is:

On the plane coordinates, after any point (xa, ya) is rotated by an angle θ around a coordinate point (xb, yb), the calculation formula for setting the new coordinates to (x, y) is,

Then, calculate the midpoints of the two sides of the minimum circumscribed rectangle that are perpendicular to the axis of the welding gun. Since the welding gun nozzle of the experimental equipment used in the present invention is a standard cylindrical shape, the line connecting the midpoints of these two sides is where the axis of the welding gun is. of straight lines. By sorting the four vertices of the minimum circumscribed rectangle from small to large on the X-axis and Y-axis respectively, we can get the two sides of the minimum circumscribed rectangle that are perpendicular to the axis of the welding gun, and then get the midpoints of these two sides. Through these two midpoints, the linear equation of two variables in the pixel coordinate system of the left-view image of the welding gun axis can be obtained:

Assume that the four vertices of the rectangle surrounding the nozzle joint area are (x1, y1), (x2, y2), (x3, y3), and (x4, y4). Sorting from small to large on the y1),(x2,y2),(x4,y4),(x3,y3)]. The intersection points (x5, y5) and (x6, y6) of the welding gun axis and the minimum circumscribed rectangle can be obtained,

Among them, vector_x[0] and vector_y[0] represent the first elements of vector_x and vector_y respectively, that is, (x4, y4) and (x1, y1). vector_x[-1] and vector_y[-1] represent the last elements of vector_x and vector_y respectively, namely (x2, y2) and (x3, y3). "/2" means dividing each element by 2.

From (x5, y5), (x6, y6), the slope of the quadratic equation of the welding gun axis in the left view pixel coordinate system can be obtained The intercept is b=y5-k·x5, and the equation expression is recorded as y=k·x+b. The flow chart is shown in Figure 2.

Step 5: Estimate the pixel position of the soldering point in the image. Intercept the welding area ROI (region of interest), estimate the pixel position of the spot to be welded in this area, and finally convert the coordinates of the welding spot on the welding area ROI image into the pixel coordinates on the left-view image.

When the axis of the welding gun is aligned with the welding seam in space and ready for welding, the axis of the welding gun intersects with the welding seam, and the position of the spot to be welded appears on the image as the intersection point of the axis of the welding gun and the welding seam. Steps include:

First, draw the welding area ROI rectangle on the corrected left view. Because the relative position of the camera and the welding gun is fixed, the pixel coordinates of the welding gun vertex are fixed. In the left-view image, draw a line segment of appropriate length perpendicular to the axis of the welding gun through the vertex of the welding gun as the width of the welding area ROI. In particular, the welding gun vertex is the midpoint of this line segment. Based on the width of the welding area ROI, the long side of the welding area ROI is parallel to the axis of the welding gun:

From step 4, we know the equation of the axis of the welding gun in the pixel coordinate system of the left-view image. Let it be y=k·x+b. Then, through the known pixel coordinates (_x,_y) of the welding gun vertex, make a line of appropriate length perpendicular to The linear equation of the axis of the welding gun is y=(-1/k)·x+(_y+(1/k)·_x). With the vertex of the welding gun as the midpoint, take a length of 20 pixels as the width of the welding area ROI; use the above as the width For the two endpoints of the line segment, make a line segment parallel to the axis of the welding gun, with a length of 60 pixels and a direction extending from the vertex of the welding gun to the point to be welded, as the long side of the welding area ROI.

Next, cut the welding area ROI. The welding area ROI obtained in the above steps is a rotated rectangle, which is not convenient for cropping the image and identifying the spots to be welded. Therefore, the original left view is rotated so that the rotation angle of the welding area ROI is 0, and then the welding area ROI is cropped from the left view. At this time, the vertical centerline of the rectangular area (that is, the cropped welding area) is the welding gun. axis.

Then, extract the spots to be soldered. The welding area ROI image is grayscaled, and the canny algorithm is used to extract the weld seam. The weld seam extraction effect is shown in Figure 8(b). The white straight line in the figure represents the weld seam position. Next, record the intersection coordinates of the vertical centerline of the weld seam and the welding area ROI image, and the intersection point is the point to be welded. The specific method is to start from the center point of the rectangular area and explore one pixel upward and downward each time. When the value of the pixel is 255, stop exploring and record the current pixel coordinates.

Finally, calculate the pixel coordinates of the spot to be soldered on the left-view image. Use the opposite operation to the above steps, that is, first restore the pixel coordinates of the to-be-soldered spot on the image of the cropped welding area ROI to the coordinates on the pixel coordinate system of the rotated left-view image, and then click on the coordinates The above steps are rotated in the opposite direction, and finally the pixel coordinates of the to-be-soldered spot on the image of the cropped welding area ROI are converted into pixel coordinates on the left view. The flow chart is shown in Figure 3.

Step 6: Convert the two-dimensional coordinates of the key points of the solder joint and the welding gun into three-dimensional coordinates. By calculating the Euclidean distance between the two points in the three-dimensional space, after excluding outliers, take the measurement average of every 40 frames as the double coordinate. The final result of visual measurement.

First, convert the two-dimensional coordinates of the key points of the welding point and the welding gun into three-dimensional coordinates,

Among them, (x _c , y _c , z _c ) are the three-dimensional coordinates of the key points, (u _c , v _c ) are the two-dimensional coordinates of the key points, f _x , f _y , c _x , c _y are the linear model of the camera internal reference. f _x , f _y are the scale factors in the u-axis and v-axis directions of the pixel coordinate axis respectively, c _x , cy _y are the optical centers, and depth is the depth.

Calculate the Euclidean distance between two points in three-dimensional space,

Among them, (x _a , y _a , z _a ) are the three-dimensional coordinates of the point at the end of the welding gun nozzle joint located on the axis of the welding gun in the camera coordinate system. (x _b ,y _b ,z _b ) are the three-dimensional coordinates of the spot to be welded in the camera coordinate system. The length is the visually measured Euclidean distance from the point (x _a , y _a , z _a ) at the end of the welding gun nozzle joint on the welding gun axis to the point to be welded (x _b , y _b , z _b ).

Then, the distance length from the end of the welding gun nozzle joint to the point to be welded minus the length of the welding gun nozzle len _weld is the distance len from the end point of the welding gun nozzle to the point to be welded. This distance is also the dry extension of the welding wire and satisfies the following formula,

len=length-len _weld

Among them, len is the final calculated dry extension of the welding wire, and len _weld is the length of the welding gun nozzle.

According to Figure 9(a), Figure 10(a), Figure 11(a), Figure 12(a), Figure 13(a), Figure 14(a), Figure 15(a), Figure 16(a), Figure 17(a) It is known that if the measurement result of a single frame is taken as the final result of binocular vision measurement, there will be great inaccuracy and instability. In 100 measurement experiments, the experimental results are as shown in the following table:

Table 1 Experimental conditions when taking the measurement results of a single frame as the final result of binocular vision measurement

According to Figure 18, taking the average value of multiple frame measurements as the final measurement result can effectively reduce the measurement error, reduce the standard deviation, and suppress sudden changes in the measurement results. Further, according to Figure 19, although the measurement error decreases as the number of frames used to calculate the average increases, the time to obtain one measurement result also increases significantly. Therefore, in order to balance the measurement error and measurement time, the measurement average of every 40 frames is selected as the final result of the binocular vision measurement. The results of 50 consecutive measurements are shown in Figure 9(b), Figure 10(b), Figure 11(b), Figure 12(b), Figure 13(b), Figure 14(b), Figure 15(b), Figure 16 (b), as shown in Figure 17(b), the experimental results are as follows:

Table 2 Experimental conditions when taking the average measurement value of every 40 frames as the final result of binocular vision measurement

True value(cm) 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 Average(cm) 1.88 2.55 3.04 3.56 4.21 4.47 4.85 5.49 5.75 Maximum value-Minimum value (cm) 0.60 0.71 0.49 1.00 1.26 1.41 1.05 0.54 1.90 standard deviation 0.13 0.15 0.11 0.21 0.26 0.31 0.19 0.14 0.35

Compared with the above experiments, the measurement average of every 40 frames is selected as the final result of the binocular vision measurement. Both in terms of fluctuation range and discrete degree, the measurement value of a single frame is selected as the final result of the binocular vision measurement. The effect is good, and the measurement time is 3 to 4 seconds, which meets the requirements of rapidity and stability. The specific process is shown in Figure 4.

Referring to Figures 5 and 6, the present invention also provides an initial welding dry elongation measurement device based on binocular vision, including a frame synchronized binocular camera module, camera fixture, computer, lighting module, welding gun and a device for moving the welding gun. device, which:

The frame synchronized binocular camera module has a focal length of 3.0mm, a baseline of 63mm, and an image resolution of 640×480, which is used for original image collection.

The camera fixture is used to fix the relative position between the binocular camera module and the welding gun, so that the baseline of the camera is approximately perpendicular to the axis of the welding gun on a certain projection plane, and to fix the camera on one side in the forward direction of the welding gun. This design reduces the number of parameters that need to be estimated separately, simplifies the measurement algorithm, improves the measurement accuracy, and speeds up the calculation speed of the algorithm.

The computer is used for real-time image acquisition, image processing, feature matching, stereo matching, and key point estimation and identification to complete the measurement of the initial welding dry elongation.

The lighting module is fixed on the device for moving the welding gun and is used to improve ambient lighting and improve measurement accuracy.

The welding gun model is Panasonic MIG/MAG welding gun. The outer diameter of the nozzle is 24mm and the total length is 73mm. The nozzle connector is a brass-colored standard nozzle connector. The welding gun is fixed together with the camera clamp on the device for moving the welding gun. The welding gun, camera, and lighting module keep their relative positions fixed.

In summary, during the welding process, using the method of the present invention, welding personnel or robots can quickly and stably obtain the welding dry elongation before welding, which helps to autonomously adjust the working parameters according to the dry elongation and improves the robot's performance. It can improve the welding quality and efficiency, improve the automation of machine welding, and reduce the labor intensity of welders.

According to the above analysis, during the welding process, the method provided by the present invention can help welding personnel or robots quickly and stably obtain the initial welding dry elongation. The dry elongation measured according to the method of the present invention can be used to autonomously adjust working parameters or independently adjust the dry elongation according to the welding working parameters, which can improve the welding quality and efficiency of the robot, significantly improve the automation of machine welding, and reduce the labor of the welder. strength.

What is listed above is only one of the specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments, and many similar modifications are possible. All modifications directly derived or thought of by those of ordinary skill in the art from the disclosure of the present invention should be considered to be within the scope of protection of the present invention.

Claims (8)

1. An initial welding dry elongation measurement method based on binocular vision, which is characterized by comprising the following steps of:

step 1, calibrating a camera; obtaining internal and external parameters of each camera module and structural parameters between two camera modules in the binocular camera;

step 2, image acquisition and image processing; carrying out distortion correction and three-dimensional correction on the acquired original RGB image to obtain a left-right visual angle RGB image and a gray image which are free of distortion and aligned in a coplanar line;

step 3, three-dimensionally matching and obtaining a parallax image and a depth image; calculating binocular parallax by utilizing a semi-global block matching algorithm SGBM, and further obtaining a point cloud image and a depth image, wherein a left view of a binocular camera is aligned with the depth image;

step 4, adaptively positioning a welding gun and a welding gun axis; the method comprises the steps of positioning a welding gun axis by positioning the position of a point at the tail end of a welding gun nozzle joint on the welding gun axis on an image, and indirectly measuring the dry extension by calculating the distance from the point at the tail end of the welding gun nozzle joint on the welding gun axis to a to-be-welded point;

step 5, estimating the pixel position of the to-be-welded point in the image; intercepting a welding region ROI, determining a to-be-welded point through an intersection point of an extension line of a welding gun axis and a welding line, and finally converting coordinates of the welding point on a welding region ROI image into pixel coordinates on a left visual angle image;

step 6, converting the coordinates of the key points and calculating the distance; converting two-dimensional coordinates of a to-be-welded point and a welding gun key point into three-dimensional coordinates, calculating Euclidean distance between the two points in a three-dimensional space, and taking a measurement average value of every 40 frames as a final result of initial welding dry extension of binocular vision measurement after abnormal values are eliminated.

2. The binocular vision-based initial welding dry elongation measurement method of claim 1, wherein the step 1 comprises:

step 1.1, two cameras on a binocular camera module shoot images of the checkerboard calibration plate at different positions at the same time;

step 1.2, performing camera calibration and three-dimensional calibration by using OpenCV; and acquiring internal and external parameters and distortion parameters of each camera module and structural parameters between two cameras, wherein the parameters comprise a rotation matrix and a translation vector between two camera coordinate systems.

3. The binocular vision-based initial welding dry elongation measurement method of claim 1, wherein the step 2 comprises:

step 2.1, converting two three-channel RGB images of left and right visual angles of a binocular camera into two single-channel gray images;

2.2, respectively carrying out distortion correction on the two gray images by using distortion parameters obtained by camera calibration;

and 2.3, carrying out three-dimensional correction on the two gray images by using structural parameters between the camera internal parameters and the binocular cameras, which are obtained by camera calibration, so as to obtain left and right visual angle binocular images aligned in a coplanar line.

4. The binocular vision-based initial welding dry elongation measurement method of claim 1, wherein the step 3 comprises:

step 3.1, calculating parallax in binocular vision by utilizing an SGBM algorithm;

step 3.2, obtaining a mapping chart by utilizing the parallax images and the re-projection matrix obtained during the stereo correction of the images; the map is a three-channel image of the same size as the disparity map, each channel storing the values of the pixel location in the X-axis, Y-axis and Z-axis, respectively, in camera coordinate system, i.e. the three-dimensional coordinates (X, Y, Z) of each pixel in camera coordinate system, wherein the value of Z represents the distance of the object to the binocular camera plane in millimeters.

5. The binocular vision-based initial welding dry elongation measurement method of claim 1, wherein the step 4 comprises:

step 4.1, converting the corrected left view angle image into an HSV color gamut image, and dividing the nozzle joint part according to colors by adjusting a H, S, V threshold value and displaying the image on a binary image;

step 4.2, filtering the small communication area to remove noise points, and then enabling the segmented image of the nozzle joint to be more complete through closing operation; obtaining a minimum circumscribed rectangle of the outline by dividing the outline of the image;

step 4.3, calculating the midpoints of two sides of the minimum circumscribed rectangle, which are perpendicular to the axis of the welding gun, wherein the connecting line of the two midpoints is the straight line where the axis of the welding gun is located, and solving an equation of the straight line where the axis of the welding gun is located under a left-view pixel coordinate system; the pixel coordinates on the corrected left view of the midpoint nearer to the weld point are recorded.

6. The binocular vision-based initial welding dry elongation measurement method of claim 1, wherein the step 5 comprises:

step 5.1, drawing a welding region ROI rectangle on the corrected left view; making a line segment perpendicular to the axis of the welding gun at the top of the welding gun in the left view angle image, taking the line segment as the width of a welding region ROI, taking the top of the welding gun as the midpoint of the line segment, and taking the width of the welding region ROI as the basis, wherein the long side of the welding region ROI is parallel to the axis of the welding gun;

step 5.2, cutting a welding region ROI; rotating the left view so that the rotation angle of the welding region ROI is 0; cutting a welding region ROI, wherein the central line of the cut rectangular region is the axis of the welding gun;

step 5.3, extracting a to-be-welded point; graying the cut welding region ROI picture, extracting a welding line by using a canny algorithm, and recording an intersection point of the welding line and a vertical center line of the welding region ROI, wherein the intersection point is a to-be-welded point;

step 5.4, calculating pixel coordinates of the to-be-welded point; the pixel coordinates of the weld spot to be welded on the cropped weld region ROI image are converted into pixel coordinates on the corrected left view using the inverse operation to the step of cropping the weld region ROI.

7. The binocular vision-based initial welding dry elongation measurement method of claim 1, wherein the step 6 comprises:

step 6.1, respectively calculating a point positioned at the tail end of a welding gun nozzle joint on the axis of the welding gun and a three-dimensional coordinate of a to-be-welded point under a camera coordinate system; the equation for calculating the euclidean distance of two points in three dimensions is as follows,

wherein, (x) _a ,y _a ,z _a ) Is the three-dimensional coordinates of the point of the tip of the welding gun nozzle at the axis of the welding gun in the camera coordinate system, (x) _b ,y _b ,z _b ) Length is the visually measured point (x) at the end of the gun nozzle joint on the gun axis for the three-dimensional coordinates of the spot to be welded in the camera coordinate system _a ,y _a ,z _a ) To the point to be soldered (x) _b ,y _b ,z _b ) Euclidean distance of (c);

step 6.2, the distance length from the point of the tip of the welding gun nozzle joint located on the welding gun axis to the point to be welded is subtracted by the length len of the welding gun nozzle _weld I.e. the distance len from the nozzle tip to the point to be welded, which is also the dry extension of the welding wire, satisfying the following formula,

len＝length-len _weld

wherein len is the final calculated dry elongation of the welding wire, len _weld Is the nozzle length;

step 6.3, judging whether the measurement distance len of the previous step is within 0 cm to 7.5 cm, otherwise, directly jumping to the next frame for measurement; when 40 frames were measured, the measurement average value of these 40 frames was taken as the final result of binocular vision measurement.

8. The binocular vision-based initial welding dry elongation measurement device for realizing the binocular vision-based initial welding dry elongation measurement method of any one of claims 1 to 7, which is characterized by comprising a frame synchronous binocular camera module, a camera fixture, a computer, an illumination module, a welding gun and a device for moving the welding gun, wherein:

the frame synchronization binocular camera module has a focal length of 3.0mm, a base line of 63mm and an image resolution of 640 multiplied by 480 and is used for acquiring an original image;

the camera clamp is used for fixing the relative position between the binocular camera module and the welding gun, so that the base line of the camera is approximately perpendicular to the axis of the welding gun on the imaging plane of the binocular camera, and the camera is fixed on one side of the advancing direction of the welding gun;

the computer is used for real-time image acquisition, image processing, feature matching, stereo matching, key point estimation and recognition so as to finish the measurement of the initial welding dry elongation;

the lighting module is fixed on the device for moving the welding gun and is used for improving the ambient illumination;

the type of the welding gun is a Panasonic MIG/MAG welding gun; the outer diameter of the nozzle is 24mm, and the total length is 73mm; the nozzle joint is a brass standard nozzle joint; the welding gun and the camera clamp are fixed on a device for moving the welding gun; the welding gun, the camera and the lighting module are kept in fixed relative positions.