JP7724164B2 - Robot Control System - Google Patents

Description

The present invention relates to a robot control system that uses a three-dimensional visual sensor to provide feedback control of a robot.

Robots are being used to automate tasks such as gripping workpieces and connecting them to a designated position. In a typical robot control system, to move the robot's hand to a target object, position information of the object is acquired using a visual sensor such as a camera, and this position information is then converted from the camera coordinate system to the world coordinate system, and then converted back into the robot coordinate system before being transmitted to the robot. However, determining the transformation matrix between the camera coordinate system and the world coordinate system requires camera calibration, which is an extremely time-consuming and costly process.

In response to this, a technology known as visual feedback control has been developed in recent years. This control method involves feedback-controlling a robot based on feature quantities obtained from images captured by a visual sensor. For example, by using the distance between an object and the robot's hand as a feature quantity and feedback-controlling the robot's movement to reduce this distance, the hand can be made to reach the object without converting the coordinates of the object's position.

Patent Document 1 describes a robot control device that performs visual feedback control of the position and orientation of a robot arm in a robot system that grips workpieces, etc. The robot control device described in Patent Document 1 first photographs the workpiece with a CCD camera attached to the robot arm while the robot arm is positioned in advance at a taught target position relative to the workpiece, and stores the feature values in the obtained image as target data in a visual recognition device. The error between the current data and the target data during the operation is then calculated, and the position and orientation of the robot arm is repeatedly controlled until the error converges to zero. The feature values are calculated by extracting multiple feature points set on the workpiece from the image and using the visual recognition device to determine the feature values based on their positions and orientations, thereby recognizing the position and orientation of the workpiece.

Patent Document 2 describes a control device that performs visual feedback control on the arm and hand of a life support robot that assists in the care of elderly people and others. The control device moves the object to a target position after the object is grasped by the hand of the robot arm. To solve the problem of the object itself obstructing the camera's field of view just before or just after grasping the object, making it impossible to identify the target position, Patent Document 2 describes a method of installing a front camera on the grasping portion of the hand and a side camera on each side, and switching the camera that forms the basis of visual feedback control from the front camera to the side camera just before or just after the hand grasps the object.

Japanese Patent Application Laid-Open No. 2003-211381 JP 2011-235386 A

However, the control device described in Patent Document 1 has the problem that it cannot be applied to tasks involving linear or other workpieces that are easily deformed and whose relative positions relative to their feature points are not constant, because the feature values that should be used as target data differ for each workpiece. Furthermore, the control device described in Patent Document 2 cannot complete a task if the object to be grasped or the target position moves and goes out of the camera's field of view.

The present invention has been made in consideration of the above, and aims to provide a robot control system that causes a robot to perform work using visual feedback control, that is applicable even when the workpiece is easily deformed, and that can continue working even if the work object moves out of the field of view of the three-dimensional visual sensor.

The robot control system of the present invention is a system for controlling a robot that moves a first target, which moves in conjunction with the movement of a robot hand, to a second target that is stationary or moves independently of the movement of the robot hand. The system includes a three-dimensional visual sensor that acquires three-dimensional information about the second target, a sensor moving means that moves the three-dimensional visual sensor, and a control device. The control device calculates the distance between the second target and the first target based on the three-dimensional information, feedback-controls the position of the robot hand to reduce the distance between the second target and the first target, determines whether the second target is within the field of view of the three-dimensional visual sensor, and if it determines that the second target is outside the field of view, searches for the second target by widening or moving the field of view of the three-dimensional visual sensor, and controls the sensor moving means so that the second target is included in the field of view of the three-dimensional visual sensor.

In a first aspect of the robot control system of the present invention, the first object is the robot hand itself, and the second object is a task target on a workpiece.
That is, a robot control system of a first aspect is a system for controlling a robot that moves a robot hand to a work target on a workpiece that is stationary or moves independently of the movement of the robot hand, and includes a three-dimensional visual sensor that acquires three-dimensional information of the work target, a sensor moving means that moves the three-dimensional visual sensor, and a control device. The control device calculates the distance between the work target and the robot hand based on the three-dimensional information, feedback-controls the position of the robot hand so as to reduce the distance between the work target and the robot hand, determines whether the work target is within the field of view of the three-dimensional visual sensor, and if it is determined that the work target is outside the field of view, searches for the work target by widening or moving the field of view of the three-dimensional visual sensor, and controls the sensor moving means so that the work target is included in the field of view of the three-dimensional visual sensor.

Here, "work" broadly means any action performed by a robotic hand on a workpiece, including, for example, grasping a workpiece by clamping or suction, or performing various processes on the workpiece, such as cutting, grinding, discharge, or soldering.

In the robot control system of the first aspect described above, preferably, the control device further calculates a working direction for the robot hand to work on the work target based on the three-dimensional information, and feedback-controls the orientation of the robot hand so as to reduce the deviation between the working direction and the orientation of the robot hand.

In the robot control system of the first aspect described above, preferably, the control device further calculates a working posture for the robot hand to work on the work target based on the three-dimensional information, and feedback-controls the posture of the robot hand so as to reduce the deviation between the working posture and the posture of the robot hand.

In a second aspect of the robot control system of the present invention, the first object is a specific portion of a workpiece grasped by the robot hand, and the second object is a connection target that connects the specific portion.
That is, a robot control system of a second aspect is a system for controlling a robot that moves a specific portion of a workpiece gripped by a robot hand to a connection target that connects the specific portion and that is stationary or moves independently of the movement of the robot hand, and includes a three-dimensional visual sensor that acquires three-dimensional information of the connection target, a sensor moving means that moves the three-dimensional visual sensor, and a control device. The control device calculates the distance between the connection target and the specific portion based on the three-dimensional information, feedback-controls the position of the robot hand so as to reduce the distance between the connection target and the specific portion, determines whether the connection target is within the field of view of the three-dimensional visual sensor, and if it determines that the connection target is outside the field of view, searches for the connection target by widening or moving the field of view of the three-dimensional visual sensor, and controls the sensor moving means so that the connection target is included in the field of view of the three-dimensional visual sensor.

Here, "connecting" broadly means moving a specific part of the workpiece to align it with a connection target, and includes, for example, connecting a connector (specific part) on the workpiece to another mating connector (connection target), or inserting the tip (specific part) of a linear or strip-shaped workpiece into a hole or slit (connection target).

In the robot control system of the second aspect described above, preferably, the control device further calculates a connection direction for connecting the specific part to the connection target based on the three-dimensional information, and feedback-controls the posture of the robot hand so as to reduce the deviation between the connection direction and the orientation of the specific part.

In the robot control system of the second aspect described above, preferably, the control device further calculates a connection posture for connecting the specific part to the connection target based on the three-dimensional information, and feedback-controls the posture of the robot hand so as to reduce the deviation between the connection posture and the posture of the specific part.

In a third aspect of the robot control system of the present invention, the first object is a workpiece grasped by the robot hand, the second object is a placement target onto which the workpiece is to be placed, and the control device further calculates a placement orientation for placing the workpiece on the placement target based on the three-dimensional information, and feedback-controls the orientation of the robot hand so as to reduce the deviation between the placement orientation and the orientation of the workpiece.
That is, a robot control system of a third aspect is a system for controlling a robot that moves a workpiece grasped by a robot hand to a placement target on which the workpiece is to be placed, the placement target being stationary or moving independently of the movement of the robot hand, and includes a three-dimensional visual sensor that acquires three-dimensional information of the placement target, a sensor moving means that moves the three-dimensional visual sensor, and a control device. The control device calculates the distance between the placement target and the workpiece based on the three-dimensional information, calculates the orientation of the placement target, feedback-controls the position of the robot hand so as to reduce the deviation between the distance between the placement target and the workpiece and the orientation of the placement target and the orientation of the workpiece, determines whether the placement target is within the field of view of the three-dimensional visual sensor, and if it is determined that the placement target is outside the field of view, searches for the placement target by widening or moving the field of view of the three-dimensional visual sensor, and controls the sensor moving means so that the placement target is included in the field of view of the three-dimensional visual sensor.

Here, the placement target is an area large enough to place the work, and placement means placing the work within the placement target.

The robot control system of the present invention allows a robot to perform work using visual feedback control, even if the workpiece is easily deformed, and the work can continue even if the work object moves out of the field of view of the 3D visual sensor.

FIG. 1 is a diagram illustrating a device configuration of a robot control system according to a first embodiment. FIG. 1 is a process flow diagram of a workpiece gripping operation using the robot control system of the first embodiment. 10A and 10B are diagrams for explaining the gripping position and gripping posture of a linear workpiece; FIG. 10 is a diagram illustrating a device configuration of a robot control system according to a second embodiment. FIG. 10 is a process flow diagram of a workpiece gripping operation using a robot control system according to a second embodiment. FIG. 10 is a diagram illustrating a device configuration of a robot control system according to a third embodiment. FIG. 11 is a process flow diagram of a connection operation using a robot control system according to a third embodiment. FIG. 10 is a diagram for explaining the connection attitude of the connection target. FIG. 10 is a diagram illustrating a device configuration of a robot control system according to a fourth embodiment. FIG. 13 is a diagram illustrating a device configuration of a robot control system according to a fifth embodiment. FIG. 10 is a process flow diagram of a placement operation using the robot control system of the fifth or sixth embodiment. FIG. 13 is a diagram illustrating a device configuration of a robot control system according to a sixth embodiment.

A first embodiment of a robot control system of the present invention will be described with reference to Figures 1 to 3. In this embodiment, when a workpiece is grasped by a robot hand, the movement of the robot hand is feedback-controlled based on information acquired by a three-dimensional visual sensor fixed to the robot hand.

The robot control system 11 of this embodiment includes a robot 20 equipped with a robot hand 24, a stereo camera 30 fixed to the robot hand, and a control device 40 that controls the entire robot control system 11. The robot control system 11 causes the robot hand 24 to grasp a workpiece, that is, a cable 50, at a grasping position 52. The grasping position 52 is the target of the grasping operation. The position and orientation of the robot hand 24 are feedback-controlled by the control device 40 based on three-dimensional information acquired by the stereo camera 30. Note that in this specification, the robot hand may be simply referred to as the "hand." Furthermore, unless otherwise specified, the position of the hand refers to the position of the tool center point (TCP) of the hand.

In this embodiment, a cable 50 is described as an example of a workpiece, but the type of workpiece grasped by the hand 24 is not particularly limited. The workpiece may be a linear object other than a cable, or a strip-shaped object other than a linear object, such as a flat cable. The workpiece may also be a harness with a connector, electronic components, food, stationery, etc. The robot control system 11 of this embodiment can be applied to flexible and easily deformed workpieces, and is particularly advantageous when the workpiece is a linear or strip-shaped object. The supply form of the cable 50 is not particularly limited. The cable 50 may be placed on a tray or the like, or may be stored in a cylindrical container with one end protruding from the container. The cable 50 may be stationary, or may be placed on a conveyor belt 55 or the like and move independently of the operation of the hand 24.

Furthermore, while this embodiment describes the example of a cable gripping operation, the operation performed by the hand 24 on the work target is not limited to gripping. For example, by selecting an appropriate end effector as the hand, various processes such as cutting, grinding, discharge, and soldering may be performed on the work target.

The robot 20 is an articulated robot, and includes an arm 21 consisting of multiple links 22 and joints 23, and a hand 24 attached to the end of the arm. The type of hand 24 is not particularly limited, and a type suitable for grasping the workpiece can be used depending on the type of workpiece. For example, the hand may hold the workpiece between a pair of fingers, or may grasp the workpiece by vacuum suction with a nozzle. In this embodiment, a stereo camera 30, which is a three-dimensional visual sensor, is fixed to the hand 24, and the robot 20 also serves as a sensor movement means for changing the position and orientation of the stereo camera.

The stereo camera 30 is a three-dimensional visual sensor that acquires three-dimensional information about the cable 50. Here, three-dimensional information about the cable is information that enables the three-dimensional coordinates of the gripping position on the cable to be derived. Specifically, the three-dimensional information acquired by the stereo camera 30 is two images captured from different viewpoints. Hereinafter, the images acquired by the stereo camera 30 will be referred to as "stereo images" or simply "images."

In this embodiment, the stereo camera 30 is fixed to the hand 24. The stereo camera is preferably installed at the base end of the hand, facing toward the tip of the hand. However, since the relative positional relationship between the stereo camera and the hand is known from design drawings, etc., it is not necessary for the hand to be within the field of view of the stereo camera.

The type of three-dimensional visual sensor is not limited to a stereo camera, and is not particularly limited as long as it can acquire images containing three-dimensional information about the workpiece. Examples of three-dimensional visual sensors include active stereo systems such as light-section methods, in which one of the stereo cameras is replaced with a projector that projects light, and Time of Flight systems. A stereo camera is preferably used as the three-dimensional visual sensor. Stereo cameras are suitable for accurately measuring objects at close range in three dimensions, and when the hand 24 grasps the cable 50, the three-dimensional coordinates of the grasping position 52 can be determined with greater accuracy as the stereo camera 30 approaches the cable 50. Note that the stereo camera may be a system in which three or more cameras are used to capture three or more images simultaneously.

The control device 40 performs various calculations and controls the entire robot control system 11. The control device 40 performs various calculations, such as image processing of images captured by the stereo camera 30, determining the three-dimensional coordinates of the gripping position 52 on the cable 50, and calculating the distance between the gripping position 52 and the hand 24. The control device 40 feedback-controls the position of the hand by feedback-controlling the robot 20 to move the hand 24 so as to reduce the distance between the gripping position 52 and the hand 24. The control device 40 also determines whether the gripping position is within the image captured by the stereo camera 30. If the gripping position 52 is outside the field of view of the stereo camera 30 and is not captured in the image, the control device 40 searches for the gripping position by changing the position, orientation, or both of the stereo camera, and controls the operation of the sensor movement means, or in this embodiment, the robot 20, so that the gripping position is included in the field of view of the stereo camera. Details of the calculations and control performed by the control device will be described later.

The physical configuration of the control device 40 is not particularly limited. The control device 40 may be physically configured as a single device, or may be configured as two or more separate parts. In addition, the control device 40 may be configured integrally with the robot control unit (not shown) that controls the operation of the joints 23 and hands 24 of the robot 20.

Next, the process of gripping the cable 50 with the hand 24 using the robot control system 11 of this embodiment will be explained in accordance with the process flow shown in Figure 2.

(S11) The stereo camera 30 captures an image of the cable 50 to obtain three-dimensional information about the cable. This three-dimensional information serves as the basis for the control device 40 to calculate the three-dimensional coordinates of the gripping position 52 on the cable 50, so the stereo camera captures at least the range in which the coordinates of the gripping position can be calculated. For example, if the gripping position is indicated by a mark, the range in which the coordinates of the gripping position can be calculated is sufficient to include the mark. If the gripping position is determined by the distance from some landmark such as a mark or the tip of the cable, it refers to the portion from the landmark to the gripping position. In Figure 3, the gripping position 52 is set at a distance d from the tip 51 of the cable 50.

Furthermore, if the orientation X of the hand 24 when grasping the cable 50 is restricted, the stereo camera 30 captures at least a range in which it is possible to calculate the grasping direction A for the hand to grasp the grasping position 52. The range in which it is possible to calculate the grasping direction A is, for example, an area that includes the grasping position when grasping the cable from the side, and is a range wide enough to allow calculation of the tangent direction T of the cable at the grasping position 52. Once the tangent direction T is determined, it is possible to calculate the grasping direction A relative to the cable.

Furthermore, as in this embodiment, when the hand 24 is not isotropic around its direction X and grips the cable 50 with two fingers, it is necessary to prevent the fingers from interfering with the cable when the tool center point TCP is aligned with the gripping position 52. Therefore, when the hand grips the cable, not only the hand direction X but also the rotation angle θ around X is limited. In such a case, the stereo camera 30 captures images of at least the range within which the gripping posture for the hand to grip the gripping position 52 can be calculated. The gripping posture is the posture required for the hand to grip the cable. The hand posture is expressed by the hand direction X and the rotation angle θ around X, and the gripping posture is expressed by the gripping direction A and the rotation angle α around A of the hand gripping the gripping position. Similar to the range within which the gripping direction A can be calculated, the range within which the gripping posture can be calculated is, for example, an area including the gripping position when gripping the cable from the side, and is wide enough to allow the tangent direction T of the cable at the gripping position 52 to be calculated.

(S12) The control device 40 determines whether the gripping position 52 is included in the stereo image acquired by the stereo camera 30. If the gripping position is not within the field of view of the stereo camera 30 and is not included in the image, the gripping position is searched for. The search process (S17) will be described later. Note that if there is no landmark near the gripping position and it is not possible to determine whether the gripping position is included in the image without calculating the three-dimensional coordinates of the gripping position, this determination process (S12) is performed after the next process (S13). If the three-dimensional coordinates of the gripping position 52 can be calculated in the next process (S13), it is determined that the gripping position is included in the image; if the three-dimensional coordinates of the gripping position 52 cannot be calculated, it can be determined that the gripping position is not included in the image.

(S13) The control device 40 processes the stereo images to recognize the three-dimensional shape of the cable 50 and calculates the three-dimensional coordinates of the gripping position 52 and the gripping orientation. These values may be calculated within a certain tolerance range. The three-dimensional coordinates and orientation calculated by the control device 40 are all values in the camera coordinate system. Recognition of the three-dimensional shape of the cable and calculation of the three-dimensional coordinates of the gripping position 52 and the gripping orientation can be performed using known methods. For example, a method for calculating the gripping position and gripping orientation when the workpiece is a strip-shaped object is disclosed in Japanese Patent Application Laid-Open No. 2020-037147 by the present applicant.

(S14) The control device 40 calculates the distance between the gripping position 52 and the hand 24, and the deviation between the gripping posture and the posture of the hand. The position and posture of the hand 24 may be calculated based on the image if the hand is within the field of view of the stereo camera 30 and appears in the image. However, in this embodiment, the stereo camera 30 is fixed to the hand 24, so the relative positional relationship between the stereo camera and the hand is known. Therefore, even if the hand does not appear in the image, the position and posture of the hand in the camera coordinate system can be known.

(S15) The control device 40 determines whether the coordinates of the gripping position 52 and the position of the hand 24, and the gripping posture and posture of the hand 24 match. If the gripping position and the position of the hand, and the gripping posture and posture of the hand match, the hand 24 grasps the gripping position (step S18). Note that if an allowable range is given for the gripping posture, the two are considered to match if the hand posture is within that range. The same applies to the three-dimensional coordinates of the gripping position.

(S16) If the grasping position 52 and the position of the hand 24, or the grasping posture and the posture of the hand 24, do not match, the control device 40 instructs the robot 20 to perform feedback control of the position and posture of the hand 24 so as to reduce the distance between the grasping position and the hand, and the deviation between the grasping posture and the posture of the hand 24. During this process, the stereo camera 30 fixed to the hand 24 also approaches the grasping position, gradually improving the accuracy of calculation of the three-dimensional coordinates of the grasping position, etc.

(S17) If it is determined in step S12 that the image acquired by the stereo camera 30 does not include the gripping position 52, the field of view of the stereo camera is widened or moved to search for a gripping position outside the field of view of the stereo camera.

One way to widen the field of view of the stereo camera is to move the hand 24 to which the stereo camera is fixed away from the cable 50. Once the gripping position 52 is found, steps S11 and subsequent steps are repeated. Alternatively, the orientation of the stereo camera 30 may be changed to bring the gripping position closer to the center of the field of view, and then steps S11 and subsequent steps may be repeated.

Another method for widening the field of view of the stereo camera is to zoom out the stereo camera 30 if the focal length of the stereo camera 30 is variable. Once the gripping position 52 is found, the orientation of the stereo camera 30 is changed to bring the gripping position closer to the center of the field of view, and then steps S11 and subsequent steps are repeated. The focal length of the stereo camera is returned to its original position after the gripping position has been brought closer to the center of the field of view, or at some stage during the process of repeating steps S11 and subsequent steps.

One method for moving the field of view of the stereo camera is to change the posture of the hand 24 to which the stereo camera is fixed, for example, to move the field of view so that the center of the line of sight describes a spiral. Once the gripping position 52 is found, the orientation of the stereo camera 30 is left in the orientation in which the gripping position 52 was found, or preferably, the gripping position is moved closer to the center of the field of view, and then steps S11 and onward are repeated.

As described above, the control device 40 controls the robot 20, which is the sensor movement means, or controls the robot 20 and the stereo camera 30, to search for the gripping position 52. Note that in this step (S17), visual feedback control can be used to move the gripping position to the center of the field of view of the stereo camera. Also, in this step (S17), it is sufficient to determine whether the gripping position 52 is within the field of view of the stereo camera 30, and the three-dimensional coordinates of the gripping position are not used. Therefore, this step can also be performed using only one image from the stereo images.

By repeating the above steps, if it is determined in step S15 that the gripping position 52 and the position of the hand 24, and the gripping posture and the posture of the hand, match, the hand 24 grips the gripping position 52 (S18), completing the operation, as described above.

Note that if the posture of the hand when gripping the workpiece is not an issue, there is no need to calculate the grip posture, calculate the deviation between the grip posture and the hand posture, or determine whether they match. Each of the above steps can be performed by focusing only on the grip position and the position of the hand 24. Similarly, if the rotation angle θ of the hand when gripping the workpiece is not an issue, each of the above steps can be performed by focusing only on the grip position, grip direction A, hand position, and hand orientation X. Furthermore, correction of the hand orientation X and rotation angle θ may be performed independently of correction of the hand position, at some stage during the repetition of the above steps. The fact that it is sufficient to focus only on the hand position and orientation X when the posture or rotation angle of the hand is not an issue is the same in the second to fourth embodiments described below. The fact that correction of the hand posture can be performed independently of correction of the position is the same in the second to sixth embodiments described below.

A second embodiment of a robot control system of the present invention will be described with reference to Figures 4-5 and 3. In this embodiment, when a workpiece is grasped by a robot hand, the movement of the robot hand is feedback-controlled based on information acquired by a three-dimensional visual sensor installed away from the robot. Below, differences from the first embodiment will be described, and explanations of similarities to the first embodiment will be omitted.

The robot control system 12 of this embodiment includes a robot 20 equipped with a robot hand 24, a stereo camera 30 installed separately from the robot, a sensor movement means 32 for moving the stereo camera, and a control device 40 for overall control of the robot control system 12. The robot control system 12 causes the robot hand 24 to grasp a workpiece, a cable 50, at a grasping position 52. During this grasping operation, the position and orientation of the robot hand 24 are feedback-controlled by the control device 40 based on the three-dimensional information acquired by the stereo camera 30.

The stereo camera 30 acquires three-dimensional information about the cable 50, and in this embodiment, also acquires three-dimensional information about the hand 24.

The sensor moving means 32 changes the position and orientation of the stereo camera 30, which is a three-dimensional visual sensor. The structure of the sensor moving means may be an articulated robot similar to that of the robot 20, but since it is only necessary to change the position and orientation of the stereo camera within a predetermined range depending on the task, it can have a simpler structure than the robot 20. For example, if the cable 50 does not move, it is only necessary to change the distance between the stereo camera pointed at the cable and the cable in accordance with the movement of the hand 24, so a linear actuator that extends and retracts a cylinder can be used. Even if the cable 50 moves and an arm with joints is used, it is often possible to use fewer joints than the robot 20.

The control device 40 processes the images captured by the stereo camera 30, calculates the three-dimensional coordinates of the gripping position 52 on the cable 50 as well as the three-dimensional coordinates of the position of the hand 24, and determines whether the gripping position 52 and the hand 24 are included in the stereo images captured by the stereo camera 30. Details of the calculations and control by the control device will be described later.

Next, the process of gripping the cable 50 with the hand 24 using the robot control system 12 of this embodiment will be explained in accordance with the process flow shown in Figure 5.

(S21) The stereo camera 30 captures images of the cable 50 and hand 24 to obtain three-dimensional information about the cable and hand.

(S22) The control device 40 determines whether the gripping position 52 and hand 24 are included in the stereo image acquired by the stereo camera 30. If either the gripping position or the hand is not within the field of view of the stereo camera 30 and is not included in the image, the gripping position or hand is searched for in a search step (S27).

(S23) The control device 40 processes the stereo image, recognizes the three-dimensional shape of the cable 50, and calculates the three-dimensional coordinates of the gripping position 52 and the gripping posture, and at the same time recognizes the three-dimensional shape of the hand 24 and calculates the position and posture of the hand.

(S24) The control device 40 calculates the distance between the gripping position 52 and the hand 24, and the deviation between the gripping posture and the hand posture.

(S25) The control device 40 determines whether the coordinates of the gripping position 52 and the position of the hand 24, and whether the gripping posture and the posture of the hand 24 match. If the gripping position and the position of the hand, and the gripping posture and the posture of the hand, match, the hand 24 grips the gripping position (step S28).

(S26) If the gripping position 52 and the position of the hand 24, or the gripping posture and posture of the hand 24, do not match, the control device 40 instructs the robot 20 to feedback control the position and posture of the hand 24 so as to reduce the discrepancy between the distance between the gripping position and the hand, and between the gripping posture and the posture of the hand 24. The control device 40 also instructs the sensor movement means 32 to control the position and posture of the stereo camera 30. Specifically, when the gripping position 52 or the hand 24 is on the periphery of the image, the field of view of the stereo camera is moved so as to move both toward the center of the image. Furthermore, as the hand approaches the cable, the distance between the stereo camera and the gripping position is reduced to improve the measurement accuracy of the gripping position and the position and posture of the hand.

(S27) If it is determined in step S22 that the gripping position 52 or the hand 24 is not included in the image acquired by the stereo camera 30, the gripping position is searched for. As in the first embodiment, the gripping position can be searched for by widening or moving the field of view of the stereo camera 30. Specifically, the field of view of the stereo camera is widened by moving the stereo camera away from the cable 50, or by controlling the stereo camera to zoom out if the focal length of the stereo camera 30 is variable, and a gripping position outside the field of view of the stereo camera is searched for. Alternatively, the field of view of the stereo camera is moved to search for a gripping position outside the field of view of the stereo camera. If the gripping position 52 or the hand 24 is found, the gripping position and the hand are brought into the field of view of the stereo camera, and then step S21 and subsequent steps are repeated.

By repeating the above steps, if it is determined in step S25 that the gripping position 52 and the position of the hand 24, and the gripping posture and the posture of the hand, match, the hand 24 grips the gripping position 52 (S28), completing the operation, as described above.

A third embodiment of a robot control system of the present invention will be described with reference to Figures 6 to 8. In this embodiment, during the task of connecting a characteristic portion of a workpiece grasped by the robot hand to a predetermined connection target, the movement of the robot hand is feedback-controlled based on information acquired by a three-dimensional visual sensor fixed to the robot hand. Below, differences from the first embodiment will be described, and explanations of similarities to the first embodiment will be omitted.

The robot control system 13 of this embodiment includes a robot 20 equipped with a robot hand 24, a stereo camera 30 fixed to the robot hand, and a control device 40 that controls the entire robot control system 13. The robot control system 13 connects a plug 61 (specific part) attached to the end of a cable 60 (workpiece) grasped by the robot hand 24 to a socket 63 (connection target) on a circuit board 62. During this connection operation, the position and orientation of the robot hand 24 are feedback-controlled by the control device 40 based on three-dimensional information acquired by the stereo camera 30.

In this embodiment, the cable 60 is described as an example of the workpiece, the plug 61 as a specific part, and the socket 63 as an example of the connection target, but these types are not particularly limited. The robot control system 13 of this embodiment can also be used for connection tasks such as passing the tip of a cable or tube through an opening or slit in a processing device, fitting a specific part of various components into a specific position inside the device, or fixing a specific part of a cover or exterior panel to a specific position on the device body. The socket 63 may be stationary, or may be placed on a conveyor belt 55 or the like and move independently of the operation of the hand 24.

In this embodiment, if the workpiece does not deform (including when the robot hand 24 directly grasps the plug 61), and the workpiece is grasped in a specific position and orientation, the relative positional relationship between the stereo camera 30 fixed to the hand 24 and the plug 61 is known. On the other hand, if the workpiece is flexible, easily deformed, or the grasped position or orientation of the workpiece is not fixed, the relative positional relationship between the stereo camera and the plug is not constant and is unknown. The following describes a case where the relative positional relationship between the stereo camera and the plug is unknown.

The stereo camera 30 acquires three-dimensional information about the socket 63 and plug 61. In this embodiment, the robot 20 also serves as a sensor movement means for changing the position and orientation of the stereo camera.

The control device 40 processes images captured by the stereo camera 30, performs various calculations, feedback control of the robot 20, searches for the socket 63, etc. The details of the calculations and controls performed by the control device will be described later.

Next, the connection work using the robot control system 13 of this embodiment will be explained according to the process flow in Figure 7.

(S31) Images of the socket 63 and plug 61 are captured with the stereo camera 30 to obtain three-dimensional information on their respective positions. Furthermore, if the orientation Y of the plug when connecting the plug to the socket and the rotation angle φ around Y are restricted, the stereo camera 30 captures images within a range that allows calculation of at least the connection direction B for connecting the plug to the socket and the rotation angle β around B. Hereinafter, the connection direction B and rotation angle β required to connect the plug to the socket will be collectively referred to as the "connection posture."

(S32) The control device 40 determines whether the socket 63 and plug 61 are included in the stereo image acquired by the stereo camera 30, and if the socket or plug is not within the field of view of the stereo camera 30 and is not included in the image, it performs a search process (S37).

(S33) The control device 40 processes the stereo image to recognize the three-dimensional shapes of the socket 63 and plug 61, and calculates the three-dimensional coordinates of the socket's position and connection attitude, as well as the three-dimensional coordinates of the plug's position and attitude.

(S34) The control device 40 calculates the distance between the socket 63 and the plug 61, and the deviation between the connection posture and the plug posture.

(S35) The control device 40 determines whether the positions of the socket 63 and plug 61 and the connection posture and plug posture match. If the positions of the socket and plug and the connection posture and plug posture match, the plug is connected to the socket (step S38).

(S36) If the positions of the socket 63 and plug 61, or the connection attitude and plug attitude, do not match, the control device 40 instructs the robot 20 to perform feedback control of the hand 24 so as to reduce the distance between the socket and plug, and the discrepancy between the connection attitude and plug attitude. During this process, the stereo camera 30 fixed to the hand 24 also approaches the socket, gradually improving the accuracy of calculations of the socket's three-dimensional coordinates, etc.

(S37) If the determination result in step S32 indicates that a socket 63 or plug 61 is to be searched for, the field of view of the stereo camera is widened or moved, as in the first embodiment, to search for a socket or plug that is out of the field of view of the stereo camera.

By repeating the above steps, if it is determined in step S35 that the positions of the socket 63 and plug 61, and the connection posture and plug posture, match, the plug is connected to the socket (S38), completing the process, as described above.

Note that if the relative positional relationship between the stereo camera 30 and the plug 61 is known, it is not necessary to capture an image of the plug in step S31, determine whether the plug is included in the image in step S32, calculate the position and orientation of the plug in step S33, and search for the plug in step S37.

A fourth embodiment of the robot control system of the present invention will be described with reference to Figures 8 and 9. In this embodiment, when a characteristic portion of a workpiece grasped by a robot hand is connected to a predetermined connection target, the movement of the robot hand is feedback-controlled based on information acquired by a three-dimensional visual sensor installed away from the robot. Below, differences from the third embodiment will be described, and explanations of similarities to the third embodiment will be omitted.

The robot control system 14 of this embodiment comprises a robot 20 equipped with a robot hand 24, a stereo camera 30 installed separately from the robot, a sensor moving means 32 for moving the stereo camera, and a control device 40 for overall control of the robot control system 14. The robot control system 14 connects a plug 61 (specific part) attached to the end of a cable 60 (workpiece) grasped by the robot hand 24 to a socket 63 (connection target) on a circuit board 62. During this connection operation, the position and orientation of the robot hand 24 are feedback-controlled by the control device 40 based on three-dimensional information acquired by the stereo camera 30.

In this embodiment, the stereo camera 30 is installed away from the robot, so the relative positional relationship between the stereo camera 30 and the plug 61 is unknown regardless of the type of workpiece. Therefore, the stereo camera 30 acquires three-dimensional information about the socket 63 and plug 61, and the control device 40 calculates the three-dimensional coordinates of the socket and plug positions based on the images acquired by the stereo camera 30. The control device 40 also determines whether the socket and plug are within the image, and searches for the socket or plug if they are not visible in the image. The sensor movement means 32 changes the position and orientation of the stereo camera 30, which is a three-dimensional visual sensor. The structure of the sensor movement means 32 is the same as that described in the second embodiment.

The connection operation method using the robot control system 14 in this embodiment is the same as in the third embodiment, where the relative positional relationship between the stereo camera 30 and the plug 61 is unknown. However, in this embodiment, the stereo camera 30 is installed away from the robot, so in step S36 of FIG. 7, the hand 24 is moved and the control device 40 instructs the sensor movement means 32 to control the position and orientation of the stereo camera 30. Specifically, when the socket 63 or plug 61 is on the periphery of the image, the field of view of the stereo camera is moved so that both are moved toward the center of the image. Furthermore, as the hand approaches the socket, the distance between the stereo camera and the socket is reduced to improve the measurement accuracy of the position and orientation of the socket and plug.

A fifth embodiment of a robot control system of the present invention will be described with reference to Figures 10 and 11. In this embodiment, when placing a workpiece grasped by a robot hand at a predetermined placement target, the movement of the robot hand is feedback-controlled based on information acquired by a three-dimensional visual sensor fixed to the robot hand. Below, differences from the first and third embodiments will be described, and descriptions of similarities to the first and third embodiments will be omitted.

The robot control system 15 of this embodiment includes a robot 20 equipped with a robot hand 24, a stereo camera 30 fixed to the robot hand, and a control device 40 that controls the entire robot control system 15. The robot control system 15 places the cable 70 held by the robot hand 24 on a tray 72. During this placement operation, the position and orientation of the robot hand 24 are feedback-controlled by the control device 40 based on three-dimensional information acquired by the stereo camera 30.

In this embodiment, the cable 70 is described as an example of the workpiece and the tray 72 as an example of the placement target, but these types are not particularly limited. The robot control system 15 of this embodiment can be applied to tasks such as placing various types of workpieces in predetermined locations or storing them in predetermined containers. Furthermore, the tray 72 may be stationary, or may be placed on a conveyor belt 55 or the like and move independently of the operation of the hand 24.

In this embodiment, as in the third embodiment, if the workpiece does not deform, the relative positional relationship between the stereo camera 30 fixed to the hand 24 and the workpiece is known. On the other hand, if the workpiece is flexible or easily deformed, the relative positional relationship between the stereo camera and the workpiece is not constant and is unknown. The following describes the case where the relative positional relationship between the stereo camera and the workpiece is unknown.

The stereo camera 30 acquires three-dimensional information about the tray 72 and cable 70. The robot 20 also serves as a sensor movement means for changing the position and orientation of the stereo camera. The control device 40 processes the images acquired by the stereo camera 30, performs various calculations, feedback control of the robot 20, searches for the tray 72, etc. The calculations and control performed by the control device will be described later.

Next, the connection work using the robot control system 15 of this embodiment will be explained according to the process flow in Figure 11.

(S41) The stereo camera 30 captures images of the tray 72 and cable 70 to obtain three-dimensional information that enables calculation of their respective positions and orientations.

(S42) The control device 40 determines whether the tray 72 and cable 70 are included in the stereo image acquired by the stereo camera 30, and if the tray or cable is not within the field of view of the stereo camera 30 and is not included in the image, it performs a search process (S47).

(S43) The control device 40 processes the stereo image to recognize the three-dimensional shapes of the tray 72 and cable 70, and calculates the position and orientation of the tray, as well as the position and orientation of the cable. Here, the position and orientation of the tray can be represented by the center position and orientation of the tray surface, and the position and orientation of the cable can be represented by the center position and orientation of a rectangle or rectangular parallelepiped that contains the cable and is large enough to fit within the tray.

(S44) The control device 40 calculates the distance between the tray 72 and the cable 70, as well as the deviation in posture.

(S45) The control device 40 determines whether the position and orientation of the tray 72 and the cable 70 match. If the position and orientation of the tray and the cable match, the cable is placed on the tray (step S48).

(S46) If the positions or postures of the tray 72 and cable 70 do not match, the control device 40 instructs the robot 20 to perform feedback control of the hand 24 to reduce the deviation in the distance and posture between the tray and cable.

(S47) If the determination result in step S42 indicates that a tray 72 or cable 70 needs to be searched for, the field of view of the stereo camera is widened or moved, as in the first embodiment, to search for a tray or cable that is out of the field of view of the stereo camera.

By repeating the above steps, if it is determined in step S45 that the positions and connections of the tray 72 and cable 70 match, as described above, the cable is placed on the tray (S48) and the work is completed.

A sixth embodiment of the robot control system 16 of the present invention feedback-controls the movement of a robot hand based on information acquired by a three-dimensional visual sensor installed away from the robot during the task of placing a workpiece grasped by the robot hand at a predetermined placement target.

Referring to FIG. 12, the robot control system 16 of this embodiment differs from the fifth embodiment in that the stereo camera 30 is installed separately from the robot 20 and attached to a sensor movement means 32 that is independent of the robot, but the rest of the configuration is the same as in the fifth embodiment. Furthermore, the cable placement method using the robot control system 16 of this embodiment is carried out in accordance with the process flow shown in FIG. 11, and differs from the fifth embodiment in that the movement of the stereo camera is controlled by the sensor movement means 32 rather than the robot 20, but is otherwise the same as in the fifth embodiment.

The present invention is not limited to the above-described embodiments and examples, and various modifications are possible within the scope of its technical concept.

11-16 Robot control system 20 Robot 21 Arm 22 Link 23 Joint 24 Robot hand 25 Tool center point 30 Stereo camera (3D visual sensor)
32 Sensor moving means 40 Control device 50 Cable (work)
51 Cable tip 52 Grasping position (work target)
55 Conveyor belt 60 Cable (work)
61 Plug (specific area)
62 Board 63 Socket (Connection Target)
70 Cable (work)
72 Tray (placement target)
A Gripping direction B Connection direction d Distance from the cable tip to the gripping position T Tangential direction of the cable at the gripping position X Hand direction Y Plug direction α Rotation angle of the gripping position β Rotation angle of the connection position θ Rotation angle of the robot hand φ Rotation angle of the plug

Claims (3)

A system for controlling a robot that moves a robot hand to a gripping position of a flexible cable that is stationary or moves independently of the movement of the robot hand, comprising:
a stereo camera that captures an image of the gripping position to acquire a stereo image;
a sensor moving means for moving the stereo camera;
a control device;
the robot hand is capable of grasping the cable by pinching it,
The control device
Calculating three-dimensional coordinates of the gripping position in a camera coordinate system based on the stereo images, calculating a distance between the gripping position and the robot hand, and feedback-controlling the position of the robot hand so as to reduce the distance between the gripping position and the robot hand ;
calculating a tangential direction of the cable at the gripping position, a gripping direction intersecting the tangential direction, and a rotation angle around the gripping direction at which the robot hand can pinch the cable in the camera coordinate system based on the stereo images, thereby calculating a working posture for the robot hand to grip the gripping position, and feedback-controlling the posture of the robot hand so as to reduce a deviation between the working posture and the posture of the robot hand;
determining whether the gripping position is within the field of view of the stereo camera, and if it is determined that the gripping position is outside the field of view, searching for the gripping position by widening or moving the field of view of the stereo camera, and controlling the sensor moving means so that the gripping position is included in the field of view of the stereo camera;
Robot control system. the stereo camera is fixed to the robot hand,
the sensor moving means is the robot;
The robot control system of claim 1 . The stereo camera is installed at a distance from the robot.
The robot control system of claim 1 .