WO2020022040A1 - Système de commande, procédé de commande et programme - Google Patents

Système de commande, procédé de commande et programme Download PDF

Info

Publication number
WO2020022040A1
WO2020022040A1 PCT/JP2019/026958 JP2019026958W WO2020022040A1 WO 2020022040 A1 WO2020022040 A1 WO 2020022040A1 JP 2019026958 W JP2019026958 W JP 2019026958W WO 2020022040 A1 WO2020022040 A1 WO 2020022040A1
Authority
WO
WIPO (PCT)
Prior art keywords
robot
control
state
moving image
unit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2019/026958
Other languages
English (en)
Japanese (ja)
Inventor
加藤　豊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Omron Corp
Original Assignee
Omron Corp
Omron Tateisi Electronics Co
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Omron Corp, Omron Tateisi Electronics Co filed Critical Omron Corp
Publication of WO2020022040A1 publication Critical patent/WO2020022040A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
    • H—ELECTRICITY
    • H01—ELECTRIC ELEMENTS
    • H01R—ELECTRICALLY-CONDUCTIVE CONNECTIONS; STRUCTURAL ASSOCIATIONS OF A PLURALITY OF MUTUALLY-INSULATED ELECTRICAL CONNECTING ELEMENTS; COUPLING DEVICES; CURRENT COLLECTORS
    • H01R43/00—Apparatus or processes specially adapted for manufacturing, assembling, maintaining, or repairing of line connectors or current collectors or for joining electric conductors
    • H01R43/26—Apparatus or processes specially adapted for manufacturing, assembling, maintaining, or repairing of line connectors or current collectors or for joining electric conductors for engaging or disengaging the two parts of a coupling device

Definitions

  • the present technology relates to a control system, a control method, and a program for controlling a robot.
  • Non-Patent Document 1 The technique described in Non-Patent Document 1 is based on the premise that the relative position between the terminal supported by the robot hand and the camera is constant. Therefore, if the relative positional relationship between the terminal supported by the robot hand and the camera is deviated, the connection between the state of the terminal and the state of the connector may be lost, and the terminal may not be inserted into the connector.
  • the present invention has been made in view of the above problems, and an object of the present invention is to provide a control system, a control method, and a program in which the states of a plurality of objects can be changed in a coordinated manner.
  • the control system includes first to Nth robots, an imaging device for imaging the first to Nth objects, and a control system for controlling the first to Nth robots.
  • a control device. N is an integer of 2 or more.
  • the i-th robot changes the state of the i-th object.
  • i is an integer of 1 to N-1.
  • the Nth robot changes one state of the Nth object and the imaging device.
  • the other of the N-th object and the imaging device is installed at a fixed position.
  • the control device acquires change information for each of the first to Nth objects.
  • the change information corresponding to the j-th object indicates the relationship between the control amount of the j-th robot and the change amount of the state of the j-th object on the image of the imaging device.
  • the control device includes: a first process for acquiring a real image captured by the imaging device; a second process for selecting a target frame from CG moving images indicating the first to Nth object models; a real image and a target frame And a third process for controlling each of the first to Nth robots based on the above.
  • the control device brings the state of the j-th object on the real image closer to the state of the model of the j-th object on the target frame based on the change information corresponding to the j-th object.
  • the control amount of the j-th robot is calculated and the j-th robot is controlled in accordance with the calculated control amount.
  • the state of the target on the real image can be changed to the state of the model of the target on the target frame based on the real image, the target frame, and the change information.
  • the states of the first to N-th objects change in a coordinated manner according to the CG moving image.
  • the CG moving image is created based on the design data of each of the first to Nth objects.
  • the first object and the at least one object are joined from a frame indicating a state where the first object and at least one of the second to Nth objects are separated from each other. This includes up to the frame indicating the status.
  • the control device adjusts the control amount of the first robot by an adjustment amount according to a manufacturing error of the first object and at least one object with respect to the design data, and adjusts the control amount of the first robot in accordance with the adjusted control amount. Control one robot.
  • the first object and the at least one object can be joined even if a manufacturing error occurs in the first object and at least one object.
  • control system includes a force sensor that detects a force that one of the first object and the at least one object receives from the other.
  • the control device searches for a state of the first object where the force detected by the force sensor is minimal, and determines an adjustment amount based on the searched state of the first object.
  • the adjustment amount is determined based on the state of the first target at which the detection value of the force sensor is minimal. Therefore, the state of the first object can be changed in a state in which the relative displacement between the first object and at least one object caused by the manufacturing error is reduced.
  • control device is connected to the input device and acquires the adjustment amount from the input device. Thereby, the control device can easily acquire the adjustment amount.
  • the CG moving image includes a first object and at least one object from a frame indicating a state where the first object and at least one of the second to Nth objects are separated from each other. Includes the frame up to the state where the object is joined.
  • the control system includes a force sensor that detects a force that one of the first object and the at least one object receives from the other.
  • the control device performs a first mode in which the first to third processes are repeated. In the third processing in the first mode, the control device adjusts the control amount of the first robot and searches for an adjustment amount at which the force detected by the force sensor becomes a minimum.
  • the control device acquires, as the reference moving image, a moving image captured by the imaging device when the control amount of the first robot is adjusted by the searched adjustment amount.
  • the control device performs a second mode in which the first process, the fourth process of selecting a target frame from the reference moving image instead of the CG moving image, and the third process are repeated.
  • the reference moving image is an image captured when the force detected by the force sensor becomes a minimum. Therefore, even if a manufacturing error occurs in the first object and at least one object by performing the processing according to the second mode by the control device, the first object and the at least one object can be compared with each other. Can be joined.
  • the control method controls the first to Nth robots using the imaging device for imaging the first to Nth objects.
  • N is an integer of 2 or more.
  • the i-th robot changes the state of the i-th object.
  • i is an integer of 1 to N-1.
  • the Nth robot changes one state of the Nth object and the imaging device.
  • the other of the N-th object and the imaging device is installed at a fixed position.
  • the control method includes a first step of acquiring change information for each of the first to Nth robots.
  • the change information corresponding to the j-th robot indicates a relationship between the control amount of the j-th robot and the change amount of the state of the j-th object on the image of the imaging device.
  • j is an integer of 1 to N.
  • the control method includes a second step of acquiring a real image captured by the imaging device, a third step of selecting a target frame from CG moving images indicating the first to N-th object models, a real image and a target frame.
  • a program is a program for causing a computer to execute the above control method.
  • the states of a plurality of objects also change in a coordinated manner.
  • the states of a plurality of objects change in a coordinated manner.
  • FIG. 4 is a diagram illustrating an example of a real image captured by an imaging device and a first CG moving image.
  • FIG. 3 is a diagram illustrating an example of a real image and a second CG moving image captured by an imaging device 22.
  • FIG. 2 is a schematic diagram illustrating a hardware configuration of a control device included in the control system according to the embodiment.
  • FIG. 2 is a block diagram illustrating a functional configuration of a control device according to the embodiment.
  • FIG. 6 is a diagram illustrating an example of a method for creating a template.
  • FIG. 3 is a block diagram illustrating a functional configuration of a first control unit and a second control unit according to the embodiment.
  • FIG. 4 is a diagram illustrating an example of a real image captured by an imaging device and a first CG moving image.
  • FIG. 3 is a diagram illustrating an example of a real image and a second CG moving image captured by an imaging device 22.
  • FIG. 2 is a schematic diagram illustrating
  • FIG. 5 is a diagram illustrating a method of generating a first change information set in a first control unit.
  • FIG. 4 is a diagram illustrating a method of calculating a control amount by a calculation unit of a first control unit. It is a flowchart which shows an example of the flow of a process which controls a target robot so that the state of a target object may be changed along a 1st CG moving image and a 2nd CG moving image.
  • 11 is a flowchart showing the flow of processing of a subroutine of step S4 shown in FIG.
  • FIG. 3 is a diagram illustrating a relationship between a closest frame and a target frame.
  • 11 is a flowchart showing the flow of the processing of a subroutine of step S6 shown in FIG.
  • FIG. 9 is a schematic diagram illustrating an outline of a control system according to a first modification.
  • FIG. 9 is a block diagram illustrating a functional configuration of a control device according to a first modification.
  • FIG. 9 is a block diagram illustrating a functional configuration of a first control unit and a second control unit according to a first modification.
  • 15 is a flowchart illustrating an example of a flow of an offset amount determination process in Modification Example 1.
  • FIG. 13 is a block diagram illustrating a functional configuration of a control device according to a third modification.
  • FIG. 14 is a block diagram illustrating a functional configuration of a first control unit and a second control unit according to a third modification.
  • FIG. 14 is a schematic diagram illustrating an object of a control system according to Modification 4.
  • 15 is a schematic diagram illustrating an object of a control system according to Modification Example 5.
  • FIG. 15 is a schematic diagram illustrating an outline of a control system according to Modification Example 6.
  • FIG. 13 is a block diagram illustrating a functional configuration of a control device according to a modification 6.
  • 15 is a flowchart illustrating a flow of processing of a first control unit and a second control unit of a modification 6.
  • FIG. 15 is a schematic diagram illustrating another configuration example of the control system according to Modification 6.
  • FIG. 1 is a schematic diagram illustrating an outline of a control system according to an embodiment.
  • the control system 1 connects the male connector 2a and the female connector 2b by inserting the male connector 2a into the female connector 2b in, for example, an industrial product production line.
  • control system 1 includes imaging devices 21 and 22, robots 30a and 30b, robot controllers 40a and 40b, and a control device 50.
  • the imaging devices 21 and 22 capture an image of a subject present in the field of view and generate image data (hereinafter, simply referred to as “image”).
  • image image data
  • the imaging devices 21 and 22 are installed at fixed positions different from the robots 30a and 30b.
  • the imaging devices 21 and 22 capture images of the male connector 2a and the female connector 2b as subjects from different fixed positions.
  • the imaging devices 21 and 22 perform imaging according to a predetermined imaging cycle, and output a real image obtained by the imaging to the control device 50.
  • the robot 30a is a mechanism for changing the state (here, position and posture) of the male connector 2a, and is, for example, a vertical articulated robot.
  • the robot 30a has a hand 31a that supports (holds here) the male connector 2a at its tip, and changes the position and posture of the hand 31a with six degrees of freedom.
  • the robot 30a changes the position and the posture of the male connector 2a held by the hand 31a with six degrees of freedom.
  • the six degrees of freedom include translational degrees of freedom in the X, Y, and Z directions, and rotational degrees of freedom in the pitch, yaw, and roll directions.
  • the number of degrees of freedom of the hand 31a is not limited to six, and may be three to five or seven or more.
  • the robot 30a has a plurality of servo motors, and the position and posture of the male connector 2a are changed by driving the servo motors.
  • An encoder is provided for each of the plurality of servomotors, and the position of the servomotor is measured.
  • the robot 30b is a mechanism for changing the state (here, position and posture) of the female connector 2b, and is, for example, an XY ⁇ stage.
  • the robot 30b has a stage 31b supporting (put on) the female connector 2b, and changes the position and the posture of the stage 31b with three degrees of freedom. That is, the robot 30b changes the position and posture of the female connector 2b mounted on the stage 31b with three degrees of freedom.
  • the three degrees of freedom include a translational degree of freedom in the X direction and the Y direction and a rotational degree of freedom in a rotational direction ( ⁇ direction) about an axis orthogonal to the XY plane.
  • the number of degrees of freedom of the robot 30b is not limited to three, and may be four or more.
  • the robot 30b has a plurality of servomotors, and drives and changes the position and orientation of the female connector 2b.
  • An encoder is provided for each of the plurality of servomotors, and the position of the servomotor is measured.
  • the robot controller 40a controls the operation of the robot 30a according to the control command received from the control device 50.
  • the robot controller 40a receives from the control device 50 control commands for translational degrees of freedom in the X, Y, and Z directions and rotational degrees of freedom in the pitch, yaw, and roll directions. These X direction, Y direction, Z direction, pitch direction, yaw direction, and roll direction are indicated by the coordinate system of the robot 30a.
  • the robot controller 40a performs feedback control on the robot 30a so that the translation amounts of the hand 31a in the X, Y, and Z directions approach control commands for the degrees of freedom of translation in the X, Y, and Z directions, respectively. Do.
  • the robot controller 40a performs feedback control on the robot 30a so that the rotational movement amounts of the hand 31a in the pitch direction, the yaw direction, and the roll direction approach the control commands for the rotational degrees of freedom in the pitch, yaw, and roll directions, respectively. Do.
  • the robot controller 40b controls the operation of the robot 30b according to the control command received from the control device 50.
  • the robot controller 40b receives from the control device 50 control commands for the degrees of freedom of translation and rotation in the X and Y directions. These X direction, Y direction, and rotation direction are indicated by the coordinate system of the robot 30b.
  • the robot controller 40b performs feedback control on the robot 30b such that the translation amounts of the stage 31b in the X and Y directions approach control commands for the degrees of freedom of translation in the X and Y directions, respectively.
  • the robot controller 40b performs feedback control on the robot 30b such that the rotational movement amount of the stage 31b approaches the control command of the rotational degree of freedom.
  • the control device 50 controls the robots 30a and 30b via the robot controllers 40a and 40b, respectively.
  • the control device 50 stores a first CG moving image and a second CG moving image, which show samples of the male connector 2a and the female connector 2b.
  • the first CG moving image and the second CG moving image are created by simulating a model of the male connector 2a, the female connector 2b, the robots 30a and 30b, and the imaging devices 21 and 22 in a virtual space.
  • the first CG moving image and the second CG moving image show how the model of the male connector 2a and the model of the female connector 2b are connected to each other in the virtual space while changing the position and orientation.
  • the models of the male connector 2a and the female connector 2b are created based on the design data of the male connector 2a and the female connector 2b.
  • the first CG moving image is created by setting the position of the model of the imaging device 21 in the virtual space as a viewpoint.
  • the second CG moving image is created by setting the position of the model of the imaging device 22 in the virtual space as a viewpoint.
  • Each of the first CG moving image and the second CG moving image includes a plurality of frames (hereinafter, referred to as M (M is an integer of 2 or more)) arranged in time series.
  • M is an integer of 2 or more
  • the k-th frame (k is an integer from 1 to M) of the first CG moving image and the k-th frame of the second CG moving image are obtained by simultaneously using the model of the male connector 2a and the female connector 2b in a certain state in the virtual space. These are images when viewed from different directions.
  • the control device 50 acquires change information indicating a relationship between a control amount of the robot 30a and a change amount of a state of the male connector 2a on an image obtained by imaging by the imaging devices 21 and 22.
  • the change information is generated by simulating a model of the male connector 2a, the robot 30a, and the imaging devices 21 and 22 in a virtual space.
  • control device 50 acquires change information indicating the relationship between the control amount of the robot 30b and the change amount of the state of the female connector 2b on the images obtained by the imaging devices 21 and 22.
  • the change information is generated by simulating a model of the female connector 2b, the robot 30b, and the imaging devices 21 and 22 in a virtual space.
  • the controller 50 performs the following first to third processing.
  • the control device 50 repeatedly executes a series of processes including the first to third processes.
  • the first process is a process of acquiring actual images captured by the imaging devices 21 and 22.
  • the second process is a process of selecting a target frame from each of the first CG moving image and the second CG moving image.
  • the control device 50 selects the k-th frame from the second reference moving image as the target frame.
  • the third process is a process for controlling each of the robots 30a and 30b based on the actual image and the target frame.
  • the control device 50 calculates and calculates the control amount of the robot 30a for bringing the state of the male connector 2a on the actual image closer to the state of the male connector 2a on the target frame based on the change information corresponding to the robot 30a.
  • the robot 30a is controlled according to the control amount.
  • the control device 50 generates a control command indicating a control amount of the robot 30a, and outputs the generated control command to the robot controller 40a.
  • control device 50 calculates a control amount of the robot 30b for bringing the state of the female connector 2b on the actual image closer to the state of the female connector 2b on the target frame based on the change information corresponding to the robot 30b, The robot 30b is controlled according to the calculated control amount.
  • the control device 50 generates a control command indicating a control amount of the robot 30b, and outputs the generated control command to the robot controller 40b.
  • FIG. 2 is a diagram illustrating an example of a real image captured by the imaging device 21 and a first CG moving image.
  • FIG. 3 is a diagram illustrating an example of a real image captured by the image capturing device 22 and a second CG moving image.
  • FIG. 2 shows real images 90a to 93a captured by the imaging device 21 and frames 70a to 73a of the first CG moving image.
  • FIG. 3 shows real images 90b to 93b captured by the imaging device 22, and frames 70b to 73b of the second CG moving image.
  • the real images 90a and 90b are images captured at the same time.
  • the real images 91a and 91b are images captured at the same time after the real images 90a and 90b.
  • the real images 92a and 92b are images captured at the same time after the real images 91a and 91b.
  • the real images 93a and 93b are images captured at the same time after the real images 92a and 92b.
  • Each of the frames 70a and 70b is the first frame in the corresponding CG moving image.
  • Each of the frames 71a and 71b is an s-th (s is an integer of 2 or more) frame in the corresponding CG moving image.
  • Each of the frames 72a and 72b is a t-th (t is an integer greater than s) frame in the corresponding CG moving image.
  • Each of the frames 73a and 73b is a u-th (u is an integer greater than t) frame in the corresponding CG moving image.
  • the first CG moving image and the second CG moving image include a model of the hand 31a (hereinafter, referred to as “hand model”) 31a ′ and a model of the stage 31b (hereinafter, referred to as “stage model”) 31b ′. Further, the first CG moving image and the second CG moving image include a model of male connector 2a (hereinafter, referred to as “male connector model”) 2a ′ and a model of female connector 2b (hereinafter, referred to as “female connector model”) 2b ′. .
  • the first CG moving image and the second CG moving image show how the female connector model 2b 'on the stage model 31b' moves to a desired state. Further, the first CG moving image and the second CG moving image show that the male connector model 2a 'gripped by the hand model 31a' moves downward from above the female connector model 2b 'and connects to the female connector model 2b'. Is shown.
  • the control device 50 acquires real images 90a and 90b including the female connector 2b placed on the stage 31b and the male connector 2a held by the hand 31a from the imaging devices 21 and 22, respectively.
  • the ⁇ control device 50 selects, from the first CG moving image and the second CG moving image, the frames 71a and 71b when the female connector model 2b 'has moved to a desired position and posture, respectively, as target frames.
  • the control device 50 Based on the change information corresponding to the female connector 2b, the control device 50 brings the state of the female connector 2b on the real images 90a and 90b closer to the state of the female connector model 2b 'on the frames 71a and 71b, respectively. Is calculated. Then, the control device 50 outputs a control command indicating the calculated control amount to the robot controller 40b.
  • the robot controller 40b controls the robot 30b according to a control command. Thereby, as shown in the real images 91a and 91b, the position and the posture of the female connector 2b change to a desired position and posture (the position and the posture of the female connector model 2b 'shown by the frames 71a and 71b).
  • the control device 50 controls the hand 31a to bring the state of the male connector 2a on the real images 90a and 90b closer to the state of the male connector model 2a 'on the frames 71a and 71b. Calculate the amount. Then, the control device 50 outputs a control command indicating the calculated control amount to the robot controller 40a.
  • the robot controller 40a controls the robot 30a according to a control command. Thereby, as shown in the actual images 91a and 91b, the state of the male connector 2a changes to the position and posture above the female connector 2b (the positions and postures indicated by the frames 71a and 71b).
  • control device 50 selects the frames 72a and 72b when the male connector model 2a 'has moved to a position immediately above the female connector model 2b' as target frames.
  • the control device 50 uses the hand 31a for bringing the state of the male connector 2a on the real images 91a and 91b closer to the state of the male connector model 2a 'on the frames 72a and 72b based on the change information corresponding to the male connector 2a. Is calculated. Then, the control device 50 outputs a control command indicating the calculated control amount to the robot controller 40a.
  • the robot controller 40a controls the robot 30a according to a control command.
  • the position and the posture of the male connector 2a are the position and the posture immediately above the female connector 2b (the position and the posture of the male connector model 2a 'indicated by the frames 72a and 72b). Changes to
  • control device 50 selects the frames 73a and 73b when the connection between the male connector model 2a 'and the female connector model 2b' is completed as target frames.
  • the control device 50 controls the hand 31a for bringing the state of the male connector 2a on the real images 92a and 92b closer to the state of the male connector model 2a 'on the frames 73a and 73b based on the change information corresponding to the male connector 2a. Calculate the control amount. Then, the control device 50 outputs a control command indicating the calculated control amount to the robot controller 40a. The robot controller 40a controls the robot 30a according to a control command. As a result, as shown in the real images 93a and 93b, the male connector 2a moves to the position and posture at which the connection to the female connector 2b is completed (the position and posture of the male connector model 2a 'indicated by the frames 73a and 73b). .
  • control device 50 can change the states of the male connector 2a and the female connector 2b on the actual image to the states of the male connector model 2a 'and the female connector model 2b' on the target frame, respectively.
  • the states of the male connector 2a and the female connector 2b change in a coordinated manner according to the first CG moving image and the second CG moving image.
  • the control device 50 can control the robots 30a and 30b without using calibration data for associating the coordinate systems of the imaging devices 21 and 22 with the robots 30a and 30b. Therefore, the operator does not need to perform the calibration in advance. Therefore, it is possible to reduce the labor required for changing the states of the male connector 2a and the female connector 2b to desired states by the robots 30a and 30b.
  • the positions and postures of the male connector 2a and the female connector 2b in the real space are specified from the images taken by the imaging devices 21 and 22, and the robots 30a and 30b are controlled based on the positions and postures. There is a way to do it.
  • the accuracy of the calibration data is reduced in accordance with the aging of the robots 30a and 30b, and the male connector 2a and the female connector 2b cannot be connected well.
  • the male connector 2a and the female connector 2b cannot be connected properly due to a positional shift or individual difference between the male connector 2a and the female connector 2b. Even in such a case, by using this application example, the male connector 2a and the female connector 2b can be connected as in the CG moving image.
  • FIG. 4 is a schematic diagram illustrating a hardware configuration of a control device included in the control system according to the embodiment.
  • the control device 50 has a structure according to a computer architecture, and executes various programs as described below by executing a program installed in advance by a processor.
  • control device 50 includes a processor 510 such as a CPU (Central Processing Unit) or an MPU (Micro-Processing Unit), a RAM (Random Access Memory) 512, a display controller 514, a system controller 516, It includes an I / O (Input @ Output) controller 518, a hard disk 520, a camera interface 522, an input interface 524, a robot controller interface 526, a communication interface 528, and a memory card interface 530. These units are connected to each other so as to enable data communication with the system controller 516 as the center.
  • a processor 510 such as a CPU (Central Processing Unit) or an MPU (Micro-Processing Unit), a RAM (Random Access Memory) 512, a display controller 514, a system controller 516, It includes an I / O (Input @ Output) controller 518, a hard disk 520, a camera interface 522, an input interface 524, a robot controller interface 526, a communication
  • the processor 510 exchanges programs (codes) and the like with the system controller 516 and executes them in a predetermined order, thereby realizing the intended arithmetic processing.
  • the system controller 516 is connected to the processor 510, the RAM 512, the display controller 514, and the I / O controller 518 via buses, respectively, exchanges data with each unit, and performs processing of the entire control device 50. Govern.
  • the RAM 512 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory), and includes programs read from the hard disk 520, images (image data) acquired by the imaging devices 21 and 22, The processing result for the image, work data, and the like are stored.
  • DRAM Dynamic Random Access Memory
  • the display controller 514 is connected to the display unit 532, and outputs a signal for displaying various information to the display unit 532 according to an internal command from the system controller 516.
  • the I / O controller 518 controls data exchange between a recording medium connected to the control device 50 and an external device. More specifically, the I / O controller 518 is connected to the hard disk 520, the camera interface 522, the input interface 524, the robot controller interface 526, the communication interface 528, and the memory card interface 530.
  • the hard disk 520 is typically a nonvolatile magnetic storage device, and stores various information in addition to the control program 550 executed by the processor 510.
  • the control program 550 installed on the hard disk 520 is distributed while being stored in a memory card 536 or the like.
  • a semiconductor storage device such as a flash memory or an optical storage device such as a DVD-RAM (Digital Versatile Disk Random Access Memory) may be employed.
  • the camera interface 522 corresponds to an input unit that receives image data from the imaging devices 21 and 22, and mediates data transmission between the processor 510 and the imaging devices 21 and 22.
  • the camera interface 522 includes image buffers 522a and 522b for temporarily storing image data from the imaging devices 21 and 22, respectively.
  • a single image buffer that can be shared may be provided for a plurality of imaging devices, it is preferable to independently arrange a plurality of image buffers in association with each imaging device in order to speed up processing.
  • the input interface 524 mediates data transmission between the processor 510 and an input device 534 such as a keyboard, a mouse, a touch panel, and a dedicated console.
  • the robot controller interface 526 mediates data transmission between the processor 510 and the robot controllers 40a and 40b.
  • the communication interface 528 mediates data transmission between the processor 510 and another personal computer or server (not shown).
  • the communication interface 528 is typically made of Ethernet (registered trademark), USB (Universal Serial Bus), or the like.
  • the memory card interface 530 mediates data transmission between the processor 510 and the memory card 536 as a recording medium.
  • the memory card 536 circulates in a state where a control program 550 executed by the control device 50 and the like are stored, and the memory card interface 530 reads the control program 550 from the memory card 536.
  • the memory card 536 includes a general-purpose semiconductor storage device such as an SD (Secure Digital), a magnetic recording medium such as a flexible disk (Flexible Disk), and an optical recording medium such as a CD-ROM (Compact Disk-Read Only Memory). Consists of Alternatively, a program downloaded from a distribution server or the like may be installed in the control device 50 via the communication interface 528.
  • an OS for providing basic functions of the computer in addition to an application for providing functions according to the present embodiment may be installed.
  • the control program according to the present embodiment executes processing by calling necessary modules in a predetermined order and / or timing among program modules provided as a part of the OS. Is also good.
  • control program according to the present embodiment may be provided by being incorporated in a part of another program. Even in such a case, the program itself does not include a module included in another program to be combined as described above, and the process is executed in cooperation with the other program. That is, the control program according to the present embodiment may be a form incorporated in such another program.
  • part or all of the functions provided by executing the control program may be implemented as a dedicated hardware circuit.
  • FIG. 5 is a block diagram illustrating a functional configuration of the control device according to the embodiment.
  • the control device 50 includes a moving image storage unit 51, a design data storage unit 52, a CG moving image generation unit 53, a teaching range selection unit 54, an image processing unit 55, and a target frame selection unit. 56, a first control unit 57a, and a second control unit 57b.
  • the moving image storage unit 51 and the design data storage unit 52 include a hard disk 520 and a RAM 512 shown in FIG.
  • the CG moving image generation unit 53, the teaching range selection unit 54, and the image processing unit 55 are realized by the processor 510 illustrated in FIG.
  • the moving image storage unit 51 stores a first CG moving image and a second CG moving image.
  • the first CG moving image and the second CG moving image are generated by the CG moving image generating unit 53.
  • the design data storage unit 52 stores design data of the male connector 2a, the female connector 2b, the robots 30a and 30b, and the imaging devices 21 and 22.
  • the design data is, for example, 3D-CAD (Computer-Aided Design) data.
  • the design data storage unit 52 also stores arrangement data indicating installation positions of the robots 30a and 30b and the imaging devices 21 and 22 in the real space.
  • the design data storage unit 52 also stores the directions and the angles of view of the imaging devices 21 and 22.
  • the CG moving image generating unit 53 generates a first CG moving image and a second CG moving image.
  • the CG moving image generation unit 53 may generate the first CG moving image and the second CG moving image using a known CG technique.
  • the CG moving image generating unit 53 constructs a virtual space in which the models of the robots 30a and 30b and the imaging devices 21 and 22 are arranged according to the arrangement data stored in the design data storage unit 52.
  • a hand model 31a 'and a stage model 31b' are arranged in the virtual space (see FIGS. 2 and 3).
  • the virtual space is a space simulating the real space, and is represented by the same coordinate system as the real space.
  • the CG moving image generating unit 53 causes the display unit 532 to display the constructed virtual space.
  • the CG moving image generation unit 53 arranges the female connector model 2b ’in the initial posture at the initial position on the stage model 31b’ in the virtual space.
  • the female connector model 2b ' is generated based on 3D-CAD data of the female connector 2b.
  • the initial position and the initial posture are specified by the operator using the input device 534.
  • the operator sets the average position and posture of the female connector 2b as the initial position and initial posture when the female connector 2b conveyed from the upstream device is placed on the stage 31b in the real space. You can specify each one.
  • the CG moving image generation unit 53 arranges the male connector model 2a ′ held by the hand model 31a ′ at the initial position and the initial posture in the virtual space.
  • the male connector model 2a ' is generated by 3D-CAD data of the male connector 2a.
  • the initial position and the initial posture are specified by the operator using the input device 534.
  • the operator specifies the average position and posture of the male connector 2a as the initial position and the initial posture when the hand 31a grips the male connector 2a conveyed from the upstream device in the real space. I just need.
  • the CG moving image generation unit 53 generates a CG moving image having a viewpoint of the model of the imaging device 21 as a first CG moving image when each model is moved in the virtual space according to the moving program. Similarly, the CG moving image generating unit 53 generates, as a second CG moving image, a CG moving image having the viewpoint of the model of the imaging device 22 when each model is moved in the virtual space according to the moving program.
  • the CG moving image is generated based on the orientation and the angle of view of the imaging devices 21 and 22 stored in the design data storage unit 52.
  • the moving program moves the female connector model 2b ′ from the initial position and initial posture to a desired position and posture in the virtual space, and then moves the male connector model 2a ′ from above toward the female connector model 2b ′.
  • This is a program for connecting to the female connector model 2b '.
  • the moving program is created according to the moving paths of the female connector model 2b 'and the male connector model 2a' specified by the operator.
  • the teaching range selection unit 54 selects, for each object (here, the male connector 2a and the female connector 2b), a teaching range serving as a sample of the object from the first CG moving image and the second CG moving image.
  • the teaching range selection unit 54 displays a screen for prompting the user to select a teaching range on the display unit 532.
  • the operator confirms each frame of the first CG moving image and the second CG moving image, and operates the input device 534 to determine the first frame and the last frame of a series of frames in which the model of the target object performs a desired operation. specify.
  • the teaching range selection unit 54 selects the designated first frame to the last frame as the teaching range.
  • the teaching range selection unit 54 sets the female connector 2b from the first frame 70a to a frame after the s-th frame (a frame where a part of the female connector 2b starts to be cut off). Is selected as the teaching range.
  • the teaching range selection unit 54 selects a range from the first frame 70a (a frame where the entire male connector 2a starts to appear) to the u-th frame 73a as the teaching range of the male connector 2a.
  • the teaching range selection unit 54 sets the female connector from the first frame 70b to the frame after the s-th frame (the frame where a part of the female connector 2b starts to be cut off). 2b is selected as the teaching range.
  • the teaching range selection unit 54 selects from the first frame 70b (the frame where the entire male connector 2a starts to appear) to the u-th frame 73b as the teaching range of the male connector 2a.
  • the image processing unit 55 performs image processing on the target image, and detects the target from the target image using template matching.
  • template matching a template which is data representing the image feature of the target object is prepared in advance, and the degree of matching of the image feature between the target image and the template is evaluated. This is a process of detecting the position and the posture of the camera.
  • the target images on which the image processing unit 55 performs the image processing are the frames of the first CG moving image, the frames of the second CG moving image, and the actual images captured by the imaging devices 21 and 22.
  • the image processing unit 55 creates a template for each object (the male connector 2a and the female connector 2b) as advance preparation.
  • FIG. 6 is a diagram showing an example of a method for creating a template.
  • the image processing unit 55 generates a 2D model image of the female connector 2b when viewed from various directions based on the design data (3D-CAD data) of the female connector 2b.
  • the image processing unit 55 extracts a plurality of feature points of the female connector 2b and their feature amounts from each 2D model image of the female connector 2b.
  • the image processing unit 55 creates the coordinates of each of the plurality of feature points on the image and the feature amounts as a template of the female connector 2b.
  • the image processing unit 55 generates a 2D model image of the male connector 2a when viewed from various directions, based on the design data (3D-CAD data) of the male connector 2a.
  • the image processing unit 55 extracts a plurality of feature points of the male connector 2a and their feature amounts from each 2D model image of the male connector 2a.
  • the image processing unit 55 creates the coordinates of each of the plurality of feature points on the image and the feature amounts as a template of the male connector 2a.
  • a feature point is a point characterized by a corner or an outline included in an image, and is, for example, an edge point.
  • the feature quantity is, for example, luminance, luminance gradient direction, quantization gradient direction, HoG (Histogram of Oriented Gradients), HAAR-like, SIFT (Scale-Invariant Feature Transform), and the like.
  • the luminance gradient direction represents a direction (angle) of a luminance gradient in a local region centered on a feature point as a continuous value
  • the quantization gradient direction is a local region centered on a feature point.
  • the direction of the luminance gradient is represented by a discrete value (for example, eight directions are held by 1-byte information of 0 to 7).
  • the image processing unit 55 extracts a plurality of feature points and their feature amounts from the target image (the frame of the first CG moving image, the frame of the second CG moving image, and the real images captured by the imaging devices 21 and 22).
  • the image processing unit 55 detects the target in the target image by comparing the extracted feature points and feature amounts with the template of the target.
  • the image processing unit 55 outputs, for each target object (male connector 2a and female connector 2b), the coordinates of each feature point of the target object extracted from the target image on the image.
  • Target frame selection section The target frame selection unit 56 selects a target frame from the first CG moving image and the second CG moving image. However, when the k-th frame of the first CG moving image is selected as the target frame, the target frame selecting unit 56 selects the k-th frame of the second CG moving image as the target frame. A specific example of the target frame selection method will be described later.
  • the first control unit 57a controls the robot 30a via the robot controller 40a and changes the state of the male connector 2a.
  • the second controller 57b controls the robot 30b via the robot controller 40b to change the state of the female connector 2b.
  • FIG. 7 is a block diagram showing a functional configuration of the first control unit and the second control unit in the embodiment.
  • each of the first control unit 57a and the second control unit 57b includes a change information generation unit 58, a change information storage unit 59, a calculation unit 60, a command unit 61, and an end determination unit. 62.
  • the change information storage unit 59 includes the hard disk 520 and the RAM 512 shown in FIG.
  • the change information generation unit 58, the calculation unit 60, the command unit 61, and the end determination unit 62 are realized by the processor 510 illustrated in FIG.
  • the change information generating unit 58 is configured to generate, for each of the plurality of degrees of freedom, a first control indicating a relationship between a unit control amount of the target robot and a change amount of the state of the target object on the real image captured by the imaging device 21. Generate change information.
  • the change information generation unit 58 stores a first change information set 591 including a plurality of first change information generated for a plurality of degrees of freedom in the change information storage unit 59.
  • the change information generating unit 58 indicates, for each of the plurality of degrees of freedom, the relationship between the unit control amount of the target robot and the change amount of the state of the target on the real image captured by the imaging device 22.
  • the second change information is generated.
  • the change information generation unit 58 stores, in the change information storage unit 59, a second change information set 592 including a plurality of pieces of second change information generated for a plurality of degrees of freedom.
  • the target object is the male connector 2a in the first control unit 57a and the female connector 2b in the second control unit 57b.
  • the target robot is the robot 30a in the first control unit 57a and the robot 30b in the second control unit 57b.
  • the plurality of degrees of freedom are six degrees of freedom in the first control unit 57a and three degrees of freedom in the second control unit 57b.
  • the first change information and the second change information indicate the amount of change in the state of the target on the image when the target robot is controlled by the unit control amount.
  • the first change information and the second change information include an object on the image before controlling the target robot by the unit control amount, and an object on the image after controlling the target robot by the unit control amount.
  • a mapping that converts to
  • the change information generating unit 58 generates the first change information set 591 for each frame in the teaching range of the first CG moving image. Further, the change information generating unit 58 generates a second change information set 592 for each frame in the teaching range of the second CG moving image.
  • the process of generating and storing the first change information set 591 and the second change information set 592 by the change information generating unit 58 is executed as preparation.
  • a method of generating the first change information set 591 in the first control unit 57a will be described with reference to FIG. Note that the method of generating the second change information set 592 in the first control unit 57a and the method of generating the first change information set 591 and the second change information set 592 in the second control unit 57b are the same. The description of the generation method is omitted.
  • FIG. 8 is a diagram illustrating a method of generating the first change information set 591 in the first control unit.
  • FIG. 8A shows the k-th frame 84 of the first CG moving image.
  • the state (position and posture) in the virtual space of the male connector model 2a 'corresponding to the k-th frame 84 is set as a reference state.
  • FIG. 8B shows a CG image 85a with the model of the imaging device 21 as a viewpoint when the male connector model 2a ′ is translated from the reference state in the virtual space by a unit control amount to the degree of translational freedom in the Y direction.
  • FIG. 8C shows a CG image 85b with the model of the imaging device 21 as a viewpoint when the male connector model 2a 'is translated from the reference state in the virtual space by a unit control amount to the degree of translational freedom in the X direction.
  • FIG. 8D shows a CG image 85c with the model of the imaging device 21 as a viewpoint when the male connector model 2a ′ is translated from the reference state in the virtual space by a unit control amount to the translational freedom in the Z direction.
  • FIG. 8E shows a CG image 85d with the model of the imaging device 21 as a viewpoint when the male connector model 2a 'is rotated from the reference state in the virtual space by a unit control amount to the rotational degree of freedom in the pitch direction.
  • FIG. 8F shows a CG image 85e with the model of the imaging device 21 as a viewpoint when the male connector model 2a 'is rotated from the reference state in the virtual space by a unit control amount to the rotational degree of freedom in the yaw direction. Is shown.
  • FIG. 8E shows a CG image 85d with the model of the imaging device 21 as a viewpoint when the male connector model 2a 'is rotated from the reference state in the virtual space by a unit control amount to the rotational degree of freedom in the yaw direction.
  • FIG. 8G shows a CG image 85f with the model of the imaging device 21 as a viewpoint when the male connector model 2a 'is rotated from the reference state in the virtual space by a unit control amount to the rotational degree of freedom in the roll direction. Is shown.
  • the change information generating unit 58 generates CG images 85a to 85f by simulating the position and orientation of the male connector model 2a 'in the virtual space.
  • the change information generating unit 58 acquires, from the image processing unit 55, the coordinates on the image of each feature point of the male connector model 2a 'extracted from each of the frame 84 and the CG images 85a to 85f.
  • the change information generating unit 58 outputs information indicating a mapping for converting the coordinates of the feature points 4a 'to 4g' extracted from the frame 84 into the coordinates of the feature points 5a 'to 5g' extracted from the CG image 85a, respectively. It is generated as first change information corresponding to the degree of freedom of translation in the Y direction.
  • the change information generation unit 58 converts the information indicating the mapping that converts the coordinates of the feature points 4a ′ to 4g ′ into the coordinates of the feature points 5a ′ to 5g ′ extracted from the CG image 85b, using the translation in the X direction. It is generated as first change information corresponding to the degree.
  • the change information generating unit 58 converts information indicating a mapping that transforms the coordinates of the feature points 4a ′ to 4g ′ into the coordinates of the feature points 5a ′ to 5g ′ extracted from the image 85c in accordance with the degree of freedom of translation in the Z direction. Generated as first change information.
  • the change information generating unit 58 converts information indicating a mapping that transforms the coordinates of the feature points 4a ′ to 4g ′ into the coordinates of the feature points 5a ′ to 5g ′ extracted from the image 85d according to the degree of rotation in the pitch direction. Generated as first change information.
  • the change information generation unit 58 converts information indicating a mapping for converting the coordinates of the feature points 4a 'to 4g' into the coordinates of the feature points 5a 'to 5g' extracted from the image 85e, in accordance with the rotational degree of freedom in the yaw direction. Generated as first change information.
  • the change information generation unit 58 converts information indicating a mapping that transforms the coordinates of the feature points 4a ′ to 4g ′ into the coordinates of the feature points 5a ′ to 5g ′ extracted from the image 85f, according to the rotational degree of freedom in the roll direction. Generated as first change information. In this way, the change information generating unit 58 generates the first change information set 591 corresponding to the k-th frame 84 of the first CG moving image.
  • the change information generation unit 58 generates a first change information set 591 corresponding to the remaining frames in the teaching range of the first CG moving image by the same method.
  • the calculation unit 60 has a plurality of degrees of freedom for bringing the state of the target on the real image captured by the imaging devices 21 and 22 closer to the state of the model of the target on target frames of the first CG moving image and the second CG moving image, respectively. Is calculated.
  • the calculation unit 60 obtains the first change information set 591 and the second change information set 592 corresponding to the target frame from the change information storage unit 59, and stores the acquired first change information set 591 and second change information set 592 in the set.
  • the control amount is calculated based on the control amount.
  • the first change information and the second change information include an object on the image before controlling the target robot by the unit control amount and an object on the image after controlling the target robot by the unit control amount.
  • the calculation unit 60 calculates the coordinates of the feature points of the object extracted from the real image on the image and the coordinates of the feature points of the model of the object extracted from the target frame from the image processing unit 55. get.
  • the calculation unit 60 calculates a control amount of each of a plurality of degrees of freedom for mapping an object on the real image to a model of the object on the target frame based on the change information.
  • FIG. 9 is a diagram illustrating a method of calculating the control amount by the calculation unit of the first control unit.
  • the calculating unit 60 acquires, from the image processing unit 55, the coordinates on the image of the feature points 4a 'to 4g' of the male connector model 2a 'extracted from the target frame of the first CG moving image. Further, the calculation unit 60 acquires, from the image processing unit 55, the coordinates on the image of the characteristic points 4a to 4g of the male connector 2a extracted from the real image obtained by the imaging of the imaging device 21.
  • the number of feature points is not limited to seven.
  • the calculation unit 60 calculates the difference vectors 6a to 6g of each feature point.
  • the difference vectors 6a to 6g are vectors starting from the feature points 4a to 4g and ending at the feature points 4a 'to 4g', respectively.
  • the calculation unit 60 calculates the average x component ⁇ x1 and y component ⁇ y1 of the difference vectors 6a to 6g.
  • the x component and the y component are indicated by the coordinate system of the image.
  • the calculation unit 60 calculates the average x component of the difference vector between the feature point extracted from the real image obtained by the imaging of the imaging device 22 and the feature point extracted from the target frame of the second reference moving image. Calculate ⁇ x2 and y component ⁇ y2.
  • the calculation unit 60 calculates the control amounts of the three translational degrees of freedom so that the average of the difference vectors of each of the plurality of feature points is eliminated. Specifically, the calculation unit 60 uses the ⁇ x1, ⁇ y1, ⁇ x2, and ⁇ y2, the first change information set 591 and the second change information set 592 to perform translation free movement in the X, Y, and Z directions of the hand 31a. The control amount of each degree is calculated.
  • the male connector 2a When the hand 31a translates in any of the degrees of freedom of translation in the X, Y, and Z directions, the male connector 2a translates in a certain direction on the images obtained by the imaging devices 21 and 22. Therefore, the first change information corresponding to the translation degree of freedom in the first change information set 591 indicates a mapping that converts an arbitrary point on the image into a point translated in a certain direction. Similarly, the second change information corresponding to the degree of freedom of translation in the second change information set 592 indicates a mapping that converts an arbitrary point on the image into a point translated in a certain direction.
  • the first change information corresponding to the translational degree of freedom in the X direction of the first change information set 591 corresponding to the target frame indicates a mapping that converts a point (x, y) into a point (x + dX1_1, y + dY1_1).
  • the first change information corresponding to the degree of freedom of translation in the Y direction indicates a mapping that converts an arbitrary point (x, y) on the image into a point (x + dX1_2, y + dY1_2).
  • the first change information corresponding to the degree of freedom of translation in the Z direction indicates a mapping that converts an arbitrary point (x, y) on the image into a point (x + dX1_3, y + dY1_3).
  • the second change information corresponding to the degree of freedom of translation in the X direction of the second change information set 592 corresponding to the target frame converts an arbitrary point (x, y) on the image into a point (x + dX2_1, y + dY2_1).
  • the second change information corresponding to the degree of freedom of translation in the Y direction indicates a mapping that converts an arbitrary point (x, y) on the image into a point (x + dX2_2, y + dY2_2).
  • the second change information corresponding to the translation degree of freedom in the Z direction indicates a mapping that converts an arbitrary point (x, y) on the image into a point (x + dX2_3, y + dY2_3).
  • the calculation unit 60 calculates coefficients a1, a2, and a3 by solving the following four linear equations (1) to (4).
  • the first change information and the second change information indicate the amount of change in the state of the target on the image when the target robot is controlled by the unit control amount. Therefore, the calculation unit 60 determines that the control amount of the translational freedom in the X direction is a1 times the unit control amount, the control amount of the translational freedom in the Y direction is a2 times the unit control amount, and the control amount of the translational freedom in the Z direction. Is a3 times the unit control amount.
  • the control amounts of these translation degrees of freedom are control amounts for bringing the state of the male connector 2a on the real image closer to the state of the male connector 2a on the target frame by the average of the difference vectors of the plurality of feature points.
  • the calculating unit 60 calculates the control amounts of the three rotational degrees of freedom.
  • the calculation unit 60 subtracts the average x component ( ⁇ x1 or ⁇ x2) and the y component ( ⁇ y1 or ⁇ y2) from the difference vector of each feature point.
  • the calculating unit 60 calculates a control amount of three rotational degrees of freedom in which the residual of the difference vector of each feature point is closest to 0, using a search algorithm such as a hill-climbing method, for example.
  • the calculation unit 60 starts the search algorithm with a solution in which the control amounts of the rotational degrees of freedom in the pitch, yaw, and roll directions are 0 as the current solution.
  • the calculation unit 60 simulates a change in the residual of the difference vector of each feature point when the robot 30a is controlled according to each of a plurality of solutions near the current solution.
  • the calculation unit 60 replaces the neighboring solution with the current solution when there is a neighboring solution whose residual of the difference vector of each feature point is closer to 0 than the current solution based on the simulation result.
  • the calculation unit 60 searches for a solution in which the residual of the difference vector becomes an extreme value by repeating this process.
  • the command unit 61 generates a control command for moving the target robot by the control amount calculated by the calculation unit 60, and outputs the generated control command to the target robot controller.
  • the target robot controller is the robot controller 40a in the first control unit 57a and the robot controller 40b in the second control unit 57b.
  • End determination section> The end determination unit 62 calculates a deviation between the state of the object on the real image and the state of the model of the object on the last frame of the teaching range, and when the calculated deviation is less than a predetermined threshold, It is determined that the control of the target robot ends. When determining that the control of the target robot is to be ended, the end determination unit 62 outputs an end notification.
  • the deviation is, for example, the average of the distance between corresponding feature points extracted from the real image and the last frame.
  • the threshold value is set according to the accuracy required for the state of the object.
  • the threshold value is the threshold value Tha in the first control unit 57a and the threshold value Thb in the second control unit 57b.
  • the threshold value Tha and the threshold value Thb may be the same or different.
  • FIG. 10 is a flowchart illustrating an example of a process flow for controlling the target robot so as to change the state of the target object along the first CG moving image and the second CG moving image.
  • step S1 the control device 50 determines whether or not an end notification has been output from the end determination units 62 of all the control units (the first control unit 57a and the second control unit 57b). If the end notification has been output from the end determination units 62 of all the control units (YES in step S1), the process ends.
  • step S2 the control device 50 acquires the actual images captured by the imaging devices 21 and 22.
  • step S3 the image processing unit 55 detects all objects (male connector 2a and female connector 2b) from the real image by template matching, and extracts the coordinates of the feature points of each object. Further, the image processing unit 55 detects all target models (male connector model 2a 'and female connector model 2b') from the target frame by template matching, and extracts the coordinates of the feature points of each model.
  • step S4 the target frame selection unit 56 selects a target frame from the first CG moving image and the second CG moving image.
  • step S5 the target frame selection unit 56 specifies a target including the target frame within the teaching range and controls the specified target (at least one of the first control unit 57a and the second control unit 57b). And outputs a control instruction to.
  • step S6 the control unit (at least one of the first control unit 57a and the second control unit 57b) that has received the control instruction controls the target robot.
  • the process returns to step S1. If NO in step S1, a series of processing in steps S2 to S6 is repeated for each imaging cycle. If NO in step S1, step S2 for acquiring the next real image may be started while steps S2 to S6 control the target robot in step S6.
  • the target robot is continuously controlled according to the latest actual image without stopping the operation of the target robot. As a result, the state of the object can be changed quickly.
  • FIG. 11 is a flowchart showing the flow of the processing of the subroutine of step S4 shown in FIG.
  • FIG. 12 is a diagram illustrating the relationship between the closest frame and the target frame.
  • step S11 the target frame selection unit 56 acquires from the image processing unit 55 the coordinates of the feature points of the models of all the objects extracted from each frame of the first CG moving image and the second CG moving image.
  • step S12 the target frame selection unit 56 obtains, from the image processing unit 55, the coordinates of the characteristic points of all the objects extracted from the real images captured by the imaging devices 21 and 22.
  • step S13 the target frame selection unit 56 determines whether the first target frame has been selected.
  • step S14 the target frame selecting unit 56 determines that the previous target frame is the last frame in the teaching range corresponding to any one of the objects. It is determined whether or not there is.
  • step S15 the target frame selection unit 56 calculates the deviation between the state of the target on the real image and the state of the model of the target on the final frame for the target corresponding to the teaching range to which the final frame belongs. , The deviation is less than the threshold. The deviation is, for example, an average of distances between corresponding feature points in the real image and the final frame.
  • step S16 the target frame selection unit 56 selects the same frame as the previous target frame as the target frame. After step S16, the process ends.
  • step S15 If the deviation is smaller than the threshold (YES in step S15), it is determined that the state of the target has reached the state of the model of the target on the final frame, and the process proceeds to step S17. If the first target frame selection has not been completed (NO in step S13), and if the previous target frame is not the last frame (NO in step S14), the process proceeds to step S17.
  • step S17 the target frame selecting unit 56 calculates the deviation between the state of all the objects on the real image and the state of the model of all the objects on each frame, and sets the frame with the minimum deviation as the closest frame. Identify.
  • the deviation is, for example, an average of distances between corresponding feature points in the actual image and each frame.
  • the target frame selection unit 56 determines the state of all the objects on the real image captured by the imaging device 21 and the state of the model of all the objects on the k-th frame of the first CG moving image. A first deviation is calculated. Further, the target frame selection unit 56 is configured to determine the second state of the state of all the objects on the real image captured by the imaging device 22 and the state of the model of all the objects on the k-th frame of the second reference moving image. Calculate the deviation. The target frame selection unit 56 calculates the average of the first deviation and the second deviation as the deviation corresponding to the k-th frame.
  • step S18 the target frame selecting unit 56 determines whether or not there is a last frame in any of the teaching ranges up to a predetermined number of frames after the closest frame.
  • step S19 the target frame selecting unit 56 selects the final frame as a target frame. If there are a plurality of final frames up to a predetermined number of frames after the closest frame, the target frame selecting unit 56 selects the last frame having the smallest frame number among the plurality of final frames.
  • step S20 the target frame selecting unit 56 selects a frame that is a predetermined number later than the closest frame as the target frame. After step S20, the target frame selection process ends.
  • FIG. 13 is a flowchart showing the flow of the processing of the subroutine of step S6 shown in FIG.
  • step S21 the end determination unit 62 determines whether the target frame is the last frame in the teaching range.
  • step S22 the end determination unit 62 determines that the deviation between the state of the target on the real image and the state of the model of the target on the final frame is less than the threshold. It is determined whether or not.
  • step S23 the termination determination unit 62 outputs a termination notification. After step S23, the process ends.
  • step S24 If the target frame is not the last frame (NO in step S21) and if the deviation is equal to or larger than the threshold (NO in step S22), the process proceeds to step S24.
  • step S24 the calculation unit 60 converts the state of the target on the real image into the state of the model of the target on the target frame based on the first change information set 591 and the second change information set 592 corresponding to the target frame. Is calculated for each of the plurality of degrees of freedom to approach.
  • step S25 the command unit 61 generates a control command indicating the calculated control amount, and outputs the control command to the target robot controller. After step S25, the process ends.
  • the first control unit 57a and the second control unit 57b each control the target robot according to the flow shown in FIG.
  • the robot 30a is controlled such that the state of the male connector 2a on the actual image approaches the state of the male connector model 2a 'on the target frame.
  • the robot 30b is controlled such that the state of the female connector 2b on the actual image approaches the state of the female connector model 2b 'on the target frame.
  • the states of the male connector 2a and the female connector 2b change in a coordinated manner according to the target frame.
  • the control system 1 includes the robots 30a and 30b, the imaging devices 21 and 22, and the control device 50 that controls the robots 30a and 30b.
  • the robots 30a and 30b support the male connector 2a and the female connector 2b, and change the states of the male connector 2a and the female connector 2b, respectively.
  • the imaging devices 21 and 21 are installed at fixed positions different from the robots 30a and 30b, and image the male connector 2a and the female connector 2b supported by the robots 30a and 30b, respectively.
  • the control device 50 stores a first CG moving image and a second CG moving image indicating models of the male connector 2a and the female connector 2b.
  • the control device 50 acquires change information for each of the male connector 2a and the female connector 2b.
  • the change information corresponding to the male connector 2a indicates the relationship between the control amount of the robot 30a and the change amount of the state of the male connector 2a on the images of the imaging devices 21 and 22.
  • the change information corresponding to the female connector 2b indicates the relationship between the control amount of the robot 30b and the change amount of the state of the female connector 2b on the images of the imaging devices 21 and 22.
  • the control device 50 is configured to perform a first process of acquiring a real image captured by the imaging devices 21 and 22, a second process of selecting a target frame from the first CG video and the second CG video, and a process based on the real image and the target frame. And a third process for controlling each of the robots 30a and 30b.
  • the control device 50 calculates a control amount of the robot 30a for bringing the state of the male connector 2a on the real image closer to the state of the male connector model 2a 'on the target frame, based on the change information corresponding to the male connector 2a.
  • the robot 30a is controlled according to the calculated control amount.
  • the control device 50 calculates a control amount of the robot 30b for bringing the state of the female connector 2b on the actual image closer to the state of the female connector model 2b 'on the target frame based on the change information corresponding to the female connector 2b.
  • the robot 30b is controlled according to the calculated control amount.
  • the states of the male connector 2a and the female connector 2b on the actual image can be changed to the states of the male connector model 2a 'and the female connector model 2b' on the target frame. That is, the states of the male connector 2a and the female connector 2b change in conjunction with each other according to the target frame.
  • the control device 50 repeatedly executes a series of processes including the first to third processes, and starts the first process of the next series of processes while performing the third process.
  • the robots 30a and 30b are continuously controlled according to the latest actual image without stopping the operations of the robots 30a and 30b.
  • the states of the male connector 2a and the female connector 2b can be changed quickly.
  • FIG. 14 is a diagram showing the design data of the female connector and the dimensions of the female connector manufactured according to the design data. In general, depending on manufacturing conditions, dimensions of a manufactured product may deviate from design data, and a manufacturing error may occur. In the example shown in FIG. 14, a manufacturing error (systematic error) in which the position of the insertion hole 7 tends to shift in a direction away from the notch 8 is shown.
  • systematic error systematic error
  • the robot 30b is controlled such that the state of the female connector 2b on the actual image approaches the state of the female connector model 2b 'on the target frame.
  • the target frame is selected from the first GC moving image and the second CG moving image generated based on the design data. Therefore, when a manufacturing error as shown in FIG. 14 occurs, when the states of the male connector 2a and the female connector 2b are changed according to the first CG moving image and the second CG moving image, the pins (not shown) of the male connector 2a are changed. There is a possibility that the connector cannot be inserted into the insertion hole 7 of the female connector 2b.
  • the control amount calculated by the calculation unit 60 is adjusted by an offset amount according to a manufacturing error.
  • the command unit 61 generates a control command for moving the target robot by the adjusted control amount, and outputs the generated control command to the target robot controller.
  • FIG. 15 is a schematic diagram showing an outline of the control system according to the first modification.
  • the control system 1A according to the modification 1 further includes a force sensor 32 and a control device 50A instead of the control device 50, as compared with the control system 1 shown in FIG. Differs in that
  • the force sensor 32 detects a force that one of the male connector 2a and the female connector 2b receives from the other.
  • the force sensor 32 is attached between the arm of the robot 30a and the hand 31a so as to detect the force that the male connector 2a receives from the female connector 2b.
  • the control device 50A has a hardware configuration as shown in FIG. Therefore, detailed description of the hardware configuration of the control device 50A is omitted.
  • FIG. 16 is a block diagram showing a functional configuration of the control device according to the first modification.
  • the control device 50A is different from the control device 50 shown in FIG. 5 in that a first control unit 157a and a second control unit 157b are used instead of the first control unit 57a and the second control unit 57b.
  • an offset amount determination unit 63 is further provided.
  • the offset amount determination unit 63 determines an offset amount (also referred to as an “adjustment amount”) for adjusting the control amount calculated by the calculation unit 60 of at least one of the first control unit 157a and the second control unit 157b. In the first modification, the offset amount determination unit 63 determines an offset amount for adjusting the control amount calculated by the calculation unit 60 of the second control unit 57b, and outputs the determined offset amount to the second control unit 157b. I do.
  • the offset amount determination unit 63 searches for the state of the female connector 2b at which the force detected by the force sensor 32 is minimal, and determines the offset amount based on the searched state of the female connector 2b. When the force detected by the force sensor 32 is minimal, the resistance when the male connector 2a and the female connector 2b are joined is minimal. At this time, the relative displacement between the pins of the male connector 2a and the insertion holes 7 of the female connector 2b is also minimized.
  • the offset amount determination unit 63 determines each offset amount of the three degrees of freedom in the robot 30b to be controlled by the second control unit 57b. Specifically, the offset amount determination unit 63 includes an offset amount ⁇ X of the translational degree of freedom of the stage 31b in the X direction, an offset amount ⁇ Y of the degree of translational freedom of the stage 31b in the Y direction, and an offset amount of the rotational degree of freedom of the stage 31b. Determine the quantity ⁇ .
  • FIG. 17 is a block diagram showing a functional configuration of the first control unit and the second control unit in the first modification. As shown in FIG. 17, each of first control unit 157a and second control unit 157b is different from first control unit 57a and second control unit 57b shown in FIG. The difference is that a unit 161 is provided.
  • the command unit 161 When the command unit 161 receives the offset amount from the offset amount determination unit 63, the command unit 161 adjusts the control amount by adding the offset amount to the control amount calculated by the calculation unit 60. The command unit 161 generates a control command for moving the robot 30b by the adjusted control amount, and outputs the generated control command to the robot controller 40b. When the command unit 161 does not receive the offset amount from the offset amount determination unit 63, similarly to the command unit 61, the command unit 161 generates a control command for moving the robot 30b by the control amount calculated by the calculation unit 60. Then, the generated control command is output to the robot controller 40b.
  • FIG. 18 is a flowchart illustrating an example of the flow of an offset amount determination process in the first modification.
  • the offset amount determination process shown in FIG. 18 is performed at a timing at which a manufacturing error of at least one of the male connector 2a and the female connector 2b tends to fluctuate (for example, a timing at which at least one lot of the male connector 2a and the female connector 2b is changed). It is performed every time.
  • control device 50A repeatedly performs a series of steps S1 to S6 shown in FIG. 10 until “YES” in step S1.
  • the offset amount determining unit 63 acquires the detection value of the force sensor 32 during the processing in step S32.
  • the detection value is, for example, an average value or a maximum value of the force detected by the force sensor 32 during the process of step S32.
  • step S34 the offset amount determining unit 63 selects one of the neighboring solutions of the current solution, and outputs the selected neighboring solution to the second control unit 157b.
  • ( ⁇ X, ⁇ Y, ⁇ ) (X0, Y0, ⁇ 0) is the current solution
  • a1 indicates a unit control amount of the degree of freedom of translation of the stage 31b.
  • a2 indicates a unit control amount of the degree of freedom of rotation of the stage 31b.
  • step S35 control device 50A repeatedly performs a series of steps S1 to S6 shown in FIG. 10 until the result of step S1 becomes YES.
  • the robot is controlled according to the neighborhood solution selected in step S34. That is, the command unit 161 of the second control unit 157b adjusts the control amount by adding the offset amount indicated by the neighborhood solution to the control amount calculated by the calculation unit 60.
  • the command unit 161 of the first control unit 157a does not adjust the control amount calculated by the calculation unit 60 because it has not received the offset amount.
  • step S36 the offset amount determination unit 63 acquires the detection value of the force sensor 32 during the processing in step S35.
  • step S37 the offset amount determination unit 63 determines whether there is an unselected neighborhood solution. If there is an unselected neighborhood solution (YES in step S37), the offset amount determination process returns to step S34. As a result, the detection value of the force sensor 32 can be obtained even for an unselected neighborhood solution.
  • step S38 the offset amount determination unit 63 determines whether the detection value corresponding to the current solution is minimal. Specifically, when the detected value corresponding to the current solution is smaller than all the detected values (detected values corresponding to all of the neighboring solutions) acquired in step S36, the offset amount determination unit 63 sets the current It is determined that the detection value corresponding to the solution is minimal.
  • step S39 If the detected value corresponding to the current solution is not the minimum (NO in step S38), in step S39, the offset amount determination unit 63 sets the nearby solution having the smallest detected value as the current solution. After step S39, the process of determining the offset amount returns to step S34.
  • step S40 the offset amount determination unit 63 maintains the current solution of the offset amount and sets the current solution to the second control unit. 157b. After step S40, the offset amount determination processing ends.
  • the offset amount determination unit 63 may execute steps S31 to S40 for each of a plurality of different pairs of the male connector 2a and the female connector 2b to obtain an offset amount, and calculate an average value thereof. .
  • the force sensor 32 Is minimum.
  • the relative position between the pin of the male connector 2a and the insertion hole 7 of the female connector 2b may be shifted according to the manufacturing error.
  • an unintended resistance force including frictional force
  • the detection value of the force sensor 32 increases.
  • the offset amount maintained in step S40 is the amount of change from the state of the female connector model 2b 'to the state of the female connector 2b at which the detection value of the force sensor 32 is minimal.
  • the change amount corresponds to a relative displacement amount between the pin of the male connector 2a and the insertion hole 7 of the female connector 2b.
  • the displacement is an amount corresponding to a manufacturing error of at least one of the male connector 2a and the female connector 2b. That is, the offset amount at which the detection value of the force sensor 32 becomes a minimum is an amount corresponding to a manufacturing error of at least one of the male connector 2a and the female connector 2b, and the pin of the male connector 2a is inserted into the insertion hole 7 of the female connector 2b. Shows the relative positional shift amount.
  • the pin of the male connector 2a can be inserted into the insertion hole 7 of the female connector 2b.
  • the offset amount determination unit 63 determines the offset amount for adjusting the control amount calculated by the calculation unit 60 of the second control unit 57b. An offset amount for adjusting the control amount calculated by the above may be determined.
  • the offset amount determination unit 63 may acquire the offset amount from the input device 534 (see FIG. 4). The operator may measure a systematic error with respect to the design data of the male connector 2a and the female connector 2b, and input an offset amount corresponding to the systematic error to the input device 534.
  • the operator adjusts the control amount calculated by the calculation unit 60 of the first control unit 57a and the offset amount for adjusting the control amount calculated by the calculation unit 60 of the second control unit 57b. At least one of the offset amounts is input to the input device 534.
  • the operator inputs an offset amount for adjusting the control amount calculated by the calculating unit 60 of the first control unit 57a based on the systematic error of the male connector 2a. Further, the operator inputs an offset amount for adjusting the control amount calculated by the calculator 60 of the second controller 57b based on the systematic error of the female connector 2b.
  • the operator obtains the relative displacement between the pin of the male connector 2a and the insertion hole 7 of the female connector 2b from the systematic error of the male connector 2a and the female connector 2b.
  • the operator may input an offset amount for adjusting the control amount calculated by the calculation unit 60 of the first control unit 57a or the second control unit 57b according to the position shift amount.
  • control system according to the third modification differs from the control system 1A according to the first modification in that a control device 50B having a functional configuration as shown in FIG. 19 is provided instead of the control device 50A.
  • the control device 50B according to the third modification has a hardware configuration as shown in FIG. Therefore, a detailed description of the hardware configuration of the control device 50B will be omitted.
  • FIG. 19 is a block diagram showing a functional configuration of a control device according to the third modification.
  • the control device 50B is different from the control device 50A shown in FIG. 16 in that the teaching range selecting unit 54, the image processing unit 55, the target frame selecting unit 56, the first control unit 157a, and the second The difference is that a teaching range selection unit 254, an image processing unit 255, a target frame selection unit 256, a first control unit 257a, and a second control unit 257b are provided instead of the control unit 157b, and a reference moving image generation unit 64 is further provided.
  • the control device 50B controls the robot 30a, 30b using the first CG moving image and the second CG moving image, and uses the first reference moving image and the second reference moving image instead of the first CG moving image and the second CG moving image. And a second mode for controlling 30a and 30b.
  • control device 50B performs the processing shown in the flowchart of FIG. 18 to obtain the solution of the offset amount at which the detection value of the force sensor 32 is minimized.
  • the reference moving image generation unit 64 stores an image set including a plurality of images acquired for each imaging cycle from the imaging device 21 in each of steps S33 and S35 shown in FIG.
  • the reference moving image generation unit 64 reads an image set corresponding to the solution of the offset amount at which the detection value of the force sensor 32 is minimized, and sets a first reference moving image having each of a plurality of images included in the read image set as a frame.
  • the reference moving image generation unit 64 stores an image set including a plurality of images acquired for each imaging cycle from the imaging device 22 in each of step S33 and step S35.
  • the reference moving image generation unit 64 reads an image set corresponding to the solution of the offset amount at which the detection value of the force sensor 32 is minimal, and sets a second reference moving image having each of a plurality of images included in the read image set as a frame.
  • the reference moving image generation unit 64 stores the generated first reference moving image and the second reference moving image in the moving image storage unit 51.
  • the k-th frame (k is an integer from 1 to Q) of the first reference moving image and the k-th frame of the second reference moving image are images captured by the imaging devices 21 and 22 in the same imaging cycle. is there.
  • the control device 50B controls the robots 30a and 30b according to the second mode.
  • the teaching range selection unit 254 performs the teaching from the first reference moving image and the second reference moving image for each object (the male connector 2a and the female connector 2b) in addition to the processing of the teaching range selection unit 54 illustrated in FIGS. Select a range.
  • the teaching range selection unit 254 may select the teaching range for each object from the first reference moving image and the second reference moving image by the same method as that for selecting the teaching range from the first CG moving image and the second CG moving image.
  • the image processing unit 255 performs image processing on the target image and detects the target from the target image using template matching, similarly to the image processing unit 55 shown in FIGS. 5 and 16.
  • the target image on which the image processing unit 255 performs the image processing includes a frame of the first reference moving image and a frame of the first reference moving image.
  • a frame of the second reference moving image is included.
  • the target frame selection unit 256 selects a target frame from each of the first CG moving image and the second CG moving image, similarly to the target frame selection unit 56 shown in FIGS.
  • the target frame selecting unit 256 selects a target frame from each of the first reference moving image and the second reference moving image stored in the moving image storage unit 51 in the second mode.
  • the target frame selection unit 256 may select a target frame from each of the first reference moving image and the second reference moving image according to the flow illustrated in FIG. However, the target frame selection unit 256 acquires, from the image processing unit 255, the coordinates of the feature points of all the objects extracted from each frame of the first reference moving image and the second reference moving image in step S11 of FIG.
  • FIG. 20 is a block diagram illustrating a functional configuration of the first control unit and the second control unit according to the third modification. As shown in FIG. 20, each of the first control unit 257a and the second control unit 157b is different from the first control unit 157a and the second control unit 157b shown in FIG. The difference is that a change information generating unit 258, a calculating unit 260, and a command unit 261 are provided instead of the unit 60 and the command unit 161.
  • the change information generation unit 258 generates the first change information set 591 for each frame in the teaching range of the first CG moving image, and generates each of the teaching ranges of the second CG moving image in the same manner as the change information generation unit 58 illustrated in FIG. A second change information set 592 is generated for the frame.
  • the change information generation unit 258 generates a third change information set 593 for each frame in the teaching range of the first reference moving image, and generates a fourth change information set for each frame in the teaching range of the second reference moving image. 594 is generated.
  • the change information generating unit 258 acquires from the image processing unit 255 the coordinates of the feature points of the target object (the male connector 2a or the female connector 2b) extracted from each frame of the first reference moving image and the second reference moving image.
  • the change information generation unit 258 acquires from the image processing unit 255 the coordinates of the feature points of the object models (the male connector model 2a 'and the female connector model 2b') extracted from each frame of the first CG moving image and the second CG moving image. I do.
  • the change information generating unit 258 sets corresponding feature points extracted from the ith frame and the jth (j is an integer from 1 to M) frame of the first CG moving image. (For example, average distance).
  • the change information generation unit 258 specifies j at which the deviation is minimum.
  • the change information generation unit 258 sets the third change information set 593 corresponding to the i-th frame of the first reference moving image to the same content as the first change information set 591 corresponding to the specified j-th frame in the first CG moving image.
  • the change information generating unit 258 determines, for the i-th frame of the second reference moving image, the corresponding extracted from the i-th frame and the j-th frame (j is an integer of 1 to M) of the second CG moving image.
  • the deviation between the feature points (for example, the average of the distances) is calculated.
  • the change information generation unit 258 specifies j at which the deviation is minimum.
  • the change information generation unit 258 sets the fourth change information set 594 corresponding to the i-th frame of the second reference moving image to the same content as the second change information set 592 corresponding to the specified j-th frame in the second CG moving image.
  • Calculating section 260 calculates the control amount of each degree of freedom of the target robot by performing the same processing as calculating section 60 shown in FIG.
  • the calculating unit 260 reads the first change information set 591 and the second change information set 592 corresponding to the target frame from the change information storage unit 59, and reads the read first change information set 591 and second change information A control amount is calculated based on the set 592.
  • the calculation unit 260 reads the third change information set 593 and the fourth change information set 594 corresponding to the target frame from the change information storage unit 59 in the second mode, and reads the read third change information set 593 and fourth read information A control amount is calculated based on the set 594.
  • the command unit 261 outputs a control command to the target robot controller by performing the same processing as the command unit 161 shown in FIG. 17 in the first mode. That is, when receiving the offset amount from the offset amount determining unit 63, the command unit 261 adjusts the control amount using the offset amount and generates a control command to move the target robot by the adjusted control amount.
  • the command unit 261 outputs a control command to the target robot controller by performing the same processing as the command unit 61 shown in FIG. 7 in the second mode. That is, the command unit 261 generates a control command to move the target robot by the control amount received from the calculation unit 260.
  • the control device 50B adjusts the control amount calculated by the calculation unit 260 by the offset amount, and searches for the offset amount at which the force detected by the force sensor 32 is minimal. I do.
  • the control device 50B acquires images captured by the imaging devices 21 and 22 as the first and second reference moving images when the control amount is adjusted by the searched offset amount.
  • control device 50B performs processing according to the second mode. That is, control device 50B performs a process of selecting a target frame from the first reference moving image and the second reference moving image instead of the first CG moving image and the second CG moving image.
  • the first reference moving image and the second reference moving image are images captured when the force detected by the force sensor 32 is minimal. Therefore, according to the third modification, the control device 50B performs the processing according to the second mode so that the pins of the male connector 2a are inserted into the insertion holes 7 of the female connector 2b even if a manufacturing error occurs. Can be.
  • control system 1 connects (assembles) the male connector 2a and the female connector 2b.
  • control system 1 may assemble another two objects.
  • FIG. 21 is a schematic diagram showing an object of the control system according to the fourth modification.
  • the control system according to Modification 4 screws a bolt 9a into a bolt hole 13 formed in a plate-like component 9b in an industrial product production line or the like.
  • the bolt 9a is gripped by the hand 31a (see FIG. 1) of the robot 30a.
  • the plate-shaped component 9b is placed on the stage 31b of the robot 30b.
  • the number of robots provided in the control system is not limited to two, and may be three or more.
  • the control system may include three or more robots for changing the states of the three or more objects, respectively. .
  • the CG moving image indicates a change in the state of the model of three or more objects.
  • FIG. 22 is a schematic diagram illustrating an object of the control system according to the fifth modification.
  • the control system according to Modification Example 5 joins cylindrical members 10e and 10f with screws 10c and 10d in an industrial product production line or the like.
  • Screw holes 11a and 11b are formed in the cylindrical member 10e, and screw holes 12a and 12b are formed in the cylindrical member 10f. With the screw hole 11a and the screw hole 12a overlapping and the screw hole 11b and the screw hole 12b overlapping, the screw 10c is inserted into the screw holes 11a and 12a, and the screw 10d is inserted into the screw holes 11b and 12b.
  • the screw 10c is held by the hand of the first robot.
  • the screw 10d is gripped by the hand of the second robot.
  • the cylindrical member 10e is gripped by the hand of the third robot.
  • the cylindrical member 10f is mounted on the stage of the fourth robot.
  • the imaging devices 21 and 22 are arranged at positions where the screw holes 11a and 12a and the screw 10c can be imaged. However, the imaging devices 21 and 22 cannot image the screw holes 11b and 12b and the screw 10d. Therefore, the control system according to Modification 5 further includes imaging devices 23 and 24. The imaging devices 23 and 24 are arranged at positions where the screw holes 11b and 12b and the screw 10d can be imaged.
  • the control device stores four CG moving images corresponding to the imaging devices 21 to 24, respectively.
  • the control device of Modification 5 may include first to fourth control units respectively corresponding to the first to fourth robots. Each of the first to fourth control units has the same configuration as in FIG.
  • the change information generation units 58 of the first to fourth control units only need to generate four change information sets respectively corresponding to the four CG moving images.
  • the calculation unit 60 may calculate the control amount of each degree of freedom of the target robot based on the four change information sets.
  • the states of the four objects (cylindrical members 10e and 10f and screws 10c and 10d) can be changed in a coordinated manner according to the CG moving image.
  • FIG. 23 is a schematic diagram illustrating an outline of a control system according to the sixth modification.
  • one of the plurality of objects is set at a fixed position, and the state of the imaging devices 21 and 22 is changed by the robot.
  • the control system 1C sequentially processes the processing target portions 15, 16 of the large member 14e using the processing tools 14c, 14d in a production line of an industrial product or the like.
  • the large member 14j is, for example, a housing of a large device, an automobile body, or the like.
  • the processing tools 14h and 14i are, for example, a drill, an electric file, or the like.
  • the control system 1C is different from the control system 1 shown in FIG. 1 in that instead of the robots 30a, 30b, the robot controllers 40a, 40b, and the control device 50, the robots 30c, 30d, 30e, the robot controllers 40c, 40d, 40e and The difference is that a control device 50C is provided.
  • the robot 30c is a mechanism for changing the state (here, position and orientation) of the processing tool 14c, and is, for example, a vertical articulated robot.
  • the robot 30c has a hand 31c at its tip for supporting the processing tool 14c, and changes the position and posture of the hand 31c with a plurality of degrees of freedom. Further, the robot 30c includes a pedestal 33c that is movable along the rail 34 in the direction of the arrow AR.
  • the robot 30d is a mechanism for changing the state (here, position and posture) of the processing tool 14d, and is, for example, a vertical articulated robot.
  • the robot 30d has a hand 31d that supports the processing tool 14d at its tip, and changes the position and posture of the hand 31d with a plurality of degrees of freedom. Further, the robot 30d includes a pedestal 33d movable along the rail 34 in the direction of the arrow AR.
  • the robot 30e is a mechanism for changing the states (here, positions and postures) of the imaging devices 21 and 22, and is, for example, a vertical articulated robot.
  • the robot 30e has a hand 31e that supports the imaging devices 21 and 22 at the tip, and changes the position and orientation of the hand 31e with a plurality of degrees of freedom. Further, the robot 30e includes a pedestal 33e movable along the rail 34 in the direction of the arrow AR.
  • the pedestals 33c, 33d, and 33e move along the common rail.
  • a rail is provided for each of the pedestals 33c, 33d, and 33e, and each of the pedestals 33c, 33d, and 33e may move along the corresponding rail.
  • the robot controllers 40c, 40d, and 40e control the operations of the robots 30c, 30d, and 30e, respectively, according to the control commands received from the control device 50C.
  • the robot controllers 40c, 40d, and 40e change the states of the hands 31c, 31d, and 31e, respectively, and move the pedestals 33c, 33d, and 33e, respectively, according to a control command from the control device 50B.
  • the control device 50C has a hardware configuration as shown in FIG. 4, as in the first embodiment. Therefore, a detailed description of the hardware configuration of the control device 50C is omitted.
  • FIG. 24 is a block diagram showing a functional configuration of the control device according to the sixth modification.
  • the control device 50C is different from the control device 50 shown in FIG. 5 in that a first control unit 57c and a second control unit 57d are used instead of the first control unit 57a and the second control unit 57b. And a third control unit 57e.
  • the moving image storage unit 51 stores a first CG moving image and a second CG moving image showing a sample of the processing tools 14c and 14d and the large member 14e.
  • the first CG moving image and the second CG moving image indicate, for example, the following first to third scenes in order.
  • the first scene is a scene in which the models of the processing tools 14c and 14d process the model of the processing target portion 15 when the model of the processing target portion 15 is at a fixed position on the image.
  • the second scene is a scene in which the model of the large member 14e moves on the image, and the model of the processing target portion 16 of the large member 14e moves to a fixed position on the image.
  • the third scene is a scene in which the models of the processing tools 14c and 14d process the model of the processing target portion 16 when the model of the processing target portion 16 is at a fixed position on the image.
  • the teaching range selection unit 54 selects a teaching range for each of the processing tool 14c, the processing tool 14d, and the large member 14e.
  • the image processing unit 55 detects the processing tool 14c, the processing tool 14d, and the large member 14e from the target image. Since the size of the large member 14e is large, the field of view of the imaging devices 21 and 22 includes only a part of the large member 14e. Therefore, the image processing unit 55 detects a pattern formed on the surface of the large member 14e.
  • the first controller 57c controls the robot 30c via the robot controller 40c to change the state of the processing tool 14c.
  • the second control unit 57d controls the robot 30d via the robot controller 40d to change the state of the processing tool 14c.
  • the third control unit 57e controls the robot 30e via the robot controller 40e to change the state of the imaging devices 21 and 22.
  • Each of the first control unit 57c, the second control unit 57d, and the third control unit 57e includes, like the first control unit 57a and the second control unit 57b, a change information generation unit 57, a change information storage unit 59, It includes a calculation unit 60, a command unit 61, and an end determination unit 62 (see FIG. 7).
  • the target object is the processing tool 14c in the first control unit 57c, the processing tool 14d in the second control unit 57d, and the large member 14e in the third control unit 57e.
  • the target robot is the robot 30c in the first control unit 57c, the robot 30d in the second control unit 57d, and the robot 30e in the third control unit 57e.
  • the target robot controller is the robot controller 40e in the first controller 57c, the robot controller 40d in the second controller 57d, and the robot controller 40e in the third controller 57e.
  • the change information generation unit 58 of the first control unit 57c and the second control unit 57d determines the unit control amount of the target robot and the target object on the real image captured by the imaging device 21.
  • the first change information indicating the relationship with the change amount of the state is generated.
  • the change information generation unit 58 of the first control unit 57c and the second control unit 57d determines the unit control amount of the target robot and the actual control
  • the second change information indicating the relationship with the amount of change in the state of the object is generated.
  • the change information generating unit 58 of the first control unit 57c determines whether the first change information and the second change information Generate change information.
  • the change information generation unit 58 of the second control unit 57d generates the first change information and the second change information in the state of the imaging devices 21 and 22 where the processing target portion 16 of the large member 14e is at a fixed position on the image. .
  • the state of the large member 14e is not changed by the robot 30e. However, since the state of the imaging devices 21 and 22 is changed by the robot 30e, the state of the large member 14e on the real images of the imaging devices 21 and 22 is changed. Therefore, the change information generation unit 58 of the third control unit 57e performs the first change indicating the relationship between the unit control amount of the robot 30e and the change amount of the state of the large member 14e on the real image captured by the imaging device 21. Generate information. The change information generation unit 58 of the third control unit 57e performs the second change indicating the relationship between the unit control amount of the robot 30e and the change amount of the state of the large member 14e on the real image captured by the imaging device 22. Generate information.
  • the processing contents of the calculation unit 60, the command unit 61, and the end determination unit 62 in the third control unit 57e are the same as those of the first control unit 57a and the second control unit 57b according to the above embodiment.
  • the calculation unit 60 in the first control unit 57c and the second control unit 57d determines that the state of the target on the real image approaches the state of the target on the target frame only when the following start conditions are satisfied.
  • the control amount of the target robot is calculated. Start condition: The deviation between the state of the large member 14e on the actual image and the state of the large member 14e on the target frame is less than the threshold The.
  • the processing contents of the command unit 61 and the end determination unit 62 in the first control unit 57c and the second control unit 57d are the same as those of the first control unit 57a and the second control unit 57b according to the above-described embodiment.
  • the control device 50C controls the target robot so that the state of the target object changes according to the first CG moving image and the second CG moving image in accordance with the flowchart of FIG. 10 as in the above embodiment.
  • the third control unit 57e performs the process of the subroutine of step S6 shown in FIG. 10 according to the flowchart shown in FIG. 13 similarly to the above embodiment.
  • the first control unit 57c and the second control unit 57d perform the processing of the subroutine of step S6 shown in FIG.
  • FIG. 25 is a flowchart showing the flow of processing of the first control unit and the second control unit of the sixth modification. As shown in FIG. 25, the flow of processing of the first control unit 57c and the second control unit 57d of Modification 6 is different from the flow chart shown in FIG. 13 in that step S90 is provided. Therefore, only step S90 will be described.
  • step S90 it is determined whether the deviation between the state of the large member 14e on the actual image and the state of the large member 14e on the target frame is less than the threshold Thj. If the deviation is equal to or larger than the threshold Thj (NO in step S90), the process ends. If the deviation is less than the threshold Thj (YES in step S90), steps S21 to S25 are performed.
  • control device 50C controls only the robot 30e when the deviation between the state of the large member 14e on the actual image and the state of the large member 14e on the target frame exceeds the threshold Thj.
  • the controller 50C controls each of the robots 30c to 30e when the deviation between the state of the large member 14e on the actual image and the state of the large member 14e on the target frame is less than the threshold Thj.
  • the robots 30c to 30e are controlled as follows.
  • the robots 30c and 30d move until the imaging devices 21 and 22 move so that the processing target portion 15 of the large member 14e is at a fixed position on the image. It will be in a stopped state.
  • the first control unit 57c sets the target so that the state of the target on the real image approaches the state of the target on the target frame. Control the robot.
  • the processing tools 14c and 14d process the processing target portion 15.
  • the third control unit 57e controls the robot 30e such that the state of the large member 14e on the actual image approaches the state of the large member 14e on the target frame.
  • the robot 30e since the state of the large member 14e is constant, the robot 30e hardly operates, and the states of the imaging devices 21 and 22 are substantially constant.
  • the robot 30e When the target frame is selected from the second scene, the robot 30e is controlled so that the state of the large member 14e on the real image changes, and the imaging devices 21 and 22 move.
  • the robots 30c and 30d are in a stopped state until the imaging devices 21 and 22 move so that the processing target portion 16 of the large member 14e is at a fixed position on the image.
  • the first control unit 57c changes the state of the target on the real image to the target frame.
  • the target robot is controlled so as to approach the state of the target object above.
  • the processing tools 14c and 14d process the processing target portion 16.
  • the third control unit 57e controls the robot 30e such that the state of the large member 14e on the actual image approaches the state of the large member 14e on the target frame.
  • the robot 30e since the state of the large member 14e is constant, the robot 30e hardly operates, and the states of the imaging devices 21 and 22 are substantially constant.
  • FIG. 26 is a schematic diagram showing another configuration example of the control system according to the sixth modification.
  • the robots 30c and 30d may be installed on a pedestal 33e included in the robot 30e. In this case, the robots 30c and 30d move integrally with the robot 30e.
  • the calculation unit 60 performs a known model predictive control (“Adachi,“ Basics of Model Predictive Control ”, Journal of the Robotics Society of Japan, July 2014, Vol. 32, No. 6, p. 9-12) (Non-Patent Document By performing 2)), control amounts of a plurality of degrees of freedom may be calculated.
  • the target frame selecting unit 56 selects a plurality of frames included in the predicted horizon period in the teaching range as target frames.
  • the calculation unit 60 performs control during the control horizon period so as to minimize the deviation between the state of the object on the target frame and the state of the object on the images captured by the imaging devices 21 and 22 during the predicted horizon period. Calculate the amount.
  • the male connector 2a and the female connector 2b are connected by moving the male connector 2a toward the female connector 2b.
  • the male connector 2a may be placed on the stage 31b, and the female connector 2b may be gripped by the hand 31a.
  • the male connector 2a and the female connector 2b are connected by moving the female connector 2b toward the male connector 2a.
  • the moving image storage unit 51 of each of the control devices 50, 50A, 50B, and 50C stores the CG moving image.
  • external devices of the control devices 50, 50A, 50B, and 50C may store CG moving images.
  • the moving image storage unit 51 may store, instead of or in addition to the CG moving image, the coordinates and the feature amounts of the feature points of each object extracted from each frame of the CG moving image. Thus, the processing of the image processing units 55 and 255 for each frame of the CG moving image can be omitted.
  • (Configuration 1) First to Nth robots (30a to 30e); Imaging devices (21 to 24) for imaging the first to Nth objects (2a, 2b, 9a, 9b, 10c to 10f, 14c to 14e); A control system (1, 1A, 1C) including a control device (50, 50A, 50B, 50C) for controlling the first to Nth robots (30a to 30e), N is an integer of 2 or more;
  • the i-th robot (30a to 30e) changes the state of the i-th object, i is an integer of 1 to N-1,
  • the N-th robot (30a to 30e) changes one state of the N-th object and the imaging device (21 to 24),
  • the other of the N-th object and the imaging device (21 to 24) is installed at a fixed position,
  • the control device (50, 50A, 50B, 50C) acquires change information for each of the first to Nth objects,
  • the change information corresponding to the j-th object indicates a relationship between a control
  • the control amount of the j-th robot (30a to 30e) for approaching the state of the model of the j-th object on the target frame is calculated, and the j-th robot (30a to 30e) is calculated according to the calculated control amount.
  • the CG moving image is created based on design data of each of the first to Nth objects,
  • the CG moving image is obtained from a frame indicating a state in which the first object and at least one of the second to Nth objects are separated from the first object and the at least one object.
  • the control device (50A) may include, in the third processing, the first robot (30a to 30e) by an adjustment amount corresponding to a manufacturing error of the first object and the at least one object with respect to the design data.
  • the control system (1A) according to Configuration 1, wherein the control amount is adjusted, and the first robot (30a to 30e) is controlled according to the adjusted control amount.
  • the control system (1A) includes a force sensor (32) for detecting a force that one of the first object and the at least one object receives from the other,
  • the control device (50A, 50B) searches for the state of the first object where the force detected by the force sensor is minimal, and based on the searched state of the first object,
  • the control system (1A) according to Configuration 2, wherein the control system determines the adjustment amount.
  • Control system (1A) The control system (1A) according to configuration 2, wherein the control device (50A, 50B) is connected to an input device (534) and acquires the adjustment amount from the input device (534).
  • the CG moving image is obtained from a frame indicating a state in which the first object and at least one of the second to Nth objects are separated from the first object and the at least one object. And up to the frame showing the joined state,
  • the control system (50B) includes a force sensor (32) for detecting a force that one of the first object and the at least one object receives from the other,
  • the control device (50B) Performing a first mode in which the third processing is repeated from the first processing;
  • the control amount of the first robot is adjusted to search for an adjustment amount at which the force detected by the force sensor is minimal,
  • a moving image captured by the imaging device is acquired as a reference moving image,
  • the control method includes: A first step of acquiring change information for each of the first to Nth robots (30a to 30e);
  • the change information corresponding to the j-th robot (30a to 30e) includes a control amount of the j-th robot (30a to 30e), a change amount of
  • N is an integer of 2 or more;
  • the i-th robot (30a to 30e) changes the state of the i-th object, i is an integer of 1 to N-1,
  • the N-th robot (30a to 30e) changes one state of the N-th object and the imaging device,
  • the other of the N-th object and the imaging device (21 to 24) is installed at a fixed position,
  • the imaging devices (21 to 24) are installed at fixed positions different from the first to Nth robots (30a to 30e),
  • the control method includes: A first step of acquiring change information for each of the first to Nth robots (30a to 30e);
  • the change information corresponding to the j-th robot (30a to 30e) includes a control amount of the j-th robot (30a to 30e), a change amount of a state of the j-th object on an image of the imaging device, and Shows the relationship j is an integer from 1 to N;
  • the control method includes: A second step of acquiring a real image captured
  • 1, 1A, 1C control system 2a male connector, 2a 'male connector model, 2b female connector, 2b' female connector model, 7 insertion hole, 8 notch, 9a bolt, 9b plate-like component, 10c, 10d screw, 10e, 10f cylindrical member, 11a, 11b, 12a, 12b screw hole, 13 bolt hole, 14c, 14d processing tool, 14e large member, 15, 16 processing target portion, 21-24 imaging device, 30a-30e robot, 31a, 31c to 31e @ hand, 31a '@ hand model, 31b @ stage, 31b' @ stage model, 32 force sensor, 40a to 40e robot controller, 50, 50A, 50B, 50C control device, 51 moving image storage unit, 52 design data storage unit 53 CG video generator, 5 , 254 ⁇ Teaching range selector, 55, 255 ⁇ Image processor, 56, 256 Target frame selector, 57a, 57c, 157a, 257a ⁇ First controller, 57b, 57d, 157b, 257

Landscapes

  • Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Manufacturing & Machinery (AREA)
  • Manipulator (AREA)
  • Manufacturing Of Electrical Connectors (AREA)

Abstract

L'invention concerne un dispositif de commande qui réalise : un premier traitement qui obtient une image réelle capturée par un dispositif d'imagerie ; un deuxième traitement qui sélectionne une trame cible à partir d'une vidéo CG ; et un troisième traitement qui commande chacun des 1er à Nième robots sur la base de l'image réelle et de la trame cible. Dans le troisième traitement, le dispositif de commande : calcule une quantité de commande pour un jème robot afin d'amener l'état d'un jème objet cible dans l'image réelle en direction d'un état de modèle pour le jème objet cible dans le cadre cible sur la base d'informations de changement correspondant au jème objet cible ; et commande le jème robot conformément à la quantité de commande calculée. Par conséquent, un système de commande peut être obtenu, qui est capable de modifier de manière coordonnée l'état d'une pluralité d'objets cibles.
PCT/JP2019/026958 2018-07-23 2019-07-08 Système de commande, procédé de commande et programme Ceased WO2020022040A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2018-137705 2018-07-23
JP2018137705A JP6939729B2 (ja) 2018-07-23 2018-07-23 制御システム、制御方法およびプログラム

Publications (1)

Publication Number Publication Date
WO2020022040A1 true WO2020022040A1 (fr) 2020-01-30

Family

ID=69181667

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/026958 Ceased WO2020022040A1 (fr) 2018-07-23 2019-07-08 Système de commande, procédé de commande et programme

Country Status (2)

Country Link
JP (1) JP6939729B2 (fr)
WO (1) WO2020022040A1 (fr)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6763492B1 (ja) * 2019-07-19 2020-09-30 日本ゼオン株式会社 保存安定性に優れるアクリルゴムシート
JP7093881B1 (ja) * 2021-09-06 2022-06-30 Dmg森精機株式会社 システム、及び自動搬送車
JP7829358B2 (ja) * 2022-03-08 2026-03-13 本田技研工業株式会社 エンドエフェクタ制御方法
EP4289566A3 (fr) 2022-06-07 2024-01-03 Canon Kabushiki Kaisha Procédé de commande, programme de commande, support d'enregistrement, procédé de fabrication de produit et système

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH02284885A (ja) * 1989-04-27 1990-11-22 Nissan Motor Co Ltd ワーク位置決め装置の位置教示方法
JP2013146844A (ja) * 2012-01-23 2013-08-01 Seiko Epson Corp 教示画像生成装置、教示画像生成方法および教示画像生成プログラムならびにロボット制御装置、ロボット制御方法およびロボット制御プログラム
JP2013217893A (ja) * 2012-03-13 2013-10-24 Canon Inc モデル生成装置、位置姿勢推定装置、情報処理装置、モデル生成方法、位置姿勢推定方法、情報処理方法
JP2015150636A (ja) * 2014-02-13 2015-08-24 ファナック株式会社 ビジュアルフィードバックを利用したロボットシステム
JP2015223649A (ja) * 2014-05-27 2015-12-14 株式会社安川電機 ギヤ組み込みシステムおよびギヤ組み込み方法
JP2016059971A (ja) * 2014-09-16 2016-04-25 キヤノン株式会社 ロボット装置、ロボット制御方法、プログラム及び記録媒体
WO2017018113A1 (fr) * 2015-07-29 2017-02-02 株式会社オートネットワーク技術研究所 Dispositif de simulation de manipulation d'objet, système de simulation de manipulation d'objet, procédé destiné à la simulation de manipulation d'objet, procédé de fabrication destiné à un objet et programme de simulation de manipulation d'objet

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH02284885A (ja) * 1989-04-27 1990-11-22 Nissan Motor Co Ltd ワーク位置決め装置の位置教示方法
JP2013146844A (ja) * 2012-01-23 2013-08-01 Seiko Epson Corp 教示画像生成装置、教示画像生成方法および教示画像生成プログラムならびにロボット制御装置、ロボット制御方法およびロボット制御プログラム
JP2013217893A (ja) * 2012-03-13 2013-10-24 Canon Inc モデル生成装置、位置姿勢推定装置、情報処理装置、モデル生成方法、位置姿勢推定方法、情報処理方法
JP2015150636A (ja) * 2014-02-13 2015-08-24 ファナック株式会社 ビジュアルフィードバックを利用したロボットシステム
JP2015223649A (ja) * 2014-05-27 2015-12-14 株式会社安川電機 ギヤ組み込みシステムおよびギヤ組み込み方法
JP2016059971A (ja) * 2014-09-16 2016-04-25 キヤノン株式会社 ロボット装置、ロボット制御方法、プログラム及び記録媒体
WO2017018113A1 (fr) * 2015-07-29 2017-02-02 株式会社オートネットワーク技術研究所 Dispositif de simulation de manipulation d'objet, système de simulation de manipulation d'objet, procédé destiné à la simulation de manipulation d'objet, procédé de fabrication destiné à un objet et programme de simulation de manipulation d'objet

Also Published As

Publication number Publication date
JP6939729B2 (ja) 2021-09-22
JP2020015102A (ja) 2020-01-30

Similar Documents

Publication Publication Date Title
JP7326911B2 (ja) 制御システムおよび制御方法
US11207781B2 (en) Method for industrial robot commissioning, industrial robot system and control system using the same
CN111515950B (zh) 机器人坐标系变换关系确定方法、装置、设备和存储介质
CN104842352B (zh) 利用了视觉反馈的机器人系统
CN111604942B (zh) 物体检测装置、控制装置以及物体检测用计算机程序
JP6965844B2 (ja) 制御システム、解析装置および制御方法
JP5949242B2 (ja) ロボットシステム、ロボット、ロボット制御装置、ロボット制御方法、およびロボット制御プログラム
WO2020022040A1 (fr) Système de commande, procédé de commande et programme
JP7115096B2 (ja) 制御システム、制御方法およびプログラム
JP6331517B2 (ja) 画像処理装置、システム、画像処理方法、および画像処理プログラム
JP2017094406A (ja) シミュレーション装置、シミュレーション方法、およびシミュレーションプログラム
KR20190070875A (ko) 비전-기반 조작 시스템들의 교정 및 동작
CN114952832B (zh) 基于单目六自由度物体姿态估计的机械臂拼装方法及装置
JP7249221B2 (ja) センサ位置姿勢キャリブレーション装置及びセンサ位置姿勢キャリブレーション方法
CN120269580B (zh) 一种基于图像分割与物体位姿估计的物体抓取方法及系统
JP2014238687A (ja) 画像処理装置、ロボット制御システム、ロボット、画像処理方法及び画像処理プログラム
JP7707332B2 (ja) 目標物体の姿勢を識別する方法およびそれを実行する計算装置
CN119515972A (zh) 运动机构轨迹的生成与位置校准和装置
CN119784839A (zh) 机械臂位姿确定方法、设备、存储介质及程序产品
Liang et al. Research on hand-eye calibration method based on monocular vision and accuracy chain analysis
Zhang et al. Vision-guided robotic assembly using uncalibrated vision
JP6904315B2 (ja) 情報処理装置、制御システム、情報処理方法およびプログラム
WO2016151862A1 (fr) Dispositif d'enseignement d'assemblage et procédé d'enseignement d'assemblage
Woo et al. Implicit camera calibration using MultiLayer perceptron type neural network
Korayem et al. Vision based robot experiment: measurement of path related characteristics

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19840305

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19840305

Country of ref document: EP

Kind code of ref document: A1